Hack Apertus 2026 · Track 2A · OST challenge

Every claim checked against the official voting booklet. Each Yes or No comes with the booklet line behind it.

GeaNaVid reads the Swiss Federal Council's voting booklet and a claim in German, French or Italian. It answers Gea, Na or Vid, Romansh for yes, no and empty: the booklet supports the claim, contradicts it, or does not say. Every Gea and every Na points to the page and passage it rests on.

The problem

Voters hear the claim. The answer is on page 4 of 40.

Before every federal vote, each household receives the booklet: the proposal, the legal text, the arguments of the committee and the position of the Federal Council and Parliament, in German, French and Italian. During the campaign, claims about it travel fast. Checking one means finding the right paragraph in 40 to 90 pages, often in another language than the claim.

Claim · German

„Gemäss dem Abstimmungstext ist die Bevölkerung seit Einführung der Personenfreizügigkeit im Jahr 2002 um rund 500'000 Personen gewachsen.“

Booklet of 14 June 2026 · page 4 · In Kürze

„Seit Einführung der Personenfreizügigkeit im Jahr 2002 ist die Bevölkerung um rund 1,7 Millionen Personen gewachsen.“

Na · contradiction (2)

Approach

Narrow first, then let Apertus read.

Sending the whole booklet to the model costs about 20,000 input tokens per claim. GeaNaVid gives Apertus the few passages of the right proposal instead, about 1,500 to 2,200 input tokens per claim.

  1. Read the booklet

    The PDF is split into passages with page, proposal, section and author, including the note that a double page was written by the committee. No language model, no tokens.

  2. Find the proposal and the passages

    The vote title narrows the booklet to one proposal. A claim in another language is translated by Apertus into the booklet language; BM25 then picks five passages and their neighbours.

  3. Ask Apertus

    One call to Apertus v1.5 with numbered passages. The answer is strict JSON: first the passages that decide, then the label. With 8B, unsure verdicts are re-checked claim part by claim part, and a neutral verdict Apertus is not fully sure of is asked once more whether one detail is contradicted.

  4. Quote the deciding sentence

    The evidence is the booklet sentence that covers the claim best, verbatim and with its PDF page, chosen without another model call. A claim that names its source ("the summary says", "the committee thinks") is answered from that section.

Results · 14 October 2026

OST v1.1, test split

OST's benchmark MNLIoverSwissVotingBooklets v1.1, split by voting date: 493 test pairs from 7 voting dates, 21 booklets, two thirds cross-lingual. Macro-F1 over entailment, neutral and contradiction; input tokens per claim from the API. Both models are shown: switch between Apertus 8B and 70B, or compare them side by side.

Results with Apertus 70B
Macro-F1, test split (493 pairs)0.965 [0.944–0.983]
Whole booklet, same model, test split (493)0.889 at 20,242 tokens
Macro-F1, 546 dev claims outside the tuning subset0.971
Macro-F1, Beginner task, test split (493)0.980
OST evidence score, 324 gold entailments / contradictions of the test split96.3 %
Input tokens per claim, test split (493)1,461
Time per case, mean / p95 (cold, 50 dev cases)1,274 / 2,197 ms
OST's Docker run with OST's evaluate.py, task A / B macro-F1 (493 + 493 cases)0.965 / 0.976 B 0.974 in a second run
OST's Docker run, task A (493 cases): evidence found; input tokens; time mean / p95312 of 324 · 1,460 · 1,059 / 2,344 ms
  • Less context, better verdicts: 8B with the selected passages (0.959) beats 70B with the whole booklet (0.889), at 7 to 11 % of the tokens (test split, 493 pairs).
  • Translation is the largest step: cross-lingual claims (330 of the 493 test pairs) rise from 0.70 to 0.94 with 8B and from 0.74 to 0.94 with 70B (ablation step; the full defaults reach 0.943 and 0.958) once Apertus translates them into the booklet language.
  • Evidence as OST scores it: one of the first five quoted sentences lies in OST's gold passage for 97 % (8B) and 96 % (70B) of the 324 gold entailment and contradiction cases of the test split, by the rule of OST's own evaluation script.
  • Claims outside the tuning subset: 546 development claims that only confirmed options chosen on the tuning subset give 0.971 (8B) and 0.971 (70B). The test split was scored 26 times, for 24 configurations.
  • Time as OST measures it: cold run on 50 development cases, mean / p95 per claim 366 / 545 ms (8B) and 1,274 / 2,197 ms (70B); 48 ms of it is our own code. Re-checking only unsure neutral answers cut the 8B p95 by 44 % at the same F1.
  • New booklets: on a synthetic hold-out from three voting dates outside OST v1.1 (89 claims, labels checked blindly by four open-weight models) the lead over the whole booklet holds: 0.874 (second evaluation; first 0.845) vs 0.665 (8B), 0.932 vs 0.872 (70B).
  • Older booklets: OST v1.1 covers 2020–2026. On two synthetic stress sets from 1998–2018 (26 voting dates, 802 claims; claims by an open model, labels agreed by two other open models, no human check) task A first reached only 0.79 (8B) and 0.82 (70B), because older layouts name no proposal in their page headers. Reading the proposal structure from the booklet's own outline raised it to 0.84 and 0.89. One held-out half first got worse (a booklet with a single proposal); the fix was found there, so the report shows both numbers. OST results are unchanged.

OST v1.1 repeats 122 of the 493 test pairs, so the 95 % intervals resample copies together: 0.933–0.980 (8B), 0.944–0.983 (70B); without the copies the defaults score 0.957 (8B) and 0.962 (70B). 8B and 70B do not differ significantly. Variants were built on development data, but test scores were seen 26 times, for 24 configurations and decided some of them, so the 546 development claims and the synthetic hold-out are the honest checks. The rounds are in docs/ITERATIONS.md.

Open

  • open OST's weighting of F1, evidence, tokens and time
  • open Token counts of OST's proxy

Video

Two minutes with GeaNaVid

Pitch

Nine slides, one idea

The pitch deck runs in the browser. Arrow keys move between slides; a PDF version is linked inside.

First slide of the GeaNaVid pitch

Technical report

How it works, what the numbers say

Architecture, use of Apertus, data, evaluation and limitations, with sources. State of 14 October 2026 with the results on OST v1.1; open points are marked as such.

  • Model Apertus v1.5, 8B and 70B
  • Languages German, French, Italian, all nine combinations
  • Output label 0/1/2, passages with page, tokens, seconds
  • Runs Docker, no GPU, any OpenAI-compatible Apertus endpoint
Apertool logo

Team

Built by Apertool

One builder, four tools on Apertus, Switzerland's fully open language model. GeaNaVid is one of them.

Contact

Questions, ideas or feedback on GeaNaVid? Write to Team Apertool and we will answer by e-mail.

Your name, e-mail address and message are forwarded by e-mail to Team Apertool, solely to answer you. The form does not store them and does not pass them on. This site uses no analytics and sets no cookies.