Benchmarking retrieval-augmented generation on published QA benchmarks.
Two datasets (PubMedQA & HotpotQA), reporting Exact Match (EM) and F1 against
the literature.
This leaderboard compares Kurious (AIntropy AI's hybrid retrieval + RAG pipeline) against published results on two standard QA benchmarks. All numbers other than Kurious's are taken verbatim from their original papers (or, where noted, from a later paper's comparison table); every citation links directly to the source.
| Benchmark | Task | Source |
|---|---|---|
| PubMedQA | Biomedical yes/no/maybe research-question answering | Jin et al., 2019 — PubMedQA |
| HotpotQA (Fullwiki) | Multi-hop open-domain question answering over Wikipedia | Yang et al., 2018 — HotpotQA |
Kurious (AIntropy AI) — a Universal Knowledge Perception retrieval pipeline paired with Llama-3.3-70B-Instruct as the RAG answerer, zero-shot at the LLM level (no fine-tuning, no in-context examples).
PubMedQA (Table 4 — classification task). The gold answer is a single label,
yes / no / maybe. EM is exact-match accuracy: the predicted label string must match the
gold label exactly, which for this task is the same thing as classification accuracy reported by
the baselines (BioBERT, PubMedBERT, BioLinkBERT, GPT‑4). F1 is computed the same token-overlap
way as SQuAD-style QA F1, but because the gold answer is one token drawn from only three possible
classes, a wrong-class prediction has zero token overlap with gold — there is no partial credit to
recover, so F1 tracks EM closely and can come in slightly lower rather than higher. Kurious's row
shows both metrics at retrieval depth K=20, so EM and F1 are directly comparable (K=10 gives a
marginally higher F1 of 72.31%, not shown here, to keep both metrics at the same K).
HotpotQA (Table 6 — extractive multi-hop QA). EM and F1 follow the standard SQuAD/HotpotQA official-evaluation-script definitions used by every row in this table: strings are normalized (lowercased, articles and punctuation stripped), EM is 1 if the normalized prediction equals the normalized gold answer, and F1 is the token-level bag-of-words F1 (harmonic mean of precision and recall over shared tokens) between normalized prediction and gold answer. These are the "Ans‑EM" / "Ans‑F1" columns as defined in Yang et al., 2018.
Every baseline citation was independently re-verified against its primary source before publishing
this table. A few numbers required correction from earlier drafts (e.g. GoldEn Retriever, MUPPET,
DecompRC — see the † markers), and a few widely-cited figures are flagged †/‡ because they are
close approximations rather than a single verbatim in-paper cell (e.g. BioBERT's 60.2% PubMedQA
figure, or the GPT‑4 zero-shot figure which is from Nori et al., 2023, not Med‑PaLM‑2 as sometimes
cited elsewhere). Hover the linked source for each row to check the original number yourself.
Source: Jin et al., 2019 — PubMedQA. EM is exact-match against the gold yes/no/maybe label; F1 is computed the same token-overlap way as SQuAD-style QA F1 (see the methodology note above).
† / ‡ markers: flagged rows are widely-cited approximations, not a single verbatim in-paper cell — see the note above.
| #1 | Kurious (ours) | 75.60 % | 71.96 % | this work |
| #2 | GPT-4 (zero-shot) ‡ | ~74 % | – | Nori et al., 2023 — GPT-4 on Medical Challenge Problems |
| #3 | BioLinkBERT (fine-tuned) | 70.2 % | – | Yasunaga et al., 2022 |
| #4 | BioBERT (fine-tuned) † | 60.2 % | – | Jin et al., 2019 — PubMedQA |
| #5 | PubMedBERT (fine-tuned) | 55.8 % | – | Gu et al., 2021 |
Source: Yang et al., 2018 — HotpotQA. EM/F1 follow the official HotpotQA/SQuAD-style evaluation: normalized exact match and token-level bag-of-words F1 (see the methodology note above).
† markers: flagged rows were corrected from earlier drafts or are ranges from re-implementations in the literature — see the note above.
| #1 | HGN | 59.74 | 71.41 | Fang et al., 2020 — HGN |
| #2 | DecompRC † | 55.20 | 69.63 | Min et al., 2019 (EM via Fang et al., 2020 — HGN) |
| #3 | Kurious (ours) | 48.28 | 61.04 | this work |
| #4 | GoldEn Retriever (test) † | 37.92 | 48.58 | Qi et al., 2019 |
| #5 | CogQA (dev) | 37.6 | 49.4 | Ding et al., 2019 |
| #6 | DPR + reader | 37.0 | 47.0 | Karpukhin et al., 2020 |
| #7 | MUPPET (dev) † | ~30.2–31.1 | ~39.4–40.4 | Feldman & El-Yaniv, 2019 |
| #8 | BiDAF baseline (TF-IDF retrieval) † | 23.95–24.68 | 32.89–34.36 | Yang et al., 2018 — HotpotQA |
This leaderboard tracks EM and F1 on two fixed, published benchmarks — PubMedQA and HotpotQA (Fullwiki). If you'd like to add your system's results:
pqa_labeled, 1,000 questions with yes/no/maybe gold answers).Run your pipeline (retrieval + generation, or direct prompting) over the benchmark questions and collect a predicted answer string for each question.
yes/no/maybe; F1 computed the
same way (see the note above on why F1 ≤ EM for this task).Open an issue with the following template:
System: <name>
Paper / link: <optional>
Dataset: PubMedQA | HotpotQA (Fullwiki)
EM:
F1:
Evaluation script / notebook (for reproducibility):
We will review and add your entry within a few days.
@misc{omni-domain-qa-leaderboard-2026,
title = {Omni-Domain QA Leaderboard},
author = {AIntropy AI},
year = {2026},
url = {https://huggingface.co/spaces/aintropy-ai/omni-domain-qa-leaderboard-v1}
}