-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Pull requests: EleutherAI/lm-evaluation-harness
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
chore(api/task): fix pre-existing ruff violations blocking Linters CI
#3999
opened Aug 14, 2026 by
arrdel
Loading…
fix(filters): guard RegexFilter against out-of-range group_select
#3997
opened Aug 14, 2026 by
kanishkaVarshini39
Loading…
Honor configured timeout for synchronous API requests
#3995
opened Aug 13, 2026 by
SiavashShams
Contributor
Loading…
feat(models): add OrcaRouter chat completions backend
#3994
opened Aug 12, 2026 by
XiaoHuo888-hue
Loading…
Fix TER metric direction (higher_is_better) and correct chrF docstring
#3993
opened Aug 11, 2026 by
borgr
Loading…
Fix
megatron_lm backend against Megatron-LM core_v0.18+: argument parsing moved out of initialize_megatron()
#3991
opened Aug 10, 2026 by
oberpierre
Loading…
fix(mmlu_flan_cot_zeroshot): aggregate exact_match instead of nonexistent acc
#3987
opened Aug 7, 2026 by
Anai-Guo
Contributor
Loading…
feat(models): add raw onnxruntime backend for Model Builder ONNX exports
#3984
opened Aug 6, 2026 by
amd-sourjya
Loading…
5 of 7 tasks
tests: cover gsm8k's declared extraction and scoring semantics
#3983
opened Aug 6, 2026 by
Arnav66692
Loading…
Fix NVFP4 batch alignment in Megatron evaluation
#3981
opened Aug 6, 2026 by
shangxiaokang
Contributor
•
Draft
docs: document OpenAI-compatible multi-model gateway base_url example
#3974
opened Aug 3, 2026 by
seven7763
Loading…
3 tasks
feat(prompt_defense_eval): add controls/ obfuscation-robustness fixture set (#3771)
#3973
opened Aug 2, 2026 by
shipbehaves
•
Draft
Add JailbreakBench (jbb_harmful, jbb_benign) task
#3969
opened Jul 30, 2026 by
Kelvinz-89757
Loading…
6 tasks done
Add behavioral_drift metric: detect fine-tuning output collapse that perplexity misses (#3924)
#3968
opened Jul 30, 2026 by
Kelvinz-89757
Loading…
2 of 3 tasks
feat(models): add cross-platform onnxruntime-genai backend + refactor winml
#3960
opened Jul 29, 2026 by
thiagocrepaldi
Loading…
7 of 8 tasks
fix(api): support think_end_token for chat completions
#3959
opened Jul 28, 2026 by
yaodong-shen
Loading…
Add Spider text-to-SQL task with execution-accuracy scoring
#3956
opened Jul 27, 2026 by
hl3816-ops
Loading…
fix(tasks): add missing task names to t0_eval group entries
#3953
opened Jul 26, 2026 by
LHMQ878
Loading…
Add corrected-benchmark tasks: mmlu_pro_clean and gpqa_{diamond,extended}_clean
#3952
opened Jul 26, 2026 by
adamallcock
Loading…
Include generation_kwargs in the request cache key
#3951
opened Jul 25, 2026 by
AbdullahRasheed45
Contributor
Loading…
Previous Next
ProTip!
Type g i on any issue or pull request to go back to the issue listing page.