Skip to content

Add Aletheia MBPP / MBPP+ official evaluate scores - #27

Open
shaneraphel wants to merge 1 commit into
evalplus:mainfrom
shaneraphel:aletheia-mbpp-plus-20260910
Open

Add Aletheia MBPP / MBPP+ official evaluate scores#27
shaneraphel wants to merge 1 commit into
evalplus:mainfrom
shaneraphel:aletheia-mbpp-plus-20260910

Conversation

@shaneraphel

@shaneraphel shaneraphel commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Test plan

  • pip install evalplus==0.3.1 && evalplus.evaluate --dataset mbpp --samples samples.jsonl
  • Confirm results.json still parses and the leaderboard page renders

@shaneraphel

Copy link
Copy Markdown
Author

Official evalplus.evaluate 0.3.1 reprint on the same MBPP+ v0.2.0 pin, parallel=2: MBPP 100.0 / MBPP+ 86.8 (378/378 base, 328/378 plus). Entry size remains 0, open-data NONE, prompted true.

@shaneraphel

Copy link
Copy Markdown
Author

Official evalplus.evaluate 0.3.1 reprint on the same MBPP+ v0.2.0 pin, parallel=2: MBPP 100.0 / MBPP+ 87.0 (378/378 base, 329/378 plus). Entry size remains 0, open-data NONE, prompted true.

@shaneraphel

Copy link
Copy Markdown
Author

Public page: https://github.com/shaneraphel/aletheia

Completions (samples.jsonl, 378 rows): https://gist.github.com/shaneraphel/b0fbe4cde5b83185da043c9e0ac00fd0

SHA-256: ba41a115da629ac4bf55213d4e29e4545994e9a014bbee50fc1f4c384dd9f3b1

Co-authored-by: Cursor <cursoragent@cursor.com>
@shaneraphel
shaneraphel force-pushed the aletheia-mbpp-plus-20260910 branch from a3db743 to e731850 Compare September 10, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant