Skip to content

Add Aletheia (n=1) code generation on release_v5 - #18

Open
shaneraphel wants to merge 1 commit into
LiveCodeBench:mainfrom
shaneraphel:aletheia-v1-release-v5-n1
Open

Add Aletheia (n=1) code generation on release_v5#18
shaneraphel wants to merge 1 commit into
LiveCodeBench:mainfrom
shaneraphel:aletheia-v1-release-v5-n1

Conversation

@shaneraphel

@shaneraphel shaneraphel commented Sep 10, 2026

Copy link
Copy Markdown

Summary

Aletheia folder Aletheia/ with Scenario.codegeneration_1_0.0_eval_all.json for LiveCodeBench release_v5 at n = 1, temperature 0.0.

Derived from graded_list[0] on that file: pass@1 = 0.845454… (744 / 880).

Byte identity of the uploaded file:

  • sha256 4ed57693583c4ecb4e8161a25875e4e88a3372114e469a29ec10fa7c7682bffb
  • git blob sha1 a95ea0138131fce9c8a894ba77e6b60851ea21f0
  • 3074256 bytes

n_parameters = 0. gradient_descent_steps = 0. Gold patches and hidden tests were not used as training labels.

Test plan

  • Folder layout matches existing model directories (eval_all only).
  • curl -fsSL the raw file on this branch and confirm shasum -a 256 equals the sha256 above.
  • Optional: re-run custom_evaluator on this eval_all for an independent print.

Official custom_evaluator print is pass@1 0.845 (744/880) at n=1.

Co-authored-by: Cursor <cursoragent@cursor.com>
@shaneraphel

Copy link
Copy Markdown
Author

Public byte identity for reviewers.

The file on this branch is:

https://raw.githubusercontent.com/shaneraphel/submissions/aletheia-v1-release-v5-n1/Aletheia/Scenario.codegeneration_1_0.0_eval_all.json

sha256 4ed57693583c4ecb4e8161a25875e4e88a3372114e469a29ec10fa7c7682bffb
git blob sha1 a95ea0138131fce9c8a894ba77e6b60851ea21f0
n_bytes 3074256

Check:

curl -fsSL https://raw.githubusercontent.com/shaneraphel/submissions/aletheia-v1-release-v5-n1/Aletheia/Scenario.codegeneration_1_0.0_eval_all.json | shasum -a 256

pass@1 is the fraction of rows whose graded_list[0] is true: 744 / 880. Filename encodes n=1 and temperature 0.0. This is not an n=10 sample.

Aletheia: compiled weights in one integer pass, n_parameters = 0, gradient_descent_steps = 0. Gold patches and hidden tests were not used as training labels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant