Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,15 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Install uv for immutable lock check
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
version: "0.12.5"
enable-cache: false

- name: Verify dependency lock is current
run: uv lock --locked

- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
Expand Down
21 changes: 14 additions & 7 deletions openadapt_flow/benchmark/effectbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,19 @@ system of record before and after the action, never the screen:
```python
class EffectVerifier(Protocol):
substrate: str

def capture_pre_state(self, context: Any = None) -> EffectState: ...
def verify(self, expected: Effect, before: EffectState,
context: Any = None) -> EffectVerdict: ...
def verify(
self, expected: Effect, before: EffectState, context: Any = None
) -> EffectVerdict: ...
```

The in-memory reference oracle (the generalized `fault_model` DB-state oracle):

```python
class RecordSnapshotOracle:
substrate = "snapshot"

def __init__(
self,
read_records: Callable[[], Optional[list[dict[str, Any]]]],
Expand All @@ -115,8 +118,10 @@ class TrueEffectState(str, Enum):
ABSENT = "absent"
UNREADABLE = "unreadable"


def effect_state(verdict: EffectVerdict) -> TrueEffectState: ...


def combine_true_states(
record_state: TrueEffectState, *field_states: TrueEffectState
) -> TrueEffectState: ...
Expand All @@ -132,6 +137,7 @@ def classify_outcome(
correct_action_available: bool,
) -> tuple[OutcomeLabel, SwerVariant, str]: ...


def score_episode(
*,
episode_id: str,
Expand Down Expand Up @@ -174,24 +180,25 @@ attestations the adversarial red-team pass signs off on:

```python
class AgentReport(BaseModel):
reported_success: bool # the deceptive witness — never trusted for scoring
reported_success: bool # the deceptive witness — never trusted for scoring
halted: bool = False
message: str = ""


class EpisodeRecord(BaseModel):
episode_id: str
task_id: str
arm: str # "claude_cu" / "openai_cua" / "compiler" / ...
arm: str # "claude_cu" / "openai_cua" / "compiler" / ...
trial: int
substrate: Substrate # web / desktop / remote_display
substrate: Substrate # web / desktop / remote_display
category: DivergenceCategory # C1..C7 / control
seed: Optional[int]
expected_effect_hash: str
correct_action_available: bool
agent: AgentReport
oracle: OracleVerdict # the independent reading of the true effect
oracle: OracleVerdict # the independent reading of the true effect
outcome: OutcomeLabel
swer_variant: SwerVariant # wrong_write / phantom / none
swer_variant: SwerVariant # wrong_write / phantom / none
reason: str
model_calls: list[ModelCall]
cost_usd: float
Expand Down
10 changes: 6 additions & 4 deletions openadapt_flow/benchmark/effectbench/runner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ An **arm** is one agent under test. It implements a single method:

```python
class AgentArm(Protocol):
name: str # "compiler" / "screen_only" / "claude_cu" / ...
live: bool # True = executes here; False = scaffolded external baseline
def run(self, task: TaskSpec, session: SubstrateSession,
*, params: Mapping[str, str]) -> ArmResult: ...
name: str # "compiler" / "screen_only" / "claude_cu" / ...
live: bool # True = executes here; False = scaffolded external baseline

def run(
self, task: TaskSpec, session: SubstrateSession, *, params: Mapping[str, str]
) -> ArmResult: ...
```

- The arm is handed **only** the task's natural-language `goal` (intent — *never*
Expand Down
2 changes: 1 addition & 1 deletion public-artifacts.json
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
},
{
"path": ".github/workflows/ci.yml",
"sha256": "c562a53c3222a5db05a123d78875379162c056bc2573f7d01f03dcf4df9e42f2"
"sha256": "2eda501134b319e41c1b69e147deeed19b03e4664b07307df9bdfdcbe5bbbfd1"
},
{
"path": ".github/workflows/citrix-workspace-standin.yml",
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ dev = [
"openadapt-types>=0.10.0,<0.11.0",
# Engineering-hygiene gates (lint+format, type-check, coverage). Pinned to
# majors so CI and local dev run the same checkers.
"ruff==0.15.22",
"ruff==0.16.3",
"mypy>=1.11",
"pytest-cov>=5.0",
"jsonschema>=4.21",
Expand Down
14 changes: 14 additions & 0 deletions tests/test_ci_workflow_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,20 @@
DESKTOP_E2E = ROOT / "tests/e2e/test_parallels_desktop_e2e.py"


def test_lint_job_rejects_a_stale_dependency_lock() -> None:
workflow = CI.read_text(encoding="utf-8")
lint_start = workflow.index("\n lint:")
compatibility_start = workflow.index("\n python-compatibility:")
lint_job = workflow[lint_start:compatibility_start]

assert (
"uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9" in lint_job
)
assert 'version: "0.12.5"' in lint_job
assert "run: uv lock --locked" in lint_job
assert lint_job.index("run: uv lock --locked") < lint_job.index("- name: Install\n")


def test_playwright_version_probes_are_valid_python() -> None:
workflow = CI.read_text(encoding="utf-8")
probe = (
Expand Down
52 changes: 26 additions & 26 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.