Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/live-smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
name: Bounded Chat Completions and Responses smoke
if: >-
github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
(github.event_name == 'workflow_dispatch' || vars.LIVE_SMOKE_ENABLED == 'true')
vars.LIVE_SMOKE_ENABLED == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
environment: live-smoke
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@ jobs:
timeout-minutes: 10
permissions:
contents: read
# Required repository configuration: protect this environment, require reviewers, and
# configure COMETAPI_KEY plus the approved COMETAPI_LIVE_MODEL variable.
# Required repository configuration: protect this environment without required reviewers,
# and configure COMETAPI_KEY plus the approved COMETAPI_LIVE_MODEL variable.
environment: live-smoke
env:
COMETAPI_LIVE_CONCURRENCY: "1"
Expand Down
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ Before the first remote push:
violations in one run while returning non-zero if any violation exists.
Keep checks for canonical identity, contacts, repository metadata, public-
safe language, and standalone content.
4. Gate scheduled live smoke with a `LIVE_SMOKE_ENABLED` repository variable.
An unset or non-true value must prevent live execution. Keep
4. Gate scheduled and manually dispatched live smoke with a
`LIVE_SMOKE_ENABLED` repository variable. An unset or non-true value must
prevent live execution. Keep
`RELEASE_PLEASE_ENABLED` disabled through the initial manual alpha.
5. Make the release live-model setting use `gpt-5.4` when
`COMETAPI_LIVE_MODEL` is unset or empty; never allow an empty model value.
Expand Down
14 changes: 7 additions & 7 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,11 @@ The package manifest uses `authors = [{ name = "CometAPI" }]`. Remove
`.github/CODEOWNERS` and its validation dependencies; it is not required while
the project has one active maintainer.

Before the first push, require `LIVE_SMOKE_ENABLED=true` for scheduled live
execution and keep `RELEASE_PLEASE_ENABLED` disabled through the initial manual
alpha. An unset or non-true value prevents the corresponding workflow from
running. The release live-model configuration resolves an unset or empty
`COMETAPI_LIVE_MODEL` to `gpt-5.4`.
Before the first push, require `LIVE_SMOKE_ENABLED=true` for scheduled and
manually dispatched live execution, and keep `RELEASE_PLEASE_ENABLED` disabled
through the initial manual alpha. An unset or non-true value prevents the
corresponding gated job from executing. The release live-model configuration
resolves an unset or empty `COMETAPI_LIVE_MODEL` to `gpt-5.4`.

The private stage validates sanitized history, the complete local gate, and
real credential-free default-branch CI only. Do not configure or exercise
Expand Down Expand Up @@ -128,8 +128,8 @@ violations in one run and still returns non-zero when any violation exists.
is ongoing monitoring only and cannot satisfy a release gate. It is capped at
four requests, 16 output tokens
per generation, a 30-second request timeout, concurrency one, a ten-minute
workflow timeout, and stop on the first failure. Scheduled execution also
requires `LIVE_SMOKE_ENABLED=true`.
workflow timeout, and stop on the first failure. Every trigger requires
`LIVE_SMOKE_ENABLED=true`.
- `release-please.yml` maintains a human-reviewed version and changelog pull
request from Conventional Commits after maintainers enable the
`RELEASE_PLEASE_ENABLED` repository variable. Keep it disabled until the
Expand Down
5 changes: 3 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,8 +216,9 @@ All workflow files must pass local `actionlint` 1.7.12. This is static
validation only. Remote behavior remains unverified until each workflow runs
successfully in the canonical GitHub repository.

Scheduled live smoke must additionally require `LIVE_SMOKE_ENABLED=true`; an
unset or other value prevents live execution. Release Please requires
Scheduled and manually dispatched live smoke must require
`LIVE_SMOKE_ENABLED=true`; an unset or other value prevents live execution.
Release Please requires
`RELEASE_PLEASE_ENABLED=true` and remains disabled through the initial manual
alpha. Release jobs must resolve an unset or empty `COMETAPI_LIVE_MODEL` to
`gpt-5.4` rather than attempt a request with an empty model.
Expand Down
23 changes: 12 additions & 11 deletions scripts/check_workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ def _require_pattern(text: str, pattern: str, message: str) -> None:
raise CheckError(message)


def _job(text: str, name: str) -> str:
def _job(text: str, name: str, *, source: str = "publish workflow") -> str:
match = re.search(
rf"(?ms)^ {re.escape(name)}:\n(?P<body>.*?)(?=^ [a-zA-Z0-9_-]+:\n|\Z)",
text,
)
if match is None:
raise CheckError(f"publish workflow has no {name!r} job")
raise CheckError(f"{source} has no {name!r} job")
return match.group(0)


Expand Down Expand Up @@ -135,20 +135,21 @@ def check_publish_workflow(text: str, live_smoke_text: str) -> None:
if write_permissions != ["id-token"]:
raise CheckError("id-token: write on the publish job must be the only write permission")

monitoring_live = _job(live_smoke_text, "smoke", source="live-smoke workflow")
_require_pattern(
live_smoke_text,
r"(?m)^concurrency:\n group: trusted-live-smoke\n cancel-in-progress: false$",
"release and monitoring live smokes must share one non-cancelling concurrency group",
)
_require(
live_smoke_text,
"(github.event_name == 'workflow_dispatch' || vars.LIVE_SMOKE_ENABLED == 'true')",
"scheduled live smoke must require LIVE_SMOKE_ENABLED=true",
)
_require(
live_smoke_text,
"github.ref == format('refs/heads/{0}', github.event.repository.default_branch)",
"monitoring live smoke must run only against the canonical default branch",
_require_pattern(
monitoring_live,
r"(?m)^ if: >-\n"
r" github\.ref == format\('refs/heads/\{0\}', "
r"github\.event\.repository\.default_branch\) &&\n"
r" vars\.LIVE_SMOKE_ENABLED == 'true'\n"
r" runs-on:",
"monitoring live smoke must run only against the canonical default branch and "
"require LIVE_SMOKE_ENABLED=true for every trigger",
)

build = _job(text, "build")
Expand Down
36 changes: 33 additions & 3 deletions tests/test_release_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,13 +380,43 @@ def test_semantic_contract_rejects_split_monitoring_live_concurrency() -> None:
)


def test_semantic_contract_rejects_ungated_scheduled_live_smoke() -> None:
live_smoke = LIVE_SMOKE_WORKFLOW.read_text(encoding="utf-8").replace(
@pytest.mark.parametrize(
"replacement",
[
"github.event_name == 'workflow_dispatch'",
"(github.event_name == 'workflow_dispatch' || vars.LIVE_SMOKE_ENABLED == 'true')",
"vars.LIVE_SMOKE_ENABLED != 'false'",
"vars.LIVE_SMOKE_ENABLED == 'true'\n || github.event_name == 'workflow_dispatch'",
],
ids=["manual-only", "manual-bypass", "non-exact-opt-in", "continued-manual-bypass"],
)
def test_semantic_contract_rejects_live_smoke_gate_bypasses(replacement: str) -> None:
live_smoke = LIVE_SMOKE_WORKFLOW.read_text(encoding="utf-8").replace(
"vars.LIVE_SMOKE_ENABLED == 'true'",
replacement,
1,
)
with pytest.raises(RuntimeError, match="every trigger"):
check_publish_workflow(
PUBLISH_WORKFLOW.read_text(encoding="utf-8"),
live_smoke,
)


def test_semantic_contract_checks_live_smoke_gate_on_smoke_job() -> None:
live_smoke = LIVE_SMOKE_WORKFLOW.read_text(encoding="utf-8").replace(
"vars.LIVE_SMOKE_ENABLED == 'true'",
"github.event_name == 'workflow_dispatch'",
1,
)
with pytest.raises(RuntimeError, match="LIVE_SMOKE_ENABLED=true"):
live_smoke += """
decoy:
if: >-
github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
vars.LIVE_SMOKE_ENABLED == 'true'
runs-on: ubuntu-latest
"""
with pytest.raises(RuntimeError, match="every trigger"):
check_publish_workflow(
PUBLISH_WORKFLOW.read_text(encoding="utf-8"),
live_smoke,
Expand Down
Loading