ci: skip queue-verified Rust jobs on pushes to main

[kumarUjjawal]

Which issue does this PR close?

• Part of Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148. This PR does not close the issue.

Rationale for this change

Every commit reaches upstream main through the merge queue. The queue runs the Rust workflow at that commit. The push to main then runs the same 26 jobs at the same SHA.

Only seven of those jobs give new output after the merge:

• linux-build-lib, linux-cargo-check-datafusion-functions, linux-test, linux-test-example, clippy, and macos-aarch64 save the caches that later PRs restore. The workflow saves caches only on pushes to main.
• linux-test publishes coverage.
• linux-datafusion-ffi-features is not a required check in .asf.yaml, so a PR can merge when it fails. The post-merge run shows that failure on main.

From September 9 to September 16, 2026, the Rust workflow ran 116 times for pushes to main. Each of those commits had a successful merge-queue run at the same SHA before the push. The ten most recent pushes used a mean of 71.7 runner-minutes each. The 19 skipped jobs account for 47.6 of those minutes: 19.5 on GitHub-hosted Linux runners and 28.0 on RunsOn. At that merge rate, this change saves about 2,200 GitHub-hosted Linux runner-minutes and 3,100 RunsOn runner-minutes per week. The GitHub-hosted minutes count against the shared ASF limit that the issue describes.

Tradeoff: a direct push to main that bypasses the merge queue also skips the 19 jobs. A manual workflow_dispatch on main runs all 26 jobs. extended.yml already uses the same approach for main.

Post-merge runs also catch intermittent failures. In the sample week, three post-merge runs failed. Each commit had passed the same jobs in the queue. The logs show environment errors: a test memory-pool limit, a ChromeDriver port bind, and a MinIO image pull. Two of the three failed jobs are in the skipped set.

What changes are included in this PR?

.github/workflows/rust.yml:

• 19 jobs get if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}. The condition has no status function, so jobs that depend on linux-build-lib still require that job to succeed.
• The Cargo check artifact archive and upload steps in linux-build-lib get the same condition. Both consumers of that artifact are in the skipped set.
• A header comment explains the behavior.

Event	Rust jobs
Push to upstream main	7
PR or merge group	26
workflow_dispatch	26
Push to another branch, or a fork push	26

ci/scripts/check_asf_yaml_status_checks.py gets a second check. It requires that exactly the jobs in POST_MERGE_SKIPPED_JOBS carry the condition, that the artifact steps carry it, and that every job that downloads the cargo-check artifact is in the skipped set. A new job without the condition passes, so new jobs run by default.

docs/source/contributor-guide/testing.md gets one sentence on the new check.

What is the testing strategy for this PR?

python3 ci/scripts/check_asf_yaml_status_checks.py passes on this branch. The required asf-yaml-check job in dev.yml runs it on every PR and merge-group run. The job fails if a listed job loses the condition, if another job gains it, if an artifact step loses its guard, or if a job outside the skipped set downloads the cargo-check artifact.

This PR adds no unit tests for the validator. The validator checks the real workflow in a required job, and a test suite for the validator is out of scope for this change.

Hosted CI on this PR runs all 26 Rust jobs. The first push to main after the merge is the check of the reduced path: seven Rust jobs run and 19 skip.

Are there any user-facing changes?

No. This PR changes CI and the contributor guide only.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.94%. Comparing base (5a44905) to head (1a3b570).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25365      +/-   ##
==========================================
- Coverage   81.94%   81.94%   -0.01%     
==========================================
  Files        1135     1135              
  Lines      428278   428278              
  Branches   428278   428278              
==========================================
- Hits       350945   350933      -12     
- Misses      56378    56387       +9     
- Partials    20955    20958       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[comphead]

Thanks @kumarUjjawal in Comet we did the similar thing, wondering if its possible to test the flow?

[alamb]

Every commit reaches upstream main through the merge queue. The queue runs the Rust workflow at that commit. The push to main then runs the same 26 jobs at the same SHA.

Does this imply we try not running CI on commits to main at all? If it was the same SHA why not run everything in the merge commit?

[kumarUjjawal]

Does this imply we try not running CI on commits to main at all?

Yes

If it was the same SHA why not run everything in the merge commit?

Everything does run in the merge commit. What cannot move there is saving the caches that PRs restore, and that depends on the backend(github cache, runs on cache).

[kumarUjjawal]

runs on cache

I was exploring this earlier using claude:

RunsOn's Magic Cache only scopes by branch when EnableCacheIsolation is on, which is opt-in. If the ASF deployment has it off, the four RunsOn writers (linux-build-lib, linux-test, linux-test-example, clippy) could save from the merge-queue run instead, and the post-merge run would shrink to three jobs.

[kumarUjjawal]

if its possible to test the flow?

I could try to test it on my fork

[kumarUjjawal]

if its possible to test the flow?

I could try to test it on my fork

@comphead here's a run on my fork

https://github.com/kumarUjjawal/datafusion/actions/runs/35183647794