[FLINK-37155][historyserver] Decouple remote archive retention from local processing limit

[argoyal2212]

What is the purpose of the change

HistoryServerOptions.HISTORY_SERVER_RETAINED_JOBS currently gates both (a) which archives the HistoryServer polls/processes locally, and (b) which archives are deleted from the remote archive directory. This means operators cannot keep a large/unbounded remote job-archive history while only actively refreshing a small recent window locally — trimming the local processing window also deletes the "older" jobs remotely.

This closes that gap, which is what FLIP-505 originally set out to solve, before FLIP-584/585 (FLINK-39911, FLINK-40097) landed a more general pluggable ArchiveStorage backend + on-demand lazy-fetch mechanism that already covers FLIP-505's other goals (per-job on-demand fetch, avoiding local disk/inode exhaustion via a pluggable storage backend). This PR adds the one remaining piece: decoupling remote retention from the local processing limit.

Brief change log

• Added historyserver.archive.retain-remote-beyond-local-limit (boolean, default false, fully backward compatible).
• HistoryServerArchiveFetcher.scanArchives(): when enabled, archives beyond historyserver.archive.retained-jobs are cleaned up locally only (new cleanupLocalArchivesBeyondRetainedLimit), leaving the remote archive intact, instead of calling cleanupArchivesBeyondRetainedLimit (which deletes both local and remote).
• Such archives remain reachable on demand via the existing lazyFetchArchiveProactively path when historyserver.archive.load.mode=LAZY.
• HistoryServer reads and wires the new option into the job-archive fetcher (applications are unaffected — this only applies to job archives, matching FLIP-505's original scope).
• Regenerated docs/layouts/shortcodes/generated/history_server_configuration.html.

Verifying this change

This change added tests and can be verified as follows:

• Added HistoryServerArchiveFetcherTest#testArchivesBeyondRetainedLimitAreDeletedFromRemoteByDefault and #testArchivesBeyondRetainedLimitAreKeptRemotelyWhenConfigured, covering both storage backends (File/RocksDB), asserting: default behavior is unchanged (remote archive deleted beyond limit); with the flag enabled, the remote archive persists, is not locally cached, and is still fetchable on demand.
• All 24 tests in HistoryServerArchiveFetcherTest pass locally.

Does this pull request potentially affect one of the following parts?

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): yes, one new @PublicEvolving ConfigOption
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? JavaDocs + generated configuration docs

[flinkbot]

CI report:

• fa5e585 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[Copilot]

Pull request overview

This PR decouples local HistoryServer archive processing limits from remote job-archive retention through an opt-in configuration option.

Changes:

• Adds historyserver.archive.retain-remote-beyond-local-limit, defaulting to false.
• Implements local-only cleanup for archives beyond the local limit.
• Wires the option through HistoryServer.
• Adds File/RocksDB coverage and regenerates configuration documentation.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Reviewed changes
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcherTest.java	Adds tests for default deletion and configured remote retention behavior.
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java	Implements local-only cleanup. Moderate (4 votes): TTL-expired archives can be incorrectly retained remotely when the local count limit is -1; the local-only path should distinguish count-limit rejection from TTL expiry.
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java	Reads and wires the new option. Nit (4 votes): Log the effective retention behavior during initialization.
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java	Defines the new public configuration option.
docs/layouts/shortcodes/generated/history_server_configuration.html	Documents the new configuration option.

Suppressed comments (1)

flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java:295

• The new tests exercise the fetcher constructor directly, but none instantiate HistoryServer with this option. A regression in this configuration read/pass-through would leave the public option ineffective while all added tests still pass; add a HistoryServer-level test that sets the option and verifies remote retention.

        boolean retainRemoteBeyondLocalLimit =
                config.get(HistoryServerOptions.HISTORY_SERVER_RETAIN_REMOTE_BEYOND_LOCAL_LIMIT);

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.