[fix](routineload) Persist the current load definition

[0AyanamiRei]

What problem does this PR solve?

Issue Number: N/A

Related PR: #64878

Problem Summary:

Routine Load historically persists the original CREATE statement in origStmt and reparses it when an image is loaded. However, ALTER ROUTINE LOAD changes the effective load definition stored in RoutineLoadJob without changing the original CREATE statement. After ALTER followed by follower replay, checkpoint, or FE restart, the job could therefore recover the original CREATE semantics instead of the definition currently used to create tasks.

This PR treats the existing RoutineLoadJob runtime fields as the authoritative effective state and fixes both persistence paths:

• Image/checkpoint: persist partitions, column mappings, preceding/where/delete filters, separators, sequence column, merge type, execution memory limit, and memtable_on_sink_node directly on RoutineLoadJob.
• ALTER journal: persist a nullable RoutineLoadDesc delta in AlterRoutineLoadJobOperationLog, apply it under the job write lock on the leader, and apply properties and the delta in the same order during follower replay.

The design deliberately does not introduce a LoadDefinition wrapper or duplicate job/data-source snapshots:

• The fields consumed by Kafka/Kinesis task construction are the image source of truth.
• jobProperties remains authoritative for cached properties such as max filter ratio, batch parallelism, single-tablet loading, partial-update mode/policy, and CSV enclose/escape/empty_field_as_null.
• New-format images restore direct state without creating a SQL context or accessing the catalog.
• origStmt remains persisted for downgrade readability and legacy image migration; ALTER does not rewrite it.
• Old images are identified by the absence of the newly persisted merge type. The new FE reparses origStmt once, restores load clauses and exec_mem_limit, and the next checkpoint writes the direct-state format.
• Legacy memtable_on_sink_node cannot be reconstructed and retains its historical post-restart value of false.

The ALTER journal stores only clauses changed by that ALTER. Nullable RoutineLoadDesc fields preserve clauses not modified by the delta. Old logs do not contain the new field and therefore leave the current load definition unchanged during replay.

CSV ALTER properties are validated only on the leader before mutation. Replay trusts persisted journal values so validation added by a newer FE cannot block a journal written by an older FE. Runtime CSV caches are updated together with jobProperties and rebuilt from that map after image recovery.

Expr serialization dependency

Routine Load directly persists Expr objects from column mappings, preceding filters, where filters, and delete conditions. These expressions are converted back to SQL before each Nereids task is built, so every field that affects ExprToSqlVisitor output must survive Gson round-trip.

This PR hardens the shared legacy Expr serialization contract:

• Adds stable serialization for SQL-relevant state in PlaceHolderExpr, TimeV2Literal, MatchPredicate, SearchPredicate, SlotRef, and VariableExpr.
• Persists FunctionCallExpr ORDER BY metadata and nested OrderByElement state.
• Requires all concrete Expr subtypes to be present in both Gson registries and in a non-default sample set.
• Reflectively requires every Expr instance field to have @SerializedName or be explicitly classified as non-durable derived state.
• Verifies that Gson round-trip preserves both ExprToSqlVisitor(WITH_TABLE) and ExprToSqlVisitor(WITHOUT_TABLE) output for every registered concrete subtype.
• Adds analysis/AGENTS.md review guidance so future Expr changes account for metadata persistence.

Compatibility boundary:

• Old image -> new FE: supported through one-time origStmt migration. ALTER clauses that were never recorded by the old image or old journal are not recoverable; the original CREATE semantics are restored.
• New image/log -> old FE: unknown JSON fields are structurally ignorable and origStmt is still available, but semantic downgrade is not guaranteed. An old FE may restore the original CREATE clauses and ignore new ALTER deltas.
• Avoid ALTERing Routine Load load clauses until every FE has been upgraded. During a mixed-version rolling upgrade, an old follower can ignore the new journal delta, while a new follower cannot reconstruct a delta omitted by an old leader.
• Rolling back after a new-version checkpoint can lose post-upgrade ALTER semantics.

Out of scope:

• Target-table ALTER.
• Full failed-ALTER/edit-log-write atomicity refactoring.
• Pause/cancel reason persistence, automatic-recovery backoff, state-transition timestamps, and unrelated Routine Load lifecycle issues.

Tests added or extended:

• Complete direct-state image round-trip with invalid origStmt, non-default task configuration, all load clauses, and SQL-sensitive Expr values.
• Empty load-definition recovery proving new images do not fall back to SQL parsing.
• Fixed image and ALTER-log fixtures generated by the real serializer at merge base a8928245.
• Kafka/Kinesis durable-versus-derived state recovery.
• ALTER journal serialization for every load-clause category and complex nested Expr values.
• Kafka/Kinesis leader-to-replay delta application, CSV cache synchronization, legacy journal replay, and checkpoint parity.
• Full concrete Expr subtype coverage, field-classification enforcement, and SQL-semantic Gson round-trip.
• A three-FE Docker case covering follower journal replay before checkpoint, leader failover, a subsequent ALTER on the new leader, checkpoint creation, and FE restart recovery.

Validation status for the final head:

• git diff --check: passed.
• FE compile, testCompile, and checkstyle: passed.
• Focused FE unit tests: 36 passed, 0 failures, 0 errors.
• The three-FE Docker regression case was added but has not been executed locally.

Release note

Routine Load jobs now preserve the effective load definition after ALTER across follower replay, leader failover, checkpoint, and FE restart.

During a rolling upgrade, do not ALTER Routine Load load clauses until all FEs are running the new version. Rolling back after a new-version checkpoint can restore the original CREATE clauses instead of post-upgrade ALTER semantics.

Check List (For Author)

• Test

  • Regression test
    • Added a three-FE Routine Load journal replay, leader failover, checkpoint, and restart case.
    • Not executed locally.
  • Unit Test
    • 36 focused FE tests passed.
    • FE compile, testCompile, and checkstyle passed.
  • Manual test
  • No need to test or manual test

• Behavior changed:

  • No.
  • Yes. New images and ALTER journals retain the current effective Routine Load definition; legacy images migrate once from origStmt.

• Does this need documentation?

  • No.
  • Yes. Rolling-upgrade and rollback limitations must be documented.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?