[release-2.8] backend: migrate avro library from hamba/avro to twmb/avro

[r-vasquez]

Replace hamba/avro and linkedin/goavro with twmb/avro v1.8.0 across the serde and schema packages.

The serde now decodes with Schema.Decode and renders payloads with the schema-aware Schema.EncodeJSON: bytes and fixed fields encode as \u00XX strings, NaN and Infinity encode as strings instead of failing, and logical-type
durations keep their original millis or micros values. JSON input is serialized through Schema.DecodeJSON, removing the goavro codec.

Schema references now parse into a per-call avro.SchemaCache rather than a process-global cache, with a cycle guard and support for diamond-shaped reference graphs. Schemas that under-declare references and only parsed through global cache pollution now fail deterministically.

The frontend JSON viewer gains an escapeLatin1 prop, enabled for avro payloads, so \u00XX byte escapes survive display and copy-paste instead of being converted to UTF-8 glyphs.

GetAvroSchemaByID returns *avro.Schema and ParseAvroSchemaWithReferences drops its schemaCache parameter.

Functional ports of #2271, #2351, #2606, and #2425

Example

Using rpk to show that the messages are serialized, and next to it Console successfully deserializing the payload

[graham-rp]

why change this? Is this a real difference in the libraries?

[r-vasquez]

Yes, twmb/avro does a key reordering alphabetically (fields > name > type)

{
  "fields": [
    {
      "name": "reference",
      "type": {
        "name": "schema",
        "namespace": "referenced",
        "symbols": [
          "FOO"
        ],
        "type": "enum"
      }
    }
  ],
  "name": "parent.schema",
  "type": "record"
}

While hamba/avro does not, it's always name>type>fields for them:

{
  "name": "parent.schema",
  "type": "record",
  "fields": [
    {
      "name": "reference",
      "type": {
        "name": "referenced.schema",
        "type": "enum",
        "symbols": [
          "FOO"
        ]
      }
    }
  ]
}