feat(gapic): add OpenTelemetry channel tracing to generator templates

[chalmerlowe]

Problem

Generated client libraries currently lack OpenTelemetry tracing interceptor support for gRPC channels. When OpenTelemetry is configured in client options, generated clients cannot automatically create and pass interceptors down to their transport and gRPC channel.

Solution

Updates the code in the following ways:

• Imports the centralized ClientInterceptor in the gRPC transport template and updates the transport initializer signature and docstrings to accept channel interceptors.
• Updates the client template to look up OpenTelemetry interceptors via google.api_core._observability and pass them into the gRPC transport arguments.
• Adds showcase-unit tests to the generated unit test templates to verify interceptor injection and transport application.
• Adds showcase (system tests) to the generated system test templates.

Non-Goals & Future Work

• AsyncIO Transport (grpc_asyncio): Channel tracing for grpc_asyncio and AsyncClient is intentionally deferred to an upcoming follow-up PR to maintain a focused review scope and ensure isolated test coverage.
• Streaming RPC Assertions: While gRPC client interceptors intercept streaming calls, dedicated verification tests for client-, server-, and bidirectional-streaming spans will be added in a subsequent observability milestone.

Notes for Reviewers

• The changes here are based on the feat/otel-tracing-t4-resource-attributes branch which introduces the required core observability helpers.

[gemini-code-assist]

Code Review

This pull request adds OpenTelemetry tracing capabilities (Tier 3 observability) for gRPC channels in the GAPIC generator and google-api-core. It integrates client-side span creation and attribute extraction for non-streaming gRPC calls. The review feedback highlights two important improvements: handling potential ValueError exceptions gracefully when parsing malformed ports from api_endpoint in _observability.py, and supporting client_options when passed as a dictionary when resolving the custom tracer_provider in method.py.

[chalmerlowe]

Added do-not-merge

Until the most recent version of google-api-core is published to PyPI, the ClientInterceptor object is unavailable. While working on this PR, this is a temporary workaround to enable testing, etc. Will be removed before merge.

[daniel-sanche]

This is a good change, but looking at this more, I'm thinking we should probably just append the custom interceptors inside the Transport.__init__, instead of trying to build the interceptor list here. I forgot that this same method is shared for sync/async/rest, which all have different interceptor formats. And this logic loses the interceptors if a Callable is passed

What do you think?

[daniel-sanche]

suggestion: If you wanted to make sure this is present, you could make use of the _compat file until we get the right version of api_core in place

[chalmerlowe]

Not recommended

Regarding _compat.py: Because OpenTelemetry tracing requires google-api-core >= 2.36.0 anyway (for _observability and method spans), generated clients on older api_core versions will never emit or pass interceptors. Since interceptors is a new parameter that didn't exist in older client releases, the getattr fallback lambda safely avoids AttributeError without needing to generate and maintain a duplicate polyfill in _compat.py across 100+ libraries. When google-api-core >= 2.36.0 is eventually set as a minimum dependency in setup.py, we can drop the getattr entirely.

[chalmerlowe]

This is a good change, but looking at this more, I'm thinking we should probably just append the custom interceptors inside the Transport.init, instead of trying to build the interceptor list here. I forgot that this same method is shared for sync/async/rest, which all have different interceptor formats. And this logic loses the interceptors if a Callable is passed

What do you think?

I made this change.

[daniel-sanche]

Looking good for the most part, but take a look at my comments before merging. Particularly around client_options

[daniel-sanche]

nit: can't this just be if _WRAP_METHOD_SUPPORTS_TRACING? It doesn't seem like it should be tied to the transport

[daniel-sanche]

If the point of the wrapper is just to strip out unsupported args, we should be able to avoid the extra indirection, and do something like this:

try:
    kind = self.kind
except NotImplementedError:
    kind = None

self.export_assets: self._wrap_method(
    self.export_assets,
    default_timeout=60.0,
    client_info=client_info,
    **({
           "method_name": "google.cloud.asset.v1.AssetService/ExportAssets",
           "kind": kind,
        } if _WRAP_METHOD_SUPPORTS_TRACING and kind else {}
    )
),

[daniel-sanche]

I wonder if it would be worth adding a value for the base class, so we don't have to worry about this edge case. Maybe an empty string

[daniel-sanche]

relying on an internal property like this seems a little ugly, since it could change at any time. I'd say this check probably isn't worth it, and we can just trust our inputs

But this is protected by a getattr guard, so not a big deal either way

[daniel-sanche]

this seems to be unused

[daniel-sanche]

Does this need to be added to http too, to be consistent?

We should at least add a **kwarg, so new arguments can be passed through?

[daniel-sanche]

Do we have to be so selective about which transports we pass client_options to? I was thinking by moving the interceptor to the transport, we could pass the same init args to them all

(I wonder if grpc_transport_name here could cause problems specifically, if there are any http-only clients without this set)