Expired authentication credentials are reported as HTTP 503 (Service Unavailable) errors

[GergelyKalmar]

Determine this is the right repository

• I determined this is the correct repository in which to report this bug.

Summary of the issue

Context
Whenever we try to use the SDK clients with expired authentication credentials our programs halt for 60 seconds. This is because such requests get a 503 HTTP response back (which is not the correct response), so the library keeps retrying until the default timeout (which is 60 seconds).

Expected Behavior:
We expect SDK clients to fail immediately when credentials are expired.

Actual Behavior:
SDK clients do not fail immediately when credentials are expired.

API client name and version

google-cloud-secret-manager==2.30.0

Reproduction steps: code

See https://github.com/logikal-io/stormware/blob/main/stormware/google/secrets.py#L71.

Reproduction steps: supporting files

No response

Reproduction steps: actual results

The errors seen are as follows:

  File ".../lib/python3.12/site-packages/stormware/google/secrets.py", line 71, in __getitem__
    response = self.client.access_secret_version(name=name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1875, in access_secret_version
    response = rpc(
               ^^^^
  File ".../lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 373, in __call__
    result = wrapped_func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 295, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 157, in retry_target
    next_sleep = _retry_error_helper(
                 ^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 230, in _retry_error_helper
    raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 Getting metadata from plugin failed with error: Reauthentication is needed. Please run `gcloud auth application-default login` to reauthenticate.

Reproduction steps: expected results

We'd expect these calls to fail immediately.

OS & version + platform

No response

Python environment

No response

Python dependencies

No response

Additional context

No response