feat: Enable Bound Token for Agentic Identities - #13873
macastelaz wants to merge 31 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Agent Identity token binding support for Cloud Run. It adds AgentIdentityUtils to resolve, load, and verify certificates and private keys, and updates ComputeEngineCredentials to request bound tokens via POST requests when a valid certificate chain is present. The review feedback suggests a cohesive improvement to implement a single-read pattern for certificate files. By reading the certificate chain once, caching it in CertInfo, and passing it to parseCertificate and getBoundTokenPayload, the implementation can avoid redundant disk I/O and prevent potential race conditions during certificate rotation.
macastelaz
changed the title
| // Environment variables | ||
| static final String GOOGLE_API_CERTIFICATE_CONFIG = "GOOGLE_API_CERTIFICATE_CONFIG"; | ||
| static final String GOOGLE_API_PREVENT_TOKEN_SHARING_FOR_GCP_SERVICES = | ||
| "GOOGLE_API_PREVENT_TOKEN_SHARING_FOR_GCP_SERVICES"; |
There was a problem hiding this comment.
Note that based on googleapis/google-cloud-python#17698 (comment) this is not yet finalized
f5e81cc to
db1c39c
Compare
db1c39c to
3ada55f
Compare
1. POST request to MDS with cert-chain 2. Cert-key matching 3. Included logic to consider the user's choice by looking at GOOGLE_API_USE_CLIENT_CERTIFICATE env variable 4. Bound ID tokens. # Conflicts: # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/ComputeEngineCredentialsTest.java # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/MockMetadataServerTransport.java
…etry logic. Nit fixes. # Conflicts: # google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/ComputeEngineCredentialsTest.java
# Conflicts: # google-auth-library-java/oauth2_http/java/com/google/auth/oauth2/ComputeEngineCredentials.java
93a7c86 to
5b1c82b
Compare
…fill CI formatting rules
…n binding - Prevent 30-second polling delay and IOException on standard GCE/container environments when well-known credentials directory exists without certificate files unless mTLS is explicitly enabled. - Fail fast on malformed certificate config JSON without retrying. - Fix URI resource path decoding in AgentIdentityUtilsTest to prevent FileNotFoundException when workspace paths contain spaces. - Copy matching private key in well-known fallback test to verify full key pair loading. - Clean up static wellKnownDir state and temporary directories in ComputeEngineCredentialsTest. - Correct opt-out environment variable value in test setup from 'true' to 'false'.
73fa344 to
6181f6a
Compare
| @InternalApi | ||
| public final class AgentIdentityUtils { | ||
|
|
||
| /** Javadoc. */ |
There was a problem hiding this comment.
Remove placeholder /** Javadoc. */ everywhere and replace with real docs where applicable
| static { | ||
| List<Long> intervals = new ArrayList<>(); | ||
| for (int i = 0; i < FAST_POLL_CYCLES; i++) { | ||
| intervals.add(FAST_POLL_INTERVAL_MS); | ||
| } | ||
| long remainingTime = TOTAL_TIMEOUT_MS - (FAST_POLL_CYCLES * FAST_POLL_INTERVAL_MS); | ||
| int slowPollCycles = (int) (remainingTime / SLOW_POLL_INTERVAL_MS); | ||
| for (int i = 0; i < slowPollCycles; i++) { | ||
| intervals.add(SLOW_POLL_INTERVAL_MS); | ||
| } | ||
| POLLING_INTERVALS = Collections.unmodifiableList(intervals); | ||
| } |
There was a problem hiding this comment.
Why not do this directly in the polling loop?
There was a problem hiding this comment.
Refactored to avoid this
| static final String GOOGLE_API_CERTIFICATE_CONFIG = "GOOGLE_API_CERTIFICATE_CONFIG"; | ||
|
|
||
| /** Javadoc. */ | ||
| static final String GOOGLE_API_PREVENT_TOKEN_SHARING_FOR_GCP_SERVICES = |
There was a problem hiding this comment.
GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES for the old one.
but I think they picked a new one.
There was a problem hiding this comment.
Thanks for flagging - done
| private static final int CERT_KEY_MATCH_RETRIES = 3; | ||
|
|
||
| /** Javadoc. */ | ||
| private static final long CERT_KEY_MATCH_RETRY_INTERVAL_MS = 100; |
There was a problem hiding this comment.
AIP-4118 recommends ~5s between attempts. What does Python do?
There was a problem hiding this comment.
In Python (_agent_identity_utils.py:44-56), polling uses 100ms (50 cycles) then 500ms up to 30s.
In Java, credential discovery occurs on the synchronous token fetch path, so waiting 5s per attempt would freeze calls for 5–15s during rotation races. Using a 100ms backoff with 3 retries (CERT_KEY_MATCH_RETRIES) allows filesystem writes to settle within ~300ms without freezing the caller.
| /** Javadoc. */ | ||
| private static final List<Pattern> AGENT_IDENTITY_SPIFFE_PATTERNS = | ||
| ImmutableList.of( | ||
| Pattern.compile("^agents\\.global\\.org-\\d+\\.system\\.id\\.goog$"), |
There was a problem hiding this comment.
I think in Python @nbayati went back and added non prod patterns. Let's add that + tests?
There was a problem hiding this comment.
Added non-prod patterns and corresponding tests
| } | ||
|
|
||
| if (!matched) { | ||
| throw new IOException( |
There was a problem hiding this comment.
should this be an IOException or something else? perhaps a retryable GoogleAuthException?
There was a problem hiding this comment.
I kept IOException because Credentials#refreshAccessToken() declares throws IOException, and all other credential providers (ComputeEngineCredentials, ServiceAccountCredentials, CertificateIdentityPoolSubjectTokenSupplier) throw IOException for token retrieval/verification errors. This ensures compatibility with GAPIC client retry interceptors without breaking interface contracts.
| transportFactory.transport.getRequest(); | ||
| assertEquals("POST", transportFactory.transport.getRequestMethod()); | ||
| String body = request.getContentAsString(); | ||
| assertTrue(body.contains("certificate_chain")); |
There was a problem hiding this comment.
Let's validate that it contains the full cert in these tests
There was a problem hiding this comment.
Done - updated ComputeEngineCredentialsTest to assert that certificate_chain equals the full PEM content
| "spiffe://agents.global.org-INVALID.system.id.goog/path"; | ||
|
|
||
| private TestEnvironmentProvider envProvider; | ||
| private Path tempDir; |
| * <p>To handle transient race conditions during certificate rotation on disk, this method employs | ||
| * a retry mechanism with backoff when reading the configuration and certificate files. | ||
| * | ||
| * @return A {@link CertInfo} object containing the loaded certificate and its path, or {@code |
There was a problem hiding this comment.
Updated - thanks for flagging
…n binding - Replace placeholder Javadoc comments across AgentIdentityUtils and CertInfo with descriptive documentation. - Inline polling interval calculations via getSleepIntervalMs(), removing the static POLLING_INTERVALS list. - Support GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN as primary environment variable with fallback to GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES. - Add early check for GOOGLE_API_USE_CLIENT_CERTIFICATE=false in getAgentIdentityCertInfo() to exit without polling. - Parse certificate and private key blocks separately using regex to strip private keys from HTTP payloads and enable cryptographic key-pair verification on combined bundle files. - Throw IOException on AccessDeniedException for explicit configs and gracefully return null for implicit well-known discovery. - Exit early without polling for non-workload configs (hasWorkloadConfig) and configs outside the well-known directory. - Add non-production SPIFFE trust domain patterns (agents-nonprod) matching Python. - Maintain IOException exception type on credential verification failures to preserve Credentials#refreshAccessToken interface contract. - Validate full certificate content in ComputeEngineCredentialsTest. - Use JUnit 5 @tempdir in AgentIdentityUtilsTest.
| private static final String SPIFFE_SCHEME_PREFIX = "spiffe://"; | ||
|
|
||
| /** Javadoc. */ | ||
| private static String wellKnownDir = "/var/run/secrets/workload-spiffe-credentials/"; |
There was a problem hiding this comment.
Doesn't this cause issues for other non-run environments?
There was a problem hiding this comment.
No problems with the current implementation - also note that "run" here is runtimes, not cloud run specifically.
| /** Javadoc. */ | ||
| private static final List<Pattern> AGENT_IDENTITY_SPIFFE_PATTERNS = | ||
| ImmutableList.of( | ||
| Pattern.compile("^agents\\.global\\.org-\\d+\\.system\\.id\\.goog$"), |
There was a problem hiding this comment.
Added non-prod patterns and corresponding tests
| private static final int CERT_KEY_MATCH_RETRIES = 3; | ||
|
|
||
| /** Javadoc. */ | ||
| private static final long CERT_KEY_MATCH_RETRY_INTERVAL_MS = 100; |
There was a problem hiding this comment.
In Python (_agent_identity_utils.py:44-56), polling uses 100ms (50 cycles) then 500ms up to 30s.
In Java, credential discovery occurs on the synchronous token fetch path, so waiting 5s per attempt would freeze calls for 5–15s during rotation races. Using a 100ms backoff with 3 retries (CERT_KEY_MATCH_RETRIES) allows filesystem writes to settle within ~300ms without freezing the caller.
| LOGGER, | ||
| org.slf4j.event.Level.WARN, | ||
| Collections.emptyMap(), | ||
| "Permission denied reading certificate config file. Falling back to unbound" |
There was a problem hiding this comment.
Good catch — you're completely right. In the previous implementation, catching AccessDeniedException and returning null left configExists = true and certsPresent = false, causing shouldEnableMtls to throw an IOException("Certificate intent inferred via config, but cert files are missing") despite the log claiming to fall back.
To fix this cleanly per go/sdk-mtls-by-default-cert-discovery:
- Explicit Configuration (GOOGLE_API_CERTIFICATE_CONFIG is set): If permission is
denied reading an explicitly configured file/directory, we now fail fast and propagate
an IOException directly (rather than logging a fallback). - Implicit Discovery (well-known directory): If access to the well-known credentials
directory is denied, configExists now evaluates to false (tied to paths.
hasWorkloadConfig()), so shouldEnableMtls(false, false) cleanly evaluates to false and
execution truly falls back to returning null (unbound token) without throwing an
exception.
Added unit tests in AgentIdentityUtilsTest covering both paths:
• getAgentIdentityCertInfo_explicitConfigFileNotReadable_throwsIOException
• getAgentIdentityCertInfo_explicitConfigDirectoryAccessDenied_throwsIOException
• getAgentIdentityCertInfo_implicitWellKnownAccessDenied_returnsNull
| ResolvedCertAndKeyPaths paths = extractPathsFromConfig(certConfigPath); | ||
| if (paths != null | ||
| && !Strings.isNullOrEmpty(paths.getCertPath()) | ||
| && checkExistsOrAccessDenied(Paths.get(paths.getCertPath()))) { |
There was a problem hiding this comment.
Done - polling is now only done for paths in the well known directory
| "spiffe://agents.global.org-INVALID.system.id.goog/path"; | ||
|
|
||
| private TestEnvironmentProvider envProvider; | ||
| private Path tempDir; |
| * <p>To handle transient race conditions during certificate rotation on disk, this method employs | ||
| * a retry mechanism with backoff when reading the configuration and certificate files. | ||
| * | ||
| * @return A {@link CertInfo} object containing the loaded certificate and its path, or {@code |
There was a problem hiding this comment.
Updated - thanks for flagging
| transportFactory.transport.getRequest(); | ||
| assertEquals("POST", transportFactory.transport.getRequestMethod()); | ||
| String body = request.getContentAsString(); | ||
| assertTrue(body.contains("certificate_chain")); |
There was a problem hiding this comment.
Done - updated ComputeEngineCredentialsTest to assert that certificate_chain equals the full PEM content
| */ | ||
| private static ResolvedCertAndKeyPaths getPathsFromConfigWithRetry(final String certConfigPath) | ||
| throws IOException { | ||
| boolean shouldPoll = isPathInWellKnownDir(certConfigPath); |
There was a problem hiding this comment.
isPathInWellKnownDir(certConfigPath) checks if the config file itself is inside /var/run/secrets/workload-spiffe-credentials/. In practice, GOOGLE_API_CERTIFICATE_CONFIG points to a path like /etc/google-cloud-sdk/certificate_config.json, while workload.cert_path inside that JSON points to /var/run/secrets/workload-spiffe-credentials/certificates.pem.
Because certConfigPath is outside the well-known directory, shouldPoll evaluates to false. If the certificate file is not ready on startup, getPathsFromConfigWithRetry exits on cycle 0 and returns hasWorkloadConfig = true with certsPresent = false. Then shouldEnableMtls(false, true) immediately throws an IOException instead of polling.
Check isPathInWellKnownDir(paths.getCertPath()) after parsing the config, and do not fail fast on cycle 0 when the target certificate path is inside the well-known directory. Also remove the early throw on "Failed to parse Agent Identity config JSON" at line 509 when polling is active, since non-atomic writes during container startup can temporarily produce partial or empty JSON files.
There was a problem hiding this comment.
Good call - I've updated this to make sure to check the cert_path and key_path as well for being located in the wellknown directory then startup polling will be enabled (even if GOOGLE_API_CERTIFICATE_CONFIG points outside of the wellknown directory). I have also gated the early throw so that it won't happen when polling is active.
Added "getAgentIdentityCertInfo_configOutsideWellKnownDir_targetCertInWellKnownDir_pollsUntilReady" as well to cover this.
|
|
||
| // 3) Retry loop for rotation/transient absence when explicitly enabled: | ||
| boolean warned = false; | ||
| for (int cycle = 0; cycle < TOTAL_POLL_CYCLES; cycle++) { |
There was a problem hiding this comment.
You noted in the thread that credential discovery runs on the synchronous token refresh path, so 5s retries would freeze callers. However, when GOOGLE_API_USE_CLIENT_CERTIFICATE=true and /var/run/secrets/workload-spiffe-credentials/ is an empty directory, getWellKnownCertificatePathWithRetry runs all 100 cycles and blocks refreshAccessToken() for 30 seconds.
Because AgentIdentityUtils is stateless, every subsequent token refresh repeats this 30-second sleep.
We should only poll for 30 seconds once during initial startup, or record that the startup poll timed out so subsequent refreshes fail fast or fall back immediately.
There was a problem hiding this comment.
Made sure we only poll once during startup (with private static volatile boolean initialStartupCompleted) which ensures the 30s sleep can only possibly happen once during startup.
| + " failure after %d retries.", | ||
| CERT_KEY_MATCH_RETRIES)); | ||
| } | ||
| } else if (!Strings.isNullOrEmpty(certPath)) { |
There was a problem hiding this comment.
If certificates.pem exists on disk but private_key.pem is missing (Files.exists(Paths.get(keyPath)) is false), loadAndVerifyCredentials enters the else if (!Strings.isNullOrEmpty(certPath)) branch, skips private key verification, and returns a valid CertInfo.
getBoundTokenPayload() then sends that certificate to the Metadata Server and mints a bound token. Every API call using that token will fail because the client does not have the private key for mTLS.
Require a valid private key whenever loading credentials for token binding. Remove the cert-only branch or return null / throw an IOException when the private key is missing.
There was a problem hiding this comment.
Good call - issuing a known-to-fail bound token definitely isn't what we'd want. When config is setup to explicitly indicate desire for a bound token if we can't validate the private key, we now fail fast and otherwise return none to fallback to unbound tokens (but in either case we won't return a broken bound token).
| public void | ||
| getAgentIdentityCertInfo_missingConfigOutsideWellKnownDir_returnsNullImmediatelyWithoutPolling() | ||
| throws Exception { | ||
| Path outsideDir = Files.createTempDirectory("outside_well_known"); |
There was a problem hiding this comment.
This test still calls Files.createTempDirectory("outside_well_known") with manual deletion in finally. Pass a second @TempDir Path outsideDir parameter to the test method so JUnit cleans up the directory automatically even when it is non-empty.
There was a problem hiding this comment.
Done. Thanks for catching it!
|
|
||
| /** Retrieves the bound token payload (certificate chain) if applicable. */ | ||
| static String getBoundTokenPayload() throws IOException { | ||
| CertInfo info = getAgentIdentityCertInfo(); |
There was a problem hiding this comment.
getBoundTokenPayload() calls getAgentIdentityCertInfo(), which runs loadAndVerifyCredentials(certPath, keyPath) before checking shouldRequestBoundToken(info.getCertificate()). On non-agent SPIFFE workloads where shouldRequestBoundToken(cert) is false, every token refresh reads private_key.pem from disk, parses PKCS#8, and runs Signature.sign() and Signature.verify() just to discard the result. Also, CertInfo is never cached across refreshes, even though its Javadoc says it caches certificate content in memory to avoid repeated disk reads.
Check shouldRequestBoundToken(cert) right after parseCertificateContent(certContent), before reading the private key or calling verifyKeyPair(). Cache the verified CertInfo so we do not re-read and re-verify unchanged files on every token refresh.
There was a problem hiding this comment.
Done. Moved the shouldRequestBoundToken(cert) check in loadAndVerifyCredentials() immediately after parseCertificateContent(certContent). If the certificate does not have an Agent Identity SPIFFE URI SAN, we cache the negative result and return null immediately without reading keyPath or running cryptographic signature verification.
Note that I've also addeda unit test: "getAgentIdentityCertInfo_nonAgentSpiffeCert_returnsNullWithoutReadingKey."
There was a problem hiding this comment.
Also note that I've added caching of the verified CertInfo (and added unit test: "getAgentIdentityCertInfo_cachesVerifiedCertInfoAndInvalidatesOnRotation")
|
|
||
| /** Sets the environment variable reader for testing. */ | ||
| @VisibleForTesting | ||
| public static void setEnvReader(EnvReader reader) { |
There was a problem hiding this comment.
wellKnownDir, envReader, and timeService are non-volatile static fields read by concurrent token refresh threads without synchronization. Also, setEnvReader(EnvReader) and EnvReader are public on a public final class despite @VisibleForTesting.
Make wellKnownDir, envReader, and timeService volatile, and change setEnvReader and EnvReader to package-private.
There was a problem hiding this comment.
Thanks for flagging this! Made the required field volatile and setEnvReader/EnvReader package private.
…d key validation - Check isPathInWellKnownDir(paths.getCertPath()) after parsing workload config so startup polling is enabled even when GOOGLE_API_CERTIFICATE_CONFIG resides outside the well-known directory. - Avoid throwing early on malformed config JSON when polling is active to handle non-atomic container startup writes. - Record startup polling timeout (startupPollTimedOut) so subsequent token refreshes fail fast or fall back immediately without repeating 30-second sleep loops. - Require a valid matching private key whenever loading Agent Identity credentials for token binding, removing the cert-only fallback branch. - Check shouldRequestBoundToken(cert) immediately after parsing certificate content before reading private keys or executing signature verification. - Cache verified CertInfo (and negative non-agent SPIFFE results) across token refreshes based on file metadata (mtime, size, fileKey), invalidating on file rotation. - Make static fields (wellKnownDir, envReader, timeService) volatile and restrict EnvReader and setEnvReader visibility to package-private. - Use JUnit 5 @tempdir parameter in AgentIdentityUtilsTest and add unit tests covering all new behaviors.
| @InternalApi | ||
| public final class AgentIdentityUtils { | ||
|
|
||
| private static final Logger LOGGER = LoggerFactory.getLogger(AgentIdentityUtils.class); |
There was a problem hiding this comment.
LoggerFactory.getLogger runs in the static initializer, but oauth2_http/pom.xml marks slf4j-api optional in the default slf4j2x profile. Since ComputeEngineCredentials.refreshAccessToken calls getBoundTokenPayload unconditionally, consumers without SLF4J on the classpath will hit NoClassDefFoundError on their first token refresh before the env var opt-out runs. Can we switch this to LoggerProvider.forClazz and LoggingUtils.log like the rest of the package?
There was a problem hiding this comment.
Switched to use LoggerProvider.forClazz and LoggingUtils.log
|
|
||
| @AfterAll | ||
| static void tearDownAll() { | ||
| LoggingUtils.setEnvironmentProvider(LoggingUtils.SystemEnvironmentProvider.getInstance()); |
There was a problem hiding this comment.
LoggingUtils.SystemEnvironmentProvider.getInstance() does not compile because SystemEnvironmentProvider is a top-level class rather than a nested type inside LoggingUtils. This is hidden right now because the default Maven profile excludes LoggingTest.java, so running mvn test -P "!slf4j2x,slf4j2x-test" will fail to compile.
There was a problem hiding this comment.
Updated to use SystemEnvironmentProvider directly (not nested type) in tearDownAll() - verified compilation with mvn test -P "!slf4j2x,slf4j2x-test" -Dtest=LoggingTest
| return paths; | ||
| } | ||
| } | ||
| } else if (!shouldPoll) { |
There was a problem hiding this comment.
When GOOGLE_API_CERTIFICATE_CONFIG points inside the well-known directory and the file never arrives, the first refresh polls for 30 seconds and throws an exception telling the user to set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN to false. Because finally sets initialStartupCompleted to true, the very next refresh hits line 636 and silently returns an unbound token without anyone changing the env var. Line 786 has the same issue when GOOGLE_API_USE_CLIENT_CERTIFICATE is true and no certs arrive after 30 seconds. Should steady-state refreshes continue throwing here so they match the fail-closed behavior at line 894?
There was a problem hiding this comment.
Makes sense to me - thanks for catching this! Updated this so that if GOOGLE_API_CERTIFICATE_CONFIG points inside the wellknown dir or GOOGLE_API_USE_CLIENT_CERTIFICATE is true subsequent steady-state refreshes will skip the 30s startup poll and fail close instead of silently falling back to an unbound token. Added unit tests to cover this too ("getAgentIdentityCertInfo_startupPollTimeout_subsequentRefreshDoesNotPollAndThrowsIOException" and "getAgentIdentityCertInfo_configInWellKnownDirTimeout_subsequentRefreshDoesNotPollAndThrowsException"
| } | ||
| } | ||
|
|
||
| boolean explicitConfigOrMtls = |
There was a problem hiding this comment.
When an agent identity certificate is present on disk during implicit discovery but the private key is missing, unreadable, or mismatched, this logs a warning after 3 retries and returns null, which causes OAuth2Credentials to cache an unbound token. Since the negative result is not cached in cachedCredentials, every subsequent refresh repeats the 3 retries and logs 4 stack traces at WARN level. Should implicit discovery fail closed with an IOException when an agent certificate exists on disk without a valid key, or should we cache the negative result so we do not retry on every token fetch? Also, lines 478 to 494 are unreachable in implicit mode because line 410 always sets a non-empty keyPath.
There was a problem hiding this comment.
I agree that after shouldRequestBoundToken(cert) confirms an Agent Identity cert is on disk, having a missing/unreadable/mismatched private key should fail instead of logging (after 3 short retries), even in implicit discovery mode, which aligns with go/sdk-mtls-by-default-cert-discovery.
Corresponding unit tests have been updated ("loadAndVerifyCredentials_implicitDiscovery_bundleWithMismatchedKey_throwsIOException" and "getAgentIdentityCertInfo_implicitDiscovery_missingPrivateKey_throwsIOException")
Also removed the reachable branch.
| } | ||
|
|
||
| if (Strings.isNullOrEmpty(keyPath)) { | ||
| boolean explicitConfigOrMtls = |
There was a problem hiding this comment.
Nit: The explicitConfigOrMtls boolean expression is repeated three times at lines 479, 511, and 543 instead of being computed once before the retry loop.
There was a problem hiding this comment.
Refactored loadAndVerifyCredentials which removed the redundancy
| throws IOException { | ||
| try { | ||
| if (!Strings.isNullOrEmpty(certConfigPath)) { | ||
| java.nio.file.Path configPath = Paths.get(certConfigPath); |
There was a problem hiding this comment.
Nit: java.nio.file.Path and BasicFileAttributes are written with fully-qualified names at lines 379, 566, and 569 even though both are already imported at the top of the file. Same for inline fully-qualified names java.nio.file.AccessDeniedException, org.slf4j.event.Level.WARN, java.io.ByteArrayInputStream at line 951, and java.util.Map in ComputeEngineCredentials.java at lines 432 and 512.
There was a problem hiding this comment.
Removed all fully-qualified names used unnecessarily
| getMetadataResponse(createTokenUrlWithScopes(), RequestType.ACCESS_TOKEN_REQUEST, true); | ||
| String tokenUrl = createTokenUrlWithScopes(); | ||
|
|
||
| String boundTokenPayload = AgentIdentityUtils.getBoundTokenPayload(); |
There was a problem hiding this comment.
Nit: The 13-line block building the "certificate_chain" POST payload in refreshAccessToken() is duplicated verbatim in idTokenWithAudience() at lines 508 to 523. Also, line 554 takes a nullable jsonContent parameter in a @NullMarked class without @Nullable, and lines 561 to 565 manually serialize JSON into ByteArrayContent instead of using JsonHttpContent(OAuth2Utils.JSON_FACTORY, payload) like IamUtils.
There was a problem hiding this comment.
Extracted into "getMetadataResponseForToken" and now use JsonHttpContent. Also added @Nullbale to HttpContent content
| /** Test case for {@link IdTokenCredentials}. */ | ||
| class IdTokenCredentialsTest extends BaseSerializationTest { | ||
|
|
||
| private static class TestEnvironmentProvider { |
There was a problem hiding this comment.
Nit: The private TestEnvironmentProvider inner class here, in AgentIdentityUtilsTest.java:1119, and in ComputeEngineCredentialsTest.java:1408 duplicates and shadows the existing top-level package-private TestEnvironmentProvider in javatests/com/google/auth/oauth2/TestEnvironmentProvider.java.
There was a problem hiding this comment.
Removed the private inner duplicate copy.
| class LoggingTest { | ||
|
|
||
| @BeforeEach | ||
| void setUp() { |
There was a problem hiding this comment.
Nit: Adds an instance @BeforeEach void setUp() method right above the pre-existing private static void setup() helper at line 122, which is easy to confuse when reading the test.
There was a problem hiding this comment.
Renamed to be more functionally descriptive (e.g. "disableBoundTokensByDefault")
| return false; | ||
| } | ||
| // Case 2: Explicitly disabled via environment variable | ||
| else if ("false".equalsIgnoreCase(useClientCert)) { |
There was a problem hiding this comment.
When GOOGLE_API_USE_CLIENT_CERTIFICATE is false and certificate files exist on disk, shouldEnableMtls returns false before caching anything, which causes this warning to log on every single token refresh. Can we log this at most once per process, or lower the log level?
There was a problem hiding this comment.
Good catch! Added a mtlsDisabledWarningLogged guard in shouldEnableMtls(...) so this is logged at most once per process. Also added the unit test getAgentIdentityCertInfo_mtlsDisabledWithCertsPresent_logsWarningAtMostOnce. I also lowered the log level to FINE to be consistent with other unbound fallback.
| && latestCached.certInfo != null) | ||
| ? latestCached | ||
| : initialCached; | ||
| if (fallbackCached != null |
There was a problem hiding this comment.
Because fallbackCached is always populated after startup, this fallback triggers even when both files exist on disk and permanently fail verifyKeyPair. That makes the IOException throw below unreachable after startup and causes every future refresh after a bad key rotation to sleep 200ms and return stale credentials forever.
Can we add lastException != null to this condition so we only fall back when files are transiently unreadable or missing on disk?
There was a problem hiding this comment.
Thanks for catching this! We now reset lastException = null at the start of each retry attempt and gate the cache fallback on lastException != null, so a permanent key-pair mismatch on readable files throws an IOException rather than returning stale cached credentials.
Also added unit test getAgentIdentityCertInfo_steadyStatePermanentKeyMismatch_throwsIOExceptionInsteadOfReturningStaleCache.
| } catch (java.net.URISyntaxException e) { | ||
| throw new IOException("Failed to load test resource", e); | ||
| } | ||
| java.nio.file.Path certTarget = tempDir.resolve("certificates.pem"); |
There was a problem hiding this comment.
java.nio.file.Path and MockLowLevelHttpRequest are already imported at the top of the file, so we can drop the inline package prefixes in setupCertAndKeyConfig and the new bound token tests. Same for importing Paths, URISyntaxException, and StandardCopyOption instead of qualifying them inline. Thanks!
There was a problem hiding this comment.
added top-level imports for Paths, URISyntaxException, and StandardCopyOption and removed all inline package prefixes in setupCertAndKeyConfig and the bound token tests.
| lastParseException); | ||
| } | ||
|
|
||
| return new ResolvedCertAndKeyPaths(null, null, false); |
There was a problem hiding this comment.
If GOOGLE_API_CERTIFICATE_CONFIG points to a missing file outside wellKnownDir, this skips the throw above and returns null paths here, silently minting an unbound token. Since this method only runs when GOOGLE_API_CERTIFICATE_CONFIG is explicitly set, should we throw an IOException instead of returning null?
There was a problem hiding this comment.
Since getPathsFromConfigWithRetry only runs when GOOGLE_API_CERTIFICATE_CONFIG is explicitly set (and valid non-workload configs already return early at line 635), we now unconditionally throw IOException if the configured file or its workload credential files cannot be found.
Updated getAgentIdentityCertInfo_missingConfigOutsideWellKnownDir_throwsIOExceptionImmediatelyWithoutPolling accordingly.
| return paths; | ||
| } | ||
| } | ||
| } catch (AccessDeniedException e) { |
There was a problem hiding this comment.
If cert_path or key_path throws AccessDeniedException, this catch block still blames certConfigPath in the error message. Can we use e.getFile() instead so it names the actual file that failed?
There was a problem hiding this comment.
Updated the catch block to use e.getFile() != null ? e.getFile() : certConfigPath so the error message names the exact file that failed permission checks.
| * Resolves the paths for the certificate and private key based on the config path or well-known | ||
| * locations. | ||
| */ | ||
| static ResolvedCertAndKeyPaths resolveCertAndKeyPaths(final String certConfigPath) |
There was a problem hiding this comment.
Nit: This single-argument resolveCertAndKeyPaths overload has no callers in production or tests and can be removed.
There was a problem hiding this comment.
Removed the unused single-argument overload and attached its Javadoc to the two-argument method.
| Collections.emptyMap(), | ||
| "Token binding protection is disabled because mTLS was explicitly disabled" | ||
| + " via GOOGLE_API_USE_CLIENT_CERTIFICATE."); | ||
| return false; |
There was a problem hiding this comment.
Nit: Redundant return false; here inside if (certsPresent) right before the method returns false anyway.
There was a problem hiding this comment.
Removed the redundant return false;
| if (Strings.isNullOrEmpty(keyPath)) { | ||
| throw e; | ||
| } | ||
| lastException = e; |
There was a problem hiding this comment.
Nit: catch (IOException e) and catch (Exception e) have identical bodies here and can be combined.
There was a problem hiding this comment.
Thanks for flagging - combined!
| HttpRequest request; | ||
| if ("POST".equals(method)) { | ||
| request = | ||
| transportFactory.create().createRequestFactory().buildPostRequest(genericUrl, content); |
There was a problem hiding this comment.
Nit: transportFactory.create().createRequestFactory() is duplicated in both branches and can be hoisted above the if.
There was a problem hiding this comment.
Hoisted HttpRequestFactory requestFactory = transportFactory.create().createRequestFactory(); above the if/else.
| } | ||
|
|
||
| @Test | ||
| public void getAgentIdentityCertificate_enableRuntimeBoundTokenFalse_returnsNullImmediately() |
There was a problem hiding this comment.
Nit: Ten test methods still use the old getAgentIdentityCertificate_* prefix after the method under test was renamed to getAgentIdentityCertInfo().
There was a problem hiding this comment.
Renamed all ten test methods from getAgentIdentityCertificate_* to getAgentIdentityCertInfo_*
| AccessToken token = credentials.refreshAccessToken(); | ||
|
|
||
| assertNotNull(token); | ||
| com.google.api.client.testing.http.MockLowLevelHttpRequest request = |
There was a problem hiding this comment.
nit: Why are we using the full path?
There was a problem hiding this comment.
Updated to use MockLowLevelHttpRequest directly.
| private static final LoggerProvider LOGGER_PROVIDER = | ||
| LoggerProvider.forClazz(AgentIdentityUtils.class); | ||
|
|
||
| // Environment variables |
| private static volatile String wellKnownDir = "/var/run/secrets/workload-spiffe-credentials/"; | ||
|
|
||
| @VisibleForTesting | ||
| static void setWellKnownDir(final String dir) { |
There was a problem hiding this comment.
shouldn't this be below the static member vars?
There was a problem hiding this comment.
Thanks for flagging - moved to keep ordering as expected
| } | ||
| } | ||
|
|
||
| private static final class CachedCredentials { |
There was a problem hiding this comment.
lets add javadocs for consistency sake
There was a problem hiding this comment.
Added - thanks for flagging the gap here.
4 participants
This PR introduces a feature which enables the auth library to acquire bound access-tokens and bound id-tokens in Agentic Environments.
We detect certs in default paths and check if they match the SPIFFE format for agents.
If 1. is a yes then we call the MDS endpoint in a POST request with the certificate in the body.
Note this PR was based on #13169
Manual Testing & End-to-End Verification
We verified this feature end-to-end across both a Live Cloud Run Agent Identity environment (testing against the live Google Metadata Server, Security Token Service, and Vertex AI with the Java Agent Development Kit (
com.google.adk:google-adk:1.9.0)) and a 10-Scenario Local Mock MDS Simulation Harness (testing exact HTTP request payloads, true cryptographic certificate/key rotation, combined bundle private-key stripping, well-known directory discovery, non-agent SPIFFE fallback, environment variable precedence, non-atomic rotation retries, and asynchronous container startup polling).1. Live Cloud Run Agent Identity Verification (
<PROJECT_ID>,us-central1)We deployed a containerized Java test application built against this branch (
google-auth-library-oauth2-http:1.50.0-SNAPSHOT@27776f6d62a) + Java ADK (com.google.adk:google-adk:1.9.0) to Cloud Run with Agent Identity enabled (--functional-type=agent --identity-type=agent-identity).Execution A: Default Bound Token Acquisition (
GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKENunset / defaulttrue)agent-bound-token-java-test-cjrgp(latest run with Step 4D) &agent-bound-token-java-test-dd9gm/var/run/secrets/workload-spiffe-credentials/credentials.jsonspiffe://agents.global.org-<ORG_NUMBER>.system.id.goog/resources/run/projects/<PROJECT_NUMBER>/locations/us-central1/jobs/agent-bound-token-java-testx5t#S256):5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8ComputeEngineCredentials):POSTto Metadata Server (ya29.d.c0AZ4bNp...).IdTokenCredentials) & Cryptographic Binding Verification (RFC 8705 § 3.1):https://example-target-service.run.app.textPayload) and verified live Google STS embedded thecnf(Confirmation) claim containing the SHA-256 thumbprint (x5t#S256) of the workload's leaf X.509 certificate:[PASS] JWT x5t#S256 thumbprint EXACTLY matches local leaf certificate SHA-256!).com.google.adk:google-adk:1.9.0) +google-genai& mTLS Proof-of-Possession Verification (Steps 4A, 4B, 4C, 4D):HttpClientFactoryNon-mTLS Transport): Inspected ADK's sharedOkHttpClient(sun.security.ssl.SSLSocketFactoryImpl, no client cert). Calling Google APIs over this non-mTLS channel with the bound token is rejected at the auth layer withHTTP 401 UNAUTHENTICATED.LlmAgent+GeminiTurn on Vertex AI): InvokingInMemoryRunner.runAsync(...)withGemini(gemini-2.5-flash) fails on turn 1 withcom.google.genai.errors.ClientException: 401 . Request had invalid authentication credentials, confirming the known limitation wheregoogle-genaisends bound tokens over a non-mTLS channel.OkHttpClient(configured with/var/run/secrets/workload-spiffe-credentials/certificates.pem+private_key.pem,x5t#S256 = 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8) tohttps://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>passes authentication (HTTP 403 PERMISSION_DENIEDIAM check instead of401 UNAUTHENTICATED), proving Google API Frontend verified the token binding against the TLS client certificate handshake.https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>over an mTLSOkHttpClientconfigured with a different X.509 client certificate (x5t#S256 = -57ZjYVm89oczfdO02Hf3Sz-FaYQIVH1DD3c9zaBIKA) is rejected at the auth layer withHTTP 401 UNAUTHENTICATED, confirming that Google API Frontend enforces cryptographic thumbprint matching (cnf.x5t#S256 == SHA256(TLS client cert)).Execution B: Opt-Out Unbound Token Acquisition (
GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false)agent-bound-token-java-test-t7b98--set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true".ComputeEngineCredentialsandIdTokenCredentialsfell back to standardHTTP GETrequests against MDS and issued standard unbound tokens (decoded JWT payload confirmed absence of thecnfclaim).Step 4B): WithGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false, the unbound token works overgoogle-genai's non-mTLS channel and the live ADKLlmAgent+Geminiturn SUCCEEDS:[ADK Event] author=bound-token-verify-agent, content=Hello! Yes, as an agent designed to test bound token behavior with the Java ADK, I am expected to receive and process bound tokens.2. Local End-to-End Simulation & Wire Verification (10 Scenarios)
To verify internal wire-level, discovery, rotation, environment-variable, and error-handling behavior between the client and MDS, we executed our local simulation suite (
LocalSimulationRunner.java) spinning up a local Mock MDS (HttpServer) across 10 end-to-end scenarios:ComputeEngineCredentials+GOOGLE_API_CERTIFICATE_CONFIG): VerifiedHTTP POSTto/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform, verified JSON body{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}(serialized as a single PEM string), and verified no extra fields are included in the JSON payload.IdTokenCredentials+GOOGLE_API_CERTIFICATE_CONFIG): VerifiedHTTP POSTto/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://target.run.app(withaudiencepassed as a URL query parameter) and JSON body{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}.Gen-1Gen-2): Generated a new X.509 SPIFFE certificate and matching 2048-bit RSA key pair on disk, updated filemtime, and verifiedAgentIdentityUtils.getAgentIdentityCertInfo()invalidated its cache, verified the new key pair, and transmitted the rotatedGen-2certificate chain (!req3.body.equals(req1.body)).GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false): Verified settingGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falseswitches requests back toHTTP GETwith an empty request body.credentialbundle.pem) + Private Key Stripping: UnsetGOOGLE_API_CERTIFICATE_CONFIG, wrote a combinedcredentialbundle.pemcontaining both-----BEGIN CERTIFICATE-----and-----BEGIN PRIVATE KEY-----in the well-known directory, and verified thatAgentIdentityUtilsresolvedcertPath == keyPath, verified the key pair, and stripped thePRIVATE KEYblock from the transmittedPOSTpayload.certificates.pem+private_key.pem): UnsetGOOGLE_API_CERTIFICATE_CONFIG, removedcredentialbundle.pem, placed separatecertificates.pemandprivate_key.pemin the well-known directory, and verified boundHTTP POSTacquisition.shouldRequestBoundToken == false): Configured a valid X.509 certificate with a standard GKE Workload Identity SAN (spiffe://my-standard-gke-project.svc.id.goog/ns/default/sa/my-ksa); verifiedAgentIdentityUtilsdid not throw, cachedshouldRequestBoundToken = false, and fell back to standardHTTP GET.GOOGLE_API_USE_CLIENT_CERTIFICATEMatrix:GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false(withGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKENunset)HTTP GET.GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=trueoverrides legacyGOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=falseHTTP POST.GOOGLE_API_USE_CLIENT_CERTIFICATE=falsewith valid agent certs on diskHTTP GET) without startup polling.GOOGLE_API_USE_CLIENT_CERTIFICATE=truewith missing cert filesIOExceptionafter retries instead of silently downgrading to an unbound token.CERT_KEY_MATCH_RETRIES): Wrote a newGen-3non-prod SPIFFE certificate (spiffe://agents-nonprod.global.org-54321.system.id.goog/...) first while delaying the matchingprivate_key.pemupdate on a background thread; verifiedloadAndVerifyCredentials()retried cleanly viaCERT_KEY_MATCH_RETRIESand transmittedGen-3.TOTAL_POLL_CYCLES): Started with an empty well-known directory on initial startup (GOOGLE_API_USE_CLIENT_CERTIFICATE=true), deliveredcertificates.pem+private_key.pemasynchronously from a background thread after ~180ms, and verified the initialrefreshAccessToken()polled until the files arrived and succeeded with a boundHTTP POST.3. Reproducible Test Artifacts & Execution Logs (
gpaste- Internal Corp Access Only)cjrgp& Opt-Outt7b98) + Java ADK + Local 10-Scenario SimulationCloudRunAgentVerifyApp.javacnf.x5t#S256match, and ADK Steps 4A/4B/4C/4D)LocalSimulationRunner.javaHttpServer) testing all 10 scenariosDockerfilegoogle-genaiand overlaying our local PR JARdeploy_cloud_run_job.sh--identity-type=agent-identity, and executebuild_and_run_local.shcjrgp)gcloud logging readfor default Bound Token execution (with inline decoded JWT & ADK Steps 4A/4B/4C/4D)t7b98)gcloud logging readforGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falseexecution (showing live ADK Gemini response)Quick Reproduction Steps
To replicate the live Cloud Run test in any GCP project with Agent Identity enabled: