feat(tools): add WebSearchClient for invoking Amazon Web Search - #658
Conversation
The SDK can create a web search connector target on a gateway, but there is no
way to call the resulting tool from Python. Anything that is not already an MCP
client has to hand-roll the SigV4 signed MCP handshake to use it.
WebSearchClient closes that gap:
client = WebSearchClient(region="us-east-1", gateway_id="my-gateway-abc123")
for result in client.search("what is agentcore", max_results=5):
print(result.title, result.url)
Details:
- Transport sits behind a WebSearchBackend seam. GatewayMcpBackend speaks the
slice of MCP streamable HTTP that one tool call needs (initialize, the
initialized notification, optionally tools/list, then tools/call) using
urllib3 and botocore.auth.SigV4Auth, so no new dependency is added. If the
direct web search API arrives later, it is a second backend behind the same
search() signature.
- Both response framings are handled, since a gateway may answer a POST with
application/json or with text/event-stream.
- Tool name resolution accounts for Gateway prefixing every tool with its
target name: target_name derives "<target>___WebSearch" directly, otherwise
tools/list is walked (following nextCursor) and the WebSearch tool is picked,
erroring when the choice is ambiguous.
- Inputs are validated against the documented limits before the call: query is
required and capped at 200 characters, max_results at 1 to 25. Domain and
published-date filters need connector version 1.2.0 or later on the target.
- Results are returned as WebSearchResult with url, title and published_date
retained, because citations have to be displayable in anything shown to an
end user.
- get_gateway_mcp_endpoint validates the gateway identifier as a DNS label
before interpolating it into the hostname, matching the existing region
validation, so a crafted identifier cannot redirect the request off AWS.
- A region outside the connector's availability warns instead of failing, so a
stale constant never blocks a call to a newly added region.
… needs Web search takes no API key of its own: the caller's credentials need bedrock-agentcore:InvokeGateway on the gateway ARN and the gateway service role needs bedrock-agentcore:InvokeWebSearch on the connector. AccessDenied is almost always the first of those, so say so where a caller will look. Also spell out how request filters compose with the target's own domain rules. A result is dropped if its domain is on either exclude list, and returned only if it is on every include list that is set, so passing include_domains against a target that already has an include list narrows to the intersection and disjoint lists return nothing. That empty result is silent rather than an error, which is worth knowing before debugging it.
✅ No Breaking Changes DetectedNo public API breaking changes found in this PR. |
|
Claude Security Review: no high-confidence findings. (run) |
tejaskash
left a comment
There was a problem hiding this comment.
Reviewed the transport layer against the MCP streamable HTTP spec. Tests pass locally. Inline comments below, ordered roughly by severity. The first three matter once this runs against a real gateway.
Match every JSON-RPC reply to the id of the request it answers, so a server notification arriving ahead of the reply is not read as the answer, and require a reply to carry a result or an error. A JSON-RPC error is still surfaced whatever id it carries, since the spec allows a null id there. Parse SSE the way the specification defines it: consecutive data lines of one event join with a newline and a blank line ends the event, so a message split across lines decodes instead of being dropped. Wrap urllib3 transport failures in WebSearchError, so a connection drop or a read timeout cannot escape as a urllib3 exception from a method documented to raise WebSearchError. Treat HTTP 404 against a session we hold as the spec's signal that the session is gone, and drop it so the next call hands shake again. Roll the initialized flag back if the notifications/initialized POST fails, rather than leaving the client claiming a session the gateway never acknowledged. Apply the SDK's endpoint host check to a caller-supplied gateway_endpoint, since signed requests carry the caller's credentials either way. Reject a domain list longer than the documented maximum of 100 before calling, create the default boto3 session only when a region has to come from one, export GatewayMcpBackend, type _parse_gateway_arn as Tuple[str, str] and drop DEFAULT_TARGET_NAME, which duplicated a default that belongs to the gateway helper. Tests: replies are now numbered from the request they answer, so the fixtures no longer hard-code ids that did not match. Adds coverage for each fix above. Drops the signing test that only asserted botocore's own behavior.
|
Claude Security Review: no high-confidence findings. (run) |
Applies the findings from the review of the equivalent Python client (aws/bedrock-agentcore-sdk-python#658) to this one, since both speak the same subset of MCP streamable HTTP. - Parse SSE per the specification. Consecutive data lines belonging to one event are joined with a newline and one leading space after the colon is framing, so a reply split over several lines decodes instead of being dropped. A final event with no trailing blank line is still read. - Match a reply to the id of the request it answers, so a server notification arriving ahead of the reply is not taken for the answer. A JSON-RPC error is always surfaced, since it may carry a null id. - Raise on a reply that carries neither a result nor an error, rather than reading a missing result as an empty one. - Wrap a fetch rejection, meaning a refused connection, a DNS or TLS failure or an expired timeout, in WebSearchError with the original kept as cause. WebSearchError now takes ErrorOptions. - Forget the MCP session on HTTP 404 with a session held, which is the transport's signal that the session is gone, so the next search redoes the handshake. - Set initialized before the initialized notification, because every later request carries mcp-protocol-version, and roll it back if that POST fails. - Check that an endpoint resolves to an AWS host before anything is signed for it, for a URL built from a gateway id and for one passed in. The BEDROCK_AGENTCORE_GATEWAY_ENDPOINT override stays as given so a local mock can still be used. - Drop DEFAULT_TARGET_NAME, which nothing read.
tejaskash
left a comment
There was a problem hiding this comment.
Requesting changes for the first three items in my inline review above (#658 (review)):
- SSE decoder returns the first JSON-RPC message rather than the response matching the request id (line 459).
- urllib3 transport exceptions escape
_postdespite theWebSearchErrorcontract in the docstring (line 314). - An expired session (HTTP 404) leaves the backend permanently failing until
close()(line 327).
The remaining comments are non-blocking.
tejaskash
left a comment
There was a problem hiding this comment.
All 12 items addressed at 0ed088e. Verified locally: 171 tests pass, and I exercised _decode_jsonrpc directly with a notification ahead of the reply, a reply to another id, and a message split across two data: lines. The reordered _initialized handling with rollback is the right call given the protocol version header dependency. Approving.
* feat(tools): add WebSearchClient for AgentCore Web Search Web search is reachable today as an AgentCore Gateway connector target, which an agent calls as an MCP tool. There was no way to reach it from this SDK, so a TypeScript caller had to hand-roll SigV4 signing plus a JSON-RPC MCP conversation. WebSearchClient wraps that behind a plain search(query, options) that returns typed results. The transport sits behind a WebSearchBackend interface so a different access path can be added later without changing callers. Also adds getGatewayMcpEndpoint to _utils/endpoints, mirroring get_gateway_mcp_endpoint in the Python SDK. * fix(tools): harden the web search MCP transport Applies the findings from the review of the equivalent Python client (aws/bedrock-agentcore-sdk-python#658) to this one, since both speak the same subset of MCP streamable HTTP. - Parse SSE per the specification. Consecutive data lines belonging to one event are joined with a newline and one leading space after the colon is framing, so a reply split over several lines decodes instead of being dropped. A final event with no trailing blank line is still read. - Match a reply to the id of the request it answers, so a server notification arriving ahead of the reply is not taken for the answer. A JSON-RPC error is always surfaced, since it may carry a null id. - Raise on a reply that carries neither a result nor an error, rather than reading a missing result as an empty one. - Wrap a fetch rejection, meaning a refused connection, a DNS or TLS failure or an expired timeout, in WebSearchError with the original kept as cause. WebSearchError now takes ErrorOptions. - Forget the MCP session on HTTP 404 with a session held, which is the transport's signal that the session is gone, so the next search redoes the handshake. - Set initialized before the initialized notification, because every later request carries mcp-protocol-version, and roll it back if that POST fails. - Check that an endpoint resolves to an AWS host before anything is signed for it, for a URL built from a gateway id and for one passed in. The BEDROCK_AGENTCORE_GATEWAY_ENDPOINT override stays as given so a local mock can still be used. - Drop DEFAULT_TARGET_NAME, which nothing read. * fix(tools): address review on the web search client Handshake, signing and discovery fixes from review of #257: - A second search now waits for the initialized notification instead of overtaking it, so a tool call can no longer go out on a session that is about to be reset. - The endpoint query string is part of the canonical request, so a gateway URL that carries one signs correctly. - Endpoints must be HTTPS, since signed headers carry an access key id and a session token. - clientInfo reports the SDK version, read from package.json, rather than repeating the protocol version. - A region that disagrees with the region in a gateway ARN throws instead of one of them silently winning, and an ARN for another service is rejected. - Concurrent first searches share one tools/list, and pagination stops after 50 pages or on a repeated cursor. - Dropped the console.warn for an unlaunched region, which callers cannot silence. KNOWN_REGIONS is still exported for anyone who wants to check. --------- Co-authored-by: Sundar Raghavan <sdraghav@amazon.com>
Description
The SDK can already create a web search connector target on a gateway (
GatewayClient.create_web_search_target, #656), but there is no way to call the resulting tool from Python. Today anything that is not already an MCP client has to hand-roll the SigV4 signed MCP handshake against the gateway endpoint to use web search.This adds
WebSearchClient:Design notes
No new dependency. The transport sits behind a
WebSearchBackendseam.GatewayMcpBackendspeaks only the slice of MCP streamable HTTP that one tool call needs (initialize, thenotifications/initializedack, optionallytools/list, thentools/call) usingurllib3andbotocore.auth.SigV4Auth, both already core dependencies.mcp-proxy-for-awswould do the signing for us, but it is currently dev/test only here and it is async, so it would change the SDK's dependency surface and the calling style for one capability. If reviewers would rather take that dependency as a published extra, the backend is one file to swap and the publicsearch()signature does not move.Two response framings. A gateway may answer a POST with
application/jsonor withtext/event-stream. Both are parsed, since which one comes back for a singletools/callis not something the client can assume.Tool name resolution. Gateway prefixes every tool with the name of the target it came from, delimited by three underscores, so the tool the agent sees is
<targetName>___WebSearch, notWebSearch. Passingtarget_namederives the name directly and skips a round trip. Otherwisetools/listis walked (followingnextCursor) and the WebSearch tool is picked, with a clear error when more than one target exposes one.Validation before the call.
queryis required and capped at 200 characters,max_resultsat 1 to 25, per the documented tool schema. The domain and published-date filters require connector version 1.2.0 or later on the target, which the docstring says.Citations are preserved.
WebSearchResultkeepsurl,titleandpublished_datealongsidetext, because the acceptable use terms require source citations and links to be retained and displayed in anything surfaced to an end user. Dropping them in the response type would make compliant use harder than it needs to be.Endpoint safety.
get_gateway_mcp_endpointvalidates the gateway identifier as a DNS label before interpolating it into the hostname, matching the region validation already in_utils/endpoints.py, so a crafted identifier cannot redirect the request off AWS. Theconnectionheader is excluded from signing, which otherwise produces a signature mismatch server-side.Region handling. The connector is offered in us-east-1, eu-west-1 and ap-northeast-1. Calling from another region logs a warning rather than raising, so a stale constant in the SDK never blocks a call to a region that was added after the release.
Testing
uv run pytest tests -qpasses: 3501 passed, 10 skipped, 4 xpassed. 97 new unit tests (81 for the client, 16 for the gateway identifier validation) cover the SigV4 header set, the request sequence, session reuse, both response framings, tool-name resolution including pagination and ambiguity, theisErrorand JSON-RPC error paths, ARN parsing, and the gateway identifier validation.ruff checkandruff format --checkare clean. Statement coverage of the new module is 100%.Not verified against a live service. The web search connector is enabled per account and this repo's integration test account is not entitled to it, so the invoke path has been exercised only against mocked HTTP responses. The request shapes come from the public tool schema and response format documentation, and the MCP sequence from the streamable HTTP transport spec, but nothing here has completed a real search.
tests_integ/tools/test_web_search_client.pyis included and will exercise the real path once run against an entitled account withWEB_SEARCH_GATEWAY_IDset; it skips rather than fails without one. Reviewers with an entitled account running that file would be the useful confirmation.Confirmed with the service team
The API surface this is built on has since been confirmed by the Web Search Tool team, and matches what is implemented here: SigV4 (AWS_IAM) with no search-specific API key or credential; availability in us-east-1, eu-west-1 and ap-northeast-1; access scoped through IAM policies on the gateway ARN rather than any key; and both
domainFilter(include and exclude, 100 domains per list) andpublishedDateFilter(from/to, ISO-8601 UTC, inclusive) at target and request level from connector version 1.2.0.Two clarifications from that exchange are now in the docstrings: the caller's credentials need
bedrock-agentcore:InvokeGatewayon the gateway ARN while the gateway service role needsbedrock-agentcore:InvokeWebSearchon the connector, and request-level include lists intersect with the target-level include list rather than replacing it, so disjoint lists return no results with no error raised.Checklist