On 19 June 2026, AWS made Web Search on Bedrock AgentCore generally available. We had been evaluating AgentCore for other components in an agent project, and the Web Search tool was interesting enough that we wanted to spend some time understanding how it actually behaves. This post walks through what AWS says about the tool, the small experiment we designed to test some of those claims, and what we found.
A quick note before we start: Since AWS regularly ships updates to AgentCore and the Web Search is one of the newer components, this blog reflects only our experience with how the tool behaved in mid-July 2026. You might notice small differences if you try it now.
What AWS says about it
The tool is exposed as a managed target on the AgentCore Gateway, invoked via the standard Model Context Protocol. In AWS's own framing, it is a "fully managed tool that enables agents to ground responses in current, cited web knowledge with zero data egress from customer's secured AWS environment".
Two things about that description are worth pulling out. The first is that AWS is explicit that the tool is not a wrapper over a third-party search API. The launch materials describe it as backed by "a purpose-built Amazon-operated web index rather than a wrapper over a third-party search engine", one that spans tens of billions of documents and "refreshes continually, reflecting new content within minutes". The second is that the tool is priced simply: $7 per 1,000 queries, no upfront commitment, currently available only in us-east-1.
The mechanics are straightforward, too. Your agent calls "tools/call with a natural language query (up to 200 characters)", the Gateway routes the request internally within AWS, and the tool returns "semantically relevant text snippets, URLs, titles, and publication dates in both text and structured JSON formats".
In short, AWS says that it’s providing a managed, MCP-compliant, Amazon-indexed search tool with transparent pricing and a clear API contract. We wanted to find out how it performs when put to the test with real-world questions.
Designing the experiment
We did not want to run a benchmark. A benchmark typically asks questions about the quality of a given feature and expects some scalar quantitative answer. Our goal was a little different: we wanted to explore the tool’s behavioural quirks that a marketing page often does not reveal to users. But these quirks are still important and often decide how we wire the tool into an agent. Therefore, we needed a small set of queries chosen to probe specific characteristics, rather than a large corpus of representative traffic.
We designed six tests, each targeting one property of the tool:
- Baseline (Test 1): Does asking for recent AWS content actually return recent AWS content? The least interesting test and also the most necessary, because if the answer is no, nothing else matters.
- Determinism (Test 2): Do we get the same top-10 results back if we run the same query three times, 30 seconds apart? Determinism matters when the same tool call might be repeated inside a reasoning loop or between test setups.
- Query-length limit (Test 3): The official AWS docs state that the query is capped at 200 characters, but they do not specify what happens when you exceed that limit. We wanted to know whether the service rejects the request cleanly at the HTTP layer or lets it through and silently returns nothing.
- Phrasing sensitivity (Test 4): Three phrasings of the same information intent, each run three times. Combined with Test 2, this tells us how much of the tool's behaviour is stable per query versus how much is sensitive to the query's wording.
- Older content (Test 5): AWS describes an index that refreshes continuously to reflect the current content. We were curious whether that recency focus meant the index would struggle with older material, so we asked about the 2018 AWS Lambda launch.
- Response shape (Test 6): A passive audit. The docs mention both text and structured JSON formats, so we saved every raw response and tallied the shapes at the end.
The whole thing ran in about ten minutes. Total spend: $0.105 across sixteen paid queries. We saved the full audit trail for every call, so any observation below can be traced back to a specific response on disk.
Determinism holds, at least at short timescales
The determinism observation turned out to be the strongest finding, as it was the most consistent throughout the run. Every time we sent the same query 30 seconds later, we got the same ten URLs back, in the same order. When we did the same thing with three different queries in the fourth test, each of them was also deterministic across its three repeats - twelve data points in total, and no drift in any of them.
Whether AgentCore's underlying index, which AWS describes as "purpose-built and Amazon-operated", is genuinely stable at this timescale or whether the Gateway is caching aggressively, we cannot tell from the client side. Both behaviours would produce the same observation. What we can say is that within a short evaluation window, the tool does not re-rank or re-select its top-10. Whether that holds over hours, days, or weeks is a separate question we did not test.
This matters for how you write agent code in three specific ways:
- If your test suite runs the same query in setup and teardown, you will not see drift within a single session, which is convenient for deterministic tests but might mask the real variance you would see across sessions.
- If your agent retries a failed tool call quickly, it will get the same results back, unlike many raw search APIs that would re-rank.
- If your agent's reasoning chain issues the same query twice, hoping for different information the second time, it will be disappointing. You would need to vary the wording.
The 200-character limit is a silent failure, not a loud one
The docs say the query is capped at 200 characters. Our test sent a 154-character query (which should succeed) and a 220-character query (which should fail). Both came back with HTTP 200. Neither returned an error status code.
The 154-character query worked as expected and returned ten results in the standard shape. However, with the 220-character query, we noticed something interesting. Although the response was well-formed at the HTTP layer, the body contained no results field in any shape we know about. As a result, our parser saw zero results but no errors.
This is a subtle failure mode. If you had built your agent to treat non-200 status codes as the signal that a tool call failed, a too-long query would slip past that check, return an empty result set, and your agent would proceed as if it had asked a valid question and simply not found anything. That is a very different bug from a clean 400 rejection. It will show up in your logs as "tool call succeeded but returned no useful information," and you will spend a while trying to fix the wrong problem.
Our takeaway from this test isn’t that "the limit is exactly 200 characters," because we did not test the exact boundary. Instead, this test shows that we need to keep our queries short, and if we get a suspiciously empty result set, we need to check the query length before assuming the tool just did not know the answer.
Small wording changes produce very different result sets
This one surprised us the most. The fourth test compared three phrasings of the same information intent: "AWS Bedrock new features 2026", "what's new in AWS Bedrock", and "AWS Bedrock AgentCore release notes". A human reading those three queries would consider them closely related. The tool did not.
The two keyword-style phrasings (variants 1 and 2) share 67% of their URLs. That is real overlap; the queries are pulling from the same corner of the index. But adding the word "AgentCore" to variant 3 reduced the URL overlap with the other two to 6%. And that is not a re-ranking. It is a different corpus.
To make that concrete:
- Variant 1 ("AWS Bedrock new features 2026") surfaced generic Bedrock announcements: OpenAI model launches on Bedrock, the redesigned console optimised for OpenAI and Anthropic APIs, weekly roundup posts, and general what's-new pages.
- Variant 3 ("AWS Bedrock AgentCore release notes") surfaced almost exclusively AgentCore-specific pages: AgentCore Harness general availability, AgentCore Runtime, new AgentCore features, and a GitHub release-notes file from a third-party agentcore package.
The tool responded to the word "AgentCore" as if it were a routing token, moving the search to a different neighbourhood of the index rather than narrowing within the previous one.
Combined with the determinism observation, this has a real design implication. If your agent constructs its own queries, small wording differences that a human would consider synonymous will not be treated as synonymous by the tool. And because a given query is deterministic, an agent that gets stuck on one phrasing will never see the content a different phrasing would surface. If you want breadth, you have to vary the wording explicitly. The tool will not do it for you.
The index leans hard on Amazon's own channels for AWS queries
The baseline test asked "latest AWS Bedrock announcements 2026" and returned ten URLs. Eight of them were from aws.amazon.com or aboutamazon.com. When we asked about the 2018 AWS Lambda launch, seven of ten again came from those two domains, with the rest split between Wikipedia, Computerworld, and a Palo Alto Networks blog.
This is not a criticism. AWS's launch materials are explicit that the tool is "backed by a web index that Amazon builds and operates directly". If you are asking about AWS products, having the top results come from AWS's own channels is exactly right. But for questions where you would want a third-party perspective, or anything comparative, the index will pull you toward Amazon-published content by default. Worth being explicit about your assumption before you point the tool at every retrieval task.
Two smaller things we saw
On the response shape: We saw a single form across all sixteen queries. Every successful call returned its results inside a JSON-encoded string within a content-block of type text, not directly under a structuredContent field. The docs describe both text and structured JSON formats, so the other shape is presumably legal too, but we did not see it appear from normal search calls. If you are writing a parser, start with the text-block path and handle the structured shape defensively for the case where it does appear.
On the 2018 Lambda query: The results were actually about the 2018 launch: the original AWS blog announcement, the corresponding Amazon press release, a Wikipedia entry for AWS Lambda, and a "Lambda turns ten" retrospective. The index does not seem to have a strong recency bias that hides older material, even though AWS describes it as refreshing continually for current content.
What this experiment did not tell us
Despite running sixteen queries in a single session, there are many things that this experiment did not answer:
- Whether determinism holds over hours or days rather than seconds?
- How does the tool behave under concurrent load?
- How does the maxResults parameter change the picture?
- Whether the wording-sensitivity finding generalises to query domains we did not probe, such as technical documentation or news?
- What happens if you invoke it from a region other than us-east-1, and how do the answers change when the underlying index refreshes?
These questions are very important, and the answers to any of these could shift what we would say. However, after our experiment, we can confidently say that AgentCore Web Search is a well-behaved managed tool with a few characteristics: short-window determinism, silent failure on overly long queries, real sensitivity to query wording, and an index that leans heavily on Amazon's own channels for AWS content. We feel that these results provide additional information and help shape how a user can effectively wire the tool into an agent flow, more than the launch blog lets on.
Cost is low enough that you can run a similar experiment yourself in an afternoon and see what your workload actually looks like against it. That is probably the most useful thing you can do before designing around any managed search tool, not just this one.
References
[1] Announcing Web Search on Amazon Bedrock AgentCore: Ground your AI agents in current, accurate web knowledge. https://aws.amazon.com/blogs/aws/announcing-web-search-on-amazon-bedrock-agentcore-ground-your-ai-agents-in-current-accurate-web-knowledge/
[2] Web Search Tool, Amazon Bedrock AgentCore Developer Guide. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-target-connector-web-search-tool.html
[3] Introducing Web Search on Amazon Bedrock AgentCore. https://aws.amazon.com/blogs/machine-learning/introducing-web-search-on-amazon-bedrock-agentcore/
[4] Amazon Bedrock AgentCore Pricing. https://aws.amazon.com/bedrock/agentcore/pricing/