AI Agent Memory: Add Semantic Search Without a Vector Database

📦 Clone and ⭐ stop-ai-agents-losing-memory-sample-for-aws

The agent's memory holds the answer. The user asks the question. And retrieval returns nothing.

stored:   dietary_notes: "Vegetarian; severe shellfish allergy, strictly no
          crustaceans or mollusks."

asked:    "What should I avoid eating when I go out for dinner on this trip?"

keyword scan: 4 hits, answer found: False

That's a real run, not a thought experiment. The question names no key and shares no words with the stored note, so the key-value memory from the previous post never finds it. The answer was in the store the whole time.

This is the dividing line for semantic search: do you know the key, or only the intent? When questions stop matching keys, you retrieve by meaning: embed each memory once, embed the question, return the nearest neighbors by cosine similarity. This post measures two things (whether semantic search finds what keyword search misses, and which vector store fits your deployment) using the same embeddings and the same memories in the companion repo.

(Post 2 of a series; the intro maps all the memory types. The code uses Strands Agents, an open source SDK; the pattern carries over to any agent framework.)

Why Strands Agents for this demo?

Strands makes comparing vector backends straightforward. The demo tests three vector stores (FAISS, S3 Vectors, DynamoDB Vector Search) against the same memories and same embeddings, so the comparison isolates storage and retrieval performance, not the agent framework.

Adding semantic search to an agent is just a tool:

from strands import Agent, tool

@tool
def recall_memory(query: str) -> str:
    """Search memory by meaning, not keywords."""
    # Embed the query, find nearest neighbors
    results = vector_store.search(query, top_k=3)
    return "\n".join(results)

agent = Agent(
    model=model,
    tools=[search_flights, recall_memory],
)

The recall_memory tool wraps the vector store. Swap FAISS for S3 Vectors or DynamoDB, and the agent code stays the same.

The pattern shown here (semantic recall as a tool) works in any agent framework. Strands just makes it simple to plug different backends and measure them.

Why does key-value memory miss the question?

Because a key-value read is a lookup someone designed in advance, and this question maps to no key. The demo stores 10 memories about a traveler (profile facts, notes, episodes) and asks the dinner question against three stores. The key-value store has exactly two moves, and both fail honestly:

1. Keyword scan: match question words against keys and values. It returns 4 hits, none of them the allergy note, because "avoid eating at dinner" shares no words with dietary_notes or "shellfish". Answer found: False.
2. Dump-all fallback: give the model the entire memory and let it read. It works, at a price that grows with every memory you add. For these 10 memories that's 647 characters per question; for hundreds of notes it's thousands of tokens, every single question, forever.

This isn't a bug in key-value memory. Profile lookups ("what's my preferred cabin?") stay exact, instant, and free of embedding costs, which is why the previous post built them that way. The limit only appears when the question is semantic. That's the signal to add a second way in, not to replace the first.

How does semantic search find it?

By comparing meanings instead of words. Every memory is embedded once at write time into a vector (here: Amazon Titan Text Embeddings V2, 1,024 dimensions). At query time, the question is embedded and the store returns the nearest neighbors by cosine similarity:

top hit: "Vegetarian; severe shellfish allergy, strictly no crustaceans
          or mollusks."  (score 0.231)
answer found: True

No shared words between question and note. They're close in meaning, and meaning is what got indexed. Both backends below return this same top hit, because they use the same embeddings; what differs is everything around the query.

Two implementations: FAISS to prototype, S3 Vectors to persist

Both are embedding vector stores. They use the same model (Titan V2), the same algorithm (cosine similarity), and they return the same top hit with the same score. The accuracy is identical, this is not a quality trade-off.

These are two implementations of the same idea for two different moments. FAISS is an in-process library: zero infrastructure, a pip install, running local to the process. It is how you prototype semantic search on your machine (in this demo the index is rebuilt from scratch each run; FAISS can persist to disk with faiss.write_index, but that is still a file you manage). Amazon S3 Vectors is the managed step: the index lives in a cloud bucket, reachable from any process with AWS credentials, surviving restarts with no cluster to run or scale. You reach for it when the memory has to outlive the process.

The demo uses the same AWS credentials for both: Titan embeddings via Bedrock and S3 Vectors via boto3; the same aws configure setup powers both, which is why this requires no extra setup inside a Strands Agents workflow. The demo self-provisions the bucket and index on first run: create_vector_bucket → create_index (1,024 dims, cosine) → put_vectors / query_vectors.

The cost both implementations share: embedding the question costs ~510 ms with Titan V2 in this demo. The vector query itself is small next to that, so the embedding call, not the store, is what to budget for in any latency-sensitive path.

So, do you need a vector database?

It depends on the query pattern. AWS positions S3 Vectors as "ideal for workloads where queries are less frequent", which describes agent memory exactly: an agent queries a user's memories a handful of times per conversation, not thousands of times per second.

The decision:

What this demo does not cover: FAISS and S3 Vectors are storage backends. They store vectors and retrieve by similarity. Building what to remember (extracting specific facts from conversations, deduplication, structured memory across sessions) is handled by managed memory services like Amazon Bedrock AgentCore Memory. That technique is the topic of a future post in this series.

How does the agent choose between key lookup and semantic search?

From the tool docstrings, on its own. The demo's last test attaches both recall tools to one Strands agent:

@tool
def recall_by_key(key: str) -> str:
    """Recall a memory when the question maps to a known identifier.
    Use when the user asks about a stored field: "my preferred cabin",
    "my home airport"..."""

@tool
def recall_semantic(question: str, top_k: int = 3) -> str:
    """Recall memories by meaning when no key is obvious.
    Use for open questions: "what should I avoid eating on this trip?"..."""

Asked the dinner question, the agent calls recall_semantic; asked "what cabin do I prefer?", it calls recall_by_key. No routing logic, no prompt engineering. The when to use this sentence at the top of each docstring is what the model reads to decide. Write that sentence carelessly and the agent pays embedding latency for profile lookups.

How do you ask an AI coding assistant to build this?

The quality of the semantic search implementation your assistant builds depends on the decisions you name in the prompt. Unnamed, it will default to embedding everything and querying one big index. These five instructions encode what this post measured:

1. "Add semantic search only for questions that don't map to keys; keep profile facts in key-value state." Otherwise the assistant defaults to embedding every query, including exact lookups that already have a known key.
2. "Embed each memory once, at write time; only the question gets embedded at query time." Assistants love re-embedding the whole store per query.
3. "Use one embedding function for storage and queries, and state the model and dimensions." Mixed embedders produce silent garbage similarity scores.
4. "Give me two recall tools with 'when to use' docstrings: by key, and by meaning." The agent routes per question from those sentences; no router code.
5. "Make persistence explicit: in-process index for a prototype, managed vector storage for anything that must survive a restart, and prove it with a fresh-client test that still sees every vector."

The companion repo implements and measures all five. Run it to see each decision play out.

How do you run the demo?

git clone https://github.com/elizabethfuentes12/stop-ai-agents-losing-memory-sample-for-aws
cd stop-ai-agents-losing-memory-sample-for-aws/02-vector-memory-demo
uv venv && uv pip install -r requirements.txt
uv run python test_vector_memory.py

Needs AWS credentials (aws configure) for Titan embeddings and S3 Vectors. The demo creates the vector bucket and index automatically if they don't exist. OPENAI_API_KEY is only needed for the agent conversation in the notebook (or swap one line for Amazon Bedrock); the retrieval measurements run without any LLM.

FAQ

Is a vector database the same as AI agent memory?
No. A vector database is one possible backend for one memory type (retrieval by meaning). Agent memory is the whole system: key-value state, vector or graph storage, selection rules, and hygiene. Many production agents need vector retrieval without a vector database.

Can I use a vector database as agent memory?
Yes, for memories you'll query by meaning. But route keyed facts (preferences, settings) to key-value storage first: a direct lookup costs nothing, while every vector query pays the question-embedding call (~510 ms with Titan V2) before the index is even touched.

When do I need something beyond S3 Vectors?
When your query pattern changes. Dedicated vector databases such as OpenSearch, Qdrant, Weaviate, Milvus, pgvector, and Chroma are built for high QPS, hybrid keyword+vector search, aggregations, and advanced filtering. S3 Vectors is purpose-built for infrequent queries: it handles up to 2 billion vectors per index with subsecond latency, which covers agent memory workloads well past prototype scale.

Is the vector store my latency bottleneck?
No. In this demo the vector query is small next to embedding the question (~510 ms with Titan V2), which both implementations pay. Whether you prototype with FAISS in-process or persist to S3 Vectors, budget for the embedding call, not the index lookup.

Why did my semantic search return the wrong memories?
The most common causes: the store and the queries use different embedding models or dimensions, memories were embedded with stale text, or keyed facts polluted the index. Keep one embedder for everything, embed at write time, and keep profile facts out of the vector store.

Resources

• Companion repo: demo 02 with the measured tests and notebook
• Amazon S3 Vectors User Guide and limitations
• Amazon Titan Text Embeddings V2
• FAISS, Meta's similarity search library
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory, Rasmussen et al., 2025
• From RAG to Memory: Non-Parametric Continual Learning for LLMs (HippoRAG 2), 2025

Gracias!

🇻🇪 Dev.to Linkedin GitHub Twitter Instagram Youtube

Elizabeth Fuentes LFollow

I help developers build production-ready AI applications through hands-on tutorials and open-source projects.