Why semantic search failed our AI memory system (and what we replaced it with)

Hey everyone, been a Product Hunt visitor for years but never signed up, so I thought I'd start by sharing some recent learnings that hopefully other builders will find useful.

For context, we're building FanBase Copilot, an AI assistant for content creators that learns their voice and context over time. The memory layer is critical. It's what makes the AI actually useful after the first conversation.

Our first approach seemed sound: store memories as small facts, embed them, run semantic search against user queries, pass top matches to an LLM for relevance filtering.

It worked good, but not great. Here's why:

Problem 1: Facts don't embed well "User streams on Tuesdays" or "Audience is mostly Brazilian" are sparse semantic content. Embeddings need richer context to match meaningfully.

Problem 2: Every query required an LLM pass Just to filter results. Expensive and slow.

Problem 3: Reducing everything to facts loses information Tweets, conversations, uploaded docs all flattened into the same structure. The transformation itself was lossy.

The rebuild: enrich and expand, don't reduce and replace

Now, for every piece of content (tweet, conversation, document), we extract structured metadata:

• Entities are the named nouns involved

• Tags are useful keywords

• Categories are general topics

We store the full original content alongside this metadata.

The key shift: instead of the agent dispatching verbose semantic queries like "Discord community information" (hard to match), it requests keyword searches using tags/entities/categories: [Discord, Community]

This trivially expands to everything we have on each term. Much easier matching.

We also maintain a repository of the creator's existing entities/tags/categories. The LLM reuses them when possible, which naturally creates relationships between memories over time.

Too many matches? Then we pass to an LLM to refine. But now that's the exception, not the default path.

tl;dr for anyone working on AI memory:

• Semantic search on facts is a trap

• Structured metadata + keyword matching scales better than embedding everything

• Save LLM calls for refinement, not filtering

Hope this saves someone the weeks we spent learning it the hard way.