When I built the first serious RAG for semantic search in JXBS, I spent days comparing embedding models and trying ever-bigger LLMs on top of the retriever. The real quality jump didn't come from there. It came from fixing how I split documents before indexing them. Chunking —that boring part everyone solves with a character split and moves on from— is the decision that most determines whether your RAG answers well or hallucinates with confidence. This article is what I learned by retrieving useless fragments the hard way.
The retriever can only return what you indexed well
A RAG has two halves: retrieve the relevant fragments and generate an answer from them. The second half gets all the attention because that's where the flashy LLM lives. But generation is bounded by what reaches it: if the retriever brings back three mediocre fragments, no model, however large, will invent the missing context. At best it will fake it, which is worse.
And what the retriever can return depends entirely on how you cut the text. An embedding represents the meaning of a whole fragment. If that fragment mixes two distinct ideas, its vector lands at an in-between point that represents neither well, and similarity search skips it exactly when you need it. The problem isn't the embedding model: it's that you handed it a piece of text with no clear idea inside.
Splitting by characters is the default mistake
The recipe almost every tutorial ships is: cut every 1000 characters with 200 of overlap. It's convenient and it's what I did at first. The problem is that 1000 characters mean nothing semantically: the cut falls mid-sentence, separates a definition from its example, or puts the end of one section and the start of the next in the same chunk. You end up with vectors representing fragments split down the middle.
The first cheap fix is to split by structure, not by length. Documents already come with semantic boundaries: paragraphs, headings, list items. Cutting along those boundaries makes each chunk hold a reasonably complete idea.
// Instead of blindly chopping every N characters,
// respect the document's natural boundaries
function chunkByStructure(markdown: string): string[] {
// Split by section headings first
const sections = markdown.split(/\n(?=#{1,3}\s)/);
return sections.flatMap((section) => {
// A short section is a whole chunk
if (section.length <= 1200) return [section];
// A long one subdivides by paragraphs, not characters
return section
.split(/\n\n+/)
.reduce<string[]>((acc, para) => {
const last = acc[acc.length - 1];
if (last && (last + '\n\n' + para).length <= 1200) {
acc[acc.length - 1] = last + '\n\n' + para;
} else {
acc.push(para);
}
return acc;
}, []);
});
}
It isn't sophisticated, but the quality jump over blind cutting is immediate because each vector now represents a unit of meaning rather than an arbitrary slice.
Chunk size is a trade-off, not a magic number
There's no correct value here, there's a tension you have to resolve for your case. Small chunks give very precise embeddings —the vector represents a concrete idea— but they fragment context: the answer to a question can be spread across five slices and the retriever only brings back the three most similar. Large chunks preserve context but dilute the embedding: the more text you cram in, the more the meaning averages out and the less the search discriminates.
My practical rule is to start with medium chunks, the size of a couple of paragraphs covering a single subtopic, and adjust by watching what the system retrieves on real queries. If I see correct answers getting cut off, I raise the size; if I see fragments that talk about several things at once, I lower it. Overlap between adjacent chunks helps so an idea straddling two slices isn't lost, but overlap doesn't fix a cut made in the wrong place.
💡 Don't optimize chunk size in the abstract. Write ten real queries from your product, look at which fragments the system retrieves for each, and adjust from there. A manual eval of ten cases tells you more than any heuristic.
Metadata and context: the chunk doesn't live alone
A fragment ripped from the middle of a document loses where it came from. "The term is thirty days" means nothing without knowing the term of what. That's why I attach metadata to every chunk —document title, section, date— and where I can I prepend a line of context to the text that gets embedded, so the vector also captures what the fragment belongs to. That small header makes ambiguous fragments retrievable by the right query.
Metadata also gives you pre-filtering. In pgvector I can combine similarity search with a WHERE over regular columns, narrowing the search space before comparing vectors.
SELECT id, content
FROM chunks
WHERE document_type = 'contract'
AND language = 'en'
ORDER BY embedding <=> $1
LIMIT 5;
Filtering by metadata before ordering by distance doesn't just improve precision: it shrinks the set the vector search runs over, and with it the latency.
Where I put the effort today
If I had to divide up the time again, I'd spend the bulk of it on the ingestion phase —how I split, what metadata I store, how I enrich each chunk with its context— and far less trying models. The embedding model matters, and the generator LLM matters, but both operate on what ingestion prepares for them. I chose to invest in chunking instead of a bigger model because chunking is cheap to iterate and its effect is a multiplier: it improves every query the system serves, today and in the future. In exchange, it's unglamorous work that's hard to show off. It's still worth it to me.
