RAG Chunking Strategies for Messy Real-World Documents

RAG systems break on real documents. Naive character splitting turns tables into gibberish and code blocks into syntax errors. This article covers chunking strategies that survive PDFs, multi-column layouts, mixed code and prose, and other messiness you'll actually encounter in production.

Why Naive Character Splitting Fails

Most tutorials start with text.split(".") or a fixed character count. That works for clean paragraphs from Wikipedia. Real documents contain tables, code, headers, footers, and page numbers. A character splitter doesn't know what a table row is. It will slice a row across two chunks, losing column alignment. When you later retrieve that chunk, the LLM sees orphaned numbers and no headers—useless.

Consider a PDF invoice:

| Item          | Qty | Price |
|---------------|-----|-------|
| Widget A      | 2   | $10   |
| Widget B      | 1   | $15   |

A 200-character split might cut after "Widget A | 2 | $". The second chunk gets "$10 |\n| Widget B | 1 | $15 |". No headers, no context. Retrieval returns a fragment that can't answer "how many Widget A were ordered?".

The same happens with code blocks. A Python function:

def calculate_discount(price, rate):
    if rate > 1:
        rate = rate / 100
    return price * (1 - rate)

A naive split at 100 characters might cut after if rate > 1:. The second chunk starts with rate = rate / 100. No function signature, no indentation context. The embedding model sees a floating line that could mean anything. The trade-off is clear: simple implementation buys you nothing if retrieval quality is garbage. You need structure-aware splitting.

Recursive Splitting with Separators

LangChain's RecursiveCharacterTextSplitter (and its equivalents in other frameworks) is the pragmatic default. It tries a list of separators in order: double newline, single newline, space, character. This preserves paragraph boundaries first, then sentence boundaries, then word boundaries. You configure the separators per document type.

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
# For prose-heavy documents: prioritize paragraph and sentence breaks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)
 
chunks = text_splitter.split_text(document_text)

For code, adjust separators to respect function boundaries:

code_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\nclass ", "\ndef ", "\n\tdef ", "\n\n", "\n", " "],
)

Chunk size and overlap are empirical. 500–1000 tokens (not characters) is a common range. Overlap of 10–20% prevents context loss at boundaries—the first few tokens of the next chunk appear at the end of the previous one. But overlap increases the number of embeddings, raising storage and retrieval cost. Measure against your corpus; I've seen teams use 200-token overlap on 1000-token chunks and still miss context because the separator list didn't match the document structure.

Semantic Chunking: Splitting by Topic Boundaries

Recursive splitting respects layout but not semantics. Two paragraphs separated by a blank line may still belong to the same topic. Semantic chunking uses sentence embeddings to detect topic shifts: encode each sentence, then compute cosine similarity between consecutive sentences. When similarity drops below a threshold, split.

from sentence_transformers import SentenceTransformer
import numpy as np
 
model = SentenceTransformer("all-MiniLM-L6-v2")
 
def semantic_chunk(text, threshold=0.5, min_chunk_size=3):
    sentences = text.split(". ")
    embeddings = model.encode(sentences)
    chunks = []
    current = [sentences[0]]
    for i in range(1, len(sentences)):
        sim = np.dot(embeddings[i-1], embeddings[i]) / (
            np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
        )
        if sim < threshold and len(current) >= min_chunk_size:
            chunks.append(" ".join(current))
            current = []
        current.append(sentences[i])
    if current:
        chunks.append(" ".join(current))
    return chunks

Libraries like semantic-text-splitter (Rust/JS) and LangChain's experimental semantic chunker implement this with rolling windows. The trade-off is compute cost: encoding every sentence adds latency. For static documents, pre-chunk offline. For dynamic content, evaluate whether the coherence gain justifies the latency budget. Semantic chunking shines on reports and articles with clear section breaks; it struggles on dense technical docs where topics interleave (e.g., a troubleshooting guide that jumps between error codes and solutions).

Document-Specific Parsing Before Chunking

The biggest chunking failures come from raw text extraction artifacts. PDFs with multi-column layouts, headers, and page numbers produce a single text stream that reads "left column line 1, right column line 1, left column line 2..." unless you use layout-aware parsers. Tools like PyMuPDF, pdfplumber, and Unstructured.io extract text with bounding boxes and reconstruct reading order.

import pdfplumber
 
with pdfplumber.open("invoice.pdf") as pdf:
    for page in pdf.pages:
        # Extract text with layout preservation
        text = page.extract_text(layout=True)
        # Or extract tables directly as DataFrames
        tables = page.extract_tables()

For HTML, strip markup but preserve heading hierarchy (convert <h1> to #, <h2> to ##) and table structure to markdown. For code files, keep code fences intact and split at function or class boundaries. This pre-processing step eliminates many chunking failures caused by page numbers, running headers, and column misordering. Unstructured.io provides a pipeline that handles PDFs, images (via OCR), HTML, and more, outputting structured elements you can chunk by type.

Handling Tables, Code, and Mixed Content

Tables and code need special treatment because they are not linear text.

Tables: Keep the table as a markdown table or convert to key-value pairs. If the whole table fits within the chunk size, keep it intact. If not, split by logical rows but repeat column headers in each chunk. For wide tables, consider transposing or summarizing. Example: a financial report table with 50 rows—split every 10 rows with headers repeated.

Code: Use file-level chunking that includes imports and function signatures. For inline code in text (e.g., markdown code fences), preserve the fences and avoid splitting mid-block. Set the separator for code blocks to \n```\n or \nclass , \ndef as shown earlier.

Mixed content (e.g., Jupyter notebooks, documentation with interleaved code and explanation): split by cell type but keep neighboring cells together if they form a narrative. A markdown cell explaining a function followed by a code cell that implements it should be one chunk. Merge them based on proximity or use a sliding window that spans cell boundaries.

Chunk Overlap Strategies and Retrieval Implications

Overlap is not one-size-fits-all. Two common modes:

• Sliding window: Duplicate content across chunks. The last 100 tokens of chunk N appear as the first 100 tokens of chunk N+1. This increases the chance that a query matches the boundary region, but it also duplicates embeddings, raising storage and retrieval cost. Deduplication at retrieval time (e.g., max marginal relevance) can help, but adds complexity.

• Centered window: Keep chunk boundaries at semantic breaks (paragraph ends, section headings). No duplication, but you risk missing context if the semantic break is poorly chosen. This is cheaper and often sufficient for well-structured documents.

Overlap size: 10–20% of chunk length. Too small (<5%) and you miss context; too large (>30%) and you waste tokens. Measure precision/recall on your own corpus. A common pitfall is using the same overlap for all document types—code and tables often need less overlap than prose because their boundaries are clearer.

Gotchas and Failure Modes

• Chunking after OCR: OCR errors produce fragmented text. Clean with regex (e.g., remove stray hyphens, fix common misreads) or use layout-aware OCR that outputs structured blocks. Tesseract's tsv output with bounding boxes can help you merge lines before chunking.

• Very short chunks: Headings, page numbers, standalone numbers are semantically empty. Merge them with the following chunk or filter them out entirely. A heading like "## Introduction" alone is useless; attach it to the first paragraph underneath.

• Language mix: Code comments in Spanish surrounded by English prose confuse embedding models. Consider separate chunking for code vs. prose, or use a language detection pass to split by language before chunking.

• Performance: Semantic chunking is slower than recursive splitting. For static documents, pre-chunk offline and store the embeddings. For dynamic content (e.g., live crawling), evaluate latency budget—you may need to fall back to recursive splitting with a generous overlap.

• Token counting: Chunk sizes are often specified in characters, but LLM cost is driven by tokens. A 500-character chunk of Chinese text may be 200 tokens, while 500 characters of JSON might be 600 tokens. Use a tokenizer (e.g., tiktoken) to measure chunk size, not character count. See Token counting is wrong: measure actual LLM cost for why this matters.

• Embedding model selection: The chunking strategy interacts with the embedding model. Models fine-tuned for code (e.g., code-search-msmarco) handle code blocks better than general-purpose models. If you chunk code poorly, even a good embedding model can't fix it. See Selecting an Embedding Model for Code Search for guidance.

Key takeaways

• Naive character splitting destroys structure. Always use a separator-based or layout-aware approach.
• Recursive splitting with tuned separators is the safe default. Adjust separators for code, tables, and prose.
• Semantic chunking improves coherence but costs more. Use it for static documents with clear topic shifts.
• Parse before you chunk. Layout-aware PDF parsers and HTML-to-markdown conversion eliminate extraction artifacts.
• Overlap is a tuning knob, not a fix for bad separators. Measure precision/recall on your data to set the right size.