A Guide to Chunking Strategies for Retrieval Augmented Generation (RAG)

Introduction

Retrieval Augmented Generation (RAG) has been a hot topic in understanding, interpreting, and generating text with AI for the last few months. It's like a wonderful union of retrieval-based and generative models, creating a playground for researchers, data scientists, and natural language processing enthusiasts, like you and me.

To truly control the results produced by our RAG, we need to understand chunking strategies and their role in the process of retrieving and generating text. Indeed, each chunking strategy enhances RAG's effectiveness in its unique way.

The goal of chunking is, as its name says, to chunk the information into multiple smaller pieces in order to store it in a more efficient and meaningful way. This allows the retrieval to capture pieces of information that are more related to the question at hand, and the generation to be more precise, but also less costly, as only a part of a document will be included in the LLM prompt, instead of the whole document.

Let's explore some chunking strategies together.

The methods mentioned in the article you're about to read usually make use of two key parameters. First, we have [chunk_size]— which controls the size of your text chunks. Then there's [chunk_overlap], which takes care of how much text overlaps between one chunk and the next.

Character Chunking

First things first, we have Character Chunking. This strategy divides the text into chunks based on a fixed number of characters. Its simplicity makes it a great starting point, but it can sometimes disrupt the text's flow, breaking sentences or words in unexpected places. Despite its limitations, it's a great stepping stone towards more advanced methods.

Now let’s see that in action with an example. Imagine a text that reads:

Dive into the joy of chunking, where each piece is a puzzle of its own. As you assemble them, a mosaic of understanding takes shape. This engaging mental exercise sparks creativity and hones analytical skills. It's like solving a puzzle, finding satisfaction in each arrangement. Approach chunking with curiosity and a playful spirit. Let it be an intellectual playground, making the process not only enjoyable but deeply satisfying. Happy chunking!

If we decide to set our chunk size to 100 and no chunk overlap, we'd end up with the following chunks. As you can see, Character Chunking can lead to some intriguing, albeit sometimes nonsensical, results, cutting some of the sentences in their middle.

By choosing a smaller chunk size,  we would obtain more chunks, and by setting a bigger chunk overlap, we could obtain something like this:
‍

Also, by default this method creates chunks character by character based on the empty character [’ ’]. But you can specify a different one in order to chunk on something else, even a complete word! For instance, by specifying [' '] as the separator, you can avoid cutting words in their middle.

Recursive Character Chunking

Next, let's take a look at Recursive Character Chunking. Based on the basic concept of Character Chunking, this advanced version takes it up a notch by dividing the text into chunks until a certain condition is met, such as reaching a minimum chunk size. This method ensures that the chunking process aligns with the text's structure, preserving more meaning. Its adaptability makes Recursive Character Chunking great for texts with varied structures.

Again, let’s use the same example in order to illustrate this method. With a chunk size of 100, and the default settings for the other parameters, we obtain the following chunks:

Document Specific Chunking

Document Specific Chunking is a strategy that respects the document's structure. Rather than using a set number of characters or a recursive process, it creates chunks that align with the logical sections of the document, like paragraphs or subsections. This approach maintains the original author's organization of content and helps keep the text coherent. It makes the retrieved information more relevant and useful, particularly for structured documents with clearly defined sections.

Document Specific Chunking can handle a variety of document formats, such as:

• Markdown
• HTML
• Python
• etc

Here we’ll take Markdown as our example and use a modified version of our first sample text:

# Unraveling the Joy of Chunking: A Puzzle of Delight 🧩

Dive into the joy of **chunking**, where each piece is a *puzzle* of its own. As you `assemble` them, a mosaic of understanding takes shape. This engaging mental exercise sparks creativity and hones analytical skills, turning your mind into a playground of discovery.

> Chunking is not just a task; it's an intellectual adventure, a series of small victories that culminate in a deep sense of accomplishment.

Approach chunking with curiosity and a playful spirit. Experiment with different strategies:

- **Character Chunking**: Divide the text based on a fixed number of characters.
- **Recursive Character Chunkiang**: Take it a step further by dividing until a certain condition is met.
- **Document Specific Chunking**: Respect the document's structure for a coherent breakdown.
- **Semantic Chunking**: Group chunks based on semantic relationships for meaningful insights.
- **Agent Chunking**: Imagine having an assistant navigate and chunk the information for you.

Let chunking be an _intellectual playground_, making the process not only enjoyable but deeply satisfying. Happy chunking! 🧩

‍
The result is the following:

You can see here that with a chunk size of 105, the Markdown structure of the document is taken into account, and the chunks thus preserve the semantics of the text!

Token-based Chunking

Language models used in the rest of your possible RAG pipeline have a token limit, which should not be exceeded. When dividing your text into chunks, it's advisable to count the number of tokens. Plenty of tokenizers are available. To ensure accuracy, use the same tokenizer for counting tokens as the one used in the language model.

Consequently, there are also splitters available for this purpose.

For instance, by using the [SpacyTextSplitter] from LangChain, the following chunks are created:
‍

  
Dive into the joy of chunking, where each piece is a puzzle of its own.
  
As you assemble them, a mosaic of understanding takes shape.
  
This engaging mental exercise sparks creativity and hones analytical skills.
  
It's like solving a puzzle, finding satisfaction in each arrangement.
  
Approach chunking with curiosity and a playful spirit.
  
Let it be an intellectual playground, making the process not only enjoyable but deeply satisfying.
  
Happy chunking!

Semantic Chunking

Semantic Chunking considers the relationships within the text. It divides the text into meaningful, semantically complete chunks. This approach ensures the information's integrity during retrieval, leading to a more accurate and contextually appropriate outcome.

Semantic chunking involves taking the embeddings of every sentence in the document, comparing the similarity of all sentences with each other, and then grouping sentences with the most similar embeddings together.

By focusing on the text's meaning and context, Semantic Chunking significantly enhances the quality of retrieval. It's a top-notch choice when maintaining the semantic integrity of the text is vital.

However, this method does require more effort and is notably slower than the previous ones.

On our example text, since it is quite short and does not expose varied subjects, this method would only generate a single chunk.

Agent Chunking

Have you ever wondered how we, humans, would chunk? Here's a breakdown of a possible way a human would process a new document:

   1. We start at the top of the document, treating the first part as a chunk.
   2. We continue down the document, deciding if a new sentence or piece of information belongs with the first chunk or should start a new one.
   3. We keep this up until we reach the end of the document.

The ultimate dream? Having an agent do this for you. But slow down! This approach is still being tested and isn't quite ready for the big leagues due to the time it takes to process multiple LLM calls and the cost of those calls. There's no implementation available in public libraries just yet. However, Greg Kamradt has his version available here.

Conclusion

And there you have it! These chunking strategies are like a personal toolbox when it comes to implementing Retrieval Augmented Generation. They're a ton of ways to slice and dice text, each with its unique features and quirks. This variety gives you the freedom to pick the strategy that suits your project best, allowing you to tailor your approach to perfectly fit the unique needs of your work.

To put these strategies into action, there's a whole array of tools and libraries at your disposal. For example, llama_index is a fantastic tool that lets you create document indices and retrieve chunked documents. Let's not forget LangChain, another remarkable tool that makes implementing chunking strategies a breeze, particularly when dealing with multi-language data. Diving into these tools and understanding how they can work in harmony with the chunking strategies we've discussed is a crucial part of mastering Retrieval Augmented Generation.

By the way, if you're eager to experiment with your own examples using the chunking visualisation tool featured in this blog, feel free to give it a try! You can access it right here. Enjoy, and happy chunking! 😉