Introducing llm-splitter — a fast, lightweight, text chunker for embeddings, LLMs, and more!

We develop impactful AI applications at Nearform, each underpinned by rigorous data insights and real-world utility. We regularly wrangle enormous text-based document stores, that we process, normalize, and refine into usable formats for upstream usage (e.g., supporting similarity search or providing specific context to LLM-based applications).

In our AI apps, we’re typically working with vector embeddings — numerical representations of text inputs which can be used to retrieve semantically similar data. A fundamental (if seemingly boring) task along the way is dividing large text documents into smaller, more manageable pieces. It’s tricky, chunks that are too large lose specificity and precision in similarity calculations, while overly small chunks lose contextual relationships that inform meaning. Separately, embedding models often have limits that require sending smaller parts of a document. Thus, deciding how to slice text for maximum semantic usefulness in an upstream application is a matter of both science and art as applied to specific use cases.

We’re going to skip over the arc of determining the optimal chunking strategy and focus on the practical end of things — once you know your chunking strategy, what is the best tool to transform your text into the right-sized chunks for your application?

Just looking in the JavaScript ecosystem, there are a number of reliable text chunking solutions. For a quick introduction to several libraries, take a peek at Phil Nash’s "How to Chunk Text in JavaScript for Your RAG Application” and then check them out at this online demo. To better understand this space, let’s look at two popular open source libraries:

@langchain/textsplitters

The popular LangChain project provides a text splitting library with a rich set of different splitters. Some particularly useful ones include the RecursiveCharacterTextSplitter which preserves language structure and TokenTextSplitter which uses the OpenAI-style tiktoken library to split text into appropriate chunks.

LlamaIndex

LlamaIndex is another popular project that provides a TypeScript framework with various text parsers and splitters. Some useful ones here include SentenceSplitter for splitting text into sentences, MarkdownNodeParser for handling markdown text, and CodeSplitter, for — you guessed it — source code.

These two libraries are powerful, flexible, and fantastic to use if you’re already using the framework behind them in your application. But they’re also quite heavy — @langchain/textsplitters brings 21MB of dependencies when installed, and llamaindex is even larger with a 36MB node_modules install impact.

There are many other excellent open source JavaScript text chunking libraries — and our team evaluated them across library size, chunking options (size, overlap, LLM token flexibility, paragraph/sentence support), execution speed and other criteria. But after much research and tire-kicking, we found that the collection of features we wanted most weren’t available from a single open source JavaScript library. That gap motivated us to create a new entrant in the text processing ecosystem, which we're pleased to share with the community — llm-splitter.

While llm-splitter is intentionally minimal, it's also built for extensibility and storage efficiency. Need model-specific chunking? Just plug in libraries like tiktoken to apply token-based outputs tailored to specific models.

Let's look at how it works!

Installation

First, let's install the library:

$ npm install llm-splitter

A basic example

Now, in our JavaScript code we can get to work with a simple example:

import { split } from "llm-splitter";

// Some short text we want to split!
const text = "Hello world! This is a test.";

const chunks = split(text, {
  chunkSize: 10, // split into 10 character chunks
});

console.log(chunks);
// =>
[
  { "text": "Hello worl", "start": 0, "end": 10 },
  { "text": "d! This is", "start": 10, "end": 20 },
  { "text": " a test.", "start": 20, "end": 28 }
]

Flexible options for your text-slicing use cases

With that introduction, let’s dive into the various options for split():

splitter

A function to split text into tokens.

The default is that each character becomes a "token" when creating chunks. So in the example above, tokens are each character, which we assemble into chunks of 10 characters. If we switched to something like a naive word-splitter (e.g., (text) => text.split(/\\s+/).filter(Boolean)), then each token would be a word ("Hello", "world") of varying character sizes.

A common use case for OpenAI embedding models is counting tokens with tiktoken to ensure inputs fit within model context limits, preventing embedding API calls failures. llm-splitter works well for this purpose — we even have a specific example for plugging tiktoken into a splitter function!

chunkSize

The maximum number of tokens per chunk.

This defaults to 512 tokens, but it's important to note that your token sizes change with the splitter (e.g. character vs. word-based), so adjust accordingly.

chunkOverlap

Number of tokens to include from the previous chunk for overlap.

chunkStrategy

Either character (default) or paragraph. For the character strategy, text will be split into as many tokens as can fit in a chunk without any other considerations.

By contrast, the paragraph strategy first splits text into paragraphs (at the \n\n boundary) and then when assembling chunks, won't include tokens from a new paragraph in an existing chunk unless the entire paragraph can fit. Thus, we can better preserve the semantic meaning of paragraphs more well encapsulated in chunks by reducing partial paragraphs in output chunks.

Outputs, positional data, and efficient storage!

Let’s put these options together in an example where we’ll take a string, split it into word tokens, group into paragraphs and then aim to get up to 15 words in a chunk with 2 words of overlap from the previous chunk.

Note that llm-splitter takes a greedy approach that performs a single pass on the input for speed. This means that the chunks created might not be the optimal set like LangChains’s RecursiveCharacterTextSplitter. But the approach is simple and fast — and that’s the tradeoff we make.

import { split } from "llm-splitter";

const text = `
Some say the world will end in fire,
Some say in ice.

From what I've tasted of desire
I hold with those who favor fire.

But if it had to perish twice,
I think I know enough of hate

To say that for destruction ice
Is also great
And would suffice.
`.trim();

const chunks = split(text, {
  chunkOverlap: 2,
  chunkSize: 15,
  chunkStrategy: "paragraph",
  splitter: (text) => text.split(/\s+/g),
});

console.log(JSON.stringify(chunks, null, 2));
// =>
[
  {
    "text": "Some say the world will end in fire,\nSome say in ice.",
    "start": 0,
    "end": 53
  },
  {
    "text": "in ice.\n\nFrom what I've tasted of desire\nI hold with those who favor fire.",
    "start": 46,
    "end": 120
  },
  {
    "text": "favor fire.\n\nBut if it had to perish twice,\nI think I know enough of",
    "start": 109,
    "end": 177
  },
  {
    "text": "enough of hate\n\nTo say that for destruction ice\nIs also great\nAnd would suffice.",
    "start": 168,
    "end": 248
  }
]

Examining our outputs here, we can see that the first chunk has fewer than 15 words, but couldn’t fit all words from the second paragraph in that same chunk, so started the second paragraph word tokens in the next chunk. We can also see the overlap of two words in the second chunk from the previous chunk.

The output format has two key parts — the text portion of the entry is the chunk text, similar to all other chunking libraries. The second portion, start and end, are unique features to llm-splitter — they represent positional data that can recreate the exact text in the text field when given the original input. Using this approach, we can look at our last chunk and recreate it from only the start and end with getChunk:

import { getChunk } from "llm-splitter";

const text = // ... our previous string above ...

const chunk = getChunk(text, 168, 248);
console.log(chunk);
// =>
`enough of hate

To say that for destruction ice
Is also great
And would suffice.`

The start/end index data feature can be useful for situations in which you want to store a chunk value — typically an embedding numerical array derived from the chunk — but you don’t want to store the actual text from the chunk. An exemplary situation for this scenario would be an application that needs full documents in a data store, and separately uses chunks for relevance search and context for an LLM knowledge application. For example, the documents are stored in a source of truth like S3 and the embeddings are stored in a dedicated vector store like pgvector, Pinecone, AstraDB, etc. In cases where storing the actual chunk text is duplicative and unnecessary, llm-splitter allows storage of much smaller data (two integer indexes) instead of variable-length text data.

For use cases requiring efficient storage without text duplication, llm-splitter provides an ideal solution.

llm-splitter isn’t fancy or complicated, but it offers a solid set of base options with the flexibility to cover most text chunking needs, especially for creating embeddings for use in AI applications. You can find the source code on GitHub at: https://github.com/nearform/llm-splitter. You can also see it in action on our demo page: https://nearform.github.io/llm-splitter.

At Nearform, we love both building AI applications and contributing back to the open source community. We hope that llm-splitter provides the features and flexibility you need for the next time you process large volumes of text data. We’d love to hear how your usage is going — send us an issue if you’re running into any challenges or reach out to us if you’re looking for expert help with shipping your AI applications.