RecursiveCharacterTextSplitter

Idle

Algorithm (Python Logic)

def _split_text(text, separators):
# 1. Choose Separator
separator = get_separator(text, separators)
 
# 2. Split Text
splits = text.split(separator)
_good_splits = []
 
# 3. Iterate Splits
for s in splits:
if len(s) < chunk_size:
_good_splits.append(s)
else:
# If buffer has items, we MUST flush them
# before processing the big chunk 's'
if _good_splits:
merge_splits(_good_splits, separator)
_good_splits = []
 
# 4. Recursion on big chunk
if new_separators:
_split_text(s, new_separators)
else:
force_emit(s)
 
# 5. Flush Remaining
if _good_splits:
merge_splits(_good_splits, separator)
 
 
def merge_splits(splits, separator):
current_doc, total = [], 0
for d in splits:
if total + len(d) > chunk_size:
emit_chunk(current_doc)
while total > overlap: 
current_doc.pop(0)
current_doc.append(d)
emit_chunk(current_doc)

Execution Log Step: 0

Ready to split...

_good_splits (Input Buffer)

Buffer is empty

Merge Process (Accumulator)

Length: 0

Waiting for merge_splits()...

Final Chunks 0 chunks

Input Text

Chunk Size 100

Overlap 20