Hour 3 - Understanding Chunks

Lecture Notes: 

---

1. Concepts

In Ollama, chunks refer to segments of data or text that are processed by the model for training or inference. Breaking large inputs into manageable chunks ensures efficient computation and prevents memory overflow.

Why Chunks are Important:

• Efficiency: Allows processing of large datasets by dividing them into smaller parts.
• Accuracy: Helps the model maintain context within its processing limits.
• Compatibility: Ensures inputs fit within the model's context window.

Chunking Strategies:

1. Token-based Chunking: Divides text based on the number of tokens.
2. Sentence-based Chunking: Divides text at sentence boundaries for better coherence.
3. Custom Chunking: Tailored to specific tasks like splitting code blocks or paragraphs.

---

2. Key Aspects

Key Components of Chunking:

• Token Limit: Each model has a context window, e.g., 2048 or 4096 tokens. Inputs exceeding this must be chunked.
• Overlap: Adding overlapping text between chunks maintains context.
• Chunk Size: Balance between efficiency and coherence; usually a few hundred tokens.

---

3. Implementation

Step-by-Step: Chunking Text for Ollama

1. Choose a Chunking Strategy:
   Decide between token-based, sentence-based, or custom chunking based on the task.

2. Set the Context Window:
   Identify the model's token limit (use ollama show model_name).

3. Implement Chunking:
   Use a script to divide the text into chunks within the token limit.

4. Run Chunks through Ollama:
   Process each chunk sequentially and combine the outputs.

---

4. CLI Commands for Working with Chunks

Command	Description	Example
ollama show	Displays model details, including the context window size.	ollama show llama3.1
ollama run	Runs a model on a single chunk or input.	ollama run llama3.1 --prompt "Hello"
ollama run --format	Outputs results in JSON format, useful for processing chunked outputs.	ollama run llama3.1 --format json
ollama create	Creates a model optimized for specific chunk sizes or use cases.	ollama create chunk_model -f ./modelfile

---

5. Real-Life Example

Scenario: Processing a Large Document for Summarization

Suppose you have a large article that exceeds the context window of the llama3.1 model. You can split the text into chunks, process each chunk, and combine the summaries.

---

6. Code Examples

Token-Based Chunking in Python

from transformers import GPT2Tokenizer

# Initialize tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Define text and token limit
text = "Large text input that needs to be chunked..." * 100
token_limit = 2048

# Split into chunks
def chunk_text(text, token_limit):
    tokens = tokenizer.encode(text)
    chunks = [tokens[i:i+token_limit] for i in range(0, len(tokens), token_limit)]
    return [tokenizer.decode(chunk) for chunk in chunks]

chunks = chunk_text(text, token_limit)

# Print chunk sizes
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: {len(tokenizer.encode(chunk))} tokens")

Processing Chunks with Ollama

# Save chunks to a file
echo "Chunk 1 text" > chunk1.txt
echo "Chunk 2 text" > chunk2.txt

# Process each chunk
ollama run llama3.1 --prompt "$(cat chunk1.txt)"
ollama run llama3.1 --prompt "$(cat chunk2.txt)"

Combining Outputs

outputs = ["Summary of chunk 1", "Summary of chunk 2"]
combined_summary = " ".join(outputs)
print("Combined Summary:", combined_summary)

---

7. Summary

• Concepts Covered: Importance of chunks, chunking strategies, and context windows.
• Key Aspects: Token limits, overlap, and chunk size considerations.
• CLI Commands: Commands for inspecting models and processing chunks.
• Real-Life Example: Summarizing large documents by chunking.
• Code Examples: Implementing chunking and processing in Python and Bash.

---

8. Homework/Practice

1. Use ollama show to check the context window of a model on your system.
2. Implement a chunking script in Python or another language.
3. Process a large document by dividing it into chunks and running each through Ollama.
4. Experiment with different chunk sizes and overlaps to observe their effects on the output.

---

These lecture notes provide a hands-on understanding of chunking in Ollama with practical examples and real-world scenarios.