How Knowledge Sources Work

When you upload a document to your Chipp app, something powerful happens behind the scenes. Your files get transformed into a format that AI can search and understand semantically—not just by keywords, but by meaning. This guide explains how it works in plain terms.

The Big Picture

Here’s what happens when you add a knowledge source:

1. You upload a file (PDF, document, spreadsheet, URL, etc.)
2. We extract the text from your file
3. We split it into chunks (smaller, manageable pieces)
4. We create embeddings (numerical representations of meaning)
5. We store everything in a searchable database
6. Your AI gets a search tool. When it decides a question needs your content, it searches the knowledge base and reads the most relevant chunks before answering

Let’s break down each step.

Step 1: Text Extraction

Different file types need different handling. We support:

The goal is to get clean, readable text regardless of the original format.

Step 2: Chunking

AI models have limits on how much text they can process at once. Plus, when answering a question, you don’t need an entire 50-page document—you need the relevant paragraph.

Chunking splits your text into smaller pieces (typically a few pages worth each). We try to split at natural boundaries—between paragraphs or sections—so each chunk contains a complete thought.

Example: A 20-page PDF might become 15-20 chunks, each containing related content that can stand on its own.

Step 3: Creating Embeddings

This is where the magic happens.

What’s an Embedding?

An embedding is a list of numbers that represents the meaning of text. Think of it as coordinates in a giant map of concepts.

Simple analogy: Imagine a map where:

• “Dog” and “puppy” are placed close together (similar meaning)
• “Dog” and “cat” are somewhat close (both pets)
• “Dog” and “refrigerator” are far apart (unrelated concepts)

Embeddings work the same way, but in thousands of dimensions instead of two. Text with similar meaning gets similar numbers.

Why This Matters

Traditional search looks for exact keyword matches. If your document says “automobile” but someone searches “car,” traditional search might miss it.

Semantic search using embeddings understands that “automobile” and “car” mean the same thing. It finds relevant content based on meaning, not just matching words.

The Numbers

Each chunk of text becomes a vector of ~3,000 numbers. These numbers encode:

• What the text is about
• The context and tone
• Relationships to other concepts

You don’t need to understand the math—just know that similar content produces similar numbers, which makes searching incredibly powerful.

Step 4: Storage

We store both:

• The original text chunks (so we can show them to the AI)
• The embeddings (so we can search by meaning)

This lives in a specialized vector database optimized for finding similar embeddings quickly.

Step 5: Retrieval (Search as a Tool)

Retrieval works as a tool the AI calls, not as an automatic step bolted onto every message. When your app has knowledge sources, its AI is given a searchKnowledge tool and told your knowledge base exists. Then, for each message, the AI decides whether it needs your content:

1. The AI Decides to Search

“Hi!” doesn’t need a knowledge lookup; “What’s your refund policy?” does. When the question calls for your content, the AI calls the search tool with a query it writes itself. For complex questions it can search multiple times with refined queries, the way a person would try a second search after skimming the first results.

2. Hybrid Search

Each search combines two strategies at once: semantic search (comparing the query’s embedding against your chunk embeddings, so “car” finds “automobile”) and keyword search (classic full-text matching, so exact names, part numbers, and rare terms aren’t missed). By default the blend weighs meaning at 70% and keywords at 30%.

3. Relevance Scoring

Each chunk gets a combined score. Chunks below the relevance threshold are dropped; the top results (up to your configured chunk limit) are returned.

4. The AI Reads the Results

The search results come back to the AI as tool output: the matching chunks with their source documents. If the results don’t answer the question, it can search again with a different query.

5. AI Response

The AI answers grounded in what it actually found in your content—not just its general training data—and it knows which document each passage came from.

Why tool-based retrieval is better than searching on every message: no wasted searches (and tokens) on messages that don’t need your content, the AI writes better search queries than raw user messages (“it broke again” becomes a search for the product discussed earlier), and it can iterate when the first search misses.

Why This Approach Works

Semantic Understanding

Users don’t need to know the exact words in your documents. Ask about “pricing” and find content about “costs,” “fees,” or “rates.”

Accuracy

By giving the AI specific, relevant content, answers are grounded in your actual information rather than generic responses.

Scalability

Whether you have 10 documents or 10,000, the search remains fast because embeddings can be compared efficiently.

Relevance

Instead of dumping entire documents into the AI (which would exceed limits and include irrelevant content), we surface just the pieces that matter.

Document Boosting: Smarter Retrieval for Large Knowledge Bases

When you have dozens or hundreds of knowledge sources, finding the right information becomes more challenging. A simple chunk-by-chunk search might return text that sounds relevant but comes from the wrong document entirely.

This is where document boosting comes in.

The Problem with Pure Chunk Search

Imagine you have 50 documents uploaded:

• Employee Handbook
• API Documentation
• Password Reset Guide
• System Administration Manual
• …and 46 more

A user asks: “How do I reset my password?”

Pure chunk search might find:

1. “Reset API tokens in the developer console” (API docs)
2. “Reset your password in account settings” (Password Guide)
3. “Reset deployment configuration” (Admin Manual)

The first result has high text similarity to “reset” but comes from the wrong context entirely. The user wanted account passwords, not API tokens.

How Document Boosting Solves This

We use a two-tier approach:

Tier 1: Document-Level Understanding

When you upload a document, we also generate an AI summary of the entire document and create an embedding for it. This captures what the document is about at a high level.

For example:

• Employee Handbook → “Company policies covering PTO, benefits, conduct…”
• API Documentation → “Technical reference for developer integrations…”
• Password Reset Guide → “Instructions for resetting user account passwords…”

Tier 2: Combined Scoring

When document boosting is enabled and the AI runs a search, we:

1. First find which documents are most relevant to the query
2. Then search for chunks, but boost chunks from relevant documents

For “How do I reset my password?”:

• Password Reset Guide gets a high document relevance score (0.92)
• API Documentation gets a low score (0.45)

A chunk from the Password Guide with slightly lower text similarity (0.78) now outranks a chunk from API docs with higher text similarity (0.85)—because it comes from the right context.

Why Document Summaries Matter

Document summaries serve as a “table of contents” for your entire knowledge base. They help the system understand:

• What each document is about
• Which documents are relevant to a query
• How to prioritize chunks from contextually appropriate sources

This is especially powerful when you have:

• Many similar documents (multiple policy manuals, product guides)
• Overlapping terminology (technical terms used in different contexts)
• Large knowledge bases (hundreds of documents with thousands of chunks)

The Result

Document boosting delivers more accurate answers by understanding context at both the document and chunk level. Users get responses from the right source, not just text that happens to contain similar words.

Real-World Example

Scenario: You’ve uploaded your company’s 50-page employee handbook.

User asks: “How many vacation days do I get?”

What happens:

1. The AI decides this question needs your content and calls its search tool
2. The search runs against your handbook chunks (meaning + keywords combined)
3. Finds the chunk about PTO policy (high score)
4. Also finds related chunks about holiday schedule (moderate score)
5. The AI reads the results
6. AI responds: “According to the employee handbook, full-time employees receive 15 vacation days per year, plus 10 company holidays…”

The AI didn’t have to read all 50 pages—just the relevant sections.

Tips for Better Results

1. Quality Content

The AI can only find what’s in your documents. Make sure your knowledge sources contain the information users will ask about.

2. Clear Writing

Well-organized, clearly written content produces better embeddings. If a human would struggle to understand your document, the AI will too.

3. Comprehensive Coverage

If users frequently ask questions you can’t answer, consider adding more knowledge sources to cover those topics.

4. Keep Sources Updated

Outdated documents lead to outdated answers. Regularly refresh your knowledge sources with current information.

Supported File Types

Using with MCP

You can manage knowledge sources programmatically using the Chipp MCP Server, enabling bulk operations and automated workflows.

Available Tools

Example Prompts

"Add https://docs.example.com to my FAQ app with crawl depth 2"
"List all knowledge sources for app 123"
"Add this YouTube video to my training bot"
"Refresh all outdated knowledge sources"

Workflow: Bulk Knowledge Migration

You: List knowledge sources in my old FAQ app (ID 100)

AI: [Calls list_knowledge_sources]
Found 8 sources:
- docs.example.com (URL, COMPLETED)
- Product Manual.pdf (FILE, COMPLETED)
- FAQ-2024.docx (FILE, COMPLETED)
...

You: Add all the URLs to my new FAQ app (ID 200)

AI: [Calls add_url_source for each URL]
Added 5 URL sources to app 200. Processing will complete
in a few minutes.