Labs Learning - April 2026
This is Part 2 of a two-part series on building AI knowledge bases. Read Part 1: Your AI Knowledge Base Is Only as Good as What You Feed It
In the last post, I talked about the data quality problem: how 87% of my AI knowledge base turned out to be noise, and why preparing your documents is the most important step in building an AI-powered search system.
This post is about what comes next. Once your data is clean, how do you actually build a system that reliably finds the right information and gives you trustworthy answers?
I'll share what I learned from building and testing a complete system against my own business content - six published books, training materials, client proposals, and research notes. Some of what I found confirmed the expert advice. Some of it contradicted it.
How an AI knowledge base actually works
Before we get into what I learned, it helps to understand the basic process. It's simpler than most technical explanations make it sound.
Step 1: Break your documents into pieces. A 200-page book is too large for an AI to process as a single unit. So the system splits it into smaller, overlapping chunks - typically a few paragraphs each. This splitting process is called "chunking," and how you do it matters a lot.
Step 2: Create a mathematical fingerprint of each piece. The system converts each chunk of text into a list of numbers (called an "embedding") that captures the meaning of the text. Chunks with similar meanings get similar numbers, which is how the system can later find content that's relevant to a question, even if the exact words are different.
Step 3: Store everything in a searchable database. All the chunks and their mathematical fingerprints go into a specialised database designed for this kind of similarity search.
Step 4: When someone asks a question, the system converts the question into the same kind of mathematical fingerprint, searches the database for the most similar chunks, and feeds those chunks to an AI model that generates a natural language answer based on what it found.
Each of these steps involves choices that affect the quality of the final result. Here's what I learned about making those choices well.
How you split your documents matters more than which AI model you use
This was the single most impactful finding. The way you break documents into chunks has a bigger effect on result quality than which AI model you use or which database you choose.
I built what I call a "chunking router" - different splitting strategies for different types of content. My books and structured documents get split by their heading structure. The system knows that a chunk came from "Chapter 3 > Pricing Strategy > Value-Based Pricing" rather than just "somewhere in the middle of a long document." That structural context makes search results dramatically more relevant.
Podcast transcripts - long conversational text with no headings - get split differently. Instead of looking for formatting, the system uses the AI to detect where topics naturally shift in the conversation and splits there.
Everything else gets a standard approach that respects sentence and paragraph boundaries.
The takeaway for business owners: if someone is building a RAG system for you and they're using a one-size-fits-all approach to splitting documents, they're leaving quality on the table. Different content types benefit from different strategies. It's not the most glamorous part of the project, but it's one of the most important.
Don't believe everything you read about what AI search can't do
The expert literature on RAG systems has a consistent warning: AI semantic search is bad at finding exact terms. If your documents mention a specific programme name, company name, or technical term, the AI search might miss it because it's looking for meaning rather than exact words. The standard advice is to add a second, keyword-based search system to catch what the AI misses.
I built that second system. Then I tested it.
For my corpus, the AI search found Irish-specific terms like "Skillnets" and "Local Enterprise Office" perfectly well on its own. The keyword search didn't change the results at all.
Why? Because my documents discuss these organisations in enough context that the AI captures the meaning, not just the word. A paragraph about Skillnets training subsidies gives the AI enough surrounding information to understand what "Skillnets" means and retrieve it when someone asks about it.
This doesn't mean the expert advice is wrong. In a sparser dataset - say, a database of invoices where "Skillnets" only appears in a company name field with no surrounding context - you'd almost certainly need keyword search. But it does mean you should test your specific situation rather than assuming the worst case applies to you.
Build the capability, but measure whether it's needed. Testing takes minutes; unnecessary architecture costs weeks.
How to tell if your system is actually working
This is where many RAG projects go wrong. They build the system, run a few queries, see reasonable-looking answers, and declare success.
Here's what I did instead: before running the first index, I wrote ten test queries with expected answers. I noted which documents each question should draw from and what key points the answer should include. I also wrote one "negative test" - a question the system should NOT be able to answer (in my case, "What are my bank account details?").
Writing the test queries before seeing any results is important. If you write them afterwards, you'll unconsciously pick questions that work and avoid questions that don't. You need to test the failure modes, not just the happy path.
My results: seven of ten queries retrieved exactly the right source documents. The remaining three retrieved different but related documents that actually contained better answers (more focused, more specific) than the ones I expected. The negative test passed perfectly - instead of real financial data, the system returned the fictional placeholder values I'd inserted during the data preparation step.
Zero hallucination across all ten queries. The AI never made anything up. When it didn't have good information, it said so rather than guessing.
You probably need less technology than the experts suggest
Most of the current guidance on building AI knowledge bases is written for large organisations with enterprise-scale requirements: thousands of users, millions of documents, complex security and access control, high availability.
If you're a small or mid-sized business with a few thousand documents and a handful of users, most of that advice will lead you toward unnecessary complexity.
I'm running my system on a small home server using straightforward, well-understood tools. The database handles 38,000 chunks with no performance issues. The search pipeline is deterministic and predictable - no fancy "agentic" loops where the AI retries queries on its own. The whole thing produced zero hallucination across my test queries.
Could I make it more sophisticated? Of course. There are frameworks for building systems where the AI automatically rewrites your query if the first search doesn't find good results, grades its own retrieval quality, and loops until it's satisfied. These are useful capabilities for complex enterprise use cases.
But for a consulting-scale knowledge base, the simpler approach works and has significant advantages: it's easier to debug when something goes wrong, it's cheaper to run, and it's more predictable. You can always add complexity later if your evaluation shows you need it. Removing complexity after the fact is much harder.
The principle I'd encourage: start with the simplest system that gives you good results. Add sophistication only when you can point to a specific problem it solves.
What I'd recommend if you're considering this
Based on everything I've learned building and testing this system, here's what I'd suggest for a business exploring AI-powered knowledge search:
Start with your data, not the technology. Audit what you have. What formats are your documents in? How much sensitive information is in there? How well-organised are your files? The answers to these questions will determine 80% of your project timeline.
Be realistic about formats. Not every document type works well for AI search. Text-based formats are ideal. Presentations and PDFs may need conversion. Spreadsheets generally don't belong in a knowledge base without additional context.
Write your test questions first. Before building anything, write down the ten questions you most want the system to answer. Include at least one question it should refuse to answer. This gives you a clear success criterion and prevents the project from becoming an endless optimisation exercise.
Don't over-engineer for scale you don't have. A small business with 500 documents doesn't need the same infrastructure as a bank with 50 million documents. Choose tools that match your actual scale, not your theoretical future scale.
Plan for maintenance. A knowledge base isn't a one-time build. Documents change, new files are added, old ones become irrelevant. You need rules for what goes in and what doesn't, and a process for keeping the corpus clean over time. Without this, quality degrades silently.
The honest summary
Building a good AI knowledge base is more achievable than most technical guides make it sound, but also more dependent on careful data preparation than most vendors will tell you. The AI part is impressive - zero hallucination, accurate source attribution, natural language answers grounded in your actual documents. But that quality is entirely a function of what you put into the system and how carefully you prepare it.
The technology works. The question is whether you're willing to do the unglamorous preparation work that makes it work well.
Read the previous post: Part 1 - Your AI Knowledge Base Is Only as Good as What You Feed It
I help organisations adopt AI practically and responsibly. If you're thinking about building an AI knowledge base and want to understand whether it's the right fit for your situation, the AI Opportunity Audit is a 90-minute session where we work through that question together. Details at humanspark.ai
