Levelling up: from Chatbots to Custom AI with RAG and Local LLMs
The following is an AI generated summary of the video
_See my June 2026 blog post for more recent thoughts on this topic_
AI as a Force Multiplier: From Chatbots to Personal Knowledge Systems
Like many people in technology, I've spent the last couple of years experimenting with AI.
Some engineers are excited.
Some are sceptical.
Some are worried.
Most seem to be all three at different times.
What I've found most useful is thinking of AI not as a replacement for engineers, but as a force multiplier.
Used well, it can dramatically increase what a single person can accomplish.
Level One: Talking to a Chatbot
Most people's journey with AI starts with a chatbot.
You ask questions.
It explains concepts.
It writes snippets of code.
It summarises documentation.
It helps you get unstuck.
For software professionals, this is already enormously valuable.
Need a quick script to transform some data?
Need a summary of a cloud service?
Need help understanding an unfamiliar technology?
A chatbot can often get you moving faster than a search engine.
In many ways, AI has become the replacement for the search experience many of us used to rely on.
Writing Code Was Never the Main Job
One insight that keeps surfacing is that coding itself is only a small part of software development.
Software professionals spend time:
- Understanding requirements.
- Designing systems.
- Communicating with stakeholders.
- Troubleshooting problems.
- Reviewing changes.
- Maintaining infrastructure.
- Investigating incidents.
- Learning new technologies.
Writing code is important, but it's only one component of a much larger process.
That's why AI-generated code can be useful without replacing the profession.
The bottleneck was rarely typing speed.
Level Two: AI-Assisted Development
The next step is integrating AI directly into the development workflow.
Rather than copying code into a chatbot, AI tools can analyse repositories, suggest changes, identify bugs, and generate pull requests.
I've experimented with this on personal projects and found it surprisingly useful.
The most interesting cases weren't writing new code.
They were discovering things I hadn't noticed:
- Misconfigured CSS.
- Minor bugs.
- Small inconsistencies.
- Repetitive maintenance tasks.
It's less like hiring a senior engineer and more like having an enthusiastic assistant who never gets tired.
You still need to review everything.
But it can save a significant amount of effort.
Level Three: Running Models Locally
One concern many engineers have is privacy.
Uploading personal or proprietary information to external services isn't always appropriate.
That's what led me to experiment with running language models locally using Ollama.
The experience was eye-opening.
Modern language models can run directly on consumer hardware.
No cloud service.
No external API.
No internet connection required.
Just a local model responding to questions on your own machine.
The quality varies depending on the model size, but even relatively small models can be remarkably capable.
Prompt Engineering and Personality
Once you're running models locally, you can start customising them.
This usually begins with system prompts.
A system prompt defines how the model should behave.
For example, I created a chatbot for my children based on our family hamster.
The model was instructed to behave like a friendly pet while answering questions appropriately for younger users.
It sounds silly.
And it absolutely is.
But it also demonstrates how easily these systems can be adapted to specific audiences and use cases.
Running AI on a Home Lab
Naturally, I couldn't resist taking things further.
If a model runs on my laptop, could it run on my Kubernetes cluster?
The answer was yes.
Even small Raspberry Pi clusters are capable of hosting lightweight language models.
They're not particularly fast compared to modern desktop hardware, but they're more than capable of supporting personal projects and experimentation.
The fact that a conversational AI can run on hardware costing a few hundred pounds would have seemed extraordinary only a few years ago.
Level Four: Building a Knowledge Assistant
The most interesting step came when I stopped thinking about AI as a chatbot and started thinking about it as a knowledge retrieval system.
Initially, I assumed this meant training a custom model.
It turns out that's usually unnecessary.
Instead, I discovered Retrieval-Augmented Generation (RAG).
A RAG system combines:
- A language model.
- A vector database.
- A retrieval layer.
Rather than memorising information during training, the system retrieves relevant information from a knowledge base and provides it to the model at query time.
This makes it possible to build assistants that understand specific documents, websites, wikis, or datasets without retraining a model from scratch.
Teaching an AI About My Website
As an experiment, I built a simple RAG system around my professional website.
The process was surprisingly straightforward:
- Scrape the website.
- Break the content into chunks.
- Generate vector embeddings.
- Store them in a vector database.
- Retrieve relevant content when questions are asked.
The result was an assistant capable of answering questions about my experience, projects, and skills.
Not perfectly.
But surprisingly well.
Internal Knowledge Has Huge Potential
The more interesting use case wasn't my website.
It was internal documentation.
Using a small subset of team documentation, I built a prototype assistant capable of answering onboarding questions.
Things like:
- What tools should I install?
- What does this team do?
- Where can I find documentation?
The answers weren't flawless, but they were useful.
And that's what makes this approach interesting.
Many organisations already possess enormous amounts of knowledge.
The challenge isn't creating more information.
It's helping people find it.
AI Works Best as a Multiplier
After all this experimentation, I've become increasingly convinced that AI's greatest strength isn't replacing expertise.
It's amplifying it.
A knowledgeable engineer with AI assistance can move faster.
Learn faster.
Experiment faster.
Communicate faster.
The technology doesn't eliminate the need for understanding.
If anything, it makes understanding more important.
Because someone still needs to evaluate the answers.
Someone still needs to recognise when the model is wrong.
Someone still needs to decide what problem is worth solving.
Final Thoughts
AI has already become part of my workflow.
Not because it's magical.
Not because it's always correct.
And certainly not because it replaces engineering judgement.
It's useful because it removes friction.
It helps answer questions, explore ideas, automate repetitive work, and unlock knowledge that would otherwise remain buried.
The tools will continue to evolve.
The models will improve.
But the underlying lesson remains simple:
The real value isn't the AI itself.
It's what a skilled person can accomplish when they learn how to use it effectively.