Tags: llm, rag, chatbot
Automated chatbots and RAG are everywhere now (and none of them are the same!). I wanted to understand how they actually work - so I built one of each. Both have led me to the same architectural conclusion.
Chatbot
chatbot is an automated customer-support bot for an e-commerce store, built with Gemini function calling. It answers customers’ FAQs (“how much is shipping?”) and looks up order and product information. Inside, it runs a function-calling loop: the model asks for a tool, the tool runs, the result goes back, then repeats until it gets an answer. The tools are deterministic and return facts only. When there’s nothing to answer with, the model just says “I don’t know” instead of hallucinating. It is read-only: it cannot do irreversible actions (e.g. cancel or refund an order).
RAG
rag-learning-ts is a simple RAG pipeline over community chat history. The engine answers a member’s question from a community’s past conversations. Like the chatbot, it has a guardrail against hallucination (a relevance threshold on the similarity score).
One important lesson
The biggest takeaway was an architectural one: minimize the non-deterministic surface — for testability, efficiency, and cost. Many parts of an “LLM system” don’t actually need the LLM. LLM is probabilistic. If applied in the wrong place, it makes things worse — slower and less stable. So I think the core design question is which parts genuinely need an LLM — and the answer differs per project. In the RAG, the LLM only writes the grounded answer; retrieval, scoring, and even the decision to refuse are deterministic code. In the chatbot, the model does a bit more — it drives the orchestration, choosing which tool to call next and when to stop. Everything around that is deterministic.