Most generative AI projects that stall do so for the same reason: the first question was “which model should we use?” instead of “which job should this do?”. The model is the easy part. The hard parts are choosing a task worth automating, getting the right data in front of the model, and proving the answers are good enough before real customers see them.
This guide walks through the decisions in the order we make them with clients.
Start with one job, not a platform
Pick a single task with a clear owner and a result you can measure. Good first projects look like this:
- Answer customer or staff questions from your own help articles, policies or product manuals.
- Draft first replies to support tickets for an agent to check and send.
- Pull fields out of invoices, contracts or forms into your system.
- Summarize long reports, call notes or case files into a fixed format.
Each of these has an obvious “before” to compare against: minutes per ticket, error rate in data entry, time to find an answer. If you cannot say how you will know the project worked, it is not ready to start.
Three ways to build it
1. Prompting a hosted model
You send instructions and the user’s input to a model such as GPT, Claude or Gemini and use the answer. This is the fastest route and often enough for drafting, rewriting, classification and summarizing. Its limit is knowledge: the model only knows what it learned in training plus whatever you put in the prompt.
2. Retrieval-augmented generation (RAG)
When answers must come from your own documents, RAG is usually the right choice. At question time the system searches your content, picks the most relevant passages and gives them to the model with an instruction to answer only from them. A typical RAG pipeline has these parts:
- Ingestion: loading documents from your sources and keeping them in sync as they change.
- Chunking and embeddings: splitting documents into passages and turning each into a vector for search.
- Retrieval: a vector or hybrid search that finds the best passages, filtered by what the user is allowed to see.
- Answering: a prompt that asks the model to cite its sources and to say “I don’t know” when the passages do not cover the question.
RAG keeps answers current without retraining anything: update the document and the next answer reflects it.
3. Fine-tuning
Fine-tuning trains a model further on your own examples. It is good at teaching a consistent style, a strict output format or a narrow classification task. It is a poor way to teach facts, because facts change and a fine-tuned model can still mix them up. Fine-tuning also needs hundreds to thousands of good, labeled examples, which many teams do not have on day one.
A useful rule of thumb:
- Needs your knowledge? Use RAG.
- Needs a particular behavior or format, and prompting is not consistent enough? Consider fine-tuning.
- Neither? Start with prompting and measure before adding complexity.
What drives the cost
Running costs come mostly from the number of requests multiplied by the size of each request, plus the price of the model you choose. Long prompts with many retrieved passages cost more than short ones. Other costs to plan for are the search infrastructure for RAG, the time people spend reviewing answers, and ongoing upkeep as your documents and the models change.
The cheapest way to get a reliable estimate is a short prototype on your real data. A week of measuring actual prompt sizes and volumes beats any spreadsheet guess.
Measure accuracy before launch
Before anyone outside the project team uses the system, build an evaluation set: a list of real questions or documents with the answer you expect. Fifty to two hundred examples is a good start. Run the system against it after every change and track:
- Correctness: is the answer right?
- Grounding: does it come from the retrieved sources, with citations?
- Honesty: does it say it does not know when the sources do not cover the question?
For high-stakes work such as medical, legal or financial answers, keep a person in the loop who approves the output before it is used.
Plan for privacy from the start
Decide early which data may leave your systems and which must not. Check each model provider’s data retention and training settings, and choose a provider and region that suit your obligations under India’s Digital Personal Data Protection Act, 2023, the GDPR or your sector’s rules. In a RAG system, filter retrieval by the user’s permissions so nobody receives an answer drawn from a document they could not open themselves. Log prompts and answers for quality review, but strip personal data you do not need.
A realistic project shape
- Discovery: agree on the task, the success measure and the data sources.
- Prototype: build the simplest version that works on your real data.
- Evaluate and harden: grow the evaluation set, fix failure cases, add guardrails and monitoring.
- Pilot: release to a small group and compare against the “before” numbers.
- Roll out: widen access, then pick the next task.
How we can help
Our team builds generative AI and LLM solutions: RAG pipelines, chatbots and workflow automation on GPT, Claude, Llama and other models, connected to the systems you already use. We also run AI inside our own product: in Smart Class Sync, a daily briefing flags attendance dips, fee defaulters and at-risk students for school principals.
If you have a task in mind, tell us about it and we will help you decide whether prompting, RAG or fine-tuning is the right fit.