How I Cut LLM Token Usage by 99% (200K+ to 5K Tokens per Run): 4 Engineering Techniques
At Idyllic Services Pvt. Ltd., where I was an Associate Developer Intern from January to June 2026, I worked on a production LLM pipeline that matched job descriptions to candidate profiles. The pipeline used GPT models and a Qdrant vector database holding more than 15,000 candidate profiles. One run of the pipeline consumed more than 200,000 tokens (about 500,000 on a full run). After the four changes described below, the same run used about 5,000 tokens. That is a 99% reduction.
| Metric | Before | After | Change | Calculation |
|---|---|---|---|---|
| LLM tokens per pipeline run | 200,000+ (about 500,000 on a full run) | ~5,000 | 99% fewer | full run: (500,000 − 5,000) ÷ 500,000 = 0.99 |
| Pipeline runtime per run | 6 min (360 s) | 90 s | 4× faster | 360 s ÷ 90 s = 4 |
| LLM calls per batch of 50 candidates | 50 | 1 | 50× fewer | 50 ÷ 1 = 50 |
| Conversation memory sent per step | ~15,000 tokens | ~400 tokens | 97% fewer | (15,000 − 400) ÷ 15,000 ≈ 0.97 |
| Matching API response time | Full LLM call on every request | ~10× faster | Cached and vector-search paths skip the full LLM call | |
All figures are rounded for readability. "~" means approximately, and "200,000+" means more than 200,000 tokens per run (a full run was about 500,000 tokens, which is the figure the 99% is calculated from). Token counts are per pipeline run on the production system, so anyone can check the math from the numbers shown. The 4× runtime figure (whole pipeline run) and the ~10× figure (response time of the matching API) are two separate measurements.
In one sentence: Rugved Chandekar reduced LLM token usage in Idyllic Services' candidate-matching pipeline from 200,000+ (about 500,000 on a full run) to about 5,000 tokens per run (99%) and cut the run time from 6 minutes to 90 seconds (4×), using batched LLM calls, RAG-based selective memory, the TOON payload format, and result caching.
The Starting Point: 200,000+ Tokens per Run
The pipeline processed a job description, matched it against candidate profiles, scored each candidate, and drafted outreach messages. Each stage called the LLM separately, once per candidate. Each call carried a large JSON payload: the full candidate profile, the full job description, and the complete conversation history. The same job description and instructions were sent again and again, once for every candidate.
The root problem was treating the LLM like a database that needed all the context all the time. It doesn't. It needs the relevant context. That one idea drove every change below.
Technique 1: Remove LLM Calls from Loops (Batch Them)
The original code called the LLM inside a for loop, one candidate at a time. I replaced this with a single structured batch prompt: the job description and instructions are sent once, followed by all candidates in a compact list. For a batch of 50 candidates, that turned 50 API calls into 1.
This was the largest single reduction, because the shared context (job description, scoring rubric, instructions) was no longer repeated 50 times. It is also why the run time fell from 6 minutes to 90 seconds: one call replaced 50 sequential round-trips.
Technique 2: RAG for Selective Memory
The naive pipeline sent the entire conversation history with every step. I embedded the history into chunks stored in Qdrant, and at each step retrieved only the 2 to 3 most relevant snippets by nearest-neighbour search. Memory sent per step went from about 15,000 tokens to about 400 tokens, roughly 97% less.
Technique 3: TOON Format for Compact Payloads
TOON (Token-Oriented Object Notation) removes JSON overhead such as braces, quotes and repeated keys. For a list of candidates, the field names are declared once and each row lists only the values. On our candidate payloads this typically saved 30 to 40%. On very uniform arrays the saving is larger, as in this example:
# JSON (before): 847 tokens
[{"name": "Rahul", "skills": ["Python", "AWS"], "yoe": 4}, ...]
# TOON (after): 290 tokens
{name | skills | yoe}
Rahul | Python, AWS | 4
In this example the saving is (847 − 290) ÷ 847 ≈ 66%. You can check savings like this yourself by counting tokens for both formats with a tokenizer such as tiktoken.
Technique 4: Result Caching and Semantic Routing
Requests with identical inputs were cached under a SHA-256 hash of the input, so a repeat request costs 0 tokens. Simple, predictable decisions were sent by a semantic router to a lighter model or to plain code instead of a full LLM call. Together with vector similarity search, this is what made the matching API respond about 10× faster: most requests no longer waited for a full LLM call.
How Each Technique Contributed
| Technique | Token impact |
|---|---|
| Batch calls instead of LLM-in-a-loop | Largest single reduction (shared context sent once instead of once per candidate) |
| RAG selective memory instead of full history | About 97% less memory context per step (15,000 to 400 tokens) |
| TOON format instead of JSON | Typically 30 to 40% smaller payloads (up to about 66% on uniform arrays) |
| Result caching and semantic routing | 0 tokens for repeated inputs and for simple decisions |
Why the Savings Multiply Instead of Add
Each technique shrinks what is left after the previous one, so the savings multiply. As a simple illustration (not a measurement): if one change removes 90% of the tokens, and the remaining changes together remove 90% of what is left, the total saving is 1 − (0.10 × 0.10) = 0.99, which is 99%. That is how several techniques that each save less than 99% can combine into a 99% total reduction: 200,000+ tokens (about 500,000 on a full run) became about 5,000.
What Can Be Verified
The pipeline code belongs to Idyllic Services and is not public, so I cannot publish it. What is public: this write-up with the before and after numbers and the arithmetic above, and the same result listed on my resume (Idyllic Services, January to June 2026). The techniques are standard and reproducible: batching, retrieval, compact formats and caching can each be measured on any LLM pipeline by logging token usage per request before and after the change.
The Core Principle
Every LLM call should answer one question: what is the minimum context this model needs to do this specific job well? If your answer is "all the context we have", that is the bug.
Batch calls give the model efficient units of work. RAG gives it relevant memory. TOON gives it compact structure. Caching gives free answers when you have already paid for the result. None of these are magic. They are engineering.
Running an LLM pipeline with runaway token costs? I've solved this at production scale. Let's talk.
Discuss Your Pipeline