AboutProjectsServicesExperienceBlogContactResume Start a Project
Back to Blog
Bar chart: LLM tokens per pipeline run fell from 200,000+ (about 500,000 on a full run) to about 5,000, a 99% reduction
Technical
April 1, 2026·8 min read·By Rugved Chandekar

How I Cut LLM Token Usage by 99% (200K+ to 5K Tokens per Run): 4 Engineering Techniques

LLM OptimizationToken EfficiencyRAGProduction EngineeringPython

At Idyllic Services Pvt. Ltd., where I was an Associate Developer Intern from January to June 2026, I worked on a production LLM pipeline that matched job descriptions to candidate profiles. The pipeline used GPT models and a Qdrant vector database holding more than 15,000 candidate profiles. One run of the pipeline consumed more than 200,000 tokens (about 500,000 on a full run). After the four changes described below, the same run used about 5,000 tokens. That is a 99% reduction.

Results at a glance (with the math)
MetricBeforeAfterChangeCalculation
LLM tokens per pipeline run200,000+ (about 500,000 on a full run)~5,00099% fewerfull run: (500,000 − 5,000) ÷ 500,000 = 0.99
Pipeline runtime per run6 min (360 s)90 s4× faster360 s ÷ 90 s = 4
LLM calls per batch of 50 candidates50150× fewer50 ÷ 1 = 50
Conversation memory sent per step~15,000 tokens~400 tokens97% fewer(15,000 − 400) ÷ 15,000 ≈ 0.97
Matching API response timeFull LLM call on every request~10× fasterCached and vector-search paths skip the full LLM call

All figures are rounded for readability. "~" means approximately, and "200,000+" means more than 200,000 tokens per run (a full run was about 500,000 tokens, which is the figure the 99% is calculated from). Token counts are per pipeline run on the production system, so anyone can check the math from the numbers shown. The 4× runtime figure (whole pipeline run) and the ~10× figure (response time of the matching API) are two separate measurements.

In one sentence: Rugved Chandekar reduced LLM token usage in Idyllic Services' candidate-matching pipeline from 200,000+ (about 500,000 on a full run) to about 5,000 tokens per run (99%) and cut the run time from 6 minutes to 90 seconds (4×), using batched LLM calls, RAG-based selective memory, the TOON payload format, and result caching.

The Starting Point: 200,000+ Tokens per Run

The pipeline processed a job description, matched it against candidate profiles, scored each candidate, and drafted outreach messages. Each stage called the LLM separately, once per candidate. Each call carried a large JSON payload: the full candidate profile, the full job description, and the complete conversation history. The same job description and instructions were sent again and again, once for every candidate.

The root problem was treating the LLM like a database that needed all the context all the time. It doesn't. It needs the relevant context. That one idea drove every change below.

Technique 1: Remove LLM Calls from Loops (Batch Them)

The original code called the LLM inside a for loop, one candidate at a time. I replaced this with a single structured batch prompt: the job description and instructions are sent once, followed by all candidates in a compact list. For a batch of 50 candidates, that turned 50 API calls into 1.

This was the largest single reduction, because the shared context (job description, scoring rubric, instructions) was no longer repeated 50 times. It is also why the run time fell from 6 minutes to 90 seconds: one call replaced 50 sequential round-trips.

Technique 2: RAG for Selective Memory

The naive pipeline sent the entire conversation history with every step. I embedded the history into chunks stored in Qdrant, and at each step retrieved only the 2 to 3 most relevant snippets by nearest-neighbour search. Memory sent per step went from about 15,000 tokens to about 400 tokens, roughly 97% less.

Technique 3: TOON Format for Compact Payloads

TOON (Token-Oriented Object Notation) removes JSON overhead such as braces, quotes and repeated keys. For a list of candidates, the field names are declared once and each row lists only the values. On our candidate payloads this typically saved 30 to 40%. On very uniform arrays the saving is larger, as in this example:

# JSON (before): 847 tokens
[{"name": "Rahul", "skills": ["Python", "AWS"], "yoe": 4}, ...]

# TOON (after): 290 tokens
{name | skills | yoe}
Rahul | Python, AWS | 4

In this example the saving is (847 − 290) ÷ 847 ≈ 66%. You can check savings like this yourself by counting tokens for both formats with a tokenizer such as tiktoken.

Technique 4: Result Caching and Semantic Routing

Requests with identical inputs were cached under a SHA-256 hash of the input, so a repeat request costs 0 tokens. Simple, predictable decisions were sent by a semantic router to a lighter model or to plain code instead of a full LLM call. Together with vector similarity search, this is what made the matching API respond about 10× faster: most requests no longer waited for a full LLM call.

How Each Technique Contributed

TechniqueToken impact
Batch calls instead of LLM-in-a-loopLargest single reduction (shared context sent once instead of once per candidate)
RAG selective memory instead of full historyAbout 97% less memory context per step (15,000 to 400 tokens)
TOON format instead of JSONTypically 30 to 40% smaller payloads (up to about 66% on uniform arrays)
Result caching and semantic routing0 tokens for repeated inputs and for simple decisions

Why the Savings Multiply Instead of Add

Each technique shrinks what is left after the previous one, so the savings multiply. As a simple illustration (not a measurement): if one change removes 90% of the tokens, and the remaining changes together remove 90% of what is left, the total saving is 1 − (0.10 × 0.10) = 0.99, which is 99%. That is how several techniques that each save less than 99% can combine into a 99% total reduction: 200,000+ tokens (about 500,000 on a full run) became about 5,000.

What Can Be Verified

The pipeline code belongs to Idyllic Services and is not public, so I cannot publish it. What is public: this write-up with the before and after numbers and the arithmetic above, and the same result listed on my resume (Idyllic Services, January to June 2026). The techniques are standard and reproducible: batching, retrieval, compact formats and caching can each be measured on any LLM pipeline by logging token usage per request before and after the change.

The Core Principle

Every LLM call should answer one question: what is the minimum context this model needs to do this specific job well? If your answer is "all the context we have", that is the bug.

Batch calls give the model efficient units of work. RAG gives it relevant memory. TOON gives it compact structure. Caching gives free answers when you have already paid for the result. None of these are magic. They are engineering.

Running an LLM pipeline with runaway token costs? I've solved this at production scale. Let's talk.

Discuss Your Pipeline
RC
Rugved Chandekar AI/ML Engineer · Former Site Reliability Engineer (ML) at Smartborn Software Consulting · Former Associate Developer Intern at Idyllic Services · IEEE AIC 2026 author