How I Cut LLM Token Usage by 99% — 4 Engineering Techniques That Actually Work

At Idyllic Services, I was working on a production agentic AI pipeline · an automated candidate sourcing system that ran LLM calls at scale. One iteration consumed 200,000+ tokens. That's ?15·20 per run, multiple times a day. I was asked to fix it. Here's exactly what I did · four engineering techniques that combined to reduce token usage by ~99%, bringing each run down to around 5,000 tokens.

The Starting Point: 200K Tokens Per Run

The pipeline processed job descriptions, matched them against candidate profiles, scored candidates, and generated outreach drafts. Each stage called the LLM individually. Each call carried a fat JSON payload · full candidate profiles, full job descriptions, complete conversation history. Every call was redundant. Every payload was bloated.

The root problem: I was treating the LLM like a database that needed all context at all times. It doesn't. It needs relevant context. That one insight unlocked everything.

Technique 1: RAG Integration, Semantic Routing & Result Caching

The first major architectural fix was selective context management and intelligent routing. The naive pipeline dumped entire conversation histories and bloated documents into the context window.

1. RAG Memory Retrieval: I embedded memory chunks into Qdrant vector database. Instead of 15,000 tokens of raw history, nearest-neighbor retrieval pulled only the top 2–3 relevant snippets (~400 tokens) for each step.

2. Semantic Routing: Instead of invoking expensive LLMs for simple deterministic decisions, I implemented a fast semantic router to direct requests to lighter models or local code execution.

3. SHA-256 Result Caching: Repeat requests with identical inputs were cached with SHA-256 hashes — serving repeat queries instantly at 0 tokens.

Technique 2: TOON Format · 30–40% Payload Reduction

TOON (Token-Oriented Object Notation) strips JSON syntactic overhead (braces, quotes, repeated keys). For candidate arrays, header fields are declared once and rows are listed with clean delimiters:

# JSON (before) · 847 tokens:
[{"name": "Rahul", "skills": ["Python", "AWS"], "yoe": 4}, ...]

# TOON (after) · 290 tokens:
{name | skills | yoe}
Rahul | Python, AWS | 4

Technique 3: Removing LLMs from Loops · Batch API Calls

Replacing sequential single-candidate LLM calls inside for-loops with a single structured batch prompt reduced 50 API calls to 1 single pass — taking runtime from 6 minutes down to 90 seconds.

The Combined Result

Technique	Token Impact
TOON format · compact payload	-30 to -40%
Remove LLM from loops → batch calls	largest single reduction
RAG selective memory vs full context	-90 to -97% on memory tokens
Result caching · repeat run elimination	-70 to -100% on cached steps

Final numbers: 200,000 tokens per iteration → ~5,000 tokens. ~99% reduction. Runtime dropped from 6 minutes to 90 seconds because batch calls replaced the sequential per-candidate loop.

The Core Principle

Every LLM call should answer one question: what is the minimum context this model needs to do this specific job well? If your answer is "all the context we have" · that's the bug.

TOON gives it compact structure. Batch calls give it efficient work units. RAG gives it relevant memory. Caching gives it free answers when you've already paid for the result. None of these are magic. They're engineering.