RAG is the difference between an LLM that hallucinates and one that answers with your actual data. Here's the complete production architecture I built on AWS — semantic search, vector storage, LLM generation, and all the edge cases that tutorials skip.

What RAG Actually Is (Beyond the Buzzword)

Retrieval-Augmented Generation = retrieve relevant context from your data → augment the LLM prompt with it → generate a grounded response. It prevents hallucination by giving the model real information instead of asking it to remember things.

The three-step flow:

Query → [Retrieval] → Relevant Chunks → [LLM] → Grounded Answer
          (OpenSearch)                  (Bedrock)

The Indexing Pipeline (Offline)

Before anyone can query, you need to index your documents. My pipeline:

1. Document loading — Parse PDFs, DOCX, plain text via LangChain document loaders
2. Chunking — Split into 512-token overlapping chunks (128-token overlap to preserve context across boundaries)
3. Embedding — Each chunk embedded using Amazon Titan Embeddings (1536-dim vectors)
4. Indexing — Vectors stored in Amazon OpenSearch Serverless k-NN index

from langchain.text_splitter import RecursiveCharacterTextSplitter
import boto3
import json

bedrock = boto3.client('bedrock-runtime')

def embed(text):
    response = bedrock.invoke_model(
        modelId='amazon.titan-embed-text-v1',
        body=json.dumps({'inputText': text})
    )
    return json.loads(response['body'].read())['embedding']

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=128
)
chunks = splitter.split_text(document_text)
for chunk in chunks:
    vector = embed(chunk)
    opensearch_client.index(index='docs', body={
        'text': chunk,
        'embedding': vector
    })

The Query Pipeline (Online / Real-Time)

When a user asks a question:

1. Embed the query using the same Titan model
2. K-NN search in OpenSearch — retrieve top-5 semantic matches
3. Build a prompt with retrieved chunks as context
4. Call Claude via Amazon Bedrock — stream the response back

def answer_query(query: str) -> str:
    # Step 1: Embed the query
    query_vector = embed(query)
    
    # Step 2: Semantic search
    results = opensearch_client.search(
        index='docs',
        body={
            'knn': {'field': 'embedding', 'query_vector': query_vector, 'k': 5}
        }
    )
    context = "\n\n".join([r['_source']['text'] for r in results['hits']['hits']])
    
    # Step 3: LLM generation with context
    prompt = f"""Answer based only on the context below:
    
Context:
{context}

Question: {query}

Answer:"""
    
    response = bedrock.invoke_model(
        modelId='anthropic.claude-3-sonnet-20240229-v1:0',
        body=json.dumps({'prompt': prompt, 'max_tokens': 1024})
    )
    return json.loads(response['body'].read())['completion']

The AWS Deployment Stack

• Amazon ECS (Fargate) — Flask API deployed as a containerized service, auto-scaling
• Amazon OpenSearch Serverless — Managed vector database, scales to zero when idle
• Amazon Bedrock — Claude 3 Sonnet for generation, Titan for embeddings
• ECR — Docker image registry for the Flask app
• ALB — Application Load Balancer in front of ECS for HTTPS termination

The Token Cost Problem (And How We Solved It)

At scale, the biggest RAG cost is tokens — every query sends context that can be 2,000+ tokens. At my internship, we reduced token usage by ~99% through:

• Hierarchical retrieval — Retrieve document summaries first, only fetch full chunks when needed
• Caching embeddings — Same query twice? Return cached vector, skip re-embedding
• Prompt compression — Use a small model to compress retrieved context before sending to Claude