LangChain vs LlamaIndex vs Plain SDKs: What Should You Build LLM Apps With?

"Should we use LangChain?" is one of the most common questions in AI engineering teams, and one of the most polarised. Some engineers swear by frameworks for speed; others have ripped them out after fighting abstractions in production. Both camps are describing real experiences. The way through is to look at what each layer actually does for you, write the same app three ways, and decide based on the shape of your problem rather than on hype or backlash.

Framework APIs change fast. The code below targets the current major versions (LangChain 1.x, recent LlamaIndex releases and current provider SDKs); verify import paths against the official docs before copying into production.

The Short Answer

The rest of this article justifies those choices.

What Each Option Actually Is

Plain provider SDKs

The official client libraries from model providers (OpenAI, Anthropic, Google and others) give you typed access to their APIs: messages, streaming, tool calling, structured outputs, embeddings, batch jobs. Modern SDKs already handle retries, timeouts and streaming. What they do not give you: document loaders, vector store integrations, provider-agnostic interfaces, or an agent loop beyond basic tool-calling helpers. See our LLM APIs guide for the fundamentals.

LangChain

LangChain began in 2022 as a library of composable "chains" and grew a huge set of integrations. The 1.0 release in late 2025 refocused it: the core abstraction is now an agent created with create_agent, built on LangGraph's runtime, with middleware hooks for things like summarising long conversations, human-in-the-loop approval and PII handling. Model interfaces are standardised so you can swap providers, and older chain-style APIs moved into a separate legacy package. The surrounding ecosystem includes LangGraph (orchestration) and LangSmith (tracing and evaluation, usable without LangChain).

LlamaIndex

LlamaIndex is a data framework for LLM applications. Its centre of gravity is ingestion and retrieval: readers for many sources, node parsers (chunkers), indexes, retrievers, rerankers, query engines that combine retrieval and synthesis, and evaluation helpers. It also has an event-driven Workflows abstraction and agents. Its commercial products focus on document parsing and managed ingestion, which reflects where it is strongest.

Others you will hear about

Haystack (pipelines, strong in search), DSPy (programmatic prompt optimisation), Pydantic AI (type-safe agents), and provider agent SDKs from model vendors, plus TypeScript-first options such as the Vercel AI SDK. The arguments in this article apply to them too: every framework trades control for convenience.

The Comparison Task

To compare fairly, we build the same thing three times: answer employee questions about HR policies (leave, notice period, reimbursements) from a folder of text files, citing which file the answer came from. Same chunking idea, same top-k, same prompt intent. This is the core of most internal RAG assistants at Indian companies. If RAG itself is new to you, start with RAG systems.

Version 1: Plain SDK, No Framework

import math
from pathlib import Path
from openai import OpenAI

client = OpenAI()
EMBED_MODEL = "text-embedding-3-small"
CHAT_MODEL = "gpt-5-mini"        # any chat model you have access to

def chunk(text, size=800, overlap=100):
    step = size - overlap
    return [text[i:i + size] for i in range(0, max(len(text) - overlap, 1), step)]

def embed(texts):
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    return [d.embedding for d in resp.data]

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

# --- ingest (run once; persist in a vector DB in real life) ---
chunks = []
for path in Path("policies").glob("*.txt"):
    for piece in chunk(path.read_text(encoding="utf-8")):
        chunks.append({"source": path.name, "text": piece})
vectors = embed([c["text"] for c in chunks])

# --- query ---
def answer(question, k=4):
    q = embed([question])[0]
    ranked = sorted(zip(chunks, vectors), key=lambda cv: cosine(q, cv[1]), reverse=True)[:k]
    context = "\n\n".join(f"[{c['source']}]\n{c['text']}" for c, _ in ranked)
    resp = client.responses.create(
        model=CHAT_MODEL,
        instructions=("Answer only from the context. Cite sources in square brackets. "
                      "If the answer is not in the context, say you don't know."),
        input=f"Context:\n{context}\n\nQuestion: {question}",
    )
    return resp.output_text

print(answer("How many days of earned leave can I carry forward?"))

About 40 lines, every step visible. You can print the exact prompt, log the retrieved chunks, and change the chunker without reading anyone else's source code. What is missing: a real vector database (swap in pgvector or another store; see the vector database comparison), PDF and DOCX parsing, batching for large ingests, and provider portability. Each is a few dozen lines or one focused library.

Version 2: LangChain

from pathlib import Path
from langchain.chat_models import init_chat_model
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

llm = init_chat_model("openai:gpt-5-mini")      # provider:model string
store = InMemoryVectorStore(OpenAIEmbeddings(model="text-embedding-3-small"))

docs = [Document(page_content=p.read_text(encoding="utf-8"), metadata={"source": p.name})
        for p in Path("policies").glob("*.txt")]
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
store.add_documents(splitter.split_documents(docs))

def answer(question, k=4):
    hits = store.similarity_search(question, k=k)
    context = "\n\n".join(f"[{d.metadata['source']}]\n{d.page_content}" for d in hits)
    msg = llm.invoke([
        {"role": "system", "content": "Answer only from the context. Cite sources in "
                                      "square brackets. If unsure, say you don't know."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ])
    return msg.content

print(answer("How many days of earned leave can I carry forward?"))

Similar length, and noticeably different in what it gives you. The recursive splitter respects paragraph and sentence boundaries better than our naive slicer. Swapping to another provider is a one-string change in init_chat_model, and swapping the in-memory store for pgvector, Qdrant or another backend is a different class with the same interface. If you turn on LangSmith tracing, every call is recorded without extra code.

The agent version

Where LangChain 1.x shines more is when retrieval becomes one tool among several and the model decides what to call:

from langchain.agents import create_agent
from langchain.tools import tool

@tool
def search_policies(query: str) -> str:
    """Search HR policy documents. Use for leave, notice period, reimbursement questions."""
    hits = store.similarity_search(query, k=4)
    return "\n\n".join(f"[{d.metadata['source']}]\n{d.page_content}" for d in hits)

@tool
def leave_balance(employee_id: str) -> str:
    """Return the employee's current leave balance."""
    return '{"earned": 12, "sick": 6}'            # call your HRMS API here

agent = create_agent(
    model="openai:gpt-5-mini",
    tools=[search_policies, leave_balance],
    system_prompt="You are an HR assistant. Cite policy sources in square brackets.",
)
result = agent.invoke({"messages": [{"role": "user",
                       "content": "I'm E1023. Can I carry my leave into next year?"}]})
print(result["messages"][-1].content)

Writing this loop yourself with a raw SDK is perhaps 40–60 more lines (call model, detect tool calls, execute, append results, repeat, with a step limit). The value of the framework grows when you add middleware: approval before a sensitive tool, conversation summarisation when context fills up, retries and fallbacks between models.

Version 3: LlamaIndex

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-5-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)  # token-based

docs = SimpleDirectoryReader("policies").load_data()   # also reads PDF, DOCX, etc.
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(similarity_top_k=4)

resp = query_engine.query("How many days of earned leave can I carry forward?")
print(resp)
for node in resp.source_nodes:
    print(round(node.score, 3), node.node.metadata.get("file_name"))

The shortest version, and the one that handles real-world ingestion best: the directory reader parses many file types and attaches file metadata automatically, and the query engine returns source nodes with scores for citations. The trade-off is that the final prompt is assembled by LlamaIndex's default templates. You can override them, but you have to know they exist, which is exactly the kind of hidden behaviour that surprises teams later.

When Frameworks Genuinely Help

• Many data sources and formats. PDFs with tables, Confluence, Google Drive, Notion, SharePoint, Slack exports. Writing and maintaining loaders is real work, and LlamaIndex and LangChain both have large integration libraries.
• Retrieval experimentation. Trying hybrid search, parent-document retrieval, sentence-window retrieval, query rewriting and rerankers is faster when each is a component you can swap. LlamaIndex is especially good here.
• Provider portability. If you genuinely need to route between providers (cost, latency, fallback when one is down, data residency), standard model interfaces save effort.
• Stateful, long-running agents. Checkpointing, resuming after a crash, pausing for human approval, and time-travel debugging are hard to build correctly. LangGraph provides them.
• Team onboarding. A well-known framework gives new engineers shared vocabulary and documentation.
• Observability out of the box. Tracing integrations are usually one environment variable.

When Frameworks Hurt

• Debugging through layers. When an answer is wrong, you need to see the exact prompt, the retrieved chunks and the raw model response. Deep abstraction stacks make that harder unless you set up tracing from day one.
• Hidden defaults. Default prompts, chunk sizes, retry policies and truncation behaviour are decisions someone else made for a generic case.
• Version churn. These frameworks evolved rapidly; APIs, package names and recommended patterns changed several times between 2023 and 2025. Pin versions and budget for upgrades.
• Lagging provider features. New model capabilities (new tool types, caching controls, reasoning settings, new content types) appear in provider SDKs first. Frameworks catch up, but there is often a gap.
• Dependency weight. Large dependency trees slow cold starts in serverless functions and widen your supply-chain surface.
• Over-abstraction of simple things. Wrapping a single API call in three classes makes code harder to read, not easier.

In code reviews I see a recurring pattern: a team adopts a framework for a prototype, the prototype becomes production, and six months later the most-modified files are workarounds for framework behaviour. That is not an argument against frameworks; it is an argument for choosing them deliberately and isolating them.

Agents, Workflows and Where LangGraph Fits

Most "agent" use cases in companies are actually workflows: a known sequence of steps with an LLM making decisions at a few points, for example classify a support ticket, fetch the order, draft a reply, send to a human for approval if the refund exceeds ₹5,000. For these, explicit orchestration beats a free-running agent: it is cheaper, more predictable and easier to test.

LangGraph models this as a graph of nodes that read and write shared state, with conditional edges. Its value is not the graph syntax but the runtime: state is checkpointed after each step, so a workflow can pause for days waiting for a human, survive a deployment, and resume. You could build this with a job queue and a database (see designing a job scheduler), or with a general durable-execution engine; LangGraph packages it specifically for LLM steps.

If your agents need to call tools owned by other teams, combine this with MCP servers; both LangChain and LlamaIndex can load MCP tools through adapters.

Decision Table: Choosing for Your Project

Mixing is normal and healthy. A common production shape is LlamaIndex (or a parsing library) for ingestion jobs, a plain SDK for the request path, and LangSmith or another tracing tool for observability.

The Thin-Layer Pattern: Use Frameworks Without Getting Stuck

Whatever you choose, define a tiny interface that your application code depends on, and keep the framework behind it.

from dataclasses import dataclass
from typing import Protocol

@dataclass
class Passage:
    text: str
    source: str
    score: float

class Retriever(Protocol):
    def search(self, query: str, k: int) -> list[Passage]: ...

class LlamaIndexRetriever:
    def __init__(self, index):
        self._r = index.as_retriever(similarity_top_k=8)
    def search(self, query: str, k: int) -> list[Passage]:
        nodes = self._r.retrieve(query)[:k]
        return [Passage(n.node.get_content(), n.node.metadata.get("file_name", "?"),
                        n.score or 0.0) for n in nodes]

class PgVectorRetriever:
    def __init__(self, conn, embed):
        self.conn, self.embed = conn, embed
    def search(self, query: str, k: int) -> list[Passage]:
        vec = self.embed(query)
        rows = self.conn.execute(
            "SELECT content, source, 1 - (embedding <=> %s::vector) AS score "
            "FROM chunks ORDER BY embedding <=> %s::vector LIMIT %s",
            (str(vec), str(vec), k)).fetchall()
        return [Passage(t, s, sc) for t, s, sc in rows]

Your answer-generation code, evals and API handlers only see Retriever and Passage. Swapping implementations becomes a configuration change, and you can A/B test them with the same eval suite. That eval suite, not the framework, is what tells you whether a change made answers better.

How This Comes Up in Interviews

AI engineer interviews at Indian product companies and GCCs often include "walk me through a RAG or agent system you built". If your project used a framework, expect probing questions like these:

• "What does as_query_engine() or create_agent() do under the hood?" You should be able to describe the steps: embed query, retrieve top-k, build prompt, call model; or call model, parse tool calls, execute, loop.
• "Why did you choose LangChain over calling the API directly?" A strong answer names a concrete need (provider fallback, integrations, LangGraph checkpointing) and acknowledges a cost you managed.
• "Your answers got worse after upgrading the library. How do you find out why?" Compare traces of final prompts and retrieved chunks before and after, run the eval set, check changed defaults.
• "How would you remove the framework if needed?" Point to your interface boundaries.

A red flag for interviewers is a candidate who can only describe their system in framework vocabulary ("I used a RetrievalQA chain") without explaining chunking, embedding choice, top-k, prompt construction or how quality was measured. Learn the fundamentals once and any framework becomes a convenience rather than a crutch. The RAG deep dive is a good place to solidify them.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.