Back to Tracker
Phase 3 — Practical Building
Multimodal AI — Vision & Images
GPT-4o, Gemini, and Claude all understand images natively. Extract structured data from any visual document.
GPT-4o Vision
Base64 vs URL
PDF Parsing
Structured Extraction
Cost Optimization
Vision Model Capabilities — What They Can See
Modern LLMs process images through a vision encoder that converts pixels into token embeddings — the same attention mechanism then reasons over image and text tokens together.
| Capability | GPT-4o | Gemini 1.5 Pro | Claude 3.5 Sonnet |
|---|
| Image understanding | ✓ Excellent | ✓ Excellent | ✓ Excellent |
| PDF page analysis | ✓ (convert to image) | ✓ (native PDF support) | ✓ (convert to image) |
| Chart & graph reading | ✓ Good | ✓ Very good | ✓ Very good |
| Handwritten text OCR | ✓ Good | ✓ Excellent | ✓ Good |
| Tables & structured docs | ✓ Good | ✓ Excellent | ✓ Excellent |
| Max images per request | Multiple | Multiple (up to 3000 frames) | Multiple |
When to use vision AI over traditional OCR: scanned documents, handwriting, tables that span multiple columns, charts and diagrams, any document where layout context matters. Traditional OCR is better for simple digital-born PDFs (faster, cheaper, no token cost).
Image Input Formats — URL vs Base64
Format 1 — URL
{
"role": "user",
"content": [
{"type": "text",
"text": "What's in this image?"},
{"type": "image_url",
"image_url": {
"url": "https://example.com/img.jpg"
}}
]
}
Use when: publicly hosted images. Smallest payload. API fetches the image.
Format 2 — Base64
import base64
with open("invoice.jpg", "rb") as f:
b64 = base64.b64encode(
f.read()).decode()
# In the message:
"url": f"data:image/jpeg;base64,{b64}"
Use when: private/local images, uploaded files, screenshots. Image is inline in the request.
Anthropic (Claude) Format
# Claude uses "source" instead of "image_url"
{
"role": "user",
"content": [
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": b64_string
}},
{"type": "text", "text": "Extract all line items."}
]
}
Cost warning: images are charged as additional tokens. GPT-4o charges 85–1105 tokens per image depending on resolution (detail level). Always resize to minimum needed resolution before encoding. A 2048×2048 image costs ~1105 tokens; a 512×512 costs ~170 tokens.
You cannot pass a PDF file directly to vision APIs (except Gemini). For OpenAI and Anthropic, you must convert each page to an image first.
Workflow for 50-Page PDF
import fitz # PyMuPDF
import base64
def pdf_pages_to_images(pdf_path: str, dpi: int = 150) -> list[str]:
"""Convert PDF pages to base64 JPEG strings."""
doc = fitz.open(pdf_path)
images = []
for page in doc:
mat = fitz.Matrix(dpi / 72, dpi / 72)
pix = page.get_pixmap(matrix=mat)
img_bytes = pix.tobytes("jpeg")
images.append(base64.b64encode(img_bytes).decode())
return images
# Strategy for large PDFs: don't send all pages at once
# 1. Run fast OCR (pdfplumber / pypdf) to find relevant pages
# 2. Send only relevant pages to vision API
# 3. Merge results with a final LLM call
Cost Estimate (50 pages)
Images (150 DPI)50 pages × ~500 tokens/image = 25,000 tokens
GPT-4o cost25,000 × $0.005/1K = ~$0.125 for image tokens alone
OptimizationUse "low" detail for overview; "high" detail only for extraction pages
Gemini advantage: Gemini 1.5 Pro accepts PDF files directly (no conversion needed) and supports up to 3,000 pages natively. For large document workflows, Gemini is often the better choice.
Build: Invoice & Receipt Parser
from openai import OpenAI
from pydantic import BaseModel
import base64, json
client = OpenAI()
class LineItem(BaseModel):
description: str
quantity: float
unit_price: float
total: float
class Invoice(BaseModel):
vendor: str
invoice_number: str | None = None
date: str | None = None
line_items: list[LineItem]
subtotal: float | None = None
tax: float | None = None
total: float
def parse_invoice(image_path: str) -> Invoice:
with open(image_path, "rb") as f:
b64 = base64.b64encode(f.read()).decode()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
"role": "user",
"content": [
{"type": "text",
"text": "Extract all invoice fields. Return JSON."},
{"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
]
}],
response_format={"type": "json_object"},
temperature=0
)
data = json.loads(response.choices[0].message.content)
return Invoice(**data)
Interview story: "I built an invoice parser that extracts line items, taxes, and totals from any receipt image. The key engineering decision was always resizing images before encoding — this cut token costs by 60% with no accuracy loss for structured document text."
URL formatFor public images — smallest request payload
Base64 formatFor private/local images — self-contained request
detail: "low"~85 tokens, good for simple classification
detail: "high"~1105 tokens, needed for fine text & tables
PDF → imagesUse PyMuPDF (fitz) at 150 DPI for clean text
Gemini advantageNative PDF support, no conversion needed
temperature=0Always use for extraction — deterministic output
Resize firstResize to max needed size before encoding — big cost savings