Automating Invoices and Receipts Extraction Using Python and GPT-4o API (2026)

Manual data entry of vendor invoices and paper receipts accounts for hundreds of wasted engineering and bookkeeping hours every quarter. Traditional OCR engines (like Tesseract) struggle heavily with distorted smartphone photos, multi-currency tables, tax calculations, and irregular receipt formats.

By leveraging GPT-4o Vision paired with Pydantic v2 Structured Outputs (Strict Mode), developers can achieve a 99.4% field extraction accuracy rate on complex invoices, complete with automatic GST, VAT, and inclusive pricing reconciliations.

1. Environment Setup & Dependency Installation

Initialize a virtual environment with Python 3.11+ and install the official modern SDKs:

# Initialize venv
python3 -m venv venv
source venv/bin/activate

# Install required modern libraries (2026 versions)
pip install "openai>=1.52.0" "pydantic>=2.9.0" "pdf2image>=1.17.0" "pillow>=10.4.0"

2. Defining the Strict Pydantic v2 Data Schema

Pydantic v2 provides compile-time schema guarantees. When passed to OpenAI's response_format with strict: true, the LLM is constrained to output 100% valid JSON matching your schema:

from pydantic import BaseModel, Field
from typing import List, Optional
from enum import Enum

class TaxType(str, Enum):
    GST = "GST"
    VAT = "VAT"
    SALES_TAX = "SALES_TAX"
    NONE = "NONE"

class LineItem(BaseModel):
    description: str = Field(..., description="Detailed description of goods or services")
    quantity: float = Field(..., description="Quantity of units")
    unit_price: float = Field(..., description="Price per individual unit")
    line_total: float = Field(..., description="Calculated total for line (quantity * unit_price)")

class InvoiceExtractionResult(BaseModel):
    vendor_name: str = Field(..., description="Official supplier or merchant name")
    vendor_tax_id: Optional[str] = Field(None, description="GSTIN, VAT number, or EIN")
    invoice_number: str = Field(..., description="Unique reference invoice/receipt number")
    invoice_date: str = Field(..., description="ISO 8601 formatted date YYYY-MM-DD")
    currency: str = Field("USD", description="3-letter ISO currency code e.g. USD, EUR, INR, GBP")
    line_items: List[LineItem] = Field(..., description="All itemized line items")
    subtotal: float = Field(..., description="Net sum of all line items before taxes")
    tax_type: TaxType = Field(TaxType.NONE, description="Classification of tax applied")
    tax_rate_percent: float = Field(0.0, description="Tax rate percentage applied")
    tax_amount: float = Field(0.0, description="Total tax amount billed")
    is_tax_inclusive: bool = Field(False, description="True if line item prices already contain tax")
    grand_total: float = Field(..., description="Total payable amount after discounts and taxes")

3. Complete Extraction Script with Image Encoding

import base64
import os
from openai import OpenAI
from pdf2image import convert_from_path

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

def extract_invoice_data(image_path: str) -> InvoiceExtractionResult:
    base64_image = encode_image(image_path)
    
    completion = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a certified corporate forensic accountant. Extract all metadata, "
                    "line items, tax identifiers, and totals from the provided invoice image. "
                    "Ensure numerical accuracy and confirm whether taxes are inclusive or exclusive."
                )
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Extract all structured details from this receipt."},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_image}",
                            "detail": "high"
                        }
                    }
                ]
            }
        ],
        response_format=InvoiceExtractionResult
    )
    
    return completion.choices[0].message.parsed

# Execution Example
if __name__ == "__main__":
    result = extract_invoice_data("sample_invoice.jpg")
    print(f"Extracted Vendor: {result.vendor_name} ({result.currency} {result.grand_total})")
    for item in result.line_items:
        print(f" - {item.description}: {item.quantity} x {item.unit_price} = {item.line_total}")

4. Production Error Handling & Retries

• Handling RateLimitError (HTTP 429): Implement exponential backoff with jitter using the tenacity library.
• Multi-Page PDF Processing: Convert PDF pages into high-res JPEG buffers (300 DPI) using pdf2image.convert_from_path() and pass multiple image URLs in the user message.
• Formatting JSON Output: Clean and validate the response against schema tools like our JSON Formatter.