10 Pandas Techniques for Processing Large Datasets with Limited RAM

Published August 2026 · Data Science · ~1,800 words

You have a 311 MB CSV file. Your laptop has 16GB RAM. The naive approach — pd.read_csv('file.csv') — loads the entire file into memory, and a 311 MB CSV becomes 422 MB in a pandas DataFrame.

That is fine for one file. But when you have multiple datasets, intermediate results, and model training — 422 MB per load adds up fast. Here are 10 techniques that keep your workflow under control.

Featured image: articles/086/featured-image.svg

Techniques comparison: articles/086/techniques-comparison.svg

The Benchmark Dataset

All measurements in this article come from a single reproducible benchmark: 2,000,000 rows × 12 columns.

Environment: Python 3.13.14, pandas 3.0.5, DuckDB 1.5.5

Technique 1: usecols — Load Only What You Need

Load 3 columns instead of 12. This is the single most effective technique.

# BAD: loads all 12 columns (422 MB)
df = pd.read_csv('large_data.csv')

# GOOD: loads only 3 columns (57 MB)
df = pd.read_csv('large_data.csv', usecols=['category', 'price', 'quantity'])

You save 86% of memory and 63% of time — before doing anything else.

Technique 2: dtypes — Use Smaller Numeric Types

int64 uses 8 bytes per value. int32 uses 4. int16 uses 2. If your data fits in a smaller type, use it.

# Default: int64, float64
df = pd.read_csv('data.csv', dtype={
    'id': 'int64', 'quantity': 'int64', 'price': 'float64',
})

# Optimized: int32, int16, float32
df = pd.read_csv('data.csv', dtype={
    'id': 'int32', 'quantity': 'int16', 'price': 'float32',
})

Our benchmark: -55% memory for 5 numeric columns. Warning: float32 loses precision — verify it matters for your use case.

Technique 3: chunksize — Process in Batches

When the file is larger than RAM, process it in chunks:

# Process 100K rows at a time
total = 0
for chunk in pd.read_csv('huge.csv', chunksize=100_000):
    total += chunk['price'].sum()

# Result is identical to loading the full file
print(f'Total: {total:,.0f}')

Peak memory: only 100K rows at a time, regardless of file size. The trade-off: slightly slower due to repeated parsing.

Technique 4: Parquet — The Best Format for Analytics

Parquet with Zstd compression produces a file 15.5x smaller than CSV and reads 34x faster:

# Convert once, benefit forever
df.to_parquet('data.parquet', compression='zstd')

# Read specific columns instantly
df = pd.read_parquet('data.parquet', columns=['category', 'price'])

Technique 5: Categoricals — For Low-Cardinality Strings

If a string column has few unique values (categories, regions, statuses), use category dtype:

df = pd.read_csv('data.csv', usecols=['category', 'region', 'status'])

# String dtype: 80 MB for 3 columns
# Categorical: 5.7 MB for 3 columns
df['category'] = df['category'].astype('category')
df['region'] = df['region'].astype('category')
df['status'] = df['status'].astype('category')

This is the highest memory reduction of any single technique — 93% for low-cardinality columns.

Technique 6: Avoid Unnecessary Copies

Chained operations create temporary DataFrames. Direct access avoids them:

# SLOW: creates intermediate copy
result = df[df['price'] > 100]['price'].mean()

# FAST: direct column access
result = df.loc[df['price'] > 100, 'price'].mean()

Measured: 1.8x faster on 2M rows. The difference grows with dataset size.

Technique 7: Vectorize — Never Use iterrows

iterrows() is a Python loop — it processes one row at a time. Vectorized operations process entire columns at once:

# SLOW: Python loop (never do this)
for idx, row in df.iterrows():
    total += row['price'] * row['quantity']

# FAST: vectorized
df['total'] = df['price'] * df['quantity']

# FAST: groupby aggregation
result = df.groupby('category')['price'].mean()

Vectorized operations use optimized C code under the hood. iterrows() is 10-100x slower for most operations.

Technique 8: Select Columns at Load Time

Combining usecols with aggregation at load time saves both memory and time:

# SLOW: load all, then group
df = pd.read_csv('data.csv')
result = df.groupby('category')['price'].sum()

# FAST: select at load, then group
result = pd.read_csv('data.csv', usecols=['category', 'price']) \\
           .groupby('category')['price'].sum()

Technique 9: DuckDB — Query Files Directly

DuckDB queries CSV and Parquet files without loading them into pandas:

import duckdb

con = duckdb.connect()

# Query Parquet directly — no DataFrame needed
result = con.execute("""
    SELECT AVG(price)
    FROM 'data.parquet'
    WHERE category = 'electronics'
""").fetchone()[0]

The biggest speedup in this article — 352x — comes from DuckDB's predicate pushdown on Parquet.

Technique 10: Row Group Reading

Parquet files are divided into row groups. You can read just one:

import pyarrow.parquet as pq

pf = pq.ParquetFile('data.parquet')
print(f'Row groups: {pf.metadata.num_row_groups}')

# Read only first row group (1M of 2M rows)
partial = pf.read_row_group(0)

# Read all
full = pf.read()

Useful for sampling, debugging, and partial processing without loading the entire file.

Combining Techniques

The biggest gains come from combining techniques:

# Maximum memory efficiency pipeline:
# 1. Convert to Parquet (one-time)
df = pd.read_csv('huge.csv', usecols=[...], dtype={...})
df['category'] = df['category'].astype('category')
df.to_parquet('optimized.parquet', compression='zstd')

# 2. Query efficiently (every time)
df = pd.read_parquet('optimized.parquet', columns=['category', 'price'])

# Or skip pandas entirely:
import duckdb
result = duckdb.execute("""
    SELECT category, AVG(price)
    FROM 'optimized.parquet'
    GROUP BY category
""").fetchdf()

Memory Impact Summary

Further Reading

• Python Data Science Optimization for 16GB RAM — the comprehensive guide to memory-efficient workflows
• Parquet vs CSV — why Parquet is the right format
• DuckDB: SQL on Your Laptop — DuckDB as a pandas complement
• Build Your First Python Data Pipeline — these techniques in a pipeline context
• Apache Arrow Explained — the columnar format powering Parquet and DuckDB
• Local Python Docker Workspace — reproducible environment for these examples
• Why ML Models Fail in Production — data loading as a production concern

Conclusion

You do not always need more RAM. Often you need fewer columns, smaller types, compressed formats, and direct file queries. Ten techniques — from usecols to DuckDB — can reduce memory usage by 86-93% and speed up reads by 30-350x.

The best practice is simple: profile first, then apply the techniques that match your specific bottleneck. Not every technique helps every workflow — but knowing them all means you can pick the right one.