Parquet vs CSV: Why Data Scientists Should Care

Published August 2026 · Data Science · ~1,800 words

Every data scientist has opened a CSV file. It is the universal format — every tool reads it, every person can open it, and every database exports to it. But when performance matters, CSV is often the wrong choice.

This article compares CSV and Parquet using real benchmark measurements, explains why the differences exist, and shows when each format is appropriate.

Featured image: articles/080/featured-image.svg

What Is CSV?

CSV (Comma-Separated Values) is a plain-text format where each row is a line and values are separated by commas.

id,category,value_a,quantity,flag
0,alpha,345.67,42,true
1,beta,892.12,17,false
2,gamma,123.45,88,true

Advantages:

• Human-readable
• Universal tool support
• No binary encoding — easy to inspect, diff, and email
• Simple to generate

Disadvantages:

• No schema — everything is text
• No compression — repeats are stored in full
• No column selection — the entire file must be read
• Text parsing overhead for every numeric value
• Data types are lost on reload

What Is Parquet?

Apache Parquet is a columnar binary format designed for analytical workloads. Instead of storing data row by row, it stores each column together, applies compression per-column, and embeds schema information.

┌──────────────────────────────────────┐
│           Parquet File               │
│  ┌─────────┐  ┌─────────┐  ┌──────┐ │
│  │ Column  │  │ Column  │  │  ... │ │
│  │  Store  │  │  Store  │  │      │ │
│  └─────────┘  └─────────┘  └──────┘ │
│         Schema + Metadata            │
│       Compression per column         │
│        Row groups for I/O            │
└──────────────────────────────────────┘

Advantages:

• Columnar storage — reads only the columns you need
• Built-in compression — Snappy, Zstd, Gzip, LZ4
• Schema and data types preserved
• Predicate pushdown — filters applied before reading
• Optimized for analytical queries

Disadvantages:

• Binary — not human-readable
• Requires library support (PyArrow, FastParquet)
• Slightly more complex for simple interchange

The Benchmark: Real Numbers

All numbers in this article come from a single reproducible benchmark run on August 2026 using a synthetic dataset of 500,000 rows × 8 columns.

Environment:

Diagrams: articles/080/benchmark-results.svg

File Size

Parquet with Zstd compression produced a file 8.6× smaller than CSV. This is not just disk savings — smaller files mean faster I/O, less network transfer, and lower storage costs.

Write Speed

Parquet writing was significantly faster in this benchmark. PyArrow's columnar encoding avoids the per-row text formatting overhead that CSV requires.

Read Full Dataset

Reading the full dataset with Parquet (Snappy) was 3.7× faster than CSV. The Zstd-compressed Parquet was even faster at decompression due to better CPU efficiency in modern pyarrow.

Column Selection (2 of 8 Columns)

This is where Parquet's columnar design shines brightest. When you only need 2 of 8 columns, Parquet skips the other 6 entirely — they are never read from disk. CSV must parse every line to extract the columns you want.

Row Filtering (Predicate Pushdown)

Parquet's predicate pushdown applies the filter category == 'alpha' at the storage level, reading only matching row groups. The result: 14.1× faster filtering with identical output.

Group-By Aggregation

Reading only the needed columns from Parquet and computing the group-by aggregation was 18.5× faster than loading the full CSV and then aggregating.

Schema Preservation

This difference is subtle but critical for production systems.

# Writing with optimized dtypes
df['category'] = df['category'].astype('category')
df['region']   = df['region'].astype('category')
df.to_parquet('data.parquet')

# Reading back — types preserved
df = pd.read_parquet('data.parquet')
print(df['category'].dtype)  # category ✓

# CSV loses this
df.to_csv('data.csv')
df = pd.read_csv('data.csv')
print(df['category'].dtype)  # object ✗

Parquet embeds schema metadata. When you save a column as category, it loads as category. CSV always reloads as text — you must manually re-apply type conversions every time.

When to Use CSV

CSV is still the right choice when:

• Human readability matters — debugging, auditing, data sharing with non-technical users
• Simple interchange — moving data between tools that may not share Parquet support
• Small datasets — where the performance difference is negligible
• Log files and exports — append-friendly, easy to inspect
• APIs and data feeds — many systems default to CSV for simplicity

When to Use Parquet

Parquet is the better choice when:

• Analytical queries — aggregation, filtering, grouping over large datasets
• Selective column access — reading a subset of columns from wide datasets
• Repeated reads — the one-time write cost pays off across many reads
• Pipeline storage — intermediate files between ETL steps
• Data lakes and warehouses — Parquet is the standard format in Spark, DuckDB, BigQuery, and S3-based data lakes
• Type-sensitive workflows — when preserving dtypes matters
• Cost-sensitive cloud storage — 8.6× less storage means 8.6× less cost per GB

How Parquet Achieves These Gains

The performance differences are not magic. They come from specific design decisions:

The Conversion Workflow

If you have CSV files and want to switch to Parquet, the conversion is straightforward:

import pandas as pd

# Read CSV
df = pd.read_csv('large_dataset.csv')

# Optimize dtypes where helpful
df['category'] = df['category'].astype('category')
df['region']   = df['region'].astype('category')

# Write Parquet
df.to_parquet('large_dataset.parquet', engine='pyarrow', compression='zstd')

# Verify
df_check = pd.read_parquet('large_dataset.parquet')
print(f'CSV:     {df.memory_usage(deep=True).sum() / 1e6:.1f} MB in memory')
print(f'Parquet: {os.path.getsize("large_dataset.parquet") / 1e6:.1f} MB on disk')

For very large files, use chunked reading:

import pandas as pd
import os

chunks = pd.read_csv('huge.csv', chunksize=100_000)
for i, chunk in enumerate(chunks):
    chunk.to_parquet(f'part_{i:04d}.parquet', engine='pyarrow')

The Parquet files can later be read individually or together with a glob pattern.

Parquet in the Data Science Ecosystem

Parquet is not a niche format. It is the default or preferred format for:

• Apache Spark — native Parquet support for distributed processing
• DuckDB — direct Parquet querying without loading into memory
• Polars — optimized Parquet reader/writer
• BigQuery, Snowflake, Redshift — all support Parquet ingestion
• AWS S3, Azure Blob, GCS — standard format for data lakes
• Hugging Face Datasets — Parquet is the default storage format
• Dask — parallel Parquet reading and writing

The In-Memory Difference

Once data is loaded into a pandas DataFrame, both formats produce similar memory usage — the DataFrame lives in RAM regardless of its source format. In our benchmark:

The small difference comes from Parquet preserving category dtype, which is more memory-efficient than object columns. The bigger advantage of Parquet is on disk and in I/O speed — not in RAM after loading.

Common Questions

Is Parquet always faster than CSV?

No. For very small files (under a few MB), the difference is negligible and CSV may be faster due to simpler parsing. Parquet's advantages grow with file size, column count, and analytical query complexity.

Can I still use CSV alongside Parquet?

Absolutely. Many pipelines use CSV for ingestion and export, and Parquet for internal storage and analytics. Convert once, benefit many times.

What about JSON?

JSON is even less efficient than CSV for tabular data because of repeated key names and nested structures. For tabular analytics, Parquet is almost always better. JSON excels for semi-structured or document-oriented data.

Does Parquet work with streaming data?

Parquet files are designed for batch writes. For streaming, you can write Parquet files in regular intervals (e.g., every 5 minutes or every N rows) or use other formats like Avro for true streaming.

Best Practices

Further Reading

• Python Data Science Optimization for 16GB RAM — when Parquet matters most for memory-constrained machines
• Build a Private Local AI Assistant — where Parquet storage plays a role in document indexing
• Local Python Docker Workspace for Students — setting up the environment to run these examples
• Local AI in 2026 — understanding hardware constraints for data processing
• Imbalanced Datasets — a workflow where Parquet preserves label distributions efficiently
• Feature Engineering in the Age of AI — Parquet as intermediate storage for feature pipelines
• Why ML Models Fail in Production — storage format as part of the production pipeline

Conclusion

CSV is simple, universal, and human-readable. It remains the best choice for quick data interchange and debugging.

But for analytical workloads — filtering, aggregation, column selection, repeated reads — Parquet is demonstrably superior. Our benchmark showed 8.6× smaller files, 3.7× faster full reads, and 18.5× faster aggregation on a 500K-row dataset.

The best practice is simple: use CSV for sharing and human inspection. Use Parquet for everything else.