iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Performance Tips

By · Reviewed & last updated · Part of NumPy & Pandas Tutorial

Pandas performance work is mostly about avoiding row-by-row Python. Vectorise with NumPy ops, push string work to .str, use categoricals for low-cardinality strings, and reach for pyarrow-backed dtypes for memory and speed. When a single machine is not enough, switch engines — polars, duckdb, or dask — without rewriting your analysis.

Vectorise, dtypes, profiling, and engine alternatives

EXAMPLE
import pandas as pd
import numpy as np
import time

# 1) Avoid apply on rows — it's a Python loop wearing a pandas suit
df = pd.DataFrame({
    'qty':   np.random.randint(1, 5, 1_000_000),
    'price': np.random.uniform(10, 100, 1_000_000),
})

# SLOW
t = time.time()
df['total_slow'] = df.apply(lambda r: r['qty'] * r['price'], axis=1)
print('apply:', time.time() - t, 's')

# FAST — vectorised, no Python loop
t = time.time()
df['total_fast'] = df['qty'] * df['price']
print('vectorised:', time.time() - t, 's')   # ~100-1000x faster

# 2) Categoricals — huge memory win on low-cardinality strings
df['region'] = np.random.choice(['Sydney', 'Melbourne', 'Brisbane'], 1_000_000)
print('before:', df.memory_usage(deep=True).sum() / 1024**2, 'MB')
df['region'] = df['region'].astype('category')
print('after:',  df.memory_usage(deep=True).sum() / 1024**2, 'MB')

# 3) PyArrow-backed dtypes (pandas 2.0+) — faster string/int, native NA
sdf = pd.DataFrame({
    'name':  ['Alice', 'Bob', None, 'Dee'],
    'score': [10, 20, None, 30],
}, dtype={'name': 'string[pyarrow]', 'score': 'Int64'})

# 4) read_csv with the right dtypes upfront — avoid the expensive inference pass
fast_df = pd.read_csv('orders.csv',
    dtype={'status': 'category', 'qty': 'int32'},
    parse_dates=['created_at'],
    engine='pyarrow',     # fastest CSV reader available
)

# 5) Profile WHERE time is spent — never optimise blind
%%timeit                   # in Jupyter
df.groupby('region', observed=True)['total_fast'].sum()
# or
import cProfile; cProfile.run('df.groupby("region").sum()')

# 6) Use .eval / .query for compiled expressions over large frames
filtered = df.query('qty >= 2 and price > 50')
df['rev'] = df.eval('qty * price * 1.10')        # adds tax

# 7) groupby + agg multiple stats in one pass — beats multiple separate aggregations
agg = df.groupby('region', observed=True).agg(
    orders=('qty', 'sum'),
    revenue=('total_fast', 'sum'),
    avg_price=('price', 'mean'),
)

# 8) When you outgrow pandas:
#    - polars:  drop-in DataFrame, multi-threaded, lazy plan
#         import polars as pl; pl.read_csv('orders.csv').group_by('region').sum()
#    - duckdb:  SQL over pandas frames / parquet / CSV with no copy
#         import duckdb; duckdb.sql('SELECT region, SUM(price) FROM df GROUP BY region')
#    - dask:    parallel pandas across cores or a cluster
#         import dask.dataframe as dd; dd.read_csv('orders/*.csv').groupby('region').sum().compute()

# 9) Read columnar files (parquet, arrow) when bytes matter
df.to_parquet('orders.parquet', engine='pyarrow', compression='zstd')
# Parquet on a typical analytical column = 5-10x smaller than CSV, faster to filter

Why it matters

On laptops, polars or duckdb beats pandas for analytical workloads by big margins. Use pandas for ergonomic prototyping and short-lived notebooks; switch to polars/duckdb when the same operation is going to run nightly in production, or when the dataset stops fitting comfortably in memory.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# Use vectorised ops, not for loops.
# Avoid object dtype (esp. strings) when you can — try 'string[python]' or 'category'.
# For huge data: read in chunks, or use PyArrow / Polars.
Try it Yourself »

Discussion

Loading…