iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Categorical

By · Reviewed & last updated · Part of NumPy & Pandas Tutorial

A pandas categorical column stores repeated string values as integer codes pointing into a small list of unique categories. Memory drops dramatically (an "active/cancelled/paid" column with 10 million rows fits in a few MB), and groupby/filter on categories is faster. Two flavours: ordered (Low < Med < High) and unordered.

Convert, order, and groupby on categoricals

EXAMPLE
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'priority': np.random.choice(['Low', 'Med', 'High'], 1_000_000),
    'status':   np.random.choice(['new', 'paid', 'cancelled'], 1_000_000),
    'amount':   np.random.uniform(10, 500, 1_000_000),
})

# Memory before
print('before:', df.memory_usage(deep=True).sum() / 1024**2, 'MB')

# 1) Convert to category — huge memory win for low-cardinality strings
df['status'] = df['status'].astype('category')

# 2) Ordered categorical: comparisons and sort respect the order you give
priority_type = pd.CategoricalDtype(['Low', 'Med', 'High'], ordered=True)
df['priority'] = df['priority'].astype(priority_type)

print('after:',  df.memory_usage(deep=True).sum() / 1024**2, 'MB')

# 3) Ordered comparisons just work
high_pri = df[df['priority'] >= 'Med']
print('high+med rows:', len(high_pri))

# 4) Groupby keeps a stable category order in the index
summary = (df.groupby(['status', 'priority'], observed=True)['amount']
             .agg(['count', 'mean', 'sum']))
print(summary.head(10))

# 5) Renaming/reordering categories without changing the data values
df['priority'] = df['priority'].cat.rename_categories({'Low': 'L', 'Med': 'M', 'High': 'H'})
df['priority'] = df['priority'].cat.reorder_categories(['H', 'M', 'L'], ordered=True)

# 6) Adding a new category before you set values to it
df['status'] = df['status'].cat.add_categories(['refunded'])
df.loc[df.index[:100], 'status'] = 'refunded'

# 7) Reading CSVs into categoricals directly to avoid the conversion pass
df2 = pd.read_csv('orders.csv',
    dtype={'status': 'category', 'priority': priority_type})

Why it matters

Pass observed=True to groupby on categoricals or pandas will emit a row for every unused category combination — fine for two columns, miserable when you have three categoricals with 50 categories each. The default behaviour was tuned for spreadsheet-style reports, not modern analytics.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
import pandas as pd
cat = pd.Categorical(['S','M','L','M'], categories=['S','M','L'], ordered=True)
df['size'] = pd.Categorical(df['size'], categories=['S','M','L','XL'], ordered=True)
Try it Yourself »

Discussion

Loading…