Quiz

# ============================================================
# Q1) Why does df['a'][i] = x raise SettingWithCopyWarning?
# ============================================================
# ANSWER: chained indexing means pandas cannot guarantee whether you are
# writing to the original frame or to a copy. Use:
#   df.at[i, 'a'] = x        # scalar
#   df.loc[i, 'a'] = x       # general
# Avoid 'df[df.x>0]['a'] = ...' patterns — they often silently no-op.

# ============================================================
# Q2) Why is .apply on rows so slow?
# ============================================================
# ANSWER: it's a Python loop wearing a pandas suit. Vectorise:
#   df['c'] = df['a'] + df['b']                       # fast
#   df['c'] = df.apply(lambda r: r['a'] + r['b'], axis=1)  # slow
# Or use numpy.where / np.select for branching logic.

# ============================================================
# Q3) groupby on 100M rows is killing memory. Now what?
# ============================================================
# ANSWER, in order:
#   - Categorical for low-cardinality string keys (huge memory win)
#   - dtype= when reading; avoid object dtype on numeric columns
#   - chunked reads (pd.read_csv(chunksize=...)) + group-merge in passes
#   - switch to polars or duckdb (sql over parquet on disk)

# ============================================================
# Q4) Joining lost 30% of my rows. What happened?
# ============================================================
# ANSWER: mismatched key dtypes ('1' as str vs 1 as int), nulls in the join
# column, or duplicate keys on one side. Add indicator=True to merge and
# inspect _merge column for left_only / right_only counts.

# ============================================================
# Q5) Date arithmetic across timezones gave wrong totals.
# ============================================================
# ANSWER: pandas datetime columns are tz-naive by default. Either:
#   - convert to UTC at the boundary: pd.to_datetime(s, utc=True)
#   - tz_localize then tz_convert when summarising by day
# Truncate to day boundaries with .dt.tz_convert('Australia/Sydney').dt.floor('D')

# ============================================================
# Q6) Reading a 5GB CSV every run is too slow.
# ============================================================
# ANSWER:
#   - Save it as parquet once: df.to_parquet('x.parquet')
#   - read_parquet is 10-50x faster and stores dtypes natively
#   - With duckdb you can SELECT … FROM 'x.parquet' WHERE ... without loading
#     the whole file into memory

# ============================================================
# Bonus — when is NumPy faster than pandas?
# ============================================================
# ANSWER: when you do pure math on a 2D array. df.values gives you the
# underlying NumPy array; do the math there, then re-wrap. Pandas overhead
# is in the index / dtype tracking that NumPy skips.

# ============================================================
# Scoring
# ============================================================
# 6 / 6 -> teach pandas review
# 4 / 6 -> bookmark the cheatsheet
# < 4   -> a focused day with the pandas user guide

# 3 questions per lesson.