HyperLogLog

# ===== Why HLL =====
# Counting uniques exactly = O(N) memory.
# HyperLogLog estimates with ~0.81% error using a fixed ~12 KB per key.
# Trade exactness for huge memory savings; great for DAU, unique IPs, etc.

# ===== Add elements =====
PFADD visitors:2024-04-10 user:1 user:2 user:3 user:1
# Returns 1 if the cardinality estimate changed; 0 otherwise.

PFADD visitors:2024-04-10 user:4 user:5
# (the same key keeps merging into the same HLL)

# ===== Count =====
PFCOUNT visitors:2024-04-10
# Returns the estimated cardinality (number of unique items).

# Multiple keys merged + counted in one shot:
PFCOUNT visitors:2024-04-10 visitors:2024-04-11

# ===== Merge =====
# Union sets without expanding to a real set:
PFMERGE visitors:weekly visitors:2024-04-10 visitors:2024-04-11 visitors:2024-04-12

PFCOUNT visitors:weekly

# Useful for rolling counts.

# ===== Memory =====
# A HLL key uses ~12 KB regardless of how many items it tracks. You can fit
# millions of distinct items in 12 KB at <1% relative error.

# ===== Real example: daily unique visitors =====
# Every request:
def visit(user_id):
    today = datetime.utcnow().strftime('%Y-%m-%d')
    redis.pfadd(f'visitors:{today}', user_id)

# Daily report:
def daily_unique(date):
    return redis.pfcount(f'visitors:{date}')

# Weekly:
def weekly_unique(start_date):
    keys = [f'visitors:{(start_date + timedelta(d)).strftime("%Y-%m-%d")}' for d in range(7)]
    return redis.pfcount(*keys)

# ===== When HLL wins =====
# - DAU / MAU / WAU
# - Unique IPs / clients in fraud signals
# - Unique device fingerprints
# - 'How many distinct X did we see between A and B'

# ===== When HLL hurts =====
# - You need EXACT counts (small sets) — use a SET
# - You need per-element TTLs (HLL is union-only)
# - You need to LIST the unique items (HLL only counts)

# ===== Patterns to internalise =====
# - HLL for unbounded uniques; SET for small bounded uniques
# - One HLL per time bucket; merge across buckets for ranges
# - Pair HLL with a Redis Stream / log if you ALSO need the raw events
# - Pin a TTL on HLL keys if they are time-bound

# ===== Pitfalls =====
# - Treating PFCOUNT as exact (it isn't; 0.81% relative error)
# - Using HLL for very small sets — exact SET is fine and gives you values
# - Forgetting that PFMERGE writes a new key (you must pre-allocate / TTL it)
# - Not realising that PFCOUNT on multiple keys MUTATES nothing (good) — single key may write a small cache slot (a Redis micro-optimisation)

PFADD visitors "ip1" "ip2" "ip3"
PFCOUNT visitors        # cardinality estimate (~12 KB / key)