iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Compile Once, Reuse

By · Reviewed & last updated · Part of RegEx Tutorial

Compiling a regex parses the pattern, builds the NFA/DFA, and caches it. Doing that inside a hot loop runs the work N times for no benefit. The fix is to hoist compilation to module level (or memoise it) so subsequent matches are pure execution. The win is largest in interpreted languages with non-trivial pattern caches.

Compile patterns once, reuse them N times

EXAMPLE
import re, time, timeit

LINE = '203.0.113.5 - - [11/Jun/2026:10:32:14 +1000] "GET /api/users HTTP/1.1" 200 1532'
PATTERN = r'^(?P<ip>\d+\.\d+\.\d+\.\d+) \S+ \S+ \[(?P<ts>[^\]]+)\] \"(?P<method>[A-Z]+) (?P<path>[^ ]+) HTTP/[\d.]+\" (?P<status>\d{3}) (?P<bytes>\d+|-)'

# 1) WRONG: recompiles every call (N times)
def slow(lines):
    out = []
    for l in lines:
        m = re.match(PATTERN, l)
        if m: out.append(m.groupdict())
    return out

# 2) RIGHT: compile once at module scope
LOG_RE = re.compile(PATTERN)
def fast(lines):
    return [m.groupdict() for l in lines if (m := LOG_RE.match(l))]

lines = [LINE] * 10_000
print('slow:', timeit.timeit(lambda: slow(lines), number=1))
print('fast:', timeit.timeit(lambda: fast(lines), number=1))

# 3) Why slow() is not always disastrous: Python caches the last ~512
#    patterns. The cache is shared per-process and is OK for very small
#    workloads — but it's a global mutex and an LRU lookup per call.
#    Explicit compile() bypasses all of that.

# 4) Dynamic patterns: cache them yourself with functools.lru_cache
from functools import lru_cache
@lru_cache(maxsize=256)
def pattern_for(field): return re.compile(rf'^{field}=(?P<value>\S+)')

def parse_kv(field, line):
    m = pattern_for(field).match(line)
    return m.group('value') if m else None

# 5) Other languages — same shape:
# JS:     const RE = /^.../;  // module-level
# Java:   private static final Pattern RE = Pattern.compile("^...");
# Go:     var RE = regexp.MustCompile("^...")        // package var
# .NET:   static readonly Regex RE = new("^...", RegexOptions.Compiled);
# Ruby:   PATTERN = /^.../          # frozen literal, compiled at parse time

# 6) Beware: do NOT compile() inside a class __init__ if instances are
#    short-lived in a tight loop — the compile still happens N times.
#    Move it to the class (not instance) level: PATTERN = re.compile(...)
class LogParser:
    PATTERN = re.compile(PATTERN)
    @classmethod
    def parse(cls, line): return cls.PATTERN.match(line)

Why it matters

For dynamic patterns built from user input or per-request, memoise the compiled regex with an LRU cache keyed on the source string. The cache size caps memory if attackers can drive cardinality, and the hit rate on legitimate traffic is usually well above 99%.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
// In hot paths, compile patterns once and reuse.
const KEY = /^[A-Z_][A-Z0-9_]*$/;
function isKey(s) { return KEY.test(s); }
Try it Yourself »

Discussion

Loading…