← Software

Redis

Software & architecture · databases · Sep 2024

Redis is an in-memory data-structure store: it keeps data in RAM and exposes strings, hashes, lists, sets, sorted sets, and streams as first-class types with atomic operations. Because everything lives in memory and the command set is tiny, operations take microseconds, which makes Redis the default choice for caching, counters, queues, and ephemeral state.

It is not usually your system of record. You put data in Redis that you can afford to lose or rebuild, and keep the durable copy in a database like Postgres, using Redis to make the hot paths fast.

How to use it

You pick the data structure that matches the job and lean on atomic commands and TTLs. The same key space serves sessions, counters, queues, and leaderboards:

SET session:abc "user42"      # a simple value
EXPIRE session:abc 3600       # TTL: auto-delete in one hour
INCR page:views               # atomic counter, no race
LPUSH queue:jobs "job1"       # list used as a queue
RPOP queue:jobs               # => "job1"
ZADD leaderboard 100 alice    # sorted set scored by points
ZREVRANGE leaderboard 0 2     # top three players

Key mechanics, and why

Redis is effectively single-threaded for command execution, which sounds like a limitation but is the source of its simplicity: every command is atomic with no locks, so an INCR or a sorted-set update can never interleave incorrectly. The why behind its speed is the combination of in-memory storage and that lock-free model. Durability is optional and tunable: RDB snapshots periodically, AOF logs every write, and you trade durability against throughput by choosing how often to fsync.

Common patterns

  • Cache-aside: check Redis, on a miss read the database and set the key with a TTL.
  • Rate limiting: INCR a per-user key and set an expiry on first use.
  • Queues: lists with LPUSH/RPOP, or streams for consumer groups.
  • Leaderboards: sorted sets give ranked queries in logarithmic time.
  • Pub/sub: lightweight fan-out messaging.

Internals worth knowing

Redis is far more than a string cache: it ships a rich set of server-side data structures, so you push computation to the data instead of round-tripping it. Beyond strings, hashes, lists, sets, and sorted sets there are streams (append-only logs with consumer groups), bitmaps, HyperLogLog (cardinality estimates in about 12 KB), and geospatial indexes. Commands execute on a single thread, which is why each is atomic; modern versions add I/O threads only for network reads and writes, not for command execution.

Durability is tunable. RDB writes compact point-in-time snapshots, while the AOF append-only file replays every write, and the appendfsync setting (always, everysec, no) trades durability for throughput. For atomic multi-step logic use Lua scripts or MULTI/EXEC transactions with WATCH for optimistic locking. Keys expire both lazily on access and via a background sampler, and an eviction policy such as allkeys-lru decides what to drop once maxmemory is hit.

  • Cluster mode shards keys across 16,384 hash slots; multi-key commands must land in one slot (use hash tags).
  • High availability: replicas plus Sentinel for failover, or Cluster's built-in failover.
  • Pipelining batches commands to amortize network round trips, usually the biggest single throughput win.

Using it from Python

import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)

r.set("session:abc", "user42", ex=3600)     # value with a 1-hour TTL
r.incr("page:views")                         # atomic counter
r.lpush("queue:jobs", "job1"); r.rpop("queue:jobs")
r.zadd("leaderboard", {"alice": 100}); r.zrevrange("leaderboard", 0, 2)

# atomic rate limit: INCR then EXPIRE in one round trip via a pipeline
p = r.pipeline()
p.incr("rate:user42"); p.expire("rate:user42", 60)
count, _ = p.execute()

Worked example

A fixed-window rate limiter is two atomic commands; the expiry makes the window roll automatically:

INCR rate:user42        # => 1 on the first request in the window
EXPIRE rate:user42 60   # window resets after 60 seconds
# allow the request while the counter is <= the limit, else return 429

In production

Redis shows up wherever microseconds matter. DoorDash built a gigascale ML feature store on Redis to serve model features in the request path, scaling it past 10 million and later tens of millions of reads per second after evaluating Cassandra, CockroachDB, and Scylla and choosing Redis for performance and cost (DoorDash Engineering). The pattern, an in-memory tier in front of slower stores, is nearly universal for caching, sessions, rate limiting, and feature serving.

Follow-up questions

  • Why is single-threaded execution an advantage? Every command runs atomically without locks, so concurrent updates like counters can never corrupt each other, and the code path stays simple and fast.
  • Is Redis durable? Optionally: RDB snapshots and AOF logging give tunable durability, traded against write throughput; by default treat it as a cache.
  • When would you not use Redis? As the primary store for data you cannot lose or rebuild, since it is memory-first and durability is a tunable add-on.
  • How do you cap memory? Set maxmemory with an eviction policy (e.g. allkeys-lru) so Redis drops the least-recently-used keys instead of failing.
  • Sorted set vs list for a leaderboard? A sorted set keeps elements ordered by score with logarithmic inserts and range queries; a list would need a full scan to rank.

References

  1. Redis Documentation.
  2. Kleppmann, Designing Data-Intensive Applications (2017).
  3. DoorDash Engineering, Building a Gigascale ML Feature Store with Redis.