Transactions and Pipelines

MULTI, EXEC, DISCARD

A Redis transaction queues commands and then runs the whole queue as one atomic unit. No other client's command can interleave, because the server is single threaded and EXEC executes the batch without yielding.

MULTI
SET  hu:user:1042:name "Ada"      # QUEUED
HSET hu:user:1042 plan free       # QUEUED
INCR hu:stats:users               # QUEUED
EXEC
# 1) OK
# 2) (integer) 1
# 3) (integer) 7

Every command between MULTI and EXEC replies QUEUED rather than running. The real replies arrive as one array from EXEC, positionally matched to the queue.

EXEC outside a transaction is ERR EXEC without MULTI, and the same for DISCARD. MULTI cannot nest.

There is no rollback

This is the fact that makes Redis transactions different from a SQL transaction, and it surprises everyone. A command that fails at runtime does not abort the transaction, and nothing before it is undone.

SET tk good
MULTI
SET   tk good2       # QUEUED
LPUSH tk x           # QUEUED, the error is not detectable yet
INCR  tk             # QUEUED
EXEC
# 1) OK
# 2) (error) WRONGTYPE Operation against a key holding the wrong kind of value
# 3) (error) ERR value is not an integer or out of range
GET tk               # "good2"  <- the successful command STUCK

The successful SET is permanent. Redis has no rollback and no undo log, so a transaction is "all commands are attempted, in order, with nothing interleaved", not "all or nothing". Your client must inspect every element of the EXEC array for an error.

A command Redis can reject at queue time is different. A syntax error or an unknown command marks the transaction dirty, and EXEC then refuses to run anything at all.

MULTI
NOSUCHCMD a          # (error) ERR unknown command 'NOSUCHCMD'
SET tk2 v            # QUEUED
EXEC
# (error) EXECABORT Transaction discarded because of previous errors.
EXISTS tk2           # (integer) 0, nothing ran

So the rule: EXECABORT means nothing happened, any other error means part of the batch landed.

Two more behaviors worth knowing. Blocking commands do not block inside MULTI, they return their empty reply immediately, so BLPOP in a transaction is pointless. And you cannot read a value mid-transaction to decide the next command, because nothing has executed yet. That is what WATCH and Lua are for.

WATCH, optimistic locking

WATCH turns a transaction into a check-and-set. If any watched key is modified by anyone between the WATCH and the EXEC, EXEC returns nil and runs nothing.

WATCH hu:account:9:balance
# read it OUTSIDE the transaction, so you actually get a value
GET   hu:account:9:balance        # "100"
# decide in your application code
MULTI
SET   hu:account:9:balance 50
EXEC
# 1) OK            <- committed
# (nil)            <- someone else touched the key, retry the whole thing

The loop, which is the pattern rather than an optimization:

def withdraw(r, key, amount):
    while True:
        with r.pipeline() as pipe:
            try:
                pipe.watch(key)                 # WATCH, connection now immediate
                balance = int(pipe.get(key) or 0)
                if balance < amount:
                    pipe.unwatch()
                    return False
                pipe.multi()                    # start queueing
                pipe.set(key, balance - amount)
                pipe.execute()                  # EXEC, raises if the key changed
                return True
            except redis.WatchError:
                continue                        # someone won the race, try again

Details that matter:

- WATCH must come before you read. Watching after the read leaves the exact window you were trying to close. - Reads happen while the connection is in immediate mode, between WATCH and MULTI. - Any successful EXEC, plus DISCARD, UNWATCH, and RESET, clears every watch. - A watched key counts as modified if it was touched, not if its value changed. Writing the same value back still aborts, and so does the key expiring. - Watching many keys under contention livelocks. This is optimistic concurrency, so it only pays off when conflicts are rare. Under heavy contention a Lua script is both simpler and faster, because it never retries.

Bounded retries, always. An unbounded while True under sustained contention is an outage.

Pipelining

Pipelining is a network optimization, not a transaction. The client writes many commands without waiting for each reply, then reads all the replies. It collapses N round trips into one.

# One RTT instead of three
printf 'SET a 1\r\nSET b 2\r\nGET a\r\n' | redis-cli --pipe

# Bulk load, the fast path for millions of keys
for i in $(seq 1 100000); do echo "SET hu:k:$i $i"; done | redis-cli --pipe
# All data transferred. Waiting for the last reply...
# Last reply received from server.
# errors: 0, replies: 100000

pipe = r.pipeline(transaction=False)     # a PLAIN pipeline, no MULTI
for i in range(1000):
    pipe.set(f"hu:k:{i}", i)
results = pipe.execute()                 # one round trip, 1000 replies

Note transaction=False in redis-py. Its pipeline() wraps the batch in MULTI/EXEC by default, which is a different thing and slightly slower.

A pipeline is not atomic. MULTI is.

The two compose. Sending MULTI ... EXEC down a pipeline is the normal way a client implements a transaction, and that gives you one round trip and atomicity.

Sizing: batch in the low thousands. A pipeline of a million commands buys nothing over one of ten thousand and inflates both the client's reply buffer and the server's output buffer. Watch client-output-buffer-limit for the same reason.

Choosing between the three

Reach for the command's own conditional flags first, since SET key v NX PX 30000 does in one command what a WATCH loop does in four round trips. Reach for Lua when the logic genuinely needs to read then decide. Reach for MULTI when you just want a fixed batch to land together, and reach for a plain pipeline when you only want the latency back.