CockroachDB "restart transaction: TransactionRetryError" in CI

CockroachDB uses SERIALIZABLE isolation. When concurrent transactions conflict, it aborts one with a retryable error and expects the client to retry. Tests that run transactions in parallel can hit this nondeterministically in CI.

What this error means

A transaction fails with "restart transaction: TransactionRetryError: ... RETRY_SERIALIZABLE" (SQLSTATE 40001), often only on some runs, which looks like flakiness.

cockroach sql

ERROR: restart transaction: TransactionRetryWithProtoRefreshError:
TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh)
SQLSTATE: 40001

Common causes

Concurrent transactions contend for the same rows

Two transactions read and write overlapping data; under SERIALIZABLE one must restart to preserve isolation.

The client does not retry 40001

Code that treats the retryable error as fatal fails the job instead of re-executing the transaction.

How to fix it

Retry transactions on SQLSTATE 40001

Catch the 40001 retry error around the transaction.
Re-run the whole transaction body, not just the failed statement.
Cap retries with backoff to avoid an infinite loop.

python

for attempt in range(5):
    try:
        with conn.transaction():
            run_statements()
        break
    except psycopg.errors.SerializationFailure:
        time.sleep(0.1 * (attempt + 1))

Reduce contention in test fixtures

Give parallel tests separate rows or databases so transactions do not collide as often.

How to prevent it

Wrap transactions in a retry helper that handles 40001.
Isolate parallel test data to cut contention.
Treat retry errors as expected, not as test failures.