TensorFlow "Failed to get convolution algorithm" (cuDNN) in CI

TensorFlow could not pick a cuDNN convolution algorithm. The most common reason is that cuDNN failed to initialize because the GPU was out of memory when TF grabbed all of it, though a cuDNN/CUDA version mismatch causes the same message.

What this error means

A convolution op fails with "Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above."

python

UnknownError: Failed to get convolution algorithm. This is probably because cuDNN
failed to initialize, so try looking to see if a warning log message was printed above.
[[node conv2d/Conv2D]]

Common causes

GPU memory exhausted at cuDNN init

TF preallocates the whole GPU by default; if another process holds memory, cuDNN cannot get a workspace and initialization fails.

A cuDNN or CUDA version mismatch

An incompatible cuDNN against the TF CUDA build makes cuDNN initialization fail with the same generic message.

How to fix it

Enable GPU memory growth

Let TF grow GPU memory on demand instead of grabbing it all, leaving room for cuDNN to initialize.

train.py

import tensorflow as tf
for g in tf.config.list_physical_devices("GPU"):
    tf.config.experimental.set_memory_growth(g, True)

Align cuDNN and CUDA with the TF build

Check the warning above the error; if it names cuDNN, install the version this TensorFlow was built against.

Terminal

python -c "import tensorflow as tf; print(tf.sysconfig.get_build_info())"

How to prevent it

Set memory growth so cuDNN always has init headroom.
Match cuDNN/CUDA to the TensorFlow build matrix.
Avoid running two GPU processes on the same card in one job.