Continuous batching auto-switches to flash_attention_3 on unsupported GPUs (Turing/T4), crashing — regression from 5.14.1

[albertvillanova]

System Info

• Affected: transformers==5.15.0 (also current main / 5.16.0.dev0). Not reproduced on the v5.14.1 release.
• Same environment also moved kernels==0.15.2 → kernels==0.16.0, and the kernels-community/vllm-flash-attn3 Hub kernel is now pulled.
• GPU: NVIDIA T4 (Turing, SM 7.5), i.e. pre-Ampere. flash_attn is not pip-installed (only transformers[kernels]).
• Container: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel.

Who can help?

cc @remi-or @IlyasMoutawwakil (continuous batching)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Running generation with continuous batching (generate_batch / use_transformers_continuous_batching=True) on a pre-Ampere GPU (NVIDIA T4).

On 5.15.0, continuous batching auto-switches the attention implementation from sdpa to flash_attention_3 and loads the kernels-community/vllm-flash-attn3 Hub kernel:

ContinuousBatchingLogger - WARNING - Continuous batching is much better when using flash attention. Switching from sdpa to flash_attention_3. If you need to use eager or sdpa, use paged|eager or paged|sdpa as the `attn_implementation`.
transformers.modeling_utils - WARNING - You do not have `flash_attn` installed, using `paged|kernels-community/vllm-flash-attn3` from the `kernels` library instead!

The generation loop then crashes, because that kernel requires newer hardware:

ContinuousBatchingLogger - ERROR - Error in generation loop: mha_fwd, /build/source/flash-attn/flash_api_stable.cpp:716, FlashAttention only supports Ampere GPUs or newer.
ContinuousBatchingLogger - ERROR - Generation thread terminated unexpectedly.
ContinuousBatchingLogger - ERROR - Requests ['req_0', 'req_1', 'req_2'] not found in results.

Because the generation thread dies, no completions are returned, which then breaks downstream code that assumes at least one completion.

The same code runs fine on v5.14.1 (with kernels==0.15.2): continuous batching does not switch to flash_attention_3, and generation completes normally.

Observed in CI on a single-GPU T4 runner (downstream project TRL):

• Passing: transformers==5.14.1 / kernels==0.15.2 — https://github.com/huggingface/trl/actions/runs/31371150907/job/93400203818
• Failing: transformers==5.15.0 / kernels==0.16.0 — https://github.com/huggingface/trl/actions/runs/31582787840/job/94069521759

Workaround: pass attn_implementation="paged|sdpa" (or paged|eager).

Expected behavior

Continuous batching should not switch to flash_attention_3 on GPUs that cannot run it. The auto-switch should be gated on device capability (or fall back to sdpa) instead of hard-crashing the generation thread, as was the case on v5.14.1.

Note: I don't yet have a minimal non-CI reproduction on non-Ampere hardware; the above is from CI logs plus a version bisect (model_runner.py is byte-identical between v5.14.1 and v5.15.0, so the change is in the attention/kernel default selection). I'll add the exact regressing change — and whether it originates in transformers or in the kernels default resolution — as a follow-up once pinned.

[jyothimalla]

Hi @albertvillanova, thanks for the detailed writeup — I dug into this and found the root cause.

is_flash_attn_3_available(kernels_fallback_ok=True) in utils/import_utils.py only checks
whether the kernels-community/vllm-flash-attn3 package can be downloaded/imported via
get_kernel() — it never checks the GPU's actual compute capability. On a T4 (Turing, SM 7.5),
the kernel package downloads and imports fine, so the function returns True, and
switch_to_cb_friendly_attn in continuous_api.py confidently switches to flash_attention_3.
The crash only happens later, at kernel launch time inside the generation thread.

I confirmed against the flash-attention README that FA2 also requires Ampere/Ada/Hopper via the
kernels-community path — Turing needs a separate flash-attention-turing repo not used here —
so both FA3 and FA2 auto-selection need the same hardware gate.

I've written a fix that adds a device-capability check in switch_to_cb_friendly_attn before
selecting FA3/FA2, falling back to sdpa on unsupported hardware instead of crashing. I'm
finishing up a regression test now and will open a PR shortly — wanted to flag I'm actively
working on this so it's not duplicated. cc @remi-or @IlyasMoutawwakil

[albertvillanova]

Thanks @jyothimalla — your analysis matches what I found.

To pin the regression: it comes from #47330 ("[CB] Automatically switch attention implementation to flash", @remi-or, merged 2026-07-16), which is why v5.14.1 works and v5.15.0 doesn't. Before it, CB only ensured paged attention (switch_to_paged_attn, over the existing sdpa/eager); #47330 added switch_to_cb_friendly_attn, which additionally auto-switches to flash (FA3 → FA2, package or kernels) when available.

The auto-switch itself is intended (a deliberate perf improvement — CB without flash pays a large attention-mask cost). The regression is a narrower side effect: is_flash_attn_3_available(kernels_fallback_ok=True) (and the FA2 variant) treat flash as "available" if the kernel can be fetched/imported via get_kernel(...), without checking the GPU's compute capability. On a T4 (SM 7.5) the kernels-community/vllm-flash-attn3 kernel downloads and imports fine, so the check returns True, switch_to_cb_friendly_attn switches to flash_attention_3, and it only crashes later at kernel-launch time.

So the fix is a device-capability gate in the is_flash_attn_*_available(kernels_fallback_ok=True) fallbacks (or in switch_to_cb_friendly_attn), so CB falls back to sdpa/paged on hardware that can't run flash while keeping #47330's behavior on supported GPUs. The kernels 0.15.2 → 0.16.0 bump only makes the FA3 kernel resolvable so the fallback returns True; it isn't the root cause.

A regression test on pre-Ampere hardware (or one that mocks is_flash_attn_*_available) would guard it. Thanks for picking it up @jyothimalla!

[jyothimalla]

@albertvillanova Thanks for pinning down the exact regression in #47330 and confirming the approach — really helpful context. Yes, I'd like to go ahead with this. I'll place the device-capability gate in switch_to_cb_friendly_attn and add a regression test that mocks the availability checks (since I don't have Turing hardware locally). Will open a PR once it's ready.