Check for attention mask in backends that don't support it

[dxqb]

What does this PR do?

Using an attention backend (https://huggingface.co/docs/diffusers/main/optimization/attention_backends) with a model that passes attention masks yields incorrect results.

This is already checked in parallel backends...

diffusers/src/diffusers/models/attention_dispatch.py

Line 978 in f6b6a71

raise ValueError("`attn_mask` is not yet supported for flash-attn 2.")

...but not yet in the regular ones.

This PR changes that.
Fixes #12605

Who can review?

@yiyixuxu and @asomoza
CC @zzlol63 @tolgacangoz

[yiyixuxu]

thanks!

[yiyixuxu]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Thanks!

But we don't even support passing attn_mask in the respective methods. So, I am not sure how this is applicable.

[dxqb]

Thanks!

But we don't even support passing attn_mask in the respective methods. So, I am not sure how this is applicable.

True, but that is the point: Those backends don't have an 'attn_mask' parameter currently. Therefore, when the caller passes an 'attn_mask' argument, the attention mask is silently ignored here:

diffusers/src/diffusers/models/attention_dispatch.py

Line 343 in 05014a3

if _AttentionBackendRegistry._checks_enabled:

if checks are disabled, which they are by default.

Ignoring extra arguments might be valid behaviour for other paremters, but not for attention masks. Ignoring an attention mask yields incorrect results. It should fail instead of returning incorrect results - that's what this PR does.

@yiyixuxu your style bot changed a line I didn't even mean to insert. Fixed.