Flash Attention 2 #64
Description
Hi @Zhihan1996,
You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.
{{ message }}
Hi @Zhihan1996,
You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.