[Feature Request] Group-Query Attention Decode

* [ ] Refactor
  * Function: group_query_attention_decode_with_kvcache
    * ref: https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#how-to-use-flashattention
    * ref: https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
    * ref: https://docs.flashinfer.ai/generated/flashinfer.decode.single_decode_with_kv_cache.html
  * Op: GroupQueryAttentionDecodeWithKVCacheOp
  * Kernel
* [ ] Test
* [ ] Benchmark
  * Baselines: flash-attention, triton

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature Request] Group-Query Attention Decode #63

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature Request] Group-Query Attention Decode #63

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions