iDzSSKrSSKJr SSKrSSKJs Jr SSKJ r SSKJ r SSK J r Jr SSKJ r \R "\5r"SS\ R&5r"SS \ R&5rS \RR&S S4S jrS \RR&S \RR&4S jrg)N)Optional)KVCache)nn) _MaskType_sdpa_or_flex_attentionc!^\rSrSrSrSSSSSSSS.S\S \S \S \S \RS \RS\RS\RS\\RS\\RS\\RS\\ S\S\ S\ SS4 U4Sjjjr S\S\ RS\SS4SjrSrS$SSS.S\ R"S\\ R"S \\S!\\ R"S\ R"4 S"jjjrS#rU=r$)%MultiHeadAttentionu NOTE: copied from Torchtune's mha.py. Should be mostly 1:1 except that SDPA is factored out so that it can be swapped for more efficient ExecuTorch-defined SDPA ops. Multi-headed attention layer with support for grouped query attention (GQA) introduced in https://arxiv.org/abs/2305.13245v1. GQA is a version of multiheaded attention (MHA) which uses fewer key/value heads than query heads by grouping n query heads for each key and value head. Multi-Query Attention is an extreme version where we have a single key and value head shared by all query heads. Following is an example of MHA, GQA and MQA with num_heads = 4 (credit for the documentation: `litgpt.Config `_). :: ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ v ││ v ││ v ││ v │ │ v │ │ v │ │ v │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ │ │ │ ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ k ││ k ││ k ││ k │ │ k │ │ k │ │ k │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ ┌──┴──┐ ┌──┴──┐ ┌────┬──┴─┬────┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ ◀──────────────────▶ ◀──────────────────▶ ◀──────────────────▶ MHA GQA MQA n_kv_heads =4 n_kv_heads=2 n_kv_heads=1 Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. Default value is 0.0. Raises: ValueError: If ``num_heads % num_kv_heads != 0`` ValueError: If ``embed_dim % num_heads != 0`` ValueError: If ``attn_dropout < 0`` or ``attn_dropout > 1`` ValueError: if q_norm is defined without k_norm or vice versa NiT)pos_embeddingsq_normk_normkv_cache max_seq_len is_causal attn_dropout embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_projr r rrrrrreturnc >[TU]5 X#-S:wa[SUSUS35eX-S:wa[SUSUS35eUS:dUS:a[SUS 35e[U 5[U 5- (a [S 5eX lX0lXlXlX@lXl Xl Xl XPl X`l XplXlXlXlXl['5Ul[+UR URURUR,(a UROS URUR(URS 9UlS Ulg)Nrz num_heads (z%) must be divisible by num_kv_heads ()z embed_dim (z") must be divisible by num_heads (zattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togetherr )rrrrr attention_fnrF)super__init__ ValueErrorboolrrrrrrrrrrrrr rr r_attention_callSDPAtraining_sdpa cache_enabled)selfrrrrrrrrr r rrrrr __class__s i/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/executorch/extension/llm/modules/attention.pyr!MultiHeadAttention.__init__[sj&   #q (i[)!!-a1   A %i[)'[+  ! |a/~i[8UVW W <$v, &@A A#("( &"!    &  , 78**nn]].2mm**nn--]]  # batch_sizedtypec URb[RS5 g[UUURUR USS9UlURUR lSUlg)a!Setup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.F)r.rrrr/transpose_cacheT)rloggerwarningInferenceKVCacherrr'r()r)r.r/rs r+ setup_cacheMultiHeadAttention.setup_cachesa == $ NNi -%'!.. % DM#'--DJJ !%D r-chURc [S5eURR5 g)zReset the key value caches.Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r RuntimeErrorreset)r)s r+ reset_cacheMultiHeadAttention.reset_caches. == P  r-)mask input_posxyr<r=c r^^^^URumpVTRU5nTRTR-nUR TUTRU-TR 5nTR bTR UTS9nTRbTRU5nUUU4SjmU4Sjn UU4Sjn TRcUcS5eT"U5upO[R"[R"U5R5R5XU45upn TRRRU 5 TRR RU 5 TRR"RU 5 TR%X{U TXSS9nTR'U5$)a Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (Optional[torch.Tensor]): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. Optional only with kv_cache enabled. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.encoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Raises: ValueError: If no ``y`` input and ``kv_cache`` is not enabled. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim r=cZ>URSnTRU5nTRU5nURTUSTR5nURTUSTR5nTR bTR UTS9nTR bTR U5nX#4$)NrrA)shaperrviewrr r)r?s_ykvbr=r)s r+ calculate_kv0MultiHeadAttention.forward..calculate_kv s''!*C AA AAq#r4==1Aq#r4==1A"".''Y'?{{&KKN4Kr-c~>TRR5nURURUR4$N)rclonek_cachev_cache kv_cache_pos)r?rr)s r+true_fn+MultiHeadAttention.forward..true_fns4}}**,H##X%5%5x7L7LL Lr-c>T"U5upTRR5nURX5 URURUR 4$rM)rrNupdaterOrPrQ)r?rGrHrrJr)s r+false_fn,MultiHeadAttention.forward..false_fn#sK?DA}}**,H OOA !##X%5%5x7L7LL Lr-zAMust provide y input or use kv_cache to enable streaming decoding)r<)rDrrrrErr r rtorchcondisnanallitemrOcopy_rPrQr'r)r)r>r?r<r=s_x_qq_per_kvrRrVrGrH cache_posoutputrIrJs` ` @@r+forwardMultiHeadAttention.forwardsbGG 3 KKN>>T%6%66 FF1c4,,x7 G    *##A#;A ;; " AA ( M M ==   SR S?DAq $jj A""$))+WOA) MM ! ! ' ' * MM ! ! ' ' * MM & & , ,Y 7A!Q7''r-)r$r'rr(rrrrrrrrrrr r rrrM)__name__ __module__ __qualname____firstlineno____doc__intrModulerrr#floatr!rXr/r5r:Tensorrrd__static_attributes__ __classcell__r*s@r+r r sB^/3&*&*&*!#H#H# H#  H#  H# H# H# H#YYH#!+H##H##H#7#H#H# !H#"#H#$ %H#H#T&&&+kk&@C& &8%)s( %),0 s( <<s( ELL !s( y! s( ELL) s( s(s(r-r c^\rSrSrSrS\S\S\S\S\SS 4 U4S jjrSS \ RS \ RS \ RS\S\S\ \ S\ R4Sjjr SrU=r$)r%i?zf TorchTune's SDPA which can be optimized and can be swapped out for a more efficient implementations. rrrrrrNc>[TU]5 XlX lX0lURUR-UlX@lXPlX`lXpl grM) r r!rrrrarr _attention_fnr) r)rrrrrrrr*s r+r! SDPA.__init__EsL (" $*;*;; (") r-r`rGrHbszseq_lenr<c XURSS5nURSS5nURSS5nURUR:waqSSURSS4nUR S5R U5R SS5nUR S5R U5R SS5nURUUUUURURSL=(a USL=(a URS9nURSS5RXES5$)NrrC)r< dropout_pr) transposerrra unsqueezeexpandflattenrtrrrreshape) r)r`rGrHrvrwr< expand_shapercs r+rd SDPA.forwardYs KK1  KK1  KK1  >>T.. .DMM2r:L A%%l3;;AqAA A%%l3;;AqAA## ''mmt+O O $ 1%--cB??r-)rtrrrrrrrarM)rfrgrhrirjrkrmr#r!rXrnrrrdrorprqs@r+r%r%?s !!! !  !  ! !6%)!@ <<!@ <<!@ << !@  !@  !@y!!@ !@!@r-r%modulercUR5Hup[U[R5(a[ UU[UR UR URURURURURURURURURUR UR"UR$UR&S95 M[)U5 M g)N)rrrrrrrrr r rrrrr)named_children isinstanceTorchTuneAttentionr setattrrrrrrrrrr r rrrrrreplace_mha_with_inference_mha)rnamechilds r+_replace_mha_with_inference_mhar}s,,.  e/BB C C "#oo#oo!&!3!3"^^ << << << % 1 1#(#7#7 << <<"^^ % 1 1#oo!&!3!3 , +5 11/r-c[U5 U$)z Replace TorchTune's MHA with an inference friendly version of MHA that separates out the inference-related parts for further optimization. )r)rs r+rrs $F+ Mr-)loggingtypingrrXtorchtune.modules.attentionmodules attentionr)executorch.extension.llm.modules.kv_cacherr4r!torchtune.modules.attention_utilsrrtorchtune.modules.kv_cache getLoggerrfr2rlr r%rrr-r+rs 88QP.   8 $f(f(R ;@299;@|2EHHOO228588??uxxr-