9i #\SSKrSSKJr SSKJr \R "5r"SS\5rg)N)PretrainedConfig)loggingc^\rSrSrSrSrSU4Sjjr\S5rSr U=r $) GPT3Configa Configuration classes for GPT-3 model. Class attributes: - **model_type** (`str`) -- An identifier for the model type, serialized into the JSON file, can be used to recreate the correct object in [`~transformers.AutoConfig`]. Args: vocab_size (`int`, *optional*, defaults to 25600): Vocabulary size of the GPT model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`GPT3Model`]. hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the decoder layers and the pooler layer. ffn_hidden_size (`int`, *optional*, defaults to None): Dimensionality of the ffn layer, None defaults to four times the hidden_size. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer decoder. hidden_act (`str` or `Callable`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder and pooler. If string, `"gelu"`, `"relu"`, `"silu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout probability for all fully connected layers in the embeddings, decoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities. max_position_embeddings (`int`, *optional*, defaults to 512): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). type_vocab_size (`int`, *optional*, defaults to 2): The vocabulary size of the `token_type_ids` passed when calling [`GPT3Model`]. layernorm_epsilon (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. bias_gelu_fusion (`bool`, *optional*, defaults to True): Whether to use gelu activation function when mixing bias. fp32_residual_connection (`bool`, *optional*, defaults to False): Whether to use fp32 for residual connection between layers to improve accuracy. sequence_parallel (`bool`, *optional*, defaults to False): Whether to use sequence parallel during training. bf16 (`bool`, *optional*, defaults to `False`): Whether to use bf16 16-bit (mixed) precision training instead of 32-bit training. Requires Ampere or higher NVIDIA architecture or using CPU (no_cuda). This is an experimental API and it may change. fp16 (`bool`, *optional*, defaults to `False`): Whether to use fp16 16-bit (mixed) precision training instead of 32-bit training. apply_query_key_layer_scaling (`bool`, *optional*, defaults to `True`): Whether to scale query and key layer parameters during training. init_method_std (`float`, *optional*, defaults to `0.02`): The standard deviation of the normal distribution for initialization process. eod_id (`int`, *optional*, defaults to `1`): The end of text label for tokenizer, also indicates the end of the generation. tokens_to_generate (`int`, *optional*, defaults to 100): Number of tokens to generate. top_k (`int`, *optional*, defaults to 0): Number of highest probability vocabulary tokens to keep for top-k-filtering that will be used by default in the `generate` method of the model. top_p (`float`, *optional*, defaults to 0.9): Value that will be used by default in the `generate` method of the model for `top_p`. If set to float < 1, only the most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. temperature (`float`, *optional*, defaults to 1.0): The value used to module the next token probabilities that will be used by default in the `generate` method of the model. Must be strictly positive. gpt3c  >[T#U]"SSU 0U D6 XlX lUcSU-OUUlX@lXPlXplX`lXl Xl Xl Xl Xl XlXlXlUUlUUlU(a U(aeUUlUUlUcX%-S:XdeX%-UlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUl [C[DRFRIS5S5n![C[DRFRIS5S5n"U!S:=(d U!S:H=(a U"S:Ul%g)Nlayer_norm_epsr. )&super__init__ vocab_size hidden_sizeffn_hidden_sizenum_hidden_layersnum_attention_heads hidden_actintermediate_sizehidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizelayernorm_epsilonbias_gelu_fusionfp32_residual_connectionsequence_parallelfp16bf16apply_query_key_layer_scalingattention_softmax_in_fp32 kv_channelsmasked_softmax_fusionattention_dropoutbias_dropout_fusion(apply_residual_connection_post_layernormhidden_dropoutinit_method_stdeod_idtokens_to_generatetop_ktop_p temperatureinttorch __version__splitno_persist_layer_norm)$selfrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0kwargs TORCH_MAJOR TORCH_MINOR __class__s$ h/var/www/html/land-doc-ocr/venv/lib/python3.13/site-packages/modelscope/models/nlp/gpt3/configuration.pyrGPT3Config.__init__hsF D(9DVD$&& !;,; !2#6 $!2#6 ,H)'>$.!2 0(@%!2  T""-J*)B&  49 99*AD %:"!2#6 4 5,. "4  &%++11#6q9: %++11#6q9: !O F q 0 E[25E "cUR(a[R$UR(a[R$[R $)N)r!r2halfr"bfloat16float)r6s r; params_dtypeGPT3Config.params_dtypes. 99::  YY>> !;; r=) r#r)r'rr$r"r(rr,rr!rrr*rrr+rr%rr&rr5rrr r0r-r.r/rr)idiN rDi gelu皙?rFig-q=TFFFFTFNTrFTFrFg{Gz?r drg?g?) __name__ __module__ __qualname____firstlineno____doc__ model_typerpropertyrB__static_attributes__ __classcell__)r:s@r;rrsL\J  "" #),$(#!%*#*.&+"&! $5: "CMG^r=r)r2 transformers.configuration_utilsrtransformers.utilsr get_loggerloggerrrr=r;rVs-  =&    g!gr=