Add IA3 Modules for Phi

[arnavgarg1]

Here's what the model architecture looks like with my proposed target modules:

>>> get_peft_model(model, ia3_config)
PeftModelForCausalLM(
  (base_model): IA3Model(
    (model): PhiForCausalLM(
      (model): PhiModel(
        (embed_tokens): Embedding(51200, 2560)
        (embed_dropout): Dropout(p=0.0, inplace=False)
        (layers): ModuleList(
          (0-31): 32 x PhiDecoderLayer(
            (self_attn): PhiAttention(
              (q_proj): Linear(
                (base_layer): Linear(in_features=2560, out_features=2560, bias=True)
                (ia3_l): ParameterDict(  (default): Parameter containing: [torch.FloatTensor of size 2560x1])
              )
              (k_proj): Linear(in_features=2560, out_features=2560, bias=True)
              (v_proj): Linear(
                (base_layer): Linear(in_features=2560, out_features=2560, bias=True)
                (ia3_l): ParameterDict(  (default): Parameter containing: [torch.FloatTensor of size 2560x1])
              )
              (dense): Linear(in_features=2560, out_features=2560, bias=True)
              (rotary_emb): PhiRotaryEmbedding()
            )
            (mlp): PhiMLP(
              (activation_fn): NewGELUActivation()
              (fc1): Linear(
                (base_layer): Linear(in_features=2560, out_features=10240, bias=True)
                (ia3_l): ParameterDict(  (default): Parameter containing: [torch.FloatTensor of size 1x2560])
              )
              (fc2): Linear(
                (base_layer): Linear(in_features=10240, out_features=2560, bias=True)
                (ia3_l): ParameterDict(  (default): Parameter containing: [torch.FloatTensor of size 1x10240])
              )
            )
            (input_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
            (resid_dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (final_layernorm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
      )
      (lm_head): Linear(in_features=2560, out_features=51200, bias=True)
    )
  )
)

This is the total number of trainable parameters using the default rank:

trainable params: 573,440 || all params: 2,780,257,280 || trainable%: 0.020625429312786478

[younesbelkada]

Thanks a lot ! 🚀

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[pacman100]

Hello @arnavgarg1, as per the T-few/IA3 paper, only fc2 feedforward layer would be considered. Therefore, left suggestions accordingly. Thank you for adding the target modules when using IA3 for Phi models!

[pacman100]

Suggested change

	"phi": ["q_proj", "v_proj", "fc1", "fc2"],
	"phi": ["q_proj", "v_proj", "fc2"],

[pacman100]

Suggested change

	"phi": ["fc1", "fc2"],
	"phi": ["fc2"],

[arnavgarg1]

Hi @pacman100! Thanks for the great catch - totally agree that this should only be applied to the fc2 feedforward layer. I just updated my PR!

[pacman100]

Thank you!