Lora Parameter Configuration in YAML File

[msmmpts]

Hi,

I would like to add custom lora rank, scaling factor and parts of the attention head that can be fine tuned.

Any idea how do I pass this as a configuration parameter in yaml config file for the same?

Refer : https://github.com/ludwig-ai/ludwig/blob/master/examples/llama2_7b_finetuning_4bit/llama2_7b_4bit.yaml

LORA_R = 8 # Rank LORA_ALPHA = 16 # Scaling Factor LORA_DROPOUT= 0.05 LORA_TARGET_MODULES = [ # Parts of attention head to be fine tuned "q_proj", "v_proj", ]

[tgaddair]

Hi @msmmpts, absolutely, you can do this as part of the Ludwig config:

adapter:
  type: lora
  r: 8
  alpha: 16
  dropout: 0.05

The one param we don't support yet is target modules, primarily because it's not clear how best to do it in a way that wouldn't be overly coupled to the underlying PyTorch module. We could definitely add it as an option if it's something you need for your use case, though.

Full details can be found here: https://ludwig.ai/latest/configuration/large_language_model/#lora

[msmmpts]

Hi @tgaddair ,

What all target modules are fine tuned by default in Ludwig?

[tgaddair]

The query and value modules for each attention block. For llama, these are q_proj and v_proj.

[msmmpts]

Thanks