[Llama Model] Run LLaMa with LoRA Error

[wlsghks4043]

Hi,

During the process of building a lora model without a modules_to_save value, the error below occurs.

Isn't the lora model based on the llama model without a modules_to_save value supported yet?

[error log]
Traceback (most recent call last): File "/code/tensorrt_llm/examples/llama/build.py", line 828, in <module> args = parse_arguments() File "/code/tensorrt_llm/examples/llama/build.py", line 499, in parse_arguments lora_config = LoraConfig.from_hf(args.hf_lora_dir, File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/lora_manager.py", line 60, in from_hf if "lm_head" in adapter_config["modules_to_save"]: TypeError: argument of type 'NoneType' is not iterable

[adapter_config.json]
{ "alpha_pattern": {}, "auto_mapping": null, "base_model_name_or_path": "/home/model/Llama2-7b-hf", "bias": "none", "fan_in_fan_out": false, "inference_mode": true, "init_lora_weights": true, "layers_pattern": null, "layers_to_transform": null, "lora_alpha": 32, "lora_dropout": 0.1, "modules_to_save": null, "peft_type": "LORA", "r": 8, "rank_pattern": {}, "revision": null, "target_modules": [ "v_proj", "q_proj", "down_proj", "k_proj", "up_proj" ], "task_type": "CAUSAL_LM" }

[byshiue]

We don't have checkpoint to verify, but I think you could just remove the related codes about modules_to_save if it is nullptr in your case.

We will also consider a more stabler way to support that.

[byshiue]

The issue should be fixed in latest main branch, please take a try.

[kaiyux]

Closing since no reply for 2 weeks. Please feel free to comment if you are still seeing the issue, thanks.