blip-2 build error

[KKwanhee]

I was following the guide for 'blip-2' to build, but encountered the following error in the section below.

trtllm-build --checkpoint_dir=./opt/2.7B/trt_ckpt/fp16/1-gpu/
--max_batch_size 8
--use_gpt_attention_plugin float16
--use_gemm_plugin float16
--max_input_len 924
--max_output_len 100
--max_beam_width 5
--output_dir ../blip2/trt_engine/blip-2-opt-2.7b/fp16/1-gpu
--max_prompt_embedding_table_size 256 # 256 = 32 (query_token number) * 8 (max_batch_size)

[12/19/2023-04:23:59] [TRT-LLM] [W] Found pynvml==11.4.1. Please use pynvml>=11.5.0 to get accurate memory usage
[12/19/2023-04:24:04] [TRT] [I] [MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 141, GPU 987 (MiB)
[12/19/2023-04:24:06] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1974, GPU +348, now: CPU 2251, GPU 1335 (MiB)
Traceback (most recent call last):
  File "/usr/local/bin/trtllm-build", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 217, in main
    build_and_save(source, build_config, args.output_dir, workers,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 154, in build_and_save
    build_and_save_shard(rank, rank % workers, ckpt_dir, build_config,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 130, in build_and_save_shard
    engine = build(build_config,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/builder.py", line 621, in build
    return build_shard_model(model, build_config)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/builder.py", line 570, in build_shard_model
    model(**inputs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/module.py", line 40, in __call__
    return self.forward(*args, **kwargs)
TypeError: DecoderModelForCausalLM.forward() got an unexpected keyword argument 'tasks'

I also tried it on the 'rel' branch, but the issue persisted. Could you help me?

[QiJune]

@KKwanhee we have a fix internally and will be synced to github main branch soon.

[jessiewiswjc]

@QiJune Thanks for your work. I have a question, does trtiton_server support blip2? Could you help me？

[kaiyux]

@jessiewiswjc Blip2 is not supported in TensorRT-LLM Triton backend yet.

[kaiyux]

@KKwanhee The issue should have been fixed on the latest main branch, please kindly check. Thanks very much.

Closing. Please let us know if you have questions.

[jessiewiswjc]

@jessiewiswjc Blip2 is not supported in TensorRT-LLM Triton backend yet.

@kaiyux Thanks for your replay. Is there a plan for this(trtion for multi-modal)?