[Question] What is the difference between gptManagerBenchmark and gptSessionBenchmark in cpp benchmarks?

[hcnhcn012]

version of TensorRT-LLM: v0.11.0

What is the difference between gptManagerBenchmark and gptSessionBenchmark in cpp benchmarks, please give me some example scenarios of using each benchmark script, thans a lot :)

[yuhengxnv]

gptSessionBenchmark seems to be simpler to use, but it's recommended to use gptManagerBenchmark.

[lfr-0531]

The gptSessionBenchmark is used to benchmark the running with GptSession runtime, which can only support the static batching and is deprecated now. The gptManagerBenchmark is used to benchmark the executor and gptManager runtime, and it can support both static batching and inflight batching. Since GptSession will be removed in a future release, we recommand using gptManagerBenchmark. For the benchmark script, plz refer to the cpp benchmark README: https://github.com/NVIDIA/TensorRT-LLM/tree/main/benchmarks/cpp#benchmark-c-runtime