Discussions

• 3 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Aug 7, 2024 in Announcements

  

  

  2
• 1 You must be logged in to vote

  🙏

  Best way to serve a 4bit model with qlora adapters in TRT

  lodm94 asked Jul 31, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  💬

  When will cuda12.5 && tritonserver 24.07-py3 be supported?

  dafu-wu started Jul 31, 2024 in General

  

  

  0
• 3 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jul 30, 2024 in Announcements

  

  

  0
• 1 You must be logged in to vote

  💬

  TensorRT-LLM redefines difficult to use，

  ZhangChuann started Jul 27, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Does tensorrt-llm optimize the diffusion model

  songh11 asked Jul 26, 2024 in Q&A · Unanswered

  

  

  0
• 4 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jul 23, 2024 in Announcements

  

  

  0
• 5 You must be logged in to vote

  📣

  TensorRT-LLM 0.11.0 Release

  kaiyux announced Jul 17, 2024 in Announcements

  

  

  0
• 5 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jul 16, 2024 in Announcements

  

  

  0
• 3 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jul 9, 2024 in Announcements

  

  

  0
• 3 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jul 4, 2024 in Announcements

  

  

  0
• 1 You must be logged in to vote

  🙏

  CLIP inference using TensorRT

  dhaivatd asked Jul 2, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  is it possible to use quantize.py on encoder models like BERT ?

  mistborn17 asked Jul 2, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Is it possible for streaming llm to support the QWEN2 model?

  lai-serena started Jul 1, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Get outputs of cpp/tensorrt_llm/kernels/cutlass_kernels/python/generate_kernels.py

  YLGH asked Jun 27, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Negative values for tokens per sec for Mixtral benchmarsk

  prasad-nair-amd asked Jun 27, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  How to use Smoothquant and FP16 or Weight Only in a LLM on the same time?

  biubiu3721 started Jun 26, 2024 in General

  

  

  0
• 4 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jun 25, 2024 in Announcements

  

  

  0
• 1 You must be logged in to vote

  💬

  No measurable difference for Lama 3 model on L4 GPU.

  jmnarloch started Jun 22, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  For TensorRT-LLM benchmarks ..whats the difference between batch_size and max_batch_size ?

  prasad-nair-amd asked Jun 18, 2024 in Q&A · Unanswered

  

  

  0
• 6 You must be logged in to vote

  📣

  Update on the development branch

  Shixiaowei02 announced Jun 18, 2024 in Announcements

  

  

  0
• 1 You must be logged in to vote

  🙏

  Parallelization on RTX 4060 Ti cards.

  AntonThai2022 asked Jun 17, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Trying to run benchmark.py script results "too many values to upack error"

  prasadwrites asked Jun 13, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Quantize llama3 70B on 3x 4090

  teis-e asked Jun 12, 2024 in Q&A · Unanswered

  

  

  0
• 5 You must be logged in to vote

  📣

  Update on the development branch

  kaiyux announced Jun 11, 2024 in Announcements

  

  

  0