questions on return-generation-logits when streaming

[binhtranmcs]

I see that when run streaming inference, the result contains generationLogits for the full sequence. Which means it will contain a tensor of shape batch_size*beam_size*max_output_length*vocab_size every time the executor returns a token, which is very inefficient. Is this expected behavior? I believe returning only the logits for the current token would be more optimal.

Please have a look. Thanks in advance!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[AdamzNV]

It's under our consideration now.