Parallel Python execution for tool completion

[yunfeng-scale]

Pull Request Summary

with some sample data (1000 prompts). on my devbox (96 CPU cores), but since i used threadpool to start subprocess i think it still might be slowed down by GIL to start processes, probably not high util for the 96 cores

467s total, 283s tool use
vs
without this change 862s total, 687s tool use

Test Plan and Usage Guide

tested with sample data
also CPU count works inside and outside container

[yunfeng-scale]

it's probably fine to use single process to start subprocesses when CPU number is low

[Georgepu1]

lgtm, thanks for investigating! btw were u able to validate the completions were close enough on the "467s total, 283s tool use" vs "862s total, 687s tool use" runs? just want to make sure nothing weird's happening here on the generation side

[yunfeng-scale]

lgtm, thanks for investigating! btw were u able to validate the completions were close enough on the "467s total, 283s tool use" vs "862s total, 687s tool use" runs? just want to make sure nothing weird's happening here on the generation side

there are actually differences, but for a hand-picked sample i can't repro either for some reason. i wonder if this is due to some tiny randomness in vLLM, or python execution. will investigate a bit more

[yunfeng-scale]

took a bit more look into this @Georgepu1 15/1000 output texts have changed, i manually checked 5 of them and i don't think python code execution results are different. the differences appear to be some randomness in sampling that sentences in both cases made sense to me