Load balancer for vLLM server instances?
Hello all, the docs for the vLLM production stack suggested autoscaling the vllm worker instances based on the number of waiting requests, but it seems like this would only help with new coming requests? We are having burst LLM calls which overwhelm ou…