You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
According to Model Server Protocol, supported model servers should expose AvailableModels. In vLLM, the /v1/models endpoint exposes such info.
The ext proc should get the AvailableModels similar as scraping metrics, and add a filter to only pick model servers that have the targetModel in the request as available.
The text was updated successfully, but these errors were encountered:
liu-cong
changed the title
Validate model/adapter before sending requests to a model server
Validate model/adapter is available on the model server before sending requests to a model server
Nov 20, 2024
According to Model Server Protocol, supported model servers should expose
AvailableModels
. In vLLM, the/v1/models
endpoint exposes such info.The ext proc should get the
AvailableModels
similar as scraping metrics, and add a filter to only pick model servers that have the targetModel in the request as available.The text was updated successfully, but these errors were encountered: