REQUEST: Create kubernetes-sigs/llm-instance-gateway

[SergeyKanzhelev]

New repo, staging repo, or migrate existing

new repository

Is it a staging repo?

no

Requested name for new repository

kubernetes-sigs/llm-instance-gateway

Which Organization should it reside

kubernetes-sigs

Who should have admin access?

ArangoGutierrez, Jeffwan, SergeyKanzhelev, terrytangyuan

Who should have write access?

ArangoGutierrez, Jeffwan, SergeyKanzhelev, terrytangyuan

Who should be listed as approvers in OWNERS?

ArangoGutierrez, Jeffwan, SergeyKanzhelev, terrytangyuan

Who should be listed in SECURITY_CONTACTS?

ArangoGutierrez, Jeffwan, SergeyKanzhelev, terrytangyuan

What should the repo description be?

LLM Instance gateway implementation.

What SIG and subproject does this fall under?

sig-apps, sig-network

Please provide references to appropriate approval for this new repo

email is sent for approval

Additional context for request

The sub project was proposed at WG Serving meeting at Aug 6, 2024: WG-Serving Agenda and Notes:

• Proposal: [PUBLIC] Dense LLM Serving (+LoRA) for Inference Platform Teams
• PoC design: [PUBLIC] Kubernetes LLM Instance Gateway PoC Design
• Slides: [PUBLIC] Inference Gateway for Multiplexing models

[aojea]

fine with me as SIG Network Lead, but delegate final LGTM to @robscott as Gateway API lead

[janetkuo]

LGTM from sig-apps. Tagging @soltysh @kow3ns for additional thoughts

[robscott]

LGTM for Gateway API (FYI @mlavacca, @shaneutt, @youngnick)

[soltysh]

👍 from sig-apps pov

[terrytangyuan]

Seems like we've got the approvals from both sig-apps and sig-network. Thanks!

👋🏻 @mrbobbytables Since you helped us create the wg-serving repo recently, would you also be able to help us create this repo? If not, could you tag others who might be able to? Thank you!

[mrbobbytables]

/assign

[ahg-g]

instance or inference gateway?

[terrytangyuan]

instance or inference gateway?

We'll start with llm-instance-gateway and rename later if needed.

[mrbobbytables]

Repo has been provisioned, once #5119 and kubernetes/community#8056, merge this should be good to go 👍

[terrytangyuan]

Just approved and merged. Thanks!

[terrytangyuan]

Repo was created: https://github.com/kubernetes-sigs/llm-instance-gateway

Thanks!
/close

[k8s-ci-robot]

@terrytangyuan: Closing this issue.

In response to this:

Repo was created: https://github.com/kubernetes-sigs/llm-instance-gateway

Thanks!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

Is Network or Apps sponsoring this long term?

x-ref: https://github.com/kubernetes/community/pull/8081/files#r1815800559

[BenTheElder]

We should pick one and update the assorted metadata, repo readme, etc.

[BenTheElder]

filed kubernetes-sigs/llm-instance-gateway#41