"AI Gateway" Working Group Proposal

Thanks Shane for putting this proposal together and to everyone for the enthusiasm. It's clear that the intersection of AI and networking on Kubernetes is an important space that is rapidly evolving, and it's great to see this Gateway API interest. 

I want to put some thoughts on this proposal, since I think there is a mismatch with the formal definition and purpose of a Kubernetes Working Group (WG). According to the official WG governance documentation, a Working Group is intended to facilitate communication and coordination across multiple SIGs to address a specific, time-limited problem that spans those SIGs.

The mission outlined in the proposal, describing the group as "effectively a 'branch' of the Gateway API and Gateway API Inference Extension (GIE) projects," suggests the work is tied to the existing Gateway API subproject within SIG Network. It sounds much more like a focused effort or a new workstream within the Gateway API community rather than a cross-SIG initiative.

The kind of collaboration being described here is something that the current community structure fully supports and encourages. You can absolutely form a focused group to drive the "AI Gateway" effort within the Gateway API subproject today.

In addition, for cross-SIG work related to serving AI/ML workloads, we already have wg-serving. If the scope of this AI Gateway workstream requires formal collaboration with other SIGs (like SIG Apps or SIG Node), wg-serving seems the best place to coordinate.

To be clear, I think this is a good idea, but I want to ensure we use the right community structures to help to keep efforts organized and avoid overlapping or redundancies with a formal Working Group.

Perhaps the best path forward would be to establish this as a dedicated focus area within the Gateway API project?

From my perspective, this WG should be a separated WG between GWAPI and WG-Serving, it solves the gap and brings focus on AI/ML workloads and kubernetes-networking in LLM routing for the LLM researcher and Networing experts. 

No matter it is the GIE or the other solution like many other  LLM routing solutions like envoy ai gateway, the inference gw introduced in AIBrix, or the semantic router introduced by RedHat, etc.

This WG brings standards and a place to gather the expertise to make the AI GW put into real-world production and solve the challenges in LLM.

Thanks, Xunzhuo

Hi!

I have only one wish:

Try to get rid of http body based routing.

For example add a header that is for now optional and publish it that vendors will opt-in to send http header in their clients (prompts, agents, whatever will come next) for routing decisions on proxy servers.

Thanks for helping http proxies to run efficiently.

Regarding your earlier point, Rob: I want to ensure that my colleagues involved in the GIE understand that this does not preclude the GIE. Adapting the GIE to support "AI Gateway" features is definitely an option, but we should take a moment to reflect on it. In brief, my goal for this proposal within our SIG is to extend an arm to more industry specialists to collaborate with us, helping us grow and identify common themes in these features. I want us to strengthen the narrative of Kubernetes as the premier platform for AI/ML workloads. "Industry Specialists" in this case certainly means everyone currently working on the GIE, but creating WG presents an opportunity for even more growth. Ultimately, the WG can determine whether these efforts should be integrated into the GIE or pursued separately based on this collaboration. Many doors open.

Sandor: Yes, this has been a source of frustration for many and I agree we should try to influence it in a better direction. Kubernetes having a stronger overall narrative on AI inference could lend to us having a stronger voice in the wider community to influence change here. I'm with you on this.

It resonates with me as an area I could definitely get involved with, as someone working on gateway level features, aka policies.

While wg-serving and GIE seem to be focused more on running of inference workloads on kubernetes, I find myself being a passive participant there at most. There's awesome work being done there, but it's not quite the right fit for where I think I can add value. (I also regularly find myself out of my depth on AI terminology and features in those meetings).

The AI Gateway WG proposal feels more approachable for someone working at the HTTP and application layer, but would also like to explore and standardise around inference features at the gateway like prompt guarding, token metrics & rate limiting, and AI specific protocols like A2A etc..

I want to also echo the comments of others here. I see this as being a great and valuable way to approach the needs and use cases in this area that go beyond routing and scheduling . 

+1 from my side as well. If meetings happen in an EU-friendly time slot, I'll definitely join. If not I'll follow along asynchronously.

Best,

Daniel