Moving from nginx-gateway-fabric to envoy-gateway

September 7, 2026

I used to run the (kubernetes-run) ingress-nginx. When they announced they’d stop updating it, I knew I wanted to move to the Gateway API/HTTPRoute method, because that seems like the future. I originally switched to nginx-gateway-fabric. That is controlled by NGINX/F5 though, and they want money, so they’re putting features I want behind a very expensive paywall. You know who doesn’t paywall features and also supports Gateway API v1.5.1 (or newer)? envoy-gateway.

History / Requirements

When ingress-nginx announced they weren’t going to be updating it after March of 2026, I started looking into GatewayAPI, knowing I wanted to jump to HTTPRoute. I tried moving over to nginx-gateway-fabric at one point, but I quickly found out a Gateway only supports 16 Listeners. This was way too little, and being a homelab, I only have one public IP to point to for all my services. Thankfully, I saw that ListenerSets were being added (or already had been) in v1.5.1 of the Gatewy API, but when looking, most implementations hadn’t implemented that version yet. So I ended up waiting.

Eventually nginx-gateway-fabric released a version with support for ListenerSets, so I made the move. While I implemented it successfully, I learned a couple things:

To increase the body size of the request (to allow large uploads for social media or whatever), you need to create a ClientSettingPolicy that is attached to an HTTPRoute for the connection you want. The following example is for making the body size 10g.

apiVersion: gateway.nginx.org/v1alpha1
kind: ClientSettingsPolicy
metadata:
  name: client-settings
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: myhttproute
  body:
    maxSize: "10g"

With ingress-nginx, I used something called OIDCGuard to let me put OIDC in front of whatever site I wanted. I just assumed NGINX will have something similar, and they do, but they put it behind a paywall. I tried contacting them to see if they offered some sort of deal/free license for homelab people like some companies do. I heard nothing back. They’re obviously looking for big money from companies, and not some dweeb who has overengineered his setup at home. The whole reason I started down the path of kube was so I could learn the skills that I would need at work. Whatever I learn at home, I end up applying to work, including what specific implementations of stuff we use. Seems like supporting people nuts enough to homelab up kube with advanced features, you’d want those people to use your software. Whatever. Anyway, as OIDC is behind a paywall, I added BasicAuth. To do that, you add an AuthenticationFilter and refer to an htpasswd file secret:

apiVersion: gateway.nginx.org/v1alpha1
kind: AuthenticationFilter
metadata:
  name: basic-auth
spec:
  type: Basic
  basic:
    secretRef:
      name: basic-auth
    realm: "Basic auth"

Then you attach that AuthenticationFilter to an HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mysite
spec:
  hostnames:
  - mysite.joyrex.net
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: ListenerSet
    name: my-ls
    sectionName: mysite
  rules:
  - backendRefs:
    - kind: Service
      name: my-service
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: ExtensionRef
      extensionRef:
        group: gateway.nginx.org
        kind: AuthenticationFilter
        name: basic-auth

Learned Requirements

I need to have ListenerSets so I can attach >16 Listeners to a gateway
I need the ability to change upload body size
I really really want OIDC authentication

Envoy Gateway

Looking around, I found envoy-gateway was another popular gateway server, supported my requirements (I thought, more later) and had good documentation.

Thankfully, gateway and HTTPProxy are core stuff not unique to nginx’s implementation, so there wasn’t much I needed to change for my gateway or applications. I just had to update the gateway name I was using and make some of the other changes. Those are talked about below. Before I did that, I disabled all the HTTPRoutes that use BasicAuth so I didn’t accidentally put stuff on the public web until I got OIDC working.

Dual-Stack

I had dual-stack (IPv4/IPv6) going with nginx and I wanted the same for this gateway. To do that, I have to use an envoy-specific manifest to make sure the Service is correctly set up:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: config
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 2
      envoyService:
        annotations:
          metallb.io/loadBalancerIPs: 192.168.1.209,2400:a842:40ae::209
  ipFamily: DualStack

The ipFamily: DualStack is the important part here, and then I use the annotations to set the IPs I want via MetalLB.

For this to be used, you attach it to the gateway’s spec:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: klopjob-gateway
spec:
  ...
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: config
  ...

OIDC Support

Using the Envoy Gateway documentation about OIDC and the Authelia Envoy Gateway documentation, I was able to get OIDC working fine. For reference, here’s an example entry for something I want to have behind OIDC.

Authelia OIDC Client Config

- client_id: 'clientid'
  client_name: 'mysite.joyrex.net'
  client_secret: 
    path: "/secrets/authelia-secrets/oidc.client.mysite-joyrex-net.value"
  consent_mode: "implicit"
  public: false
  authorization_policy: 'two_factor'
  require_pkce: false
  pkce_challenge_method: ''
  redirect_uris:
    - 'https://mysite.joyrex.net/authelia/openid_connect/callback'
  scopes:
    - 'openid'
  grant_types:
    - 'authorization_code'
  response_types:
    - 'code'
  access_token_signed_response_alg: 'none'
  userinfo_signed_response_alg: 'none'
  token_endpoint_auth_method: 'client_secret_basic'

Security Policy

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: mysite-joyrex-net-oidc
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mysite
  oidc:
    provider:
      issuer: 'https://auth.joyrex.net'
      authorizationEndpoint: 'https://auth.joyrex.net/api/oidc/authorization'
      tokenEndpoint: 'https://auth.joyrex.net/api/oidc/token'
    clientID: "clientid"
    clientSecret:
      name: "mysite-joyrex-net-oidc-secret"
    cookieDomain: 'mysite.joyrex.net'
    cookieNames:
      idToken: ''
      accessToken: ''
    scopes:
      - 'openid'
    redirectURL: "https://mysite.joyrex.net/authelia/openid_connect/callback"
    forwardAccessToken: false
    refreshToken: false
    passThroughAuthHeader: false

HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mysite
spec:
  hostnames:
  - mysite.joyrex.net
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: ListenerSet
    name: my-ls
    sectionName: mysite
  rules:
  - backendRefs:
    - kind: Service
      name: mysite
      port: 8080
    matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: x-real-ip
          value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'
  - matches:
    - path:
        type: PathPrefix
        value: /authelia/openid_connect/callback
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplaceFullPath
          replaceFullPath: /
        statusCode: 302

The x-real-ip setting is something that I’ll be describing right now.

X-Real-Ip

The X-Forwarded-For header is already set, but X-Real-Ip doesn’t get set by default. Anubis looks for this value to verify stuff is configured properly, so I wanted this attached to all my requests.

I’m supposed to be able to attach this to a gateway as a whole, but that didn’t work for me. I was tired of debugging and learning, so for right now I just attached the filter to all the httproutes I wanted it on.

    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: x-real-ip
          value: '%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%'

This is shown in place in the full httproute above.

Body Size

This is where I hit issues. I just assumed setting body size would be common enough, but it’s not. According to this ticket and this question page, the way you increase the body size is to enable requestBuffering. This has the following downsides:

breaks anything using websockets. Hopefully this gets resolved in the future.
Limits the max upload size to 4G (or more exactly, 4294967295). Depending on your use case this could suck.

That said, to set requestBuffering, you need to create a BackendPolicy that attaches to one or more HTTPRoutes.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: request-buffer
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: mysite
  requestBuffer:
    limit: 4294967295

Conclusion

Besides the requestBuffer issue, and the minor X-Real-IP header setting not applying to the Gateway as a whole, I am extremely happy with Envoy. Leaving nginx-gateway-fabric behind is great, and I am stoked I have OIDC back.