This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Envoy Gateway
Overview of the Envoy Gateway ingress in Elastx Kubernetes CaaS
This section introduces Envoy Gateway as the ingress controller in our
Elastx Kubernetes CaaS service. We manage and upgrade the controller, the
Gateway API CRDs and the cluster-scoped
GatewayClass named eg. You create the Gateway API
objects that describe your own traffic in your own namespaces.
There are companion guides for the two ways traffic typically reaches
the cluster. Pick the one that matches your setup:
Standard layout: one shared Gateway per cluster
A cluster has one shared Gateway in a dedicated namespace (for example
gateway) that serves routes from all your application namespaces through a
single OpenStack load balancer and IP. This is the standard setup and the
direct equivalent of ingress-nginx, where a single controller fronted every
host in the cluster. Each application namespace opts in with a
shared-gateway-access: "true" label and contributes its own HTTPRoutes,
and never touches the shared Gateway.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
client(["Clients"]):::client --> lb["OpenStack load balancer<br/>one LB · one IP"]:::lb
lb --> gw
subgraph gwns["gateway namespace"]
gw["Gateway 'shared'"]:::gw
ctp["ClientTrafficPolicy"]:::policy
cert["TLS certificates<br/>per hostname"]:::policy
end
ctp -.->|attaches to| gw
cert -.->|terminates TLS| gw
subgraph ta["team-a namespace"]
appa["app + HTTPRoute"]:::app
end
subgraph tb["team-b namespace"]
appb["app + HTTPRoute"]:::app
end
gw -->|"team-a.example.com"| appa
gw -->|"team-b.example.com"| appb
classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
Each Gateway provisions its own OpenStack load balancer, so a single shared
Gateway keeps your cluster on one load balancer, one IP and one
ClientTrafficPolicy: the same single-entry-point model you had with
ingress-nginx. TLS is terminated centrally in the gateway namespace, with one
certificate per hostname (Envoy serves the right one per request by SNI). The
walkthroughs below use this layout throughout. Running more than one
Gateway, a separate load balancer for a single namespace via
allowedRoutes.namespaces.from: Same, is a non-standard setup for the rare case
that genuinely needs an isolated IP or blast radius.
What you create
In the dedicated gateway namespace (once, by whoever owns ingress):
Gateway: listeners, ports, protocols, TLS; allowedRoutes selecting the
shared-gateway-access: "true" label.
ClientTrafficPolicy: controls PROXY-protocol handling, TLS parameters, timeouts.
Must live in the same namespace as the Gateway.
- TLS
Certificate / Issuer (cert-manager): one certificate per hostname you
serve (HTTP-01 in direct mode, DNS-01 in proxy mode).
- Optionally, an HTTP-to-HTTPS redirect
HTTPRoute on the http listener, to
match ingress-nginx’s ssl-redirect behaviour for every host.
In each application namespace (per team, self-service):
- The
shared-gateway-access: "true" namespace label.
HTTPRoute, GRPCRoute: routing rules, attached to the shared Gateway
via cross-namespace parentRefs.
BackendTrafficPolicy: retries, circuit breaking.
SecurityPolicy: JWT, OIDC, CORS.
BackendTLSPolicy: mTLS toward your backends.
You reference the cluster GatewayClass by its name eg from the
Gateway. You do not need to create or modify any cluster-scoped resources.
Which variant fits your setup?
The OpenStack load balancer in front of Envoy runs in TCP mode in both
cases. The variants differ in how the real client IP arrives at
Envoy, and your ClientTrafficPolicy has to match.
- Direct (PROXY-protocol) mode: clients connect straight to the
load balancer. The load balancer is configured with PROXY protocol v2
and prepends a PROXY header carrying the real client IP. Your
ClientTrafficPolicy must enable proxy-protocol parsing.
See Direct (PROXY-protocol) mode.
- Proxy (X-Forwarded-For) mode: you put your own upstream proxy
(CDN, WAF, edge proxy) in front of the load balancer. That upstream
injects the real client IP into
X-Forwarded-For; the load balancer
passes the request through unchanged. Your ClientTrafficPolicy must
trust that header with the right hop count.
See Proxy (X-Forwarded-For) mode.
Coming from ingress-nginx?
If you used our managed ingress-nginx, the two modes carry over directly;
only the names and the resources you write have changed:
| ingress-nginx |
Envoy Gateway |
When it applies |
Direct mode (use-proxy-protocol: "true") |
Direct (PROXY-protocol) mode |
Clients connect straight to our load balancer. No upstream proxy. This is the default. |
Proxy mode (use-forwarded-headers: "true") |
Proxy (X-Forwarded-For) mode |
You run your own CDN / WAF / edge proxy in front of the load balancer. |
As before, the mode is a cluster-level setting: we provision your
cluster in one mode or the other; you do not switch it from a manifest. Tell us
which fits your setup and we configure the load balancer accordingly. What you
do write is a ClientTrafficPolicy that matches that mode (see the two
guides above).
Proxy mode requires your own upstream proxy. X-Forwarded-For mode only
makes sense when a CDN, WAF, or edge proxy actually sits in front of the load
balancer and injects the header. Without one, no real client IP ever reaches
Envoy and your backends see only the load balancer. If clients connect
directly to Elastx, use direct (PROXY-protocol) mode instead. It carries
the client IP for you and needs no CDN.
Two things also changed with Kubernetes CaaS v2:
- Ingress always enters through the load balancer. Older clusters accepted
traffic directly on each worker node’s floating IP; that path is gone. DNS for
your services now points at the
Gateway’s load-balancer address.
- Floating IPs are an opt-in egress feature, used for a predictable
outbound source IP, not as an ingress path.
For the full move, including clusters that use floating IPs, see
Migrating from ingress-nginx.
TLS
Both walkthroughs terminate TLS on the shared Gateway, with cert-manager
issuing one certificate per hostname into the gateway namespace (the same
namespace as the Gateway). The validation method differs by mode: direct mode
uses ACME HTTP-01 (clients reach the load balancer directly), while proxy mode
uses DNS-01 (public DNS points at your upstream proxy, so HTTP-01 cannot reach
Envoy). If you need a guide for installing cert-manager, see
Install and upgrade cert-manager.
Advanced usage
For more advanced use cases please refer to the documentation provided by
each project or contact our support:
1 - Direct (PROXY-protocol) mode
A walkthrough of setting up Envoy Gateway when your cluster’s load balancer uses PROXY protocol v2
This guide walks through setting up Envoy Gateway in a cluster where the
OpenStack load balancer is configured in TCP mode with PROXY protocol v2.
The load balancer prepends a PROXY header to each incoming connection
carrying the real client IP. Envoy parses that header and uses it for
access logs, rate limiting and X-Forwarded-For.
Note: Your ClientTrafficPolicy must set proxyProtocol.optional: false. Without it Envoy parses the load balancer’s PROXY-v2 prefix as a malformed HTTP request and every response is HTTP 400 Bad Request.
If you are not sure which variant applies to your cluster, see the
Envoy Gateway overview.
The shared-Gateway layout
A cluster runs one shared Gateway in a dedicated namespace that serves
routes from all your application namespaces through a single load balancer
and IP. Each application namespace opts in with a label and contributes its own
HTTPRoutes. This is the standard setup, the same single-entry-point model
ingress-nginx gave you, where one controller fronted every host.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
client(["Clients"]):::client -->|PROXY protocol v2| lb["OpenStack load balancer<br/>TCP mode · one LB · one IP"]:::lb
lb --> gw
subgraph gwns["gateway namespace"]
gw["Gateway 'shared'"]:::gw
ctp["ClientTrafficPolicy<br/>proxyProtocol.optional: false"]:::policy
cert["TLS certificate<br/>team-a.example.com"]:::policy
end
ctp -.->|attaches to| gw
cert -.->|terminates TLS| gw
subgraph ta["team-a namespace (labelled)"]
appa["app + HTTPRoute"]:::app
end
subgraph tb["team-b namespace (labelled)"]
appb["app + HTTPRoute"]:::app
end
gw -->|"team-a.example.com"| appa
gw -->|"team-b.example.com"| appb
classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
What the shared Gateway gives you:
- One load balancer per cluster: a single LB and IP front all your teams,
the way a single ingress controller did before.
- One
ClientTrafficPolicy to manage, in the gateway namespace, with TLS
terminated there (one certificate per hostname you serve).
- Self-service for app teams: an app team only labels its namespace and
creates an
HTTPRoute; it never touches the shared Gateway.
Each Gateway provisions its own load balancer. Running more than one (a
separate Gateway and load balancer for a single namespace, via
allowedRoutes.namespaces.from: Same) is a non-standard setup, for the rare
case that genuinely needs an isolated IP or blast radius.
Prerequisites
- A dedicated namespace for the shared Gateway. The examples use
gateway.
- One or more application namespaces. The examples use
team-a.
- A DNS record for each hostname you serve, pointing at the load balancer’s public IP. The examples use
team-a.example.com; replace it with your own throughout.
- cert-manager in the cluster. The examples issue certificates with ACME HTTP-01, which needs no DNS-provider credentials. If you are not using our managed cert-manager, install your own.
Create the gateway namespace
kubectl create namespace gateway
kubectl label namespace gateway shared-gateway-access=true
The label lets routes created in the gateway namespace itself attach to the
shared Gateway. Two gateway-owner routes need this: cert-manager’s short-lived
HTTP-01 challenge route, and the HTTP-to-HTTPS redirect below.
Create the shared Gateway
Gateway describes the listeners. Put it in the dedicated gateway
namespace and reference the cluster GatewayClass named eg. The
allowedRoutes selector is what lets routes in other namespaces attach.
Create a file called gateway.yaml with the following content:
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shared
namespace: gateway
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"
- name: https
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: team-a-tls
The Gateway has two listeners. The https listener (port 443) terminates TLS;
it has no hostname, so it serves every host whose certificate is listed in its
certificateRefs, and Envoy picks the right one per request by SNI. The http
listener (port 80) carries plaintext requests: it serves ACME HTTP-01 challenges
and is where the HTTP-to-HTTPS redirect below attaches.
allowedRoutes.from: Selector admits routes from any namespace carrying the
shared-gateway-access: "true" label; this is the opt-in that makes the
Gateway shared. Use from: Same instead if you ever want a Gateway that only
serves its own namespace, or from: All to admit every namespace
unconditionally (not recommended, since it removes the opt-in).
Apply it:
kubectl apply -f gateway.yaml
The ClientTrafficPolicy attaches to the Gateway by name and tells Envoy
to parse the PROXY-v2 header from the load balancer. It lives in the
gateway namespace alongside the Gateway and covers the whole load
balancer; app namespaces do not need their own.
Create a file called client-traffic-policy.yaml:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: shared
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: shared
proxyProtocol:
optional: false
Note:
The policy must live in the same namespace as the Gateway. Envoy
Gateway rejects cross-namespace policy targets.
Apply it:
kubectl apply -f client-traffic-policy.yaml
Issue a TLS certificate
Terminate TLS on the shared Gateway with a certificate per hostname, issued by
cert-manager into the gateway namespace (where TLS terminates). In direct mode
clients reach the load balancer directly, so ACME HTTP-01 is the simplest
validation: Let’s Encrypt fetches a token over port 80, which the shared Gateway
already serves, and no DNS-provider credentials are needed.
Create a file called certificate.yaml:
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-http01
namespace: gateway
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: platform@example.com
privateKeySecretRef:
name: letsencrypt-http01-account
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: shared
namespace: gateway
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: team-a-tls
namespace: gateway
spec:
secretName: team-a-tls
issuerRef:
name: letsencrypt-http01
kind: Issuer
dnsNames:
- team-a.example.com
Replace the email and hostname with your own, then apply it:
kubectl apply -f certificate.yaml
To solve the challenge, cert-manager creates a short-lived HTTPRoute in the
gateway namespace, attached to the http listener. It is admitted because you
labelled the gateway namespace when you created it. The https listener stays
pending until the certificate is issued, then serves it.
Each additional hostname needs its own Certificate and a matching entry in the
https listener’s certificateRefs; Envoy then selects the right certificate
per request by SNI.
Prefer DNS-01? If you would rather validate over DNS (for example to keep
the ACME servers off port 80), use a DNS-01 Issuer instead. DNS-01 needs API
credentials for your DNS provider; see the
cert-manager DNS-01 docs.
Redirect HTTP to HTTPS
ingress-nginx redirected HTTP to HTTPS for you (the ssl-redirect default). To
keep that behaviour, attach one redirect HTTPRoute to the http listener,
created once in the gateway namespace (already labelled above). It matches
every host on port 80, so app teams do not add their own.
Create https-redirect.yaml:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: https-redirect
namespace: gateway
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
Apply it:
kubectl apply -f https-redirect.yaml
A request to http://team-a.example.com/ now returns 301 Moved Permanently
with Location: https://team-a.example.com/, and the client repeats the request
over HTTPS.
Does this break HTTP-01 certificates? No. This redirect matches the path
/, while cert-manager’s challenge route matches the longer
/.well-known/acme-challenge/ path. Gateway API gives precedence to the
longest path match, so ACME challenges are still served over plain HTTP while
everything else redirects.
Onboard an application namespace
This is all an app team does; no access to the gateway namespace is needed.
1. Label the namespace so the shared Gateway admits its routes:
kubectl label namespace team-a shared-gateway-access=true
2. Deploy the app. Create app.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: team-a
spec:
replicas: 1
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: ealen/echo-server:0.9.2
ports:
- containerPort: 80
env:
- name: PORT
value: "80"
---
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: team-a
spec:
selector:
app: echo
ports:
- port: 80
targetPort: 80
3. Route traffic to it. Create route.yaml. The parentRefs points at
the shared Gateway in the gateway namespace, and that cross-namespace
reference is what puts this app behind the shared load balancer:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo
namespace: team-a
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- team-a.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: echo
port: 80
Apply both:
kubectl apply -f app.yaml -f route.yaml
The route’s hostname (team-a.example.com) must have a certificate on the
https listener; you issued one above. The backend Service is in the same
namespace as the HTTPRoute, so no ReferenceGrant is needed; you only need
one if a route points at a Service in a different namespace.
Verify
Check that the shared Gateway got an external address and that traffic
flows for the app namespace:
kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}'
curl -v https://team-a.example.com/
Confirm the route attached to the shared Gateway:
kubectl -n team-a get httproute echo -o jsonpath='{.status.parents[0].conditions}'
Accepted: True and ResolvedRefs: True mean the cross-namespace attach
worked. The backend should see the real client IP in X-Forwarded-For and
X-Envoy-External-Address.
Common mistakes
- Namespace not labelled: the
HTTPRoute reports Accepted: False with reason NotAllowedByListeners, and traffic never reaches the app. Label the app namespace shared-gateway-access=true.
- Forgetting
namespace: in parentRefs: without it the route looks for a Gateway in its own namespace, finds none, and stays unattached. Cross-namespace routes must name the gateway namespace.
- No certificate for the hostname: if the
https listener has no certificate matching the route’s hostname, the TLS handshake fails and clients cannot connect. Issue a Certificate for each hostname and add its Secret to the listener’s certificateRefs.
- Forgetting
ClientTrafficPolicy: every request returns HTTP 400 Bad Request. The load balancer is prepending a PROXY-v2 binary header; without the policy Envoy treats those bytes as the start of an HTTP request and fails to parse it.
- Putting
ClientTrafficPolicy in another namespace: silently ignored. Must be colocated with the Gateway (here, the gateway namespace).
- Setting
proxyProtocol.optional: true: opens you up to clients that don’t send the header bypassing client-IP enforcement. Keep it false.
- Testing with
curl from outside the load balancer: PROXY-protocol traffic isn’t valid HTTP. Always go through the load balancer’s VIP.
Advanced usage
For more advanced use cases please refer to the documentation provided by
each project or contact our support:
2 - Proxy (X-Forwarded-For) mode
A walkthrough of setting up Envoy Gateway when your traffic arrives via an upstream proxy that injects X-Forwarded-For
This guide walks through setting up Envoy Gateway in a cluster where you
front the OpenStack load balancer with your own upstream proxy (for
example a CDN, WAF, or edge proxy) that terminates the client connection
and injects the real client IP into the X-Forwarded-For header. The
OpenStack load balancer itself stays in TCP passthrough; the upstream
proxy is what carries the client IP for you.
This mode requires your own upstream proxy. It is only correct when a CDN,
WAF, or edge proxy that you operate sits in front of the load balancer and
injects X-Forwarded-For. Without one, no real client IP ever reaches Envoy,
so your backends see only the load balancer’s internal IP. If clients connect
straight to Elastx with nothing in front, use direct (PROXY-protocol) mode
instead; it carries the client IP for you and needs no proxy.
Note: Your ClientTrafficPolicy must set clientIPDetection.xForwardedFor with numTrustedHops set to the number of trusted proxies in front of Envoy. Without it Envoy will not honour the incoming X-Forwarded-For header and your access logs and rate limiting will see the load balancer’s internal IP.
Note: The upstream proxy must terminate TLS to inject X-Forwarded-For; it can only read and modify headers on decrypted traffic. It then opens a fresh connection to the load balancer (which passes it through untouched) and Envoy terminates TLS again on the Gateway HTTPS listener. A pure TCP/TLS passthrough upstream cannot inject the header.
If you are not sure which variant applies to your cluster, see the
Envoy Gateway overview.
The shared-Gateway layout
A cluster runs one shared Gateway in a dedicated namespace that serves
routes from all your application namespaces through a single load balancer
and IP. Each application namespace opts in with a label and contributes its own
HTTPRoutes. This is the standard setup, the same single-entry-point model
ingress-nginx gave you, where one controller fronted every host.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
client(["Clients"]):::client --> proxy["Your upstream proxy<br/>CDN / WAF / edge<br/>injects X-Forwarded-For"]:::proxy
proxy -->|TCP passthrough| lb["OpenStack load balancer<br/>TCP mode · one LB · one IP"]:::lb
lb --> gw
subgraph gwns["gateway namespace"]
gw["Gateway 'shared'"]:::gw
ctp["ClientTrafficPolicy<br/>xForwardedFor.numTrustedHops: 1"]:::policy
cert["TLS certificate<br/>team-a.example.com"]:::policy
end
ctp -.->|attaches to| gw
cert -.->|terminates TLS| gw
subgraph ta["team-a namespace (labelled)"]
appa["app + HTTPRoute"]:::app
end
subgraph tb["team-b namespace (labelled)"]
appb["app + HTTPRoute"]:::app
end
gw -->|"team-a.example.com"| appa
gw -->|"team-b.example.com"| appb
classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
classDef proxy fill:#CFE8FF,stroke:#0041C2,color:#1E343E;
classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
What the shared Gateway gives you:
- One load balancer per cluster: a single LB and IP front all your teams,
the way a single ingress controller did before, and your upstream proxy
points at a single origin IP.
- One
ClientTrafficPolicy to manage, in the gateway namespace, with TLS
terminated there (one certificate per hostname you serve).
- Self-service for app teams: an app team only labels its namespace and
creates an
HTTPRoute; it never touches the shared Gateway.
Each Gateway provisions its own load balancer. Running more than one (a
separate Gateway and load balancer for a single namespace, via
allowedRoutes.namespaces.from: Same) is a non-standard setup, for the rare
case that genuinely needs an isolated IP or blast radius.
Prerequisites
- A dedicated namespace for the shared Gateway. The examples use
gateway.
- One or more application namespaces. The examples use
team-a.
- DNS for each public hostname (the examples use
team-a.example.com, replace with your own) pointing at your upstream proxy (CDN / WAF / edge proxy), not directly at the load balancer. If it resolves straight to the load balancer, traffic bypasses the proxy and no X-Forwarded-For is injected.
- Your upstream proxy configured with the load balancer’s public IP as its origin / backend.
- cert-manager in the cluster, with a DNS-01 capable
Issuer. In proxy mode HTTP-01 cannot reach Envoy (public DNS points at your proxy), so DNS-01 is used; it needs API credentials for your DNS provider. If you are not using our managed cert-manager, install your own.
Create the gateway namespace
kubectl create namespace gateway
Create the shared Gateway
Gateway describes the listeners. Put it in the dedicated gateway
namespace and reference the cluster GatewayClass named eg. The
allowedRoutes selector is what lets routes in other namespaces attach.
Create a file called gateway.yaml with the following content:
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shared
namespace: gateway
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"
- name: https
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
shared-gateway-access: "true"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: team-a-tls
The Gateway has two listeners. The https listener (port 443) terminates TLS
for the re-encrypted hop from your upstream proxy; it has no hostname, so it
serves every host whose certificate is listed in its certificateRefs, and
Envoy picks the right one per request by SNI. The http listener (port 80) is
where an HTTP-to-HTTPS redirect would attach, if you run one here rather than at
your proxy (see below).
allowedRoutes.from: Selector admits routes from any namespace carrying the
shared-gateway-access: "true" label; this is the opt-in that makes the
Gateway shared. Use from: Same instead if you ever want a Gateway that only
serves its own namespace, or from: All to admit every namespace
unconditionally (not recommended, since it removes the opt-in).
Apply it:
kubectl apply -f gateway.yaml
The ClientTrafficPolicy attaches to the Gateway by name and tells Envoy
how many trusted proxies sit in front of it. It lives in the gateway
namespace alongside the Gateway and covers the whole load balancer; app
namespaces do not need their own.
Create a file called client-traffic-policy.yaml:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: shared
namespace: gateway
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: shared
clientIPDetection:
xForwardedFor:
numTrustedHops: 1
numTrustedHops tells Envoy how many trusted ingress proxy hops sit in
front of it. Set it to the number of upstream proxies that prepend
entries to X-Forwarded-For. For a single CDN/WAF/edge proxy in front
of the load balancer, 1 is the right value; raise it for chains of
multiple proxies.
The setting affects what Envoy itself treats as the client IP, used in
access logs and rate limiting (and, in direct mode, the
x-envoy-external-address header).
Backends always see the full X-Forwarded-For chain that arrived
plus the load balancer’s internal IP appended on the right; Envoy does
not trim entries before forwarding the request upstream. Backend code
that needs the real client IP should parse the chain itself, typically
taking the leftmost public IP.
Note:
The policy must live in the same namespace as the Gateway. Envoy
Gateway rejects cross-namespace policy targets.
Apply it:
kubectl apply -f client-traffic-policy.yaml
Some load balancer setups forward the client IP in a different header.
Use customHeader instead; it is mutually exclusive with xForwardedFor:
clientIPDetection:
customHeader:
name: X-Real-IP
Issue a TLS certificate
Terminate TLS on the shared Gateway with a certificate per hostname, issued by
cert-manager into the gateway namespace (where TLS terminates). In proxy mode
your public DNS points at the upstream proxy rather than the load balancer, so an
HTTP-01 challenge would never reach Envoy. Use DNS-01 validation, which
proves control through a DNS record instead; it needs API credentials for your
DNS provider.
Create a file called certificate.yaml:
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-dns
namespace: gateway
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: platform@example.com
privateKeySecretRef:
name: letsencrypt-dns-account
solvers:
- dns01:
# Configure a DNS-01 solver for your DNS provider; see the
# cert-manager docs: https://cert-manager.io/docs/configuration/acme/dns01/
{}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: team-a-tls
namespace: gateway
spec:
secretName: team-a-tls
issuerRef:
name: letsencrypt-dns
kind: Issuer
dnsNames:
- team-a.example.com
Replace the email, DNS solver and hostname with your own, then apply it:
kubectl apply -f certificate.yaml
Each additional hostname needs its own Certificate and a matching entry in the
https listener’s certificateRefs; Envoy then selects the right certificate
per request by SNI.
Note: In proxy mode the upstream proxy terminates TLS for the public
client; this certificate is for the re-encrypted hop between the upstream
proxy and Envoy. Some setups instead let the proxy talk plain HTTP to the
load balancer; if so, route via the http listener and you can skip the
certificate.
Redirect HTTP to HTTPS
In this mode your upstream proxy (CDN / WAF / edge) usually performs the
HTTP-to-HTTPS redirect before traffic ever reaches the load balancer, so you
often do not need to configure one here. If you would rather have Envoy do it,
attach a redirect HTTPRoute to the http listener exactly as in the
direct-mode guide: label the gateway namespace
shared-gateway-access: "true" and apply a RequestRedirect route in it.
Onboard an application namespace
This is all an app team does; no access to the gateway namespace is needed.
1. Label the namespace so the shared Gateway admits its routes:
kubectl label namespace team-a shared-gateway-access=true
2. Deploy the app. Create app.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: team-a
spec:
replicas: 1
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: ealen/echo-server:0.9.2
ports:
- containerPort: 80
env:
- name: PORT
value: "80"
---
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: team-a
spec:
selector:
app: echo
ports:
- port: 80
targetPort: 80
3. Route traffic to it. Create route.yaml. The parentRefs points at
the shared Gateway in the gateway namespace, and that cross-namespace
reference is what puts this app behind the shared load balancer:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo
namespace: team-a
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- team-a.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: echo
port: 80
Apply both:
kubectl apply -f app.yaml -f route.yaml
The route’s hostname (team-a.example.com) must have a certificate on the
https listener; you issued one above. The backend Service is in the same
namespace as the HTTPRoute, so no ReferenceGrant is needed; you only need
one if a route points at a Service in a different namespace.
Verify
Check that the shared Gateway got an external address and that traffic
flows for the app namespace:
kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}'
curl -v https://team-a.example.com/
Confirm the route attached to the shared Gateway:
kubectl -n team-a get httproute echo -o jsonpath='{.status.parents[0].conditions}'
Accepted: True and ResolvedRefs: True mean the cross-namespace attach
worked. The backend should see the real client IP at the left of
X-Forwarded-For. Envoy forwards the full chain to the backend (including
the load balancer IP it appends on the right) and does not remove entries,
so the backend application is responsible for parsing the chain and picking
the leftmost public IP.
Common mistakes
- Namespace not labelled: the
HTTPRoute reports Accepted: False with reason NotAllowedByListeners, and traffic never reaches the app. Label the app namespace shared-gateway-access=true.
- Forgetting
namespace: in parentRefs: without it the route looks for a Gateway in its own namespace, finds none, and stays unattached. Cross-namespace routes must name the gateway namespace.
- No certificate for the hostname: if the
https listener has no certificate matching the route’s hostname, the TLS handshake fails and clients cannot connect. Issue a Certificate for each hostname and add its Secret to the listener’s certificateRefs.
- No upstream proxy in front: this variant assumes a CDN, WAF, or other proxy injects
X-Forwarded-For before traffic reaches the load balancer. Without one, no real client IP arrives, and your backend will only see the LB’s internal IP. If you have no upstream proxy, use direct (PROXY-protocol) mode instead.
- Forgetting
ClientTrafficPolicy: Envoy ignores the incoming X-Forwarded-For and treats the load balancer’s internal IP as the client. Rate limiting and access logs see the LB, not your real client.
- Putting
ClientTrafficPolicy in another namespace: silently ignored. Must be colocated with the Gateway (here, the gateway namespace).
- Wrong
numTrustedHops: too low and a caller can spoof the client IP by adding their own X-Forwarded-For entry. Too high and Envoy walks too far back into spoofable territory. Count one per trusted upstream proxy.
- Mixing
xForwardedFor and customHeader: they are mutually exclusive. Pick one.
Advanced usage
For more advanced use cases please refer to the documentation provided by
each project or contact our support:
3 - Migrating from ingress-nginx
How to move from managed ingress-nginx to Envoy Gateway, with or without floating IPs and in direct or proxy mode
This guide helps you move an existing workload from our managed
ingress-nginx to Envoy Gateway. The ingress concepts are the same
(listeners, routes, TLS, and a choice between direct and proxy mode), but the
resources you write are Gateway API objects
instead of Ingress objects.
If you are not migrating but setting up fresh, start from the
Envoy Gateway overview instead.
What changed
- Ingress always enters through the load balancer. ingress-nginx on older
clusters could accept traffic directly on each worker node (often on a node
floating IP). With Envoy Gateway, traffic always arrives through a single
OpenStack load balancer that fronts the Envoy data plane. Your public DNS
points at that load balancer, not at nodes.
- Floating IPs are now an egress feature. In Kubernetes CaaS v2 floating
IPs are removed from nodes by default and are an opt-in feature whose
purpose is a predictable outbound source IP. They are not part of the
ingress path. See Floating IPs below.
- Direct vs proxy is still a cluster-level mode, set by us: the same
choice you made with ingress-nginx (
use-proxy-protocol vs
use-forwarded-headers), just under new names.
Step 0: Confirm your mode (direct or proxy)
The mode must match how your cluster’s load balancer is provisioned. It is a
cluster-level setting that we manage; if you are unsure which one your
cluster runs, ask support before you cut over.
- Direct (PROXY-protocol) mode: clients connect straight to our load
balancer. The load balancer carries the real client IP with PROXY protocol
v2. No CDN or upstream proxy is involved. This is the default and the
equivalent of ingress-nginx
use-proxy-protocol: "true".
- Proxy (X-Forwarded-For) mode: you operate your own CDN / WAF / edge
proxy in front of the load balancer, and it injects
X-Forwarded-For. The
equivalent of ingress-nginx use-forwarded-headers: "true".
Proxy mode requires your own upstream proxy. It only makes sense if a CDN,
WAF, or edge proxy actually sits in front of the load balancer and injects the
header. Without one, no real client IP reaches Envoy and your backends see
only the load balancer. If clients connect directly to Elastx, use direct
(PROXY-protocol) mode; it needs no CDN.
Once you know your mode, the per-resource walkthrough lives in:
Resource mapping
| ingress-nginx |
Envoy Gateway / Gateway API |
IngressClass nginx |
GatewayClass eg (cluster-managed; you only reference it) |
Ingress (one object, implicit listeners) |
Gateway (explicit listeners, ports, TLS) + HTTPRoute (routing rules) |
spec.tls on the Ingress |
HTTPS listener tls.certificateRefs on the Gateway |
use-proxy-protocol / use-forwarded-headers (controller ConfigMap) |
Cluster mode (Elastx) + your ClientTrafficPolicy |
nginx.ingress.kubernetes.io/* annotations |
HTTPRoute filters, BackendTrafficPolicy, SecurityPolicy |
cert-manager Issuer with solvers.http01.ingress.class: nginx |
cert-manager Issuer with solvers.http01.gatewayHTTPRoute.parentRefs |
TCP/UDP services (tcp-services ConfigMap) |
TCPRoute / UDPRoute (see note below) |
Note: Our Envoy Gateway ships the Gateway API standard channel, which
provides only HTTPRoute and GRPCRoute. TCPRoute, TLSRoute and
UDPRoute are not installed. If you relied on the nginx tcp-services
ConfigMap, contact support before migrating those.
Migration steps
- Confirm your mode with Elastx (direct or proxy; see Step 0).
- Make sure cert-manager is available. If you are not using our managed
cert-manager, install and configure your own (with an
Issuer); see
Install and upgrade cert-manager.
- Recreate your ingress as Gateway API objects following the guide for your
mode (direct or proxy). The standard layout is a
shared
Gateway in a dedicated gateway namespace, with a matching
ClientTrafficPolicy and a TLS Certificate per hostname, that your
application namespaces attach to. Each app namespace then only carries the
shared-gateway-access: "true" label and one HTTPRoute per host. A single
ingress-nginx controller served all your hosts through one entry point
before; one shared Gateway is its direct equivalent, keeping your cluster
on a single load balancer.
- Translate annotations. Path rewrites, header manipulation and redirects
become
HTTPRoute filters; retries and circuit breaking become a
BackendTrafficPolicy; auth/CORS become a SecurityPolicy.
- Switch your cert-manager
Issuer to a Gateway-aware solver that issues a
certificate per hostname: the http01.gatewayHTTPRoute solver in direct mode,
or a DNS-01 Issuer in proxy mode (where public DNS points at your upstream
proxy). Each mode guide shows the one for that mode.
- Test before cutover without touching DNS, by resolving your hostname to
the new
Gateway address locally:
GW=$(kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}')
curl -v --resolve your.host.example.com:443:"$GW" https://your.host.example.com/
- Cut over DNS to the
Gateway’s load-balancer address. Use a low TTL (≈1
minute) beforehand so the change propagates quickly, and a CNAME where
possible so only one record needs updating.
- Decommission ingress-nginx once traffic is confirmed on Envoy Gateway.
Avoid running two controllers on the same
IngressClass during the overlap.
Floating IPs
How you migrate depends on what your floating IPs were doing:
- You used node floating IPs as your ingress entry point (older clusters).
That path no longer exists; ingress now enters through the load balancer.
Point your DNS at the
Gateway’s load-balancer address (Step 7). Nothing
about Envoy Gateway changes between “had node FIPs” and “did not”; the entry
point is the load balancer either way.
- You need a predictable egress (outbound) source IP. Floating IPs are still
available for that, as an opt-in feature; without them, egress is SNAT’ed via
the hypervisor. This is independent of ingress and of the direct/proxy choice.
If you want to keep or enable floating IPs, let support know; toggling them
recreates your nodes.
Advanced usage
For more advanced use cases please refer to the documentation provided by
each project or contact our support: