API Gateways: What They Do and When You Actually Need One

In March 2026, Kubernetes retired Ingress NGINX, and there will be no more security patches for it. None of the available alternatives to Ingress NGINX will be drop-in replacements. The steering committee has indicated that about half of all cloud native environments rely on Ingress NGINX.

This places a migration decision in front of a significant number of cloud-native teams. This is a crowded decision space. Reverse proxies, load balancers, ingress controllers, service meshes, and API gateways each govern network traffic at different layers/boundaries; their responsibilities overlap.

What does an API Gateway do, and how much overhead will using an API Gateway bring versus what benefits will it add to your organization?

Executive Summary

As a centralized entry point for incoming API requests, an API Gateway acts as a policy enforcement layer before API traffic reaches your backend services. User authentication, rate limiting, request routing and telemetry are cross-cutting concerns that move away from individual backend services into one layer you can control. One location to monitor, one location to modify.

Four key strategic arguments support this position:

• Your security and traffic rules live in one version-controlled place, so new services inherit them instead of reimplementing them.
• Your gateway handles external consumer policy while your mesh handles workload identity so large deployments typically run both together.
• Replicas protect your gateway against node failure, but a bad config pushed to all replicas takes down every API you run at once.
• Per-IP rate limits fail against AI agents because automated traffic shares NAT and cloud egress addresses so you need identity-based limits instead.

What Is an API Gateway?

A server that resides between client applications and multiple backend services, an API Gateway acts as the single entry point for API traffic. The API Gateway routes requests to the appropriate services and then returns the results. The API Gateway can handle user authentication and access control, enforce rate limits, validate payload content and generate telemetry metrics during the process.

North/South traffic flows across the boundary of your system; i.e., a mobile app calls your API or a partner pulls a report from your system. East/West traffic remains within your system; i.e., one of your services calls another service. The API Gateway controls North/South traffic flow.

While providing a long list of features helps illustrate what an API Gateway provides, its primary function is decoupling. The clients interact with a stable interface that hides complex backend systems while you refactor/merge services under the hood. An API Gateway provides a mechanism to create that interface, and without an interface in front of your services, internal changes affect client migration requirements.

How an API Gateway Works

When an API Gateway receives incoming API requests, each one goes through a series of ordered phases. Any phase may reject incoming requests before they reach your backend. If a bad request gets rejected at the gate, you won’t have spent a worker thread or database connection on your backend servers processing it. Each product has its own exact phase and order. This is an example of what many gateways look like:

One of the first things a gateway may do is terminate TLS. It can also check whether the payload is valid against a schema when configured with one you define using something like an OpenAPI definition. Malformed payloads can be immediately rejected at the edge with a 400 before your backend wastes resources processing them. Many companies also place their Web Application Firewall at this same point of entry into their systems. So when an exploit signature is recognized, it can be filtered immediately before any schema validation occurs.

Next comes authentication and authorization. Your gateway checks incoming API calls against a defined set of access control criteria, including API Keys, OAuth 2.0 Tokens, signed JWTs or OIDC Claims from trusted identity providers. Most of the time, when authentication fails, the gateway will return a 401 status code. When authorization fails, the status codes returned are often 403 (authorization failures commonly return 403).

Validating a JWT signature at the edge of the gateway lets downstream services rely on the Identity Header they receive, instead of implementing their own cryptography on the receiving end of the request. However, there are two conditions for relying on validation of a JWT signature at the edge: Clients can’t access those services unless they pass through your gateway. Additionally, your gateway should strip or overwrite any client-provided identity headers.

Next, after validating the request, is rate limiting, which protects backend services from overload. At this point, your gateway will count all requests from each unique consumer and reject anything above that threshold with a 429 status code. In cases where you’re running multiple tenants within your multi-tenant environment, one customer’s runaway script sending API requests could consume capacity that was purchased by other customers.

If a request is cacheable, your gateway can serve frequently accessed data directly back to the requesting client for those client requests and never send them to any backend.

Once caching is handled, your gateway needs to handle request routing across multiple backend services. At this point, your gateway uses logic to match each surviving request to specific API endpoints based on path or hostname. Sometimes, your gateway can also use headers to determine which service to use. Finally, your gateway must perform load balancing between all healthy instances of the selected service, distributing client requests evenly to avoid hotspots.

Once routing has occurred and your gateway has determined which service(s) to use, it can take one client request and distribute it across multiple microservices, and in high-throughput environments, efficiently handling concurrent client requests at this layer is what keeps latency low. Your gateway can then return a single combined response to the client, eliminating additional round trips that slower client connections would otherwise suffer through. Your gateway can also perform protocol translation and request transformation, so a public REST API endpoint can front internal services that speak gRPC or other protocols. Since all North-South requests flow through your gateway, it can also assign TraceIds, collect latency and error metrics and track API usage for each route.

API Gateway vs. Reverse Proxy vs. Load Balancer vs. Service Mesh

Gateway, load balancer, reverse proxy, and service mesh are all commonly used terms. Each is defined briefly, as these four tend to be interchanged in conversation.

A load balancer handles load balancing by distributing incoming traffic between multiple replicas of the same service and removes unhealthy servers from the rotation. A reverse proxy sits before your application, terminates TLS connections, and forwards requests to your application while concealing the backend from the client. A service mesh is a data plane that intercepts and governs service-to-service traffic using either a sidecar or sidecarless method based on the service mesh deployment.

Direction is a common shorthand for how gateways handle North-South traffic and how meshes handle East-West traffic. This is true and describes where things are located rather than what they can do.

Instead of looking at direction, look at what each layer typically knows about. An API gateway knows rules like “this was a call from a customer on the free tier who had 40 calls remaining this hour.” A mesh knows rules like “one workload may establish a secure connection to another, but authenticated through mutual TLS as opposed to any end user.”

While the four layers have overlapping capabilities, they are generally differentiated by defaults and operational models rather than specific functionality. Istio can validate end-user JWTs and apply request-level authorization.

As things grow larger, they can operate together. The API Gateway operates at the public boundary, authenticates the caller, and enforces quota. Then it passes traffic to the mesh, which handles encrypted routing between services.

Four Cases That Justify an API Gateway

• Multiple services under one client contract. There are two options to expose data from six services into a single client screen. You could allow the client to orchestrate six calls themselves, or you could use a gateway/BFF to do it on their behalf. You can also break up or combine those six services without updating the mobile app.
• APIs exposed to partners/clients. You need to manage keys, quotas, plans and revoke them when necessary. According to the 2025 State of the API report, which surveyed over 5700 developers, architects, and executive members, 65% of organizations generate revenue from their API program. Therefore, governance becomes less of an engineering preference and more of a billing consideration.
• Incremental legacy system migration. One way to apply Martin Fowler’s Strangler Fig pattern is to route some paths to your old system and some to your new system, then move the paths over time. A gateway provides a place to route the paths as part of this process. However, the Strangler Fig pattern does not call for a gateway. Which components you extract first is a separate decision.
• Multi-cloud hybrid infrastructure. Even though workloads are spread across two clouds and a datacenter, you still owe your clients one hostname and one authentication scheme. Gateways provide both regardless of where the workload is actually running.

Where an API Gateway Is the Wrong Tool

A gateway is another production system. It will need capacity planning, a patch schedule, on-call staff and a process for reviewing configuration changes.

You can skip using an API Gateway if you have a well-planned monolithic application behind a load balancer. A gateway adds an extra network hop and another place to misconfigure your TLS settings. NGINX in front of your application does the job.

Skip it for internal service-to-service traffic. Routing East-West calls out to an edge gateway and back in, a pattern known as hairpinning, adds latency in both directions, and a mesh generally covers the same internal concerns more directly.

Be honest about the potential blast radius. Replicas behind a load balancer will help when a single server goes down, since load balancing across healthy nodes absorbs the loss automatically. However, they won’t help if you make one incorrect configuration update and apply that same incorrect configuration update to every replica of every API you are running. Budget time for staged rollouts as well as fast rollback times the same way you would budget time for building and supporting the gateway.

How Teams Actually Use API Gateway Technology

Netflix utilizes Zuul for its edge. The first version used a threading/blocking design, using a thread for each active request until the backend completed, which tied concurrency to thread utilization. This isn’t a good fit for an edge where Netflix reported processing over a million requests per second (in 2018) through a very large number of long-lived connections.

Zuul 2 rebuilt the gateway on Netty, an asynchronous event-driven networking framework, so requests no longer hold threads while they wait. According to Netflix, this was done primarily to support persistent connections from tens of millions of devices vs. raw throughput. The concurrency model you choose for your gateway will set a limit for your whole edge. In 2016, Netflix started announcing the Zuul 2 rebuild and released the results as open source in 2018.

SoundCloud ran into another issue. They were running one general-purpose API that supported both their own web & mobile apps and third-party clients. Every time a code update was needed, it had to be coordinated across all parties involved.

Phil Calçado (then an Engineer on SoundCloud’s Core Platform Team) solved the organizational problem with a solution that also addressed the architectural issue; he applied the Backend for Frontend pattern, typically referred to as BFF. Each client has its own application-specific backend, owned by the client team. A BFF can be built using gateway technology, but most likely includes some form of presentation-oriented composition, more than just routing and policy. Sam Newman (author of Building Microservices) documented this pattern after seeing it at an Australian real estate company, REA.

A Mobile BFF will aggregate data (for example) and strip fields to reduce bandwidth usage. A web BFF will return richer payloads that a dashboard may require. By implementing this pattern, SoundCloud resolved the organizational challenge of having multiple client teams wanting to ship their own edges and doing it architecturally.

Kubernetes has been encouraging users moving applications to utilize the new Gateway API that achieved General Availability (GA) in October 2023 and shipped v1.6 in June 2026 while alternative Ingress controllers remain an option. The Gateway API was designed to create separate responsibilities for owners:

Configuration is divided among resources targeting the personas the specification names. GatewayClass defines the controller and the underlying infrastructure that supports it, which is the infrastructure provider’s responsibility. Gateway refers to the traffic-handling instance owning listeners and an address, which is the cluster operator’s responsibility.

HTTPRoute and its rules are application-facing and are the responsibility of application developers. How these map to your org chart is an implementation detail; the specification identifies personas, not team names. Ingress provided a simpler model: Ingress plus IngressClass, with controllers adding features via implementation-specific annotations and custom resource definitions. Instead, Gateway API explicitly separates gateway infrastructure and application routes, delegating what Ingress struggled to delegate cleanly.

Traffic itself has also evolved. In Postman’s 2025 report, 51% of developer respondents listed unauthorized or excessive API calls from AI agents as their #1 security concern. And Akamai’s 2026 State of the Internet report lists APIs as the primary enterprise attack surface. Daily average API attacks per organization rose from 121 in 2024 to 258 in 2025.

When developing rate limiting for human traffic, there are two presumptions about the nature of automated traffic that automated traffic strains. Automated traffic has bursts that are considered malicious based upon human activity but are simply part of completing tasks for an automation agent. When automated traffic shares IP addresses via NAT, proxies or cloud egress addresses, per-IP limitations provide a weak surrogate for user identification.

Set rate limits on authenticated consumer or agent identity where possible, rather than on IP address. The Kubernetes AI Gateway Working Group formed in March 2026. It is developing standards and guidance for AI-specific gateways, including token-based rate limiting. Track that work before building a custom solution.

Trade-Offs Worth Settling Early

Keep domain state and business logic out of the gateway.

Business logic at the edge rebuilds the monolith you were escaping, but now it sits directly in the path of every client. The gateway determines routing and enforces policy. Domain decisions belong in the service layer.

Decide where rate limiting state exists.

Per-node counters provide very low latency and allow clients to exceed their globally defined rate limit by a factor approximately equal to the number of gateway nodes you have running. Shared counters via Redis are often used as an alternative, though local counters break behind load balancers. Synchronously checking a shared counter may introduce a network round-trip delay on each incoming request.

Measure your own latency overhead rather than rely upon benchmark data.

Published latency figures don’t relate well to your workload because they are typically measured against a mock backend. AWS provides both Latency and IntegrationLatency metrics for Amazon API Gateway, and the documentation states that Latency includes IntegrationLatency plus the gateway’s own processing time. Compare aligned average values to estimate the overhead due solely to the gateway. Do not subtract the 99th percentile value from another 99th percentile value and then label the resulting value a 99th percentile. Percentiles do not work that way!

Determine who can change routing.

Centralized policy management has the side effect of centralized authority. When route changes are forced through a single platform engineer, the gateway becomes a queue. Teams start hard-coding routes into their services to get around it. Divide gateway configuration: application teams manage their routes, and the platform team manages the listener.

Choose a deployment model intentionally.

Consider them configurations, not product categorizations. All three models exist across these vendors, though combinations differ: Kong supports self-managed and cloud-runtime versions; Apigee hybrid combines a Google-managed control plane with a runtime version you manage.

Operating at high availability requires platform engineering capacity that not every organization keeps in-house. Some teams buy the managed option to sidestep the problem. Other teams engage custom software development services to create the layer and hand it over.