Building an LLM gateway on Cloudflare Workers

A Cloudflare Worker is an isolate, not a process — it starts on the request, handles it, and doesn't sit around holding memory or connections between calls. There's no server to keep warm, no fleet to scale, no region to pick; the same script runs at every Cloudflare edge location. For a gateway whose entire job is 'take a request, pick an upstream, stream the response back,' that request-scoped model is a good fit — but it also means every architectural decision has to respect isolate limits that a conventional server wouldn't have.

Two of those limits shape almost everything downstream: a 128MB memory ceiling per isolate, and a 3MiB compressed script-size cap per Worker.

The 128MB ceiling makes one common server pattern actively dangerous: reading an entire upstream response into memory before forwarding it. An LLM completion can be arbitrarily long, and await response.text() on an unbounded stream has no ceiling of its own — it grows until the isolate runs out of memory.

In practice this means chat completions, Anthropic messages, and the Responses API all return a native Response wrapping the upstream's server-sent-event stream, piped through a format translator chunk by chunk rather than accumulated and replayed:

export default (async (c) => {
  const upstream = await fetch(upstreamUrl, { method: "POST", body, headers })

  // Never buffer: pipe the upstream stream straight through,
  // translating SSE dialect chunk-by-chunk as it passes.
  return new Response(upstream.body.pipeThrough(dialectTranslator), {
    status: upstream.status,
    headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" },
  })
}) satisfies HonoHandler

A single Worker outgrew Cloudflare's 3MiB compressed script-size cap once the routing engine, the model catalog, Durable Objects, MCP server, OAuth, and the full Hono API surface all lived in one script. The fix was a hub-and-spoke split: one web Worker owns the zone-wide route and either forwards a request to a dedicated spoke over a service binding, or serves the page itself.

Because each spoke proxies its own same-origin /api/* calls to anyrouter-api over a service binding, the browser never makes a cross-origin request — no CORS configuration needed, and no extra public surface exposed per Worker.

Dispatch to the actual model provider happens through exactly one of three transport shapes, all funneled through Cloudflare's AI Gateway rather than calling a provider's API directly:

That split is invisible to the end user — every backend shares the same 'Cloudflare AI Gateway' badge in the UI regardless of which transport actually served it. The rule behind it is stricter than it sounds: every upstream must route through gateway.ai.cloudflare.com, with exactly one documented exception for a first-party backend that has no HTTP origin to proxy in the first place (it reaches a user's own device over an outbound WebSocket instead).

Running on Workers also means the platform's crypto primitives are the default, not an add-on dependency:

• crypto.getRandomValues() or crypto.randomUUID() — never Math.random() for tokens or ids
• crypto.subtle for all cryptographic operations, including per-row encryption at rest
• crypto.subtle.timingSafeEqual for secret comparison, so key checks aren't timing-attackable
• Secrets live in wrangler secret put, never hardcoded in source or committed config

Explicit error handling matters more than it would in a framework with a global exception page, too — passThroughOnException() is avoided deliberately, because it hides real bugs behind a silent fallback instead of surfacing them to Hono's error handler where they can be logged and fixed.

None of this is free. Every limit above is a constraint the architecture works around, not one it eliminates — the 3MiB cap means new features have to think about which Worker they belong in, and the streaming-only rule means there's no shortcut for post-processing a full response before it reaches the client. What it buys back is a script that runs identically at every edge location with no server fleet behind it to operate.