Bun WebSocket gateway

1. Foundational mental model

Multiplayer runs on one Bun process (apps/server). Every player holds a single WebSocket to /ws. The server owns all room state in memory; the browser only renders what it is sent. The gateway therefore has three jobs: decide who may open a socket, keep each socket honest about being alive, and keep any one socket from costing the process too much.

Admission happens once, over plain HTTP, before the upgrade. After that, every inbound frame goes through the same fixed pipeline, and every outbound frame goes through one send helper that watches the socket’s buffer.

Admission runs on the HTTP request. Nothing reaches room state until the upgrade succeeds.

Once a socket is open, the server treats a player as one of three things: connected (socket registered, heartbeat running), in grace (socket gone mid-game, seat held for 10 s), or removed (seat deleted). Most of this chapter is about the edges between those states.

Seat states as the server sees them. Grace exists only while a game is running.

The simulator below runs one room on a virtual clock with the server’s real constants. Try Player misses heartbeats in the lobby and again mid-question to see the difference grace makes.

Interactive

WebSocket state debugger

One simulated room on a virtual clock, driven by the server's real heartbeat, grace, size and rate-limit constants. Nothing touches a network.

Play Step 1s Reset

Speed

1×2×5×10×

Players

234

Seconds per question

10152030

0:00.0

Room phase 3-question demo run

1. Lobby
2. Starting
3. Question
4. Results
5. Finished

Waiting for the host Start game

• Ana Host
  Connected socket open missed 0/2 ping in 3.1s joined #1
• Ben
  Connected socket open missed 0/2 ping in 4.4s joined #2
• Chen
  Connected socket open missed 0/2 ping in 5.7s joined #3

Host disconnects Player misses heartbeats Player reconnects Send oversized message Flood answers Stop reading socket

Grace lasts 10s and only exists mid-game. In the lobby or after the game ends, a dropped player loses the seat at once.

Message log (newest first)

1. 0:00.0 room Room open with 3 players in the lobby.

Lobby. Host Ana.

Heartbeat, grace, 128 KiB message cap, 1 MiB buffer cap and sliding-window limits mirror apps/server and are checked by a parity test. Answers arrive on fixed per-player delays so every run is repeatable.

2. Implementation status & known gaps

Active invariants & limitations

• Active today: every upgrade passes origin, blocked-IP, per-session (3) and per-IP (55, or 75 for the shared unknown bucket) caps, then Arcjet, then the seat ticket check.
• Active today: every inbound frame is counted against a global 150 per 10 s budget for its connection before it is size-checked or parsed.
• Active today: the application-level heartbeat pings every 7 s and declares a socket dead after 2 unanswered pings.
• Known gap: a heartbeat death sends no close frame. The server drops its record of the socket and moves on; the client finds out only when its own socket errors or when it next tries to send.
• Known gap: SecurityUtils.validateMessage enforces a 10 000-byte MESSAGE_SIZE_LIMIT, but it serializes only { type }. The effective per-message cap is MAX_WEBSOCKET_MESSAGE_BYTES (128 KiB).
• Known gap: server-to-client message types are members of the same Zod union as client messages. A client can send NEW_QUESTION; it validates, then the router ignores it.

3. Concrete implementation

Routes and the Bun handler

app.ts registers /ws next to the HTTP routes and hands the three socket events to webSocketManager. The close handler also releases the per-IP and per-session counters.

apps/server/src/app.ts

    "/api/ws/preflight": {
      OPTIONS: handlePreflightRequest,
      GET: getWebSocketPreflight,
    },
    "/ws": {
      GET: upgradeWebSocket,
    },
  },
  websocket: {
    open: (ws: ServerWebSocket<WebSocketData>) => {
      webSocketManager.handleOpen(ws);
    },
    message: (ws: ServerWebSocket<WebSocketData>, message: string | Buffer) => {
      webSocketManager.handleMessage(ws, message);
    },
    close: (ws: ServerWebSocket<WebSocketData>, code: number, reason: string) => {
      webSocketManager.handleClose(ws, code, reason);

The websocket block sets only perMessageDeflate: false. maxPayloadLength, backpressureLimit and idleTimeout are left at Bun’s defaults, which the installed bun-types document as 16 MB, 16 MB and 120 s. The application checks below are stricter than all three, so Bun’s own limits act only as a backstop.

/api/ws/preflight runs the same WebSocketSecurity checks without Arcjet and without upgrading. The web client calls it first so it can show a specific toast (“Too many game tabs open”) instead of a bare socket error.

The upgrade gate

After the security gate and Arcjet, the handler reads the seat claim from the query string. A seat id alone is public (it appears in room snapshots), so the claim must come with the reconnect token issued at join time, bound to the same session_token cookie.

apps/server/src/lib/http/handlers/websocket.ts

  // A seat claim must come with a valid reconnect token bound to the same
  // account, otherwise any session could hijack a visible player seat id.
  if (playerSeatId) {
    const seatOk =
      !!roomId &&
      !!reconnectToken &&
      seatManager.verifySeat(roomId, playerSeatId, reconnectToken, accountId);
    if (!seatOk) {
      const origin = req.headers.get("origin");
      return createWsGateResponse(
        {
          reason: "Invalid seat credentials",
          code: WS_REJECT_CODES.FORBIDDEN,
          status: 403,
        },
        origin
      );
    }
  }

Every rejection is a JSON body { ok: false, error, code } plus an X-Flags-Reject-Reason header. The codes come from one shared list:

WS_REJECT_CODES value	Raised by	HTTP status
invalid_origin	missing or unlisted Origin	403
blocked_ip	IP in the in-memory suspicious set	403
missing_session	no session_token cookie	403
session_connection_cap	3 open sockets for this session	403
ip_connection_cap	55 sockets for this IP (75 for unknown)	403
arcjet_denied	Arcjet shield or token bucket	403
forbidden	seat claim without a matching token	403

A second missing_session branch in upgradeWebSocket returns 401, but validateConnection already rejects a missing cookie with 403 first, so that branch does not run in practice.

The inbound pipeline

handleMessage applies its checks in a fixed order. The global budget comes first, so a flood of junk costs the sender its budget before the server spends time parsing it.

apps/server/src/lib/managers/websocket-management.ts

      const connectionKey =
        ws.data?.connectionId ?? ws.data?.userId ?? ws.data?.accountId;
      if (connectionKey) {
        const globalLimit = rateLimiter.consumeGlobalMessage(connectionKey);
        if (!globalLimit.allowed) {
          const error = new AppError({
            code: ErrorCode.RATE_LIMIT_EXCEEDED,
            message: "Too many messages",
            statusCode: 429,
            details: {
              action: "GLOBAL_WS_MESSAGE",
              retryAfter: globalLimit.retryAfter,
            },
          });
          handleWebSocketError(ws, error, "rate_limit_global_message");
          return;
        }
      }

      const payloadBytes = Buffer.byteLength(message);

      if (payloadBytes > MAX_WEBSOCKET_MESSAGE_BYTES) {

The full order is:

1. Global budget: 150 frames per 10 s per connection. Over budget sends ERROR and drops the frame. The socket stays open.
2. Size: over 128 KiB sends ERROR (413) and closes with 1009.
3. Empty frame, then JSON.parse plus a JSON value schema.
4. validateMessage on { type } (string check, and the 10 000-byte check that in practice only measures the type field).
5. WebSocketMessageSchema, a Zod discriminated union on type. Failure sends VALIDATION_ERROR.
6. routeWebSocketMessage, which applies the per-action rate limit for the message type and then the handler.

Heartbeat

The heartbeat runs at the application layer, separate from Bun’s protocol pings, so the server can measure latency and detect half-open sockets on its own schedule.

apps/server/src/lib/managers/websocket-management/constants.ts

export const MAX_WEBSOCKET_MESSAGE_BYTES = 128 * 1024;
export const MAX_BUFFERED_BYTES = 1 * 1024 * 1024;
export const HEARTBEAT_INTERVAL_MS = 7000;
export const HEARTBEAT_TIMEOUT_MS = 4000;
export const HEARTBEAT_MAX_MISSED = 2;
export const INTENTIONAL_DISCONNECT_CLOSE_CODES = new Set([1000, 1001]);
export const RECONNECT_GRACE_PERIOD_MS = 10000;

Each interval sends { type: "HEARTBEAT", timestamp } and arms a 4 s timer. A HEARTBEAT_RESPONSE clears the timer and resets the miss count to zero. A timer that fires increments the count; at 2 the manager stops the heartbeat and calls handleUserDisconnect. There is no ws.close() on this path.

Heartbeat responses have their own limit (30 per 30 s). A client that exceeds it three times in a row is closed with 1008 “Heartbeat rate exceeded”.

Close and disconnect handling

handleClose splits closes three ways: replaced sockets, intentional closes, and everything else.

apps/server/src/lib/managers/websocket-management.ts

  handleClose(ws: ServerWebSocket<WebSocketData>, code?: number, reason?: string): void {
    if (!ws.data?.userId) {
      return;
    }
    if (ws.data.closedByNewSession) {
      const current = this.getConnection(ws.data.userId);
      if (current === ws) {
        this.removeConnection(ws.data.userId);
      }
      return;
    }

    if (code !== undefined && INTENTIONAL_DISCONNECT_CLOSE_CODES.has(code)) {
      logger.info("User closed connection intentionally", {
        userId: ws.data.userId,
        code,
        reason,
      });
      this.clearPendingDisconnect(ws.data.userId);
      this.evictUser(ws.data.userId);
      return;
    }

    this.handleUserDisconnect(ws.data.userId);
  }

handleUserDisconnect starts the 10 s grace timer only when gameState.isActive is true and the phase is not finished. In the lobby, or on the final scoreboard, the player is evicted at once. See Room lifecycle & host migration for what eviction does to the host role.

Close codes

Code	Reason string	Sent by	Server treats it as
1000	Seat attached elsewhere	seatManager.attachSeat on the old socket	replaced; connection record removed only
1000	User disconnected, Disposed	web client on leave or teardown	intentional; evict now, no grace
1001	(browser)	tab closed or navigated away	intentional; evict now, no grace
1006	(none)	network drop, no close frame	disconnect; grace if mid-game
1008	Heartbeat rate exceeded	server, third heartbeat-limit violation	disconnect; grace if mid-game
1009	Message too large	server, frame over 128 KiB	disconnect; grace if mid-game
1013	Backpressure	server, send buffer over 1 MiB	disconnect; grace if mid-game
4000	New session opened	server, a second socket for the same user id	replaced; the client does not reconnect
4000	Replacing socket	web client, before opening a fresh socket	replaced
4001	Unauthorized	server, open without a user id	never reached a room
4003	Invalid seat credentials	server, seat attach failed on open with no fallback id	never reached a room

The web client reconnects after any close other than 1000 and 4000, provided it still holds a room. It waits a fixed RECONNECT_DELAY_MS = 1200 between tries, with no backoff, and gives up after MAX_RECONNECT_ATTEMPTS = 6. A gate rejection during reconnect (for example forbidden because the seat was deleted) ends the loop immediately.

Rate limits

All per-action limits are keyed by user id (${action}:user:${id}). The global limit is keyed by connection id. Moderation, abuse handling and the HTTP-side limits are covered in Rate limiting & moderation.

Key	Limit	Window	On deny
Global, per connection	150	10 s	ERROR “Too many messages”, frame dropped
SUBMIT_ANSWER	50	10 s	ERROR RATE_LIMIT_EXCEEDED
SEND_REACTION	60	10 s	ERROR RATE_LIMIT_EXCEEDED
HEARTBEAT_RESPONSE	30	30 s	ignored; 1008 after 3 violations
CREATE_ROOM	5	60 s	ERROR RATE_LIMIT_EXCEEDED
JOIN_ROOM	20	60 s	same
LEAVE_ROOM	30	60 s	same
START_GAME, RESTART_GAME, STOP_GAME	10 each	60 s	same
UPDATE_ROOM_SETTINGS	30	60 s	same
UPDATE_AVATAR, KICK_USER	20 each	60 s	same
SESSION_TRUST_CONTEXT	10	60 s	same
Upgrade (Arcjet token bucket)	30 capacity	refill 10 per 60 s	403 arcjet_denied

Arcjet runs in LIVE mode only when ARCJET_MODE=live; any other value runs it as DRY_RUN, which logs decisions without blocking.

4. Internal mechanics & mathematics

Sliding window counter

rate-limiter.ts keeps two counts per key: the current fixed window and the one before it. For window size $W$, window start $s = \lfloor t / W \rfloor \cdot W$, current count $c$ and previous count $p$:

$$\hat{n}(t) = c + \operatorname{clamp}_{[0,1]}\!\left(1 - \frac{t - s}{W}\right) \cdot p$$

A request is allowed when $\hat{n}(t) < L$, and only allowed requests increment $c$. The previous count carries over only when exactly one window has elapsed; after a longer gap it resets to zero. Two consequences:

• A denied frame costs nothing, so a flooding client stays pinned at the limit instead of digging a deeper hole.
• The estimate assumes the previous window’s traffic was spread evenly. A client that sent $L$ frames at the very end of window $k$ sees $\hat{n} \approx L$ at the start of window $k+1$ and is still blocked, which is the behaviour you want from a burst limit.

The simulator uses a mirror of this counter with an explicit clock. multiplayer-constants.test.ts drives the server’s RateLimiterService and the mirror through the same 400-step schedule and asserts they allow and deny at identical instants.

Budget headroom

The config comment requires the global budget to stay above one busy window of legitimate traffic. Per 10 s: reactions $60$, answers $50$, heartbeat responses $30 \cdot \tfrac{10}{30} = 10$:

$$60 + 50 + 10 = 120 < 150$$

A normal client sends about $\tfrac{10}{7} \approx 1.4$ heartbeat responses and at most one answer per 10 s, so the global limit only trips on floods. In the simulator, Flood answers sends 160 SUBMIT_ANSWER frames at one instant: 150 pass the global check, 50 of those pass the SUBMIT_ANSWER check, 1 is accepted and 49 are rejected as ALREADY_ANSWERED.

Heartbeat detection latency

Let the client go silent at time $\sigma$ with interval $I = 7000$, timeout $T = 4000$ and $M = 2$ allowed misses. The first unanswered ping $p_1$ lands somewhere in $[\sigma, \sigma + I)$. Misses are recorded at $p_1 + T$ and $p_1 + I + T$, so death is declared at

$$D = (p_1 - \sigma) + (M - 1) I + T \in [\,I + T,\; 2I + T\,) = [\,11\,\text{s},\; 18\,\text{s}\,)$$

Mid-game, grace adds $G = 10$ s before eviction, so a half-open player holds their seat for $[21, 28)$ s after going silent. websocket-room-sim.test.ts asserts the lower and upper bounds of $D$ against the simulator.

5. Threat model, failure modes & edge cases

Where each abusive input is stopped.

• Seat hijack. Seat ids appear in every room snapshot. Claiming one needs the reconnect token (stored hashed with SHA-256) and the same session_token cookie. A stolen seat id alone gets a 403.
• Oversized frame in the lobby. 1009 is not an intentional close code, but grace only exists mid-game. A lobby player who sends one oversized frame loses the seat; the client’s automatic reconnect then fails with forbidden because the seat was deleted.
• Slow reader. Backpressure is checked in safeSendToUser, the path used for room broadcasts and direct sends. The heartbeat calls ws.send directly and does not check the buffer. A tab that stops reading in a quiet lobby therefore dies by heartbeat timeout after 11 to 18 s; in a busy game the next broadcast closes it with 1013 first.
• Half-open socket during a question. Players in grace still count as members, so the question cannot end early on “everyone answered” until they reconnect or are evicted. The timer still closes the question.
• Shared NAT. The IP cap (55) is sized for a classroom of 40 players plus host and refresh headroom. Clients whose IP cannot be resolved share one unknown bucket of 75, so a proxy misconfiguration degrades to one large shared pool instead of blocking everyone.
• Arcjet in dry run. Without ARCJET_MODE=live, Arcjet only logs. The in-process caps and rate limits still apply.
• Single process. Connection counts, rate windows, seats and grace timers live in memory. A restart drops every socket and every seat; clients’ reconnects then fail the seat check.

6. Architectural tradeoffs & non-goals

Use this gateway when a feature needs server-authoritative, low-latency state shared by a room: answers, timers, host actions.

Use HTTP instead when the data is per-user and not time-critical (solo results, leaderboards, replays). Those go through withMiddleware routes or Convex.

Only the budget headroom rule has a code comment stating its reason. The other rationales below are this chapter’s reading of the code.

Choice	Alternative	Why this one
App-level heartbeat on top of Bun pings	Rely on Bun’s idleTimeout (120 s)	Detects half-open sockets in 11 to 18 s and yields a latency sample per ping.
Grace only mid-game	Grace in every phase	A lobby seat is cheap to re-take by rejoining; holding it would keep ghost players in the member list and block “room full”.
Global budget before parsing	Parse first, then limit by type	A flood of malformed frames is rejected at the cost of one counter increment.
Fixed 1200 ms client retry, 6 attempts	Exponential backoff	Six tries spaced 1.2 s apart span about 7.2 s, inside the 10 s grace. Backoff would push later tries past the deadline.
perMessageDeflate: false	Compression on	Frames are small JSON; compression would add CPU and memory per socket for little saving.
In-memory counters	Redis or a shared store	One process serves all rooms. A shared store adds a network hop to every frame for no current benefit.

Non-goals: horizontal scaling across processes, surviving a server restart without dropping rooms, and authenticating users beyond the session cookie. Account identity comes from the web app’s session; the game server only binds seats to it.