How it’s built

Overview

Machine UI Chat is a deliberately small chat app: sign up, ask something, get a streamed reply, come back to it later. The interesting part is everything around the chat box, the parts that decide whether an app survives real users: where state lives, what happens when the model fails halfway through a sentence, how one user is kept out of another's data, and what changes when you run ten copies instead of one.

The stack, and the one-line reason for each:

One codebase, two runtimes

If you're used to a React frontend talking to a separate backend service, the biggest shift is that there is no separate service. The same repository, the same build, and often the same file contain code for two places: the browser and the server. The build splits them apart.

Three ways the browser reaches the server

Loaders are isomorphic: on the first request they run on the server during SSR; after that, client-side navigation runs them in the browser. That's why every loader here calls a server function rather than the database directly. The server function is the boundary that is always on the server.

// src/functions/conversations.functions.ts
export const getConversationFn = createServerFn({ method: 'GET' })
  .middleware([authMiddleware])               // resolves the session → context.user
  .validator(z.object({ conversationId: z.uuid() }))
  .handler(async ({ data, context }) => {
    const detail = await getConversation(context.user.id, data.conversationId)
    if (!detail) throw notFound()
    return detail
  })

// In a component or loader: a typed call. In the browser bundle this
// becomes a fetch to an RPC endpoint; the handler body never ships.
const detail = await getConversationFn({ data: { conversationId } })

Why is the chat stream a server route instead of a server function? Server functions are an RPC: typed request in, serialized value out. The chat reply is a long-lived HTTP response that the browser reads incrementally and can cancel, so a plain Request → Response handler with a ReadableStream body is the clearest fit, and it's a real HTTP API you could call with curl.

How the server code is organised

src/
├── routes/                      # file-based routes: pages AND HTTP endpoints
│   ├── __root.tsx               # <html> shell, fonts, theme script
│   ├── index.tsx                # landing (SSR)
│   ├── login.tsx, signup.tsx    # auth forms (SSR)
│   ├── architecture.tsx         # this page (prerendered at build)
│   ├── chat/route.tsx           # layout + auth gate for /chat/*
│   ├── chat/index.tsx           # new chat
│   ├── chat/$conversationId.tsx # a conversation (SSR with data)
│   └── api/
│       ├── chat.ts              # POST → Server-Sent Events
│       ├── auth/$.ts            # Better Auth catch-all
│       └── health.ts            # load balancer health check
├── functions/                   # createServerFn RPC (safe to import from UI)
├── server/                      # server-only: *.server.ts never reaches the browser
│   ├── env.server.ts            # zod-validated config
│   ├── db/                      # schema + connection pool
│   ├── auth.server.ts           # Better Auth config
│   ├── chat/                    # chat service + data access
│   └── llm/                     # provider interface, mock vendor, retries, context
├── shared/chat-protocol.ts      # request schema + stream event types (both sides)
├── features/                    # UI by feature: auth, chat, this page
├── components/ui/               # machine-ui components (installed as source)
└── start.ts                     # global middleware: CSRF, security headers

• Routes are thin. They parse input, check the session and translate results to HTTP. No SQL lives in a route file.
• The chat service is framework-free. startChatTurn() takes plain arguments and returns an async generator, so the integration tests drive it directly against a real Postgres without booting HTTP.
• .server.ts is a hard boundary. Start's import protection fails the build if browser code imports one of those files. Secrets are read in env.server.ts, lazily, through a Zod schema, so a misconfigured deploy fails at startup with every problem listed.
• Shared contracts live in one file. shared/chat-protocol.ts holds the request schema and stream event types that both the route and the browser import, so changing the protocol is a type error on both sides.

How this page was rendered

This page is static HTML generated at build time. During vite build, Nitro renders /architecture through the app and writes the result to .output/public/architecture/index.html. In production the server returns that file like any other asset, with an ETag and without running React, so it costs about as much as a CDN hit. It still hydrates, so clicking a link from here to /chat is a normal client-side navigation.

// vite.config.ts
nitro({
  prerender: {
    routes: ['/architecture'],
    crawlLinks: false,   // don't follow links to /, /chat…
    failOnError: true,   // a broken page fails the build
  },
  // prerendered files skip app middleware, so add headers here too
  routeRules: { '/architecture': { headers: securityHeaders } },
}),
tanstackStart(),

The mental-model difference from Next: in the App Router, static vs dynamic is inferred per route. Read cookies() and the route silently becomes dynamic. In Start, everything is server-rendered per request unless you opt paths into prerendering in config. It's explicit, and it's why link crawling is switched off here: / shows different buttons to signed-in users, so baking it into a file would be a bug.

Start also supports ISR-style caching: set Cache-Control: s-maxage=…, stale-while-revalidate=… on a response and let the CDN keep it. That's the same idea as Next's revalidate, expressed as plain HTTP headers instead of a framework option.

Data model

Four tables belong to Better Auth (user, session, account, verification) plus its rate_limit table. The app adds the rest:

conversation (id uuid default uuidv7(), user_id → user, title, model, created_at, updated_at)
  index (user_id, updated_at desc, id desc)          -- the sidebar query, exactly

message (id uuid default uuidv7(), conversation_id → conversation, user_id → user,
         role user|assistant, content, status streaming|complete|cancelled|error,
         model, input_tokens, output_tokens, cost_usd numeric(12,6), stop_reason, error, …)
  index (conversation_id, created_at, id)            -- load a thread in order
  index (user_id, created_at)                        -- per-user rate limiting
  unique (conversation_id) where status = 'streaming' -- one reply in flight per chat

provider_credential (user_id → user, provider, encrypted_key, key_hint, label, …)
  unique (user_id, provider)                         -- one key per provider per user

user_settings (user_id → user primary key, default_model)
app_rate_limit (key primary key, window_start, count) -- fixed-window counters

Schema changes go through versioned SQL migrations in drizzle/, generated with pnpm db:generate and applied with pnpm db:migrate. The test suite applies the same files to a separate test database, so a migration that doesn't work fails CI before it reaches production.

Anatomy of a chat turn

Phase 1: prepare (one transaction)

• Take a per-user advisory lock (pg_advisory_xact_lock) so two simultaneous sends from one user can't both pass the rate limit.
• Count this user's messages in the last minute; over the limit is a 429 with Retry-After.
• Load the conversation filtered by user id; someone else's is a 404.
• Build the model context, then insert the user message and an empty assistant row marked streaming.

Anything that fails here rolls back completely and becomes an ordinary HTTP error, because no response bytes have been sent yet. That's the reason for splitting the work into two phases.

Phase 2: stream

The response is text/event-stream. We use fetch with a streamed body on the client instead of EventSource, because EventSource can only make GET requests and can't send a JSON body. The wire format is plain SSE:

event: meta
data: {"type":"meta","conversationId":"01a1…","created":true,"userMessageId":"…","assistantMessageId":"…"}

event: delta
data: {"type":"delta","text":"This "}

event: delta
data: {"type":"delta","text":"is "}

event: done
data: {"type":"done","stopReason":"end_turn","usage":{"inputTokens":38,"outputTokens":56}}

Every reply ends in exactly one recorded state:

Building the context window

Models have a fixed context window, so the server decides what the model sees (server/llm/context.ts, unit-tested):

• Drop failed and empty turns.
• Merge consecutive same-role turns. A failed reply leaves two user messages in a row, and vendor APIs reject anything that doesn't alternate.
• Reserve room for the reply, then keep the newest turns that fit; the oldest fall off first. The window always opens on a user turn.

Token counts are estimated at about four characters per token, which is deliberately pessimistic. A real build would use the vendor's token-counting endpoint or summarise old turns instead of dropping them.

Surviving navigation

Starting a chat at /chat moves the URL to /chat/<id> as soon as the meta event arrives, while text is still streaming. The in-flight turn lives in a provider above the routes, so the navigation swaps the page without dropping the stream. When the turn ends, the client refetches the saved messages and replaces its optimistic copy, so what you see afterwards is exactly what is in the database.

The LLM boundary

The mock isn't a function that returns a string. It's built like a vendor so that everything around it is real:

• mock-api.server.ts is a fake vendor endpoint. It receives a JSON string (so anything unserialisable fails just as it would over HTTP), validates it like a real API (non-empty, alternating roles, must end on a user turn), adds latency, can fail with 429/529 at a configurable rate, and replies with SSE frames in the vendor's own format.
• mock-provider.server.ts is the adapter: it serialises our request, maps HTTP errors to typed LlmError kinds, and translates vendor frames into our events. A real adapter has exactly this shape.
• resilient.server.ts wraps any provider with an idle timeout and retries with exponential backoff and jitter, honouring retry-after. It retries only before the first token: once text has reached the user, a retry would repeat or contradict it.

// src/server/llm/types.ts
export interface LlmProvider {
  readonly name: string
  stream(request: LlmRequest, options: { signal: AbortSignal }): AsyncIterable<LlmStreamEvent>
}

export type LlmStreamEvent =
  | { type: 'text'; text: string }
  | { type: 'done'; stopReason: StopReason; usage: LlmUsage; model: string }

// One adapter per vendor, built per request with that user's credentials:
createProvider({ provider: 'openrouter', model: 'anthropic/claude-sonnet-5.5' }, { openRouterKey })

The reply quotes your message and reports how many earlier messages and tokens it received, so you can watch context handling work. Set MOCK_LLM_FAILURE_RATE=0.5 to watch retries in the server logs, or ask for "a long essay" to test Stop and max_tokens. (The "Mock model, fails half the time" option in the picker does the same from the UI.) The real adapter, for OpenRouter, has exactly the same shape as the mock one; only the wire format differs:

// src/server/llm/openrouter/provider.server.ts (excerpt)
for await (const { data } of parseSse(readTextStream(response.body))) {
  if (data === '[DONE]') break
  const chunk = JSON.parse(data)

  // Mid-stream failures arrive inside a 200 response, not as an HTTP status.
  if (chunk.error) throw new LlmError(sawContent ? 'network' : 'overloaded', chunk.error.message)

  const text = chunk.choices?.[0]?.delta?.content
  if (text) yield { type: 'text', text }
  if (chunk.usage) usage = { inputTokens: chunk.usage.prompt_tokens,
                             outputTokens: chunk.usage.completion_tokens,
                             costUsd: chunk.usage.cost }
}

Real vendors have quirks a clean mock wouldn't invent. OpenRouter sends : OPENROUTER PROCESSING comment lines as keep-alives (the SSE parser skips comments), reports errors that happen after streaming has started inside a 200 response, and puts usage and cost in one extra chunk just before [DONE]. The adapter is tested against a fake OpenRouter server on real HTTP (tests/fake-openrouter.ts) that reproduces each of those.

Models and API keys

Picking a model

Models are named <provider>:<model>, such as mock:mock-sonnet or openrouter:anthropic/claude-sonnet-5.5. Only the first colon splits them, because OpenRouter ids can contain colons themselves. The picker lists the mocks plus OpenRouter's public catalog. The server fetches that catalog once an hour, shares one request between concurrent callers, keeps serving the old list if a refresh fails, and drops models that can't do text chat (image generators, offline :batch variants).

• A conversation remembers its model, and the last model you used becomes the default for new chats. Both are stored on the server, so they follow you between devices.
• The server never trusts the picker: every turn looks the model up in the catalog (400 unknown_model otherwise) and checks you have a key for it (400 api_key_required).
• The context budget is the smaller of the model's window and ours (LLM_CONTEXT_WINDOW_TOKENS, 16K by default). A model that accepts a million tokens would otherwise resend, and bill, the whole conversation on every turn.

Storing your OpenRouter key

Each user brings their own key, so usage is billed to their OpenRouter account rather than to the app. That makes the key the most sensitive thing in the database, and it's handled accordingly:

• Checked before it's stored. Saving calls OpenRouter's GET /key; a rejected key never reaches the database, so typos fail on the settings page rather than mid-conversation.
• Encrypted at rest with AES-256-GCM under CREDENTIALS_ENCRYPTION_KEY. A database dump or backup on its own reveals nothing.
• Bound to its owner. The user id and provider are authenticated data in the encryption, so copying an encrypted value onto another user's row makes it fail to decrypt. There's a test for exactly that.
• Write-only from the browser's point of view. Pages only ever receive …abcd. The plaintext exists only inside the chat request that forwards it to OpenRouter, and is never logged.
• Key checks are rate limited (5 per 10 minutes per user). Without that, the save endpoint would let anyone with an account test stolen keys through our servers.

// src/server/crypto.server.ts
export function encryptSecret(plaintext: string, context: string): string {
  const iv = randomBytes(12)                         // never reused
  const cipher = createCipheriv('aes-256-gcm', key(), iv)
  cipher.setAAD(Buffer.from(context))                // e.g. "user:42:openrouter"
  const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()])
  return ['v1', iv, cipher.getAuthTag(), ciphertext] // "v1.<iv>.<tag>.<ct>"
    .map((p) => (typeof p === 'string' ? p : p.toString('base64url')))
    .join('.')
}

The v1. prefix is there for key rotation: add a second key, decrypt with whichever version a value names, and re-encrypt stored values in the background. The next step up is envelope encryption: a cloud KMS holds the master key and the app only ever sees per-value data keys, so a leaked environment variable is no longer enough.

Security

One deliberate trade-off: Better Auth's cookie cache keeps a signed copy of the session in a cookie for five minutes, so most requests skip the session-table lookup. The price is that a revoked session can keep working for up to five minutes. For a bank, turn it off; for a chat app, it's a reasonable deal.

Scaling

The rule that makes horizontal scaling possible: app instances hold no state that matters. Sessions, rate-limit counters, conversations and the one-reply-in-flight lock all live in Postgres, so any request can land on any instance and a load balancer can add or remove copies freely.

The Postgres-only approach is the right starting point: one fewer system to run, and transactions give correctness for free. Add Redis when a measurement says so, not before.

Performance

• First paint has data. /chat/<id> server-renders the sidebar and thread; TanStack Query hydrates from the HTML instead of refetching. No loading spinner on a hard refresh.
• Perceived latency is time to first token. Streaming shows text within a few hundred milliseconds even when the full reply takes seconds. The mock simulates both (MOCK_LLM_FIRST_TOKEN_MS, MOCK_LLM_TOKEN_MS).
• Indexes match queries. Each hot query (sidebar page, thread load, rate-limit count) is served by a composite index in the same column order as its WHERE/ORDER BY.
• Fewer round trips. The session is cached for a minute in the query cache, and Better Auth's cookie cache skips the DB on most requests. Links preload their route data on hover (defaultPreload: 'intent').
• Small client. Devtools are removed from production builds; routes are code-split. The largest chunk is about 110 kB gzipped (React + router).
• Proxies must not buffer. The chat response sets Cache-Control: no-store and X-Accel-Buffering: no, or an nginx-style proxy will hold every token until the reply ends.

What to measure in production: time to first token and tokens per second (per model), p95 latency of server functions, error rate by LlmError kind, and tokens per user per day. The structured logs (chat.completed includes tokens, duration and dropped context messages) are the raw material.

Deploying it

pnpm build produces .output/: a Node server (.output/server/index.mjs) with its dependencies bundled, plus static assets and this prerendered page in .output/public/. About 4.5 MB in total, with no node_modules install needed at runtime. Frontend and backend deploy together as one unit, so they can never be on different versions.

• Container: the multi-stage Dockerfile builds once and copies only .output/ into a slim image running as a non-root user, with a health check against /api/health.
• Migrations are a release step, not a boot step. The migrate image target runs drizzle-kit migrate once before new app containers start. If every instance migrated on boot, ten instances would race to alter the same tables.
• Config comes from the environment. .env is only for local dev; production sets real environment variables (validated at startup).
• Try it locally: docker compose -f docker-compose.prod.yml up --build runs Postgres, the migration job, then the app on port 3200.

Where to run it: any container host with managed Postgres works: Fly.io, Railway or Render for simplicity, AWS ECS/Fargate with RDS, or Cloud Run with Cloud SQL. Nitro presets also target Vercel, Netlify and Cloudflare. On edge runtimes you'd swap node-postgres for an HTTP or WebSocket Postgres driver (Neon, Hyperdrive), which the lazy db() accessor makes a one-file change.

Trade-offs and next steps

Natural next steps, roughly in order of value:

• Spending visibility: show each reply's cost and a running total per conversation, plus optional per-user budgets.
• Markdown rendering (sanitised), conversation titles generated by the model, regenerate and edit-last-message.
• Email verification and password reset; a CSP with nonces.
• OpenTelemetry traces across route → service → vendor; dashboards for time to first token and error kinds.
• Resumable streams with a job queue and Redis once replies get long or users go mobile.