HTTP over the Relay Broker

The Relay Broker carries any request and its response. @thuum/relay-http uses it to carry HTTP: an app calls a URL on its own machine, and the request reaches an HTTP server on another network, streamed both ways. Both ends only dial out to Thuum over WSS on port 443: no open port, no fixed IP, no VPN, and Thuum stores nothing.

This guide connects AI agents on edge devices, which speak the OpenAI API, to models running on your own machines with Ollama and LM Studio.

 Edge device                      Thuum                        Data center
┌───────────────────────┐       ┌──────────────┐       ┌──────────────────────────────┐
│ agent (OpenAI SDK)    │       │              │       │ relay-http server (active)   │──▶ Ollama
│   │ HTTP, localhost   │  WSS  │ Relay Broker │  WSS  │                              │──▶ LM Studio (next)
│   ▼                   │──────▶│              │◀──────│ relay-http server (standby)  │
│ relay-http client     │       │              │       │                              │
└───────────────────────┘       └──────────────┘       └──────────────────────────────┘

The two modes#

One package, two modes. Both exchange their App Key for a token, connect to the Relay Broker, and reconnect with backoff.

Client mode runs on each device, next to the apps that make the calls:

• It listens on 127.0.0.1, so only the device's own apps reach it.
• The first path segment is the topic: POST /green-lantern/v1/chat/completions becomes a request on topic green-lantern with target /v1/chat/completions.
• It streams the app's request into a head and body parts, and the response back as it arrives: Server-Sent Events work unchanged.

Server mode runs in your data center, next to the machines that answer:

• It serves each topic of its file, with a server list per topic, tried in order: on a refused connection, a connect timeout, or a 502, 503 or 504, it tries the next machine, as long as the response hasn't started.
• To retry, it keeps the request body in memory until the response starts: on your equipment, never in Thuum.
• A second instance with the same file is a standby: it takes over when the first one's connection drops.

The names of your topics, your machines and their order exist only in the server's file. Thuum learns the topics while the server is connected, and nothing else.

1. Keys and claims#

Create two App Keys, and give each a Relay Broker claim:

Neither needs a Domain Claim. A key copied from a device can send requests to its own topics and nothing else, and is revoked alone; only the servers' key can serve, so nobody else can take a topic over. With the client (or in the console's Relay Broker section):

import { PUBLISH, SUBSCRIBE } from '@thuum/client';

const agents = await acme.relayBroker.claims.create({
	name: 'Edge agents',
	topics: { 'green-lantern': PUBLISH }
});
await acme.relayBroker.claims.keys.link(agents.id, deviceSelector);

const models = await acme.relayBroker.claims.create({
	name: 'Model servers',
	topics: { 'green-lantern': SUBSCRIBE, 'iron-man': SUBSCRIBE }
});
await acme.relayBroker.claims.keys.link(models.id, serverSelector);

2. Run the server#

In the data center, on any machine that reaches the model servers, write relay-http-server.yaml:

# relay-http-server.yaml
url: { { THUUM_API_URL } }
domain: acme
key: ${THUUM_KEY} # the servers' App Key
topics:
  green-lantern:
    - http://192.168.1.21:11434 # Ollama
    - http://192.168.1.35:1234 # LM Studio, if Ollama fails
  iron-man:
    - http://192.168.1.35:1234
    - http://192.168.1.21:11434

thuum-relay-http server --config relay-http-server.yaml

Start the same command with the same file on a second machine: it serves as a standby.

3. Run the client#

On each edge device, write relay-http-client.yaml:

# relay-http-client.yaml
url: { { THUUM_API_URL } }
domain: acme
key: ${THUUM_KEY} # this device's App Key
listen: 127.0.0.1:8787

thuum-relay-http client --config relay-http-client.yaml

4. Point the agent at it#

The agent's whole change is its base URL, with the topic as the first segment:

OPENAI_BASE_URL=http://127.0.0.1:8787/green-lantern/v1
OPENAI_API_KEY=unused # any value: the Thuum App Key lives in the client app

The code stays as it was:

import OpenAI from 'openai';

const openai = new OpenAI(); // reads OPENAI_BASE_URL and OPENAI_API_KEY
const stream = await openai.chat.completions.create({
	model: 'llama3.1',
	messages: [{ role: 'user', content: 'Summarize the last shift.' }],
	stream: true
});
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? '');

The request reaches the client app on 127.0.0.1:8787, crosses Thuum on topic green-lantern, and the server app sends it to http://192.168.1.21:11434/v1/chat/completions, Ollama's OpenAI-compatible API. The events stream back as Ollama writes them. Check the path with curl http://127.0.0.1:8787/green-lantern/v1/models.

The HTTP convention#

What the two apps put in each part's data. Thuum never reads it; it is written down so you can write your own client or server, in any language, with the Relay Broker WebSocket:

• Headers are a list of [name, value] pairs, so repeated headers keep their order.
• The client app removes Authorization, Host, Cookie, and the hop-by-hop headers (Connection, Keep-Alive, Transfer-Encoding, TE, Trailer, Upgrade, Proxy-*).
• The server app adds nothing: the request reaches each machine as the app sent it.
• If the app disconnects, the client app sends cancel, and the server app aborts the machine's request.

Errors#

The apps answer errors in OpenAI's error shape ({ "error": { "message": … } }), so agents handle them as OpenAI's own: the SDKs retry 5xx, then raise.

Failover#

• Between machines. The server app tries a topic's list in order, until the response starts. Once a machine has started answering, a failure ends that response.
• Between server apps. When the active server app's connection drops, Thuum makes the standby active at once. Requests in flight on the old one fail (504 or 502); the next ones go to the new one.

End-to-end encryption#

Add the same secret to the client's and the server's files, and every data, heads and body parts alike, is encrypted with AES-256-GCM before it leaves the device or the data center. Thuum then sees topics, request ids, sizes and timing, and nothing else. It never needs more: the Relay Broker stores nothing either way.

Why it holds up#

• Long streams. Only WebSockets cross the internet, kept alive by pings; HTTP stays on the local network at both ends, so there are no proxy timeouts to tune for a response that takes minutes.
• No rate limits from Thuum. Requests and responses pass unchanged, streamed, with no queue and no 429 added on the way.
• Small blast radius. A device's key only reaches its own topics; a lost device is one key to delete.