Daemon design
Explanation · for anyone running `mesh-hub` for other processes, and for anyone who wants to know what one misbehaving client can and cannot do to the rest
The claim. One misbehaving client of the daemon, whether silent, slow, greedy or duplicated, costs the other clients nothing: every resource a connection can consume is bounded by a policy the operator sets, the number of connections is bounded by the same policy, and every failure is reported to the client that caused it.
The mechanism
The daemon is a library with a binary in front of it: the crate’s run takes a DaemonConfig (the hub name and configuration, the socket and the address to serve, the connection policy, a basket) and serves until a signal or a displacement, and the binary only parses its flags into that configuration; a host that embeds the daemon calls the same surface without the signals. A connection is one task that owns its transport in both directions and a bounded queue of outbound messages. Four timers and four sizes govern it. A WebSocket client is pinged every thirty seconds and a Unix-socket client is expected to say something itself, so a connection silent for ninety seconds is closed rather than held open forever with a name nobody can reuse. A write that does not complete in ten seconds means the peer has stopped reading, and the connection is closed rather than letting its queue grow without bound. A WebSocket upgrade that does not finish in five seconds is dropped, so a connection that never speaks cannot park a task. A frame larger than a megabyte is refused with an error the client can act on, and the daemon reads a frame incrementally rather than allocating the announced length up front, so a four-byte header cannot buy sixteen megabytes. A connection holds at most sixty-four subscriptions, because every send is matched against every live subscription and the cap bounds what one connection adds to the cost of each; the subscribe past it is refused with limit_exceeded, the connection stays open, and an unsubscribe frees a place. The daemon holds at most 256 connections across both transports, counted from accept to close: past that, a Unix-socket client is turned away with one error frame and a WebSocket upgrade is answered 503, each refusal counted, logged at the first and at each doubling, and printed at exit.
Events are the one thing a client cannot be trusted to consume at the daemon’s pace, so they go through a bounded queue, 1,024 deep by default, that the publisher writes into without waiting. When the queue is full the newest event is dropped and counted; the count is logged at the first drop and at each doubling so a log does not fill either, and it is handed back to the subscriber in the acknowledgement of its unsubscribe, so the loss is the subscriber’s to see. Acknowledgements and errors bypass the queue and are never dropped. The alternative, disconnecting a slow subscriber, would turn a brief stall into a reconnect and a resubscribe that loses the same events anyway; it remains available as a policy if a deployment wants it.
A publish the envelope store refuses, because the envelope is larger than a store entry or every slot the store tried was held, is still delivered to every matching subscriber and still acknowledged. The acknowledgement says persisted: false, which the Rust client returns from send_with_receipt, the daemon counts the refusal, logs it at the first and at each doubling, prints the count when it exits, and writes no ring signal for an envelope that is not in the store, so a process reading the ring is never pointed at an envelope that is not there. Before this rule a refusal was neither counted nor reported, and the client was acknowledged as if the envelope had been stored. A store that shared memory has no room for is refused when the daemon starts, with the reason in its log, and the daemon then runs without persistence rather than failing later with a SIGBUS at the first page it touches.
An envelope published through the daemon also reaches a process that is no client of the daemon at all, one that embeds a hub of the same name. The daemon’s hub stores the envelope and writes a signal Pulse into the ring that names it and carries the hub’s own instance identity; the other hub’s poll fetches the envelope from the store by that name and delivers it to its envelope listeners. A signal the hub wrote itself is skipped, because its listeners had the envelope when it was broadcast. A signal whose envelope the store no longer holds, overwritten since it was stored, never stored, or failing verification, delivers no envelope, and is counted (unfetched_envelopes) and logged at the first and at each doubling; a Pulse the ring overwrote before the poll reached it is counted the same way (lost_pulses) and never delivered under another index or twice. The ring has one writer, the daemon’s hub, which created it: the embedded hub reads the ring and is refused with ReadOnly if it tries to publish, broadcast or route into it, with nothing stored, signalled or dispatched, and its process publishes by sending to the daemon over the port like any other client, so two writers never race for the ring’s cursor. The signals the daemon writes carry the operating system’s monotonic clock in nanoseconds, since the daemon does not drive the hub’s clock itself. Before these rules the signal left envelope_ref and the clock at zero, so a second process saw every signal and never an envelope, and a second hub on the name could have written into the ring.
Names are exclusive while a connection holds them and released the moment it ends, whether by detach, by a closed socket, or by one of the timers, together with every subscription the connection had, so nothing leaks into the hub’s registries between a client’s death and its return. A second daemon started on the same Unix socket path probes the socket before touching it: if a daemon answers, the second refuses to start and says so; only a socket file nobody answers is removed. Before this rule a second start silently took the first daemon’s socket and left it deaf.
What it costs
A timer wheel per connection, one bounded channel per connection, and a policy the operator must think about once: the defaults suit a workstation, and a busy fan-out may want a deeper queue. A Unix-socket client written against the raw protocol must send a frame within the idle window; the Rust client does this on its own.
What would refute this
A silent connection that survives its idle window, a stalled subscriber that slows a publisher, an oversized header that allocates, a name held after its connection died, a store refusal acknowledged as a success, a sixty-fifth subscription that is accepted, a connection past the cap that is admitted, a second daemon that takes over a live socket, an embedded hub that reads the daemon’s signal and never the envelope it names, or a second hub on the daemon’s name that writes into its ring would each refute the claim. The daemon’s liveness, fan-out and embed tests are the checks, and each runs in the workspace test suite.
Spoken walkthrough
A script for the video and audio version of this page, about ninety seconds.
The question a daemon has to answer is what one bad client can do to the rest. Here the answer is nothing, because every resource a connection can consume is bounded by a policy the operator sets once.
A connection is one task with a bounded queue of outbound messages. Four timers and four sizes govern it: how long it may be silent, how long a frame may take, how large a header may be, how many subscriptions it may hold. Cross a bound and the connection is closed, and told why.
Events are the one thing a client cannot be trusted to consume at the daemon’s pace, so they go through a bounded queue the publisher never waits on. Fill it and the newest event is dropped, counted, and reported back to the subscriber when it unsubscribes. Acknowledgements and errors never queue and are never dropped.
A publish the store refuses is still delivered and still acknowledged, with persisted false, and no ring signal is written for an envelope that is not there. Names are released the instant a connection ends. A second daemon on the same socket asks before it takes it.
None of this is clever. It is a list of bounds, each with a test that tries to cross it.