Self-hosting
Running many nodes
Grow a host from one node to several, move repositories between them, back up and restore each node, retire or revoke a node, and know what happens when the control node is down.
One host can run on several nodes, each on its own machine with its own data directory. People, organizations and repositories stay one host: one address, one sign-in, one namespace for names. Most teams need one node. Add nodes when the repositories outgrow one machine.
How it fits together#
A host with many nodes has one control node and any number of repo nodes.
| The control node keeps | Each repo node keeps |
|---|---|
| accounts, sessions and tokens, invites, sign-up | its repositories, with their access settings and webhooks |
| organizations and their members | their jobs, scratch repos, logs and events |
| runners, agent profiles, preferences and notification read state | a copy of what it needs from the control node, kept current over the internal API |
| the host key, and every password account’s and agent’s key | |
| where each repository lives | |
repositories too, when its role is all |
Every node takes public traffic. A request for a repository on another node is forwarded there, so it doesn’t matter which node your load balancer picks. Lists that cover many repositories (your repositories, the inbox, notifications) ask every node and merge the answers.
Nodes talk to each other on a second listener, the internal listener, which must be on a private network. Each request between nodes is signed by the sending node’s key, and a node accepts only nodes the host has certified.
New repositories go to the node with the fewest, the control node included when its role is all, and skip a node that’s being drained. Organizations’ .moru repositories, and repositories created by linking a GitHub repository, are made on the control node. A site admin can move a repository to another node later, or move them all off a node by draining it.
Before you start#
- A private network between the machines, for the internal listener (port
7071in the examples). Never expose it publicly. - One public address in front of every node, such as a load balancer, set as
MORU_PUBLIC_URLon each node. - Nodes within one protocol level of each other. The simplest way is the same release everywhere. See Upgrading.
- A backup of your node. See Backups.
Make your node the control node#
Your existing node becomes the control node. It keeps its repositories, accounts and keys. Restart it with an internal listener and the URL the other nodes will reach it at, and keep the rest of its configuration as it is:
moru-node serve --internal-listen 10.0.0.2:7071 --internal-url http://10.0.0.2:7071 Or set MORU_INTERNAL_LISTEN and MORU_INTERNAL_URL. Its role stays all, so it keeps holding repositories and gets new ones. Settings → Network now lists it as key ID 0.
Once repo nodes have joined, you can have the control node hold only what must live there: drain it. Its repositories move to the repo nodes, organizations’ .moru repositories and those linked to GitHub stay, and it takes no new repositories from then on, without a restart.
A new host can start from an empty control node with --role control, which holds no repositories except organizations’ .moru ones and those linked to GitHub. For a control node that already holds repositories, drain it instead; it can run with --role control before or after the drain.
Add a node with a code#
On the control node, make a node code. It works once, within an hour:
moru-node node-codeOr use Add a node in Settings → Network, then Make a node code.
On the new machine, with an empty data directory and the code in a file (or in
MORU_JOIN_CODE):moru-node join --control http://10.0.0.2:7071 --internal-url http://10.0.0.3:7071 --code-file node-code.txt--controlis the control node’s internal URL,--internal-urlthis node’s.Serve it as a repo node, with the same internal URL:
moru-node serve --role repo --internal-listen 10.0.0.3:7071 --internal-url http://10.0.0.3:7071 \ --listen 0.0.0.0:7070 --public-url https://moru.example.com
A repo node prints no setup link: accounts live on the control node.
Add nodes that join by themselves#
Platforms that add machines with empty volumes leave nobody to type a code. An enrollment token lets a repo node join on its first start. Make one on the control node:
moru-node node-token It works for 30 days and for 5 nodes; --ttl and --max-joins change that. Add a node in Settings → Network, then Make an enrollment token, does the same. Then start new nodes with:
MORU_CONTROL_URL=http://10.0.0.2:7071
MORU_JOIN_TOKEN=mrj_…
moru-node serve --role repo --internal-listen [::]:7071 --internal-url http://<this node>:7071 --join-token-file reads the token from a file instead. A node whose role isn’t repo ignores both variables with a warning, so one environment can serve every machine of an app. A node that has joined doesn’t use the token again.
Each node needs its own internal URL. On Fly.io the container image sets it: when serve runs with MORU_INTERNAL_LISTEN and no internal URL, the image uses the machine’s stable private address, http://<machine ID>.vm.<app>.internal, on the internal listener’s port, so one environment fits every machine. An --internal-url flag still wins.
Anyone holding the token can add a node to your host, and a node can sign for everyone. Keep it secret, and revoke it when you’re done: moru-node node-token --list shows each token and the nodes that joined with it, and moru-node node-token --revoke <id> stops it.
Put the nodes behind one address#
Point the load balancer at every node’s public port. Set MORU_TRUSTED_PROXIES for the load balancer as on one node; requests forwarded between nodes carry the client’s address signed, so you don’t list the nodes.
On Fly.io, --replay fly (MORU_REPLAY=fly) answers small requests for another machine with Fly’s fly-replay header, so Fly’s proxy sends them straight to the right machine instead of the request passing through two. Larger requests, such as pushes, are forwarded as elsewhere.
Runners#
Register runners against the host’s public address, the one behind your load balancer (or Fly’s proxy), as with one node. Never use a node’s own address: a runner has one address for everything it does, and a runner pointed at one node stops when that node does.
Leases go to the control node, which hands out the oldest matching job from any node. A running job’s logs, events and pushes go to its repository’s node, wherever the load balancer sends them.
Upgrade runners along with the nodes: a runner from before 2026.11.7 can have a job’s first update refused for a moment after it registers or renews its token.
Settings → Network#
Settings → Network, for site admins, has three tabs: Nodes, Hosts and Activity. On Nodes a diagram draws the host’s DID at the top and its nodes in a row beneath, each joined to the host by a line for its certificate: solid while the certificate holds, amber and marked not trusted when the node is active but the certificate doesn’t hold now, dashed for a retired node, and red for a revoked one, with the time it’s revoked from. Each node shows its key ID, role, how many repositories it holds, when it last reported (about once a minute), and you’re on this node on the node that served the page. A node more than one protocol level from the control node is marked protocol too far apart. A move under way is an arrow under the row, from the node the repository leaves to the one it goes to, labelled with the repository. On a narrow screen, or with more nodes than fit in a row, the diagram turns into a list with the same facts.
Select a node to see its DID, internal URL, version, protocol level and when it joined, and its actions. Under it, while any exist, are the enrollment tokens, each with Revoke. Add a node, at the top of the page, makes a node code or an enrollment token (each shown once); on a node running on its own, it shows how to make it the control node instead.
The Hosts tab shows what each node knows about the hosts that sign in its repositories, revoked keys included: one row per host, one column per node. See How trust works.
A selected node’s menu has the actions this page describes, each with a command that does the same on the control node:
| In Settings → Network | On the command line | Offered for |
|---|---|---|
| Move all repositories off this node… | moru-node drain-node <DID> | an active node, when another node takes repositories |
| Take new repositories again | moru-node drain-node <DID> --undo | a drained node, once its drain has ended |
| Retire… | moru-node retire-node <DID> | an active node |
| Revoke… | moru-node revoke-node <DID> --at-ms <time> | an active or retired node |
| Restore its repositories from a backup… | moru-node restore-repos <backup>, on the node that takes them | a revoked or retired node that still has repositories |
The control node itself (key ID 0) has only the first two. Moving a single repository starts from the repository’s settings (or moru-node move-repo), and the Activity tab follows every drain and move. Revocations and restores you start here are followed there too, under Revocations and restores, until they finish; after a reload, the node list and moru-node moves show where things ended. The tab’s count is how many drains, moves and requests are still running.
When a node doesn’t answer in time, lists show a notice: “Some of this host’s nodes didn’t answer. Lists may be missing repos.”
Move a repository#
A site admin can move a repository to another node: to spread the load, or to empty a node before retiring it. In the web app, open the repository’s Settings → Node, which shows the node it’s on, pick a node and choose Move to another node. On the control node:
moru-node move-repo acme/timetable --to 2 --to takes the target’s DID or key ID, as moru-node nodes lists them. The command follows the move and prints each step; --no-wait hands it to the running node and returns. moru-node moves lists moves and where each stands, as the Activity tab of Settings → Network does.
While a repository moves:
- Reads keep working: clones, fetches, browsing and the API’s reads.
- Writes pause. Pushes, comments, reviews, merges and settings changes get a 503 with
Retry-After: 30saying the repository is moving. Try again once the move is done. - Jobs. Queued jobs are cancelled at once. Jobs already running may finish their work, including pushing and submitting their changes, for the drain time: 10 minutes, or
--drain(at most24h). Then they’re cancelled. - Webhooks wait. Deliveries not yet sent go with the repository and are sent from its new node.
Once nothing is running, the target node copies the repository with every ref and its settings (access, merge rules, webhooks and their secrets), the refs are compared, and the host switches every request over to the new node at once. Pages open on the repository reconnect and reload by themselves. The old node keeps its copy for 14 days in moved/, unserved, as a fallback.
What stays on the old node: the repository’s past jobs and their logs, its event history, and its webhook delivery log.
Some repositories don’t move:
- an organization’s
.morurepository, which lives on the control node; - a repository linked to GitHub, which lives on the control node too (unlink it first);
- a repository in the trash (restore it first, or let it be purged).
A move also needs a target that is active and holds repositories (role repo or all), and every active node on 2026.11.7 or later and within one protocol level of the control node (a node too far apart is refused: upgrade it first). A node being moved to can’t be retired until the move is over.
The target also needs free disk space of twice the size of the repository’s archive plus 64 MiB: it downloads the archive, reads the repository’s bundle out of it, deletes the download, then builds the repository from the bundle. It checks before the download starts; without the room, the move fails at once with “not enough free space on
When a move gets stuck#
A move that can’t go on fails and leaves the repository paused on its old node, where it keeps serving reads: when it hasn’t drained 5 minutes after its drain time, or its copy fails three times in a row (or once, for want of free space on the target), or the copied refs don’t match twice. The message in moves says which. Then either submit the same move again, which picks up where it stopped, or abort it.
To abort a move, use Abort beside it on the Activity tab of Settings → Network, or:
moru-node move-repo acme/timetable --abort Until the switch, an abort always works: the target drops its copy and the repository takes writes again on its old node. At the switch, an abort works only while the target hasn’t put its copy in place; if it has, or doesn’t answer, the abort is refused and the move goes on until it finishes. Fix the target (the message says what failed: disk space, permissions, a node that’s down), or revoke it if it’s gone for good. An abort waits for a switch call to the target that’s already under way, so a slow target can hold up its answer for up to 30 minutes. Past the switch, a move only goes forward: it retries until it’s done.
Drain a node#
To empty a node without knowing what’s on it, drain it: Moru moves every repository it can off the node, one after the other. Use it before retiring a node, to take one out for maintenance, or to leave the original node of a host holding only what has to live on the control node. In Settings → Network, select the node and choose Move all repositories off this node…, or on the control node:
moru-node drain-node 1 The node is named by its DID or key ID, as moru-node nodes lists them. The drain runs in the control node, like a move, and carries on after a restart where it was.
- One repository at a time. Each one is an ordinary move, with everything Move a repository describes: its writes pause while it moves, reads keep working, queued jobs are cancelled and running ones get 10 minutes to finish (
--drain, at most24h, or Time for running jobs to finish in the dialog). The next move starts when the last one has finished. - To the node with the fewest. Each repository goes to the active node that takes repositories (role
repoorall) and holds the fewest at that moment, counting moves on their way to it. A node added while the drain runs gets its share.--to 2,3(or Only to in the dialog) keeps it to the nodes you name. - What stays. An organization’s
.morurepository and a repository linked to GitHub live on the control node, so a drain of the control node leaves them where they are; other nodes never hold them. A repository in the trash stays too: restore it and drain again, or let it be purged. A repository left paused by an earlier move that failed stays until you submit that move again or abort it. A repository another move is already moving, or one being created on the node, is waited for. - Failures. A move that fails before its switch is rolled back, so the repository takes writes again on the node, and the drain goes on with the next one. A target that lacks the free space gets a second try on another node first. Five failures in a row stop the drain (“stopped after 5 moves in a row failed”), since something is wrong with more than one repository. Draining the node again later tries everything still on it.
From the moment the drain starts, the node takes no new repositories: new ones go to other nodes, and a move to it is refused. It stays that way after the drain finishes, so a node emptied to retire it, or to keep the control node lean, doesn’t fill up again. Settings → Network marks the node takes no new repos. To let it take repositories again, choose Take new repositories again in its menu, or:
moru-node drain-node 1 --undo That’s refused while a drain of the node is running: abort it first.
drain-node follows the drain and prints each step (“moved acme/app to node 2 (3 moved)”, “acme/old stays: in the trash: …”, “acme/web not moved: …”), then the repositories it moved, those that stay and those it didn’t move, each with why. --no-wait hands it to the running node and returns. To follow it later, use the Activity tab of Settings → Network, which lists each drain above the moves with its counts, the move in flight and, once it’s over, what stayed and what wasn’t moved; or moru-node moves, which lists drains first. A drain’s moves that went through aren’t listed one by one: the drain counts them.
To stop a drain, use Abort beside it on the Activity tab, or:
moru-node drain-node 1 --abort No more moves start, and the move in flight is rolled back if it hasn’t switched over yet; if it has, it finishes. The repositories already moved stay where they went, and the node still takes no new repositories.
A node can have one drain at a time; drains of different nodes can run side by side, one move each. A drain needs somewhere to go, so it’s refused when no other active node takes repositories, and like a move it’s refused while a node of the host is too far apart. A retired or revoked node can’t be drained: restore its repositories from a backup instead. A node being drained takes no restored repositories either: restore them on another node, or let it take new repositories again first.
Retire a node#
To take a node out of the host for good, move its repositories to other nodes by draining it, check what the drain left (repositories in the trash, and any it couldn’t move), then retire it with Retire… in Settings → Network, or on the control node:
moru-node retire-node did:key:z6Mk… The host stops trusting its key, and its tokens and requests are refused everywhere. Retiring is refused while the node still holds repositories (the page lists them, each linked to its Node settings, with Move them all off… to drain the node), has some in its trash or is creating one, and for a node that hasn’t reported since the control node’s index was rebuilt. Stop the node afterwards, and delete its volume so nothing picks up its old data.
Backups#
Back up every node: each one’s backup holds only what that node keeps. moru-node backup knows which kind of node it runs on.
- The control node’s backup holds the host key and every password account’s and agent’s key, the accounts, organizations and runners, the node list, node codes and enrollment tokens, the checkpoints its nodes reported, the host’s revocation records (
host/), its own repositories when its role isall, and its unfinished moves, revocations and restores. Encrypt it as you would any backup. - A repo node’s backup holds its repositories with their settings, webhooks and checkpoints. It leaves out the node’s key and host identity, and its copies of the control node’s state: a restored repo node joins as a new node (see Restore a repo node).
A copy on its way to a node (being moved there or restored there and not yet in place) isn’t in that node’s backup.
Restore the control node#
Restore the control node’s backup into an empty data directory as for a single node (see Restore) and serve it with the same --internal-listen and --internal-url. Where each repository lives, which jobs are queued where and the change feed are rebuilt from the repo nodes’ reports within seconds of them reaching it.
Drains, moves, revocations and restores that hadn’t finished when the backup was taken come back failed: the host carried on after the backup, so running them again from the old state isn’t safe. moru-node restore says how many. Check where each repository is with moru-node moves or the Activity tab of Settings → Network, and submit what’s still needed again. A repository a move left paused takes the same move again, and a node drained again moves what’s still on it.
Restore a repo node#
When a repo node’s machine is gone but you have its backup, restore it onto a new machine. It joins the host as a new node, with a new key, and takes over the old node’s repositories.
Make sure the old node is stopped. The control node refuses to replace a node that has reported in the last 30 seconds (three report intervals): ”… is still reporting: stop it before its backup takes over”.
Restore the backup into an empty data directory:
moru-node --data /var/lib/moru restore moru-repo-2.tgzIt says it was a repo node’s backup.
Start it as a repo node with a node code (
moru-node join … --code-file) or withMORU_CONTROL_URLand an enrollment token, as for a new node. It refuses to serve with any role butrepo.
The old node is then retired, and the new one serves its repositories. Repositories that were moved to another node since the backup are set aside in moved/ for 14 days, unserved. Repositories created on the old node after the backup aren’t in it, and stay unavailable. Like any restore, it loses the old node’s job history and the webhook deliveries it hadn’t sent yet.
A backup is restored this way once. While a move to or from the old node is under way, the join is refused until the move finishes or is aborted.
Right after the control node itself was restored or had its index rebuilt, the join can be refused for a few seconds: “the control node’s placement map is still filling from reports”. Start the node again shortly. If some other node will never report again, revoke it. When the old node’s repositories have no record at all on the control node and it never reported since, the new node takes them back and logs a warning: such a repository may have been deleted and purged after the backup, and comes back.
A revoked node’s backup can be restored the same way. The new node then writes the revocation into each repository before serving it, so they answer 404 for a moment after it starts.
Restore a revoked or retired node’s repositories on another node#
To bring a revoked (or retired) node’s repositories back on a node the host already has, use Restore its repositories from a backup… in that node’s menu in Settings → Network: pick the node to restore them on, choose the .tgz backup of the old node, and upload it. Or on the node that should take them:
moru-node restore-repos moru-repo-2.tgz
moru-node restore-repos moru-repo-2.tgz --repo acme/timetable --repo acme/site --repo restores only the repositories you name; without it, every one in the backup. The command refuses to start unless the data directory’s disk has free space of three times the backup’s size plus 64 MiB: it copies the backup there and keeps it until the restore finishes, so it can resume after a restart, while it unpacks the repositories’ bundles beside it and builds each repository from its bundle. It follows the restore and prints what happened to each repository; --no-wait returns at once, and moru-node restore-repos --abort <id> stops one (repositories it already put in place stay). A node runs one restore of a given old node’s repositories at a time: while one hasn’t finished, another from that node’s backup is refused (409 in the web app) with the first one’s ID. Wait for it, or abort it, then start the next. Restores of different nodes’ backups can run side by side.
For a revoked node, the host’s signed revocation goes into each repository before it’s served, so the old key’s ops count only as far as its revocation vouches for them (see Revoke a node). Repositories placed on another node since the backup are skipped, as are those in the old node’s trash, which backups don’t hold. The node that takes them must be active and hold repositories: a control node with role control can’t.
Uploads from the web app go to the node you picked, streamed through whichever node got the request. That node takes at most --max-restore-upload (MORU_MAX_RESTORE_UPLOAD, 20G by default), one upload at a time, and needs free disk space of three times the backup’s size plus 64 MiB, as restore-repos does. Whether another restore of the same old node’s repositories is unfinished there is known only once the backup has arrived, since the backup says which node it’s of: that 409 comes after the upload, so before uploading a large backup, make sure no earlier restore of the same node’s repositories is still running. A reverse proxy in front of the nodes needs to allow bodies that large: nginx’s client_max_body_size, for one, defaults to 1 MiB. For backups that big, restore-repos on the node itself avoids the upload.
Revoke a node#
Revoke a node whose key may be in someone else’s hands. Use Revoke… in Settings → Network, which asks for the time and for the node’s DID typed out, or on the control node:
moru-node revoke-node did:key:z6Mk… --at-ms 1762905600000 - The host stops trusting the node at once: its tokens and requests are refused everywhere.
- Moves to and from it are rolled back. A move whose target already had the repository in place goes forward instead.
- Every other node writes a revocation, signed by the host, into each of its repositories the node ever signed in, including those that moved off it. Ops the node signed stay valid as far as the newest checkpoint of it the control node had received by
--at-mslists them; the rest stop verifying. A repository in a node’s trash gets it when it’s restored. - Its own repositories are unavailable (503) until you restore them from a backup, as above.
The command follows the revocation and prints, for every node, what it wrote in each repository; --no-wait returns at once. A node that doesn’t answer is waited for, and the revocation’s status names it; it finishes once the node answers again (retire or revoke a node that’s gone for good). If a node answered with an error or a write failed, the revocation fails naming the node: fix it and run the same command again. A node more than one protocol level away (see Upgrading) isn’t asked, and the command names it: upgrade it and run the command again. Revoking an already revoked node with an earlier time widens the revocation.
The control node also keeps a record of each revocation in host/revocations.json, with a copy of the checkpoint it vouches from in host/checkpoints/ (never pruned, unlike the 30 days of reported checkpoints), and sends the records to the other nodes. Each node keeps the last records it heard and goes by them from its start, even while the control node is down: in any copy of a repository that lacks the revocation, such as an old import or a restored backup, it refuses the key from that time on, and asks the control node for the signed revocation to write into it: before serving an imported or restored copy, and as soon as it can in one it already serves. Settings → Network shows each node’s view; see How trust works.
If that record can’t be written (a full disk, a file the node can’t read), the revocation fails with ”… is revoked in the registry, but recording it in host/revocations.json failed”. The node is already untrusted everywhere by then, but nothing is written into the repositories yet: fix what the message names and run the same command again. The control node also tries to record it again each time it starts, and logs each one it can’t.
If you restore the control node from a backup taken before a revocation, run revoke-node for that node again, with the same time: the backup’s node list doesn’t have it, so the host trusts the node again until you do. The other nodes keep refusing the ops it signed after that time in their repositories meanwhile, as they never forget a revocation they heard of.
Pick the time conservatively: no later than when the key may have been taken, earlier if you aren’t sure. A time more than a minute in the future is refused.
- The control node dates each checkpoint a node reports by when it received it, not by the node’s own clock, keeps at most one per node every 5 minutes, and keeps them for 30 days. Between the real compromise and a late
--at-ms, whoever holds the key can land a forged checkpoint, and its ops would count as vouched for. - A revocation dated more than 30 days back may find no checkpoint, and then vouches for nothing: every op the node signed in other nodes’ repositories stops verifying.
- If a node is revoked while a repository is switching over to it, the switch is undone. Writes made on that node in the moment between the two are lost.
moru-node nodes lists the nodes with their DIDs. See moru-node.
When the control node is down#
These stop at once: signing in, accounts and invites, sign-up, organizations and anything the control node signs for them, creating and importing repositories, registering runners, handing out jobs to runners, drains, moves, revocations and restores, and changing preferences, agent profiles and other settings kept on the control node.
On the repo nodes, for --control-grace (MORU_CONTROL_GRACE, 5 minutes by default):
- people already signed in keep reading, pushing, cloning and making changes;
- a person’s first change on a node fails until the control node is back. The control node authorizes each node once to sign for each person, and the node keeps that for every repository it holds. An agent is authorized when its job or scratch repo is made, so it can submit while the control node is down.
After the grace, repo nodes refuse people’s tokens with a 503 until the control node is back. Running jobs go on: their runners keep sending logs and events for the jobs they already hold, and their agents keep pushing and submitting, as long as those requests reach the job’s node. Through the public address they do, except that a node that has never routed a job may need the control node to find it. Runners can’t take new jobs until the control node is back.
During the grace, Settings → Network on a repo node still opens. A banner says the control node isn’t answering, the diagram is drawn greyed out from the repo node’s own copy of the node list (without repository counts or reports), the Hosts grid fills in from the nodes that answer, and the Activity tab, Add a node and the tokens are hidden. The other pages on a repo node keep working too, without organizations: a notice under the header says the control node isn’t answering. After the grace the page fails to load like the rest of the app.
A repo node that restarts while the control node is down keeps what is left of the grace, counted from the last time it heard from the control node. It notes that time at most every 30 seconds, so after a restart it may have up to 30 seconds less than that, never more.
When the control node is back, everything resumes. Nobody is signed out.
Upgrading#
Every release has a protocol level, which goes up only when the way nodes talk to each other, or the format of what they write into repositories, changes. 2026.11.7 is level 1. Nodes within one level of each other work together, so you can upgrade one node at a time.
The control node refuses a node more than one level away from its own: such a node can’t join, its reports are refused, and once its grace runs out it refuses people’s tokens. A joining node likewise refuses a control node too far away. Settings → Network shows each node’s level, and Too far apart for one that’s refused.
- Upgrade the control node first. Keep its
--internal-listenand--internal-url. - Then the repo nodes, a few at a time.
When a release brings a new format for what nodes write into repositories, a node that already reads it refuses such writes with a 503 (“upgrade every node first”) until every node on the host reads it. Upgrading the rest clears it. See Upgrading for the steps on each platform.
Whenever you restart or upgrade the control node, keep its --internal-listen and --internal-url (or MORU_INTERNAL_LISTEN and MORU_INTERNAL_URL). Once other nodes have joined, it refuses to start without them, since it would otherwise refuse the repo nodes’ tokens and lose track of their repositories. A host can’t go back to a single standalone node.
Limits of this release#
- Repositories move one at a time, when a site admin moves them or drains a node. Nothing rebalances by itself, and a drain spreads repositories by count, not by size.
- Backups and restores are per node; there’s no backup of the whole host.
- The control node is a single machine: there’s no second one to take over.