Cathedral
A cathedral is a sanctum instance running in mode cathedral.
Peers register with it, and it then handles:
- Discovery: peers find each other through the cathedral, without knowing each other's IP addresses.
- Relay: traffic between peers goes through the cathedral when a direct path is not possible.
- Key distribution: the cathedral hands out wrapped shared secrets (ambries) to the peers in a flock.
- NAT hole punching: peers get a direct peer-to-peer tunnel when their NAT types allow it.
A cathedral never sees your traffic in plaintext. The ambries it hands out are wrapped with KEKs that only your devices hold, so a cathedral cannot read or inject session keys.
This guide sets up one cathedral with a single flock of two peers, then
connects those peers with hymn.
Want to run a larger cathedral network without managing every flock and device by hand? The reliquary framework sets up cathedral networks with an API that handles flock and device registration automatically.
Before you start
Build and install sanctum, hymn and ambry on the
cathedral host and on every peer, following the
building guide. You should be familiar with
a basic tunnel from the setup guide.
Pick your identifiers up front. This guide uses these values everywhere:
| What | Format | Example |
|---|---|---|
| Cathedral address | ip:port | 1.2.3.4:4469 |
| Flock id | 64-bit hex, must end in 00 | cafebabecafeba00 |
| Peer id (cathedral id) | 32-bit hex | deadbeef, deadbabe |
| KEK id (spi) | 8-bit hex, one per device in the flock | 01, 02 |
In the examples below, peer deadbeef uses KEK 01
and peer deadbabe uses KEK 02.
Keys at a glance
A cathedral setup uses several keys. Every one of them is created in the steps below.
| Key | Created with | Lives on | Purpose |
|---|---|---|---|
| Sync secret | dd | Cathedral(s) | Protects traffic between federated cathedrals. Required even with a single cathedral. |
| Cathedral COSK | ambry cosk-pair | Cathedral | Ed25519 key the cathedral signs its outgoing offers with. |
| KEK | ambry generate | One per peer | Unwraps the shared secrets the cathedral hands out. |
| Ambry bundle | ambry bundle | Cathedral | Wrapped shared secrets for the flock, valid for a fixed number of days. |
| Cathedral secret (CS) | dd | Cathedral and peer | 256-bit key protecting offers between a peer and the cathedral. |
| Peer COSK | ambry cosk-pair | Private: peer. Public: cathedral | Ed25519 key the peer signs its offers to the cathedral with. |
See the cryptography documentation for details.
Run ambry on a separate key-management machine that you
trust, not on the cathedral. Copy only the files listed in each step to
the cathedral and the peers, over a secure channel.
1. Create the system accounts
The cathedral runs its processes privilege separated. Create three unprivileged system accounts on the cathedral host. The exact command depends on your OS. On Linux:
# useradd -r -M -s /usr/sbin/nologin _cathedral # useradd -r -M -s /usr/sbin/nologin _purgatory-rx # useradd -r -M -s /usr/sbin/nologin _purgatory-tx
2. Create the cathedral keys
On the cathedral host, create the directory layout and the two keys that belong to the cathedral itself:
# mkdir -p /etc/cathedral/identities /etc/cathedral/ambries # dd if=/dev/urandom of=/etc/cathedral/sync.secret bs=32 count=1 # ambry cosk-pair /etc/cathedral/cosk.priv /etc/cathedral/cosk.pub
If you will federate several cathedrals later, every cathedral needs the
same sync.secret. Generate it once and copy
it to the others.
3. Generate the flock KEKs and ambry bundle
On your key-management machine, generate the KEKs for the flock:
$ ambry generate cafebabecafeba00
This creates a cafebabecafeba00/ directory. The device KEKs
are under cafebabecafeba00/kek-data/ as
kek-0x01, kek-0x02 and so on. Do not hand out
anything from internal/.
Next, create an ambry bundle for the flock, valid for 120 days:
$ ambry bundle cafebabecafeba00 cafebabecafeba00 120 ambry-cafebabecafeba00
Copy ambry-cafebabecafeba00 to the cathedral as
/etc/cathedral/ambries/ambry-cafebabecafeba00.
4. Create the peer credentials
Each peer needs a cathedral secret and a COSK key pair. For peer
deadbeef:
$ dd if=/dev/urandom of=cs-deadbeef bs=32 count=1 $ ambry cosk-pair cosk-deadbeef.priv cosk-deadbeef.pub
Repeat this for deadbabe. The cathedral expects these files
in a directory named flock-<flock id> inside its
secretdir, named after the peer id:
| Local file | Copy to the cathedral as |
|---|---|
cs-deadbeef | /etc/cathedral/identities/flock-cafebabecafeba00/deadbeef.key |
cosk-deadbeef.pub | /etc/cathedral/identities/flock-cafebabecafeba00/deadbeef.pub |
cs-deadbabe | /etc/cathedral/identities/flock-cafebabecafeba00/deadbabe.key |
cosk-deadbabe.pub | /etc/cathedral/identities/flock-cafebabecafeba00/deadbabe.pub |
Each peer then gets its own three files:
| Peer | Files |
|---|---|
deadbeef | kek-data/kek-0x01, cs-deadbeef, cosk-deadbeef.priv |
deadbabe | kek-data/kek-0x02, cs-deadbabe, cosk-deadbabe.priv |
Never copy a private COSK or a KEK to the cathedral. The cathedral only needs the cathedral secrets and the public COSKs.
5. Define the flock
Flocks and the peers allowed in them are defined in a settings file.
Create /etc/cathedral/settings.conf:
flock cafebabecafeba00 {
allow deadbeef spi 0x01 0
allow deadbabe spi 0x02 0
ambry /etc/cathedral/ambries/ambry-cafebabecafeba00
}
Each allow line has the form:
allow <peer id> spi <kek id> <bandwidth>
<peer id>is the 32-bit peer id. The cathedral looks for its.keyand.pubfiles underflock-<flock id>/.<kek id>is the KEK this peer was given, in hex.<bandwidth>is the limit for this peer in Mbit/s.0means unlimited.
The cathedral checks this file every 10 seconds and reloads it when it changes, so you can add or remove peers without a restart.
6. Configure and start the cathedral
Create /etc/cathedral/cathedral.conf:
mode cathedral instance cathedral local 1.2.3.4:4469 pidfile /run/cathedral.pid cathedral_mtu 1422 shroud yes secret /etc/cathedral/sync.secret cathedral_cosk /etc/cathedral/cosk.priv secretdir /etc/cathedral/identities settings /etc/cathedral/settings.conf run control as root run cathedral as _cathedral run purgatory-rx as _purgatory-rx run purgatory-tx as _purgatory-tx
Make sure UDP port 4469 is reachable from the internet, then
start it:
# sanctum -c /etc/cathedral/cathedral.conf
Add -d to run it as a daemon. The log should show the flock
being loaded (settings reloaded).
If the cathedral sits behind NAT with a port forward, also set
cathedral_ip to its public ip:port.
7. Configure the peers
On each peer, place its three files somewhere only root can read, for
example /etc/sanctum/, and add the tunnel with
hymn.
Tunnel names follow the form <flock>-<our kek id>-<their kek id>.
On deadbeef (KEK 01), a tunnel to KEK
02:
# hymn add cafebabecafeba00-01-02 tunnel 172.31.253.1/24 mtu 1422 \
cathedral 1.2.3.4:4469 shroud yes \
kek /etc/sanctum/kek-0x01 \
identity deadbeef:/etc/sanctum/cs-deadbeef \
cosk /etc/sanctum/cosk-deadbeef.priv
# hymn up cafebabecafeba00-01-02
On deadbabe (KEK 02), the mirror image:
# hymn add cafebabecafeba00-02-01 tunnel 172.31.253.2/24 mtu 1422 \
cathedral 1.2.3.4:4469 shroud yes \
kek /etc/sanctum/kek-0x02 \
identity deadbabe:/etc/sanctum/cs-deadbabe \
cosk /etc/sanctum/cosk-deadbabe.priv
# hymn up cafebabecafeba00-02-01
Check that the tunnel came up:
# hymn status cafebabecafeba00-01-02
Optionally, add user <name> to hymn add to
run the unprivileged processes as that user.
To connect many peers automatically instead of one tunnel at a time, use the same flock with liturgy mode. For phones, see Vigil.
Maintenance
Adding a peer
- Pick a new peer id and give the peer an unused KEK from
kek-data/. - Create its cathedral secret and COSK pair (step 4).
- Copy
<peer id>.keyand<peer id>.pubinto the flock directory on the cathedral. - Add an
allowline tosettings.conf. The cathedral picks it up within 10 seconds.
Renewing the ambry bundle
An ambry bundle expires after the number of days you gave
ambry bundle. Before it expires, generate a new one and
replace the file on the cathedral:
$ ambry bundle cafebabecafeba00 cafebabecafeba00 120 ambry-cafebabecafeba00
The cathedral only re-reads ambry files when it reloads
settings.conf. Run touch /etc/cathedral/settings.conf
after replacing the bundle.
NAT discovery and P2P
The cathedral can work out whether two peers can reach each other directly. If both peers are behind NAT types that allow hole punching, the cathedral sends both of them the information they need to set up a direct peer-to-peer tunnel. Otherwise traffic keeps going through the cathedral.
Add the following to cathedral.conf and open UDP port
4470:
cathedral_nat_port 4470 cathedral_p2p_sync yes
cathedral_nat_portis the extra port the cathedral listens on to detect a peer's NAT type.cathedral_p2p_syncshares P2P state with federated cathedrals, so P2P also works when the two peers use different cathedrals.
On each peer, set the same NAT port on the tunnel:
# hymn nat cafebabecafeba00-01-02 4470
You can also pass natport 4470 to hymn add.
Setting the port to 0 turns NAT detection, and with it
P2P, off.
Federation
Cathedrals can federate, so peers connected to different cathedrals can still reach each other. This gives you redundancy and lets you spread cathedrals across locations.
- Copy the same
sync.secretto every cathedral and pointsecretat it. - Give every cathedral the same flocks, peer files and ambry bundles (steps 3 to 5).
- In each cathedral's
settings.conf, list the other cathedrals, outside of any flock block:
federate 5.6.7.8 4469 federate 9.10.11.12 4469
You can federate with up to 31 other cathedrals. A
cathedral skips its own address, so you can use one shared
settings.conf on every cathedral.
The sync secret protects all traffic between your cathedrals. Treat it like any other long-lived shared secret.
Remembrance
With remembrance enabled, a peer's cathedral regularly tells it about the other cathedrals in the federation. The peer stores that list and fails over to another cathedral if its current one stops responding.
Turn it on for a tunnel with hymn:
# hymn remembrance cafebabecafeba00-01-02 on
In a hand-written configuration, set cathedral_remembrance
to a file where sanctum stores the list:
cathedral_remembrance /etc/sanctum/remembrance-cafebabecafeba00-01-02
Option reference
cathedral.conf
| Option | Description |
|---|---|
mode cathedral | Run this instance as a cathedral. |
local | The ip:port the cathedral listens on. |
secret | Path to the sync secret. Required. |
cathedral_cosk | Path to the cathedral's private COSK. Required. |
secretdir | Directory holding the flock-<id>/ peer directories. Required. |
settings | Path to the flock and federation settings file. |
cathedral_mtu | Expected MTU for traffic through the cathedral. |
cathedral_ip | Public ip:port when the cathedral sits behind NAT. |
cathedral_nat_port | Port used for NAT detection. |
cathedral_p2p_sync | Share P2P state with federated cathedrals. |
shroud | Turn shroud on or off. |
settings.conf
| Option | Description |
|---|---|
flock <id> { … } | Defines a flock. The id must end in 00. |
allow <peer> spi <kek> <mbit> | Allows a peer into the flock (inside a flock block). |
ambry <path> | The ambry bundle for the flock (inside a flock block). |
federate <ip> <port> | Federate with another cathedral. |
See sanctum.conf(5), ambry(1) and
hymn(1) for the full details.