SANCTUM

Cathedral

A cathedral is a sanctum instance running in mode cathedral. Peers register with it, and it then handles:

  • Discovery: peers find each other through the cathedral, without knowing each other's IP addresses.
  • Relay: traffic between peers goes through the cathedral when a direct path is not possible.
  • Key distribution: the cathedral hands out wrapped shared secrets (ambries) to the peers in a flock.
  • NAT hole punching: peers get a direct peer-to-peer tunnel when their NAT types allow it.

A cathedral never sees your traffic in plaintext. The ambries it hands out are wrapped with KEKs that only your devices hold, so a cathedral cannot read or inject session keys.

This guide sets up one cathedral with a single flock of two peers, then connects those peers with hymn.

Want to run a larger cathedral network without managing every flock and device by hand? The reliquary framework sets up cathedral networks with an API that handles flock and device registration automatically.

Before you start

Build and install sanctum, hymn and ambry on the cathedral host and on every peer, following the building guide. You should be familiar with a basic tunnel from the setup guide.

Pick your identifiers up front. This guide uses these values everywhere:

WhatFormatExample
Cathedral addressip:port1.2.3.4:4469
Flock id64-bit hex, must end in 00cafebabecafeba00
Peer id (cathedral id)32-bit hexdeadbeef, deadbabe
KEK id (spi)8-bit hex, one per device in the flock01, 02

In the examples below, peer deadbeef uses KEK 01 and peer deadbabe uses KEK 02.

Keys at a glance

A cathedral setup uses several keys. Every one of them is created in the steps below.

KeyCreated withLives onPurpose
Sync secretddCathedral(s)Protects traffic between federated cathedrals. Required even with a single cathedral.
Cathedral COSKambry cosk-pairCathedralEd25519 key the cathedral signs its outgoing offers with.
KEKambry generateOne per peerUnwraps the shared secrets the cathedral hands out.
Ambry bundleambry bundleCathedralWrapped shared secrets for the flock, valid for a fixed number of days.
Cathedral secret (CS)ddCathedral and peer256-bit key protecting offers between a peer and the cathedral.
Peer COSKambry cosk-pairPrivate: peer.
Public: cathedral
Ed25519 key the peer signs its offers to the cathedral with.

See the cryptography documentation for details.

Run ambry on a separate key-management machine that you trust, not on the cathedral. Copy only the files listed in each step to the cathedral and the peers, over a secure channel.

1. Create the system accounts

The cathedral runs its processes privilege separated. Create three unprivileged system accounts on the cathedral host. The exact command depends on your OS. On Linux:

# useradd -r -M -s /usr/sbin/nologin _cathedral
# useradd -r -M -s /usr/sbin/nologin _purgatory-rx
# useradd -r -M -s /usr/sbin/nologin _purgatory-tx

2. Create the cathedral keys

On the cathedral host, create the directory layout and the two keys that belong to the cathedral itself:

# mkdir -p /etc/cathedral/identities /etc/cathedral/ambries
# dd if=/dev/urandom of=/etc/cathedral/sync.secret bs=32 count=1
# ambry cosk-pair /etc/cathedral/cosk.priv /etc/cathedral/cosk.pub

If you will federate several cathedrals later, every cathedral needs the same sync.secret. Generate it once and copy it to the others.

3. Generate the flock KEKs and ambry bundle

On your key-management machine, generate the KEKs for the flock:

$ ambry generate cafebabecafeba00

This creates a cafebabecafeba00/ directory. The device KEKs are under cafebabecafeba00/kek-data/ as kek-0x01, kek-0x02 and so on. Do not hand out anything from internal/.

Next, create an ambry bundle for the flock, valid for 120 days:

$ ambry bundle cafebabecafeba00 cafebabecafeba00 120 ambry-cafebabecafeba00

Copy ambry-cafebabecafeba00 to the cathedral as /etc/cathedral/ambries/ambry-cafebabecafeba00.

4. Create the peer credentials

Each peer needs a cathedral secret and a COSK key pair. For peer deadbeef:

$ dd if=/dev/urandom of=cs-deadbeef bs=32 count=1
$ ambry cosk-pair cosk-deadbeef.priv cosk-deadbeef.pub

Repeat this for deadbabe. The cathedral expects these files in a directory named flock-<flock id> inside its secretdir, named after the peer id:

Local fileCopy to the cathedral as
cs-deadbeef/etc/cathedral/identities/flock-cafebabecafeba00/deadbeef.key
cosk-deadbeef.pub/etc/cathedral/identities/flock-cafebabecafeba00/deadbeef.pub
cs-deadbabe/etc/cathedral/identities/flock-cafebabecafeba00/deadbabe.key
cosk-deadbabe.pub/etc/cathedral/identities/flock-cafebabecafeba00/deadbabe.pub

Each peer then gets its own three files:

PeerFiles
deadbeefkek-data/kek-0x01, cs-deadbeef, cosk-deadbeef.priv
deadbabekek-data/kek-0x02, cs-deadbabe, cosk-deadbabe.priv

Never copy a private COSK or a KEK to the cathedral. The cathedral only needs the cathedral secrets and the public COSKs.

5. Define the flock

Flocks and the peers allowed in them are defined in a settings file. Create /etc/cathedral/settings.conf:

flock cafebabecafeba00 {
    allow deadbeef spi 0x01 0
    allow deadbabe spi 0x02 0

    ambry /etc/cathedral/ambries/ambry-cafebabecafeba00
}

Each allow line has the form:

allow <peer id> spi <kek id> <bandwidth>
  • <peer id> is the 32-bit peer id. The cathedral looks for its .key and .pub files under flock-<flock id>/.
  • <kek id> is the KEK this peer was given, in hex.
  • <bandwidth> is the limit for this peer in Mbit/s. 0 means unlimited.

The cathedral checks this file every 10 seconds and reloads it when it changes, so you can add or remove peers without a restart.

6. Configure and start the cathedral

Create /etc/cathedral/cathedral.conf:

mode cathedral
instance cathedral

local 1.2.3.4:4469
pidfile /run/cathedral.pid

cathedral_mtu 1422
shroud yes

secret /etc/cathedral/sync.secret
cathedral_cosk /etc/cathedral/cosk.priv
secretdir /etc/cathedral/identities
settings /etc/cathedral/settings.conf

run control as root
run cathedral as _cathedral
run purgatory-rx as _purgatory-rx
run purgatory-tx as _purgatory-tx

Make sure UDP port 4469 is reachable from the internet, then start it:

# sanctum -c /etc/cathedral/cathedral.conf

Add -d to run it as a daemon. The log should show the flock being loaded (settings reloaded).

If the cathedral sits behind NAT with a port forward, also set cathedral_ip to its public ip:port.

7. Configure the peers

On each peer, place its three files somewhere only root can read, for example /etc/sanctum/, and add the tunnel with hymn.

Tunnel names follow the form <flock>-<our kek id>-<their kek id>. On deadbeef (KEK 01), a tunnel to KEK 02:

# hymn add cafebabecafeba00-01-02 tunnel 172.31.253.1/24 mtu 1422 \
    cathedral 1.2.3.4:4469 shroud yes \
    kek /etc/sanctum/kek-0x01 \
    identity deadbeef:/etc/sanctum/cs-deadbeef \
    cosk /etc/sanctum/cosk-deadbeef.priv
# hymn up cafebabecafeba00-01-02

On deadbabe (KEK 02), the mirror image:

# hymn add cafebabecafeba00-02-01 tunnel 172.31.253.2/24 mtu 1422 \
    cathedral 1.2.3.4:4469 shroud yes \
    kek /etc/sanctum/kek-0x02 \
    identity deadbabe:/etc/sanctum/cs-deadbabe \
    cosk /etc/sanctum/cosk-deadbabe.priv
# hymn up cafebabecafeba00-02-01

Check that the tunnel came up:

# hymn status cafebabecafeba00-01-02

Optionally, add user <name> to hymn add to run the unprivileged processes as that user.

To connect many peers automatically instead of one tunnel at a time, use the same flock with liturgy mode. For phones, see Vigil.

Maintenance

Adding a peer

  1. Pick a new peer id and give the peer an unused KEK from kek-data/.
  2. Create its cathedral secret and COSK pair (step 4).
  3. Copy <peer id>.key and <peer id>.pub into the flock directory on the cathedral.
  4. Add an allow line to settings.conf. The cathedral picks it up within 10 seconds.

Renewing the ambry bundle

An ambry bundle expires after the number of days you gave ambry bundle. Before it expires, generate a new one and replace the file on the cathedral:

$ ambry bundle cafebabecafeba00 cafebabecafeba00 120 ambry-cafebabecafeba00

The cathedral only re-reads ambry files when it reloads settings.conf. Run touch /etc/cathedral/settings.conf after replacing the bundle.

NAT discovery and P2P

The cathedral can work out whether two peers can reach each other directly. If both peers are behind NAT types that allow hole punching, the cathedral sends both of them the information they need to set up a direct peer-to-peer tunnel. Otherwise traffic keeps going through the cathedral.

Add the following to cathedral.conf and open UDP port 4470:

cathedral_nat_port 4470
cathedral_p2p_sync yes
  • cathedral_nat_port is the extra port the cathedral listens on to detect a peer's NAT type.
  • cathedral_p2p_sync shares P2P state with federated cathedrals, so P2P also works when the two peers use different cathedrals.

On each peer, set the same NAT port on the tunnel:

# hymn nat cafebabecafeba00-01-02 4470

You can also pass natport 4470 to hymn add. Setting the port to 0 turns NAT detection, and with it P2P, off.

Federation

Cathedrals can federate, so peers connected to different cathedrals can still reach each other. This gives you redundancy and lets you spread cathedrals across locations.

  1. Copy the same sync.secret to every cathedral and point secret at it.
  2. Give every cathedral the same flocks, peer files and ambry bundles (steps 3 to 5).
  3. In each cathedral's settings.conf, list the other cathedrals, outside of any flock block:
federate 5.6.7.8 4469
federate 9.10.11.12 4469

You can federate with up to 31 other cathedrals. A cathedral skips its own address, so you can use one shared settings.conf on every cathedral.

The sync secret protects all traffic between your cathedrals. Treat it like any other long-lived shared secret.

Remembrance

With remembrance enabled, a peer's cathedral regularly tells it about the other cathedrals in the federation. The peer stores that list and fails over to another cathedral if its current one stops responding.

Turn it on for a tunnel with hymn:

# hymn remembrance cafebabecafeba00-01-02 on

In a hand-written configuration, set cathedral_remembrance to a file where sanctum stores the list:

cathedral_remembrance /etc/sanctum/remembrance-cafebabecafeba00-01-02

Option reference

cathedral.conf

OptionDescription
mode cathedralRun this instance as a cathedral.
localThe ip:port the cathedral listens on.
secretPath to the sync secret. Required.
cathedral_coskPath to the cathedral's private COSK. Required.
secretdirDirectory holding the flock-<id>/ peer directories. Required.
settingsPath to the flock and federation settings file.
cathedral_mtuExpected MTU for traffic through the cathedral.
cathedral_ipPublic ip:port when the cathedral sits behind NAT.
cathedral_nat_portPort used for NAT detection.
cathedral_p2p_syncShare P2P state with federated cathedrals.
shroudTurn shroud on or off.

settings.conf

OptionDescription
flock <id> { … }Defines a flock. The id must end in 00.
allow <peer> spi <kek> <mbit>Allows a peer into the flock (inside a flock block).
ambry <path>The ambry bundle for the flock (inside a flock block).
federate <ip> <port>Federate with another cathedral.

See sanctum.conf(5), ambry(1) and hymn(1) for the full details.