How to setup a cathedral

Prerequisites

You should have compiled and installed Sanctum according to the tunnel setup guide.

Cathedral setup

You should create three system accounts that will be used by the cathedral to properly privilege separate the control, cathedral and purgatory processes.

How you do that is left as an exercise for the reader, as that depends on your OS. A suggestion on what to name the accounts could be as follows:

• _purgatory-rx
• _purgatory-tx
• _cathedral

Cathedral configuration

You use the sanctum binary in cathedral mode to run a cathedral. Here is a minimal example configuration:

mode cathedral
instance cathedral-test

mtu 1372
shroud yes
pidfile /tmp/cathedral-test.pid
local 1.2.3.4:4469
secret /home/cathedral/sync.secret
secretdir /home/cathedral/shared/identities
settings /home/cathedral/shared/settings.conf

run control as root
run cathedral as _cathedral
run purgatory-rx as _purgatory-rx
run purgatory-tx as _purgatory-tx

Besides the paths explicitly mentioned in the configuration file, in the secretdir you have to create a directory for each flock that you will serve with the cathedral.

$ mkdir -p /home/cathedral/shared/identities
$ mkdir /home/cathedral/shared/identities/flock-<flock>

The format for their name should be flock-<flock id>. In a flock directory, you place all peers in that flock their cathedral secrets and their cosk public keys. These are used for traffic protection and offer authentication between the client and cathedral, never for normal traffic.

The cathedral secrets are 256-bit symmetrical keys shared between the cathedral and client and are stored on the cathedral in the format of <peer id>.key. The cosk public key is an ed25519 public key generated using the ambry tool on the client side. The cosk public key is stored in this directory as <peer id>.pub. The peer id is the id you will define later in settings.conf.

The synchronization key used for communication between cathedrals during federation can be generated as follows:

$ dd if=/dev/urandom of=/home/cathedral/sync.secret bs=32 count=1

Flock configuration

To create one or several flocks, they have to be defined in the file pointed to by the settings option in the cathedral configuration.

Key management

Each flock needs to receive an ambry file — these are wrapped shared secrets that will be securely distributed to your peers via the cathedral.

Use the ambry tool to generate keys for the flocks you are configuring.

$ ambry generate cafebabe
$ ambry export cafebabe cafebabe 120 /path/to/ambry.bundle

Flock definition

In our example configuration, that path is /home/cathedral/shared/settings.conf. The format for a flock defined in settings is as follows:

flock <flock id> {
    allow <peer id 1> spi <kek id 1> <bandwidth limit>
    allow <peer id 2> spi <kek id 2> <bandwidth limit>

    ambry /path/to/ambry.bundle
}

The format for the flock name is a 64-bit hex value. The peer ids are 32-bit hex strings. The kek id is the hex number of the kek associated with the peer, prefixed with 0x.

The bandwidth limit is an unsigned integer representing the limit in Mbit per second the peer will be limited to. A value of 0 is equivalent to unlimited bandwidth.

An example flock, with two peers defined, could look like this:

flock cafebabe {
    allow deadbeef spi 0x1 0
    allow deadbabe spi 0x2 0

    ambry /home/cathedral/shared/ambries/ambry-cafebabe
}

Using an ambry bundle is not strictly required for a flock if there is already a shared secret distributed to all the peers. The cathedral would then only be used to relay the traffic, but this is left as an exercise to the reader.

Peer configuration

The easiest way to configure a tunnel that uses the cathedral and flock that was just set up is to use the hymn tool.

Several keys must be generated on the client side or distributed in a secret way to the client. For example, a cosk can be generated using the ambry tool:

$ ambry cosk-pair /tmp/cosk-private /tmp/cosk-public

While the cathedral secret may be generated from /dev/urandom, for example, after which you distribute it to your cathedrals in a safe way.

$ dd if=/dev/urandom of=/tmp/cs-deadbeef bs=32 count=1

The KEK comes from your key management, using the ambry tool, and can be found under the kek-data directory for said flock.

If we were configuring the peer deadbeef, with kek id 1, to use the cathedral with hymn:

$ hymn add cafebabe-01-02 tunnel 172.31.253.1/24 mtu 1422 \
    kek /tmp/kek-0x01 cathedral 1.2.3.4:4469 shroud yes \
    identity deadbeef:/tmp/cs-deadbeef cosk /tmp/cosk-private

Note that you need /tmp/kek-0x01, /tmp/cs-deadbeef and /tmp/cosk-private on that system.

Likewise, configuring the peer deadbabe to connect to deadbeef with hymn looks similar:

$ hymn add cafebabe-02-01 tunnel 172.31.253.2/24 mtu 1422 \
    kek /tmp/kek-0x02 cathedral 1.2.3.4:4469 shroud yes \
    identity deadbabe:/tmp/cs-deadbabe cosk /tmp/cosk-private

Note that you need /tmp/kek-0x02, /tmp/cs-deadbabe and /tmp/cosk-private on that system.

NAT discovery and hole punching

Cathedral mode can detect whether it's possible for peers to connect directly to each other to get a peer-to-peer connection.

To set up the cathedral to listen for traffic from the client to determine its NAT type, specify the following configuration:

cathedral_p2p_sync yes
cathedral_nat_port 4470

The option cathedral_p2p_sync enables the cathedral to sync p2p states with other cathedrals. This allows p2p discovery even if each client is connected to a different cathedral in the setup.

The option cathedral_nat_port configures on which port the cathedral will listen for additional peer traffic to determine its NAT type.

If the cathedral learns that both peers are behind a NAT type that allows hole punching, it will inform both sides with the relevant information to do this.

Federation

Cathedral mode also has the possibility of federating with other cathedrals, which allows you to truly build distributed infrastructure.

For that to work, all cathedrals that want to federate with each other need the same federation secret.

To enable federation for a cathedral, use the following configuration in the cathedral configuration:

secret /path/to/federation.secret

Then add the federated cathedrals in the settings.conf file:

federate <ip address> <port>

You can federate to up to 32 other cathedrals.