How to setup a cathedral

This guide will help you setup your own cathedral that can be used as a discovery and key distribution point for your own flocks and peers. This guide assumes that you are familiar with your system and networking.

Prerequisites

You should have compiled and installed sanctum according to the tunnel setup guide.

Cathedral setup

You should create three system accounts that will be used by the cathedral to properly privilege separate the control, cathedral and purgatory processes.

How you do that is left as an excercise for the reader as that depends on your OS. A suggestion on what to name the accounts could be as follows:

• _purgatory-rx
• _purgatory-tx
• _cathedral

Cathedral configuration

You use the sanctum binary in cathedral mode to run a cathedral.

Here is a minimal example configuration:

mode cathedral
instance cathedral-test

pidfile /tmp/cathedral-test.pid
local 1.2.3.4:4500
secret /home/cathedral/sync.secret
secretdir /home/cathedral/shared/identities
settings /home/cathedral/shared/settings.conf

run control as root
run cathedral as _cathedral
run purgatory-rx as _purgatory-rx
run purgatory-tx as _purgatory-tx

Besides the paths that are explicitly mentioned in the configuration file, in the secretdir you have to create a directory for each flock that you will serve with the cathedral.

$ mkdir -p /home/cathedral/shared/identities
$ mkdir /home/cathedral/shared/identities/flock-<flock>

The format for their name should be flock-<flock id>. In a flock directory, you place all peers in that flock their cathedral secrets and their cosk public keys These are used for traffic protection and offer authentication between the client and cathedral, never for normal traffic.

The cathedral secrets are 256-bit symmetrical keys shared between the cathedral and client and is stored on the cathedral in the format of <peer id>.key. The cosk public key is an ed25519 public key generated using ambry tool on the client side. The cosk public key is stored in this directory as <peer id>.pub. The peer id is the id you will define later in settings.conf

These keys have to be distributed to the peers that will use the cathedral in a secure way, but that is left as an excercise to the reader because that is out of scope for this guide.

The synchronization key used for communication between cathedrals when federation data can be generated as follows:

$ dd if=/dev/urandom of=/home/cathedral/sync.secret bs=32 count=1

Flock configuration

To create one or several flocks, they have to be defined in the file pointed to by the settings option in the cathedral configuration.

Key management

Each flock needs to receive an ambry file, these are wrapped shared secrets that will be securely distributed to your peers, via the cathedral.

Use the ambry tool to generate keys for the flocks you are configuring.

$ ambry generate cafebabe
$ ambry export cafebabe cafebabe 120 /path/to/ambry.bundle

Flock configuration

In our example configuration, that path is /home/cathedral/shared/settings.conf. The format for a flock defined in settings is as follows:

flock <flock id> {
    allow <peer id 1> spi <kek id 1> <bandwidth limit>
    allow <peer id 2> spi <kek id 2> <bandwidth limit>

    ambry /path/to/ambry.bundle
}

The format for the flock name is a 64-bit hex value. The peer ids are 32-bit hex strings. The kek ids is the hex number of the kek associated with the peer, prefixed with 0x.

The bandwidth limit is a unsigned integer that represents the limit in Mbit per second that the peer will be limited to. A value of 0 is the equivalent to unlimited bandwidth.

An example flock, with two peers defined could look like this:

flock cafebabe {
    allow deadbeef spi 0x1 0
    allow deadbabe spi 0x2 0

    ambry /home/cathedral/shared/ambries/ambry-cafebabe
}

Using an ambry bundle is not strictly required for a flock, if there already is a shared secret distributed to all the peers. The cathedral would then only be used to relay the traffic but this is left as an excercise to the reader.

Peer configuration

The easiest way to configure a tunnel that uses the cathedral and flock that just was setup is to use the hymn tool.

Several keys must be generated on the client-side or distributed in a secret way to the client. For example, a cosk can be generated using the ambry tool:

$ ambry cosk-pair /tmp/cosk-private /tmp/cosk-public

While the cathedral secret may be generated from /dev/urandom for example after which you distribute it to your cathedrals in a safe way.

$ dd if=/dev/urandom of=/tmp/cs-deadbeef bs=32 count=1

The KEK comes from your key management, using the ambry tool and can be found under the kek-data directory for said flock.

If we were configuring the peer deadbeef, with kek id 1 to use the cathedral with hymn:

$ hymn add cafebabe-01-02 tunnel 172.31.253.1/24 mtu 1422 \
    kek /tmp/kek-0x01 cathedral 1.2.3.4:4500 \
    identity deadbeef:/tmp/cs-deadbeef cosk /tmp/cosk-private

Note that you need /tmp/kek-0x01, /tmp/cs-deadbeef and /tmp/cosk-private on that system.

Likewise, configuring the peer deadbabe to connect to deadbeef with hymn looks similar:

$ hymn add cafebabe-02-01 tunnel 172.31.253.2/24 mtu 1422 \
    kek /tmp/kek-0x02 cathedral 1.2.3.4:4500 \
    identity deadbabe:/tmp/cs-deadbabe cosk /tmp/cosk-private

Note that you need /tmp/kek-0x02, /tmp/cs-deadbabe and /tmp/cosk-private on that system.

NAT discovery and hole punching

The cathedral mode has the possibility of detecting if its possible for peers to connect directly to eachother to get a peer-to-peer connection.

To setup the cathedral to listen for traffic from the client to determine its NAT type, specify the following configuration:

cathedral_p2p_sync yes
cathedral_nat_port 4501

The option cathedral_p2p_sync enables the cathedral to sync p2p states with other cathedrals. This allows p2p discovery even if each client is connected to a different cathedral in the setup.

The option cathedral_nat_port configures on which port the cathedral will listen to additional peer traffic to determine its NAT type.

If the cathedral learns that both peers are behind a NAT type that allows hole punching it will inform both sides with the relevant information to do this.

Federation

The cathedral mode also has the possibilty of federating with other cathedrals which allows you to truly build distributed infrastructure.

For that to work all cathedrals that want to federate with each other need the same federation secret. How this federation secret is distributed between the cathedrals is left as an excercise to the reader.

To enable federation for a cathedral, use the following configuration in the cathedral configuration:

secret /path/to/federation.secret

Then add the federated cathedrals in the settings.conf file:

federate <ip address> <port>

You can federate to up to 32 other cathedrals.