Cathedral setup

How to setup a cathedral

This guide will help you setup your own cathedral that can be used as a discovery and key distribution point for your own flocks and peers. This guide assumes that you are familiar with your system and networking.

Prerequisites

You should have compiled and installed sanctum according to the tunnel setup guide.

Cathedral setup

You should create three system accounts that will be used by the cathedral to properly privilege separate the control, cathedral and purgatory processes.

How you do that is left as an excercise for the reader as that depends on your OS. A suggestion on what to name the accounts could be as follows:

_purgatory-rx
_purgatory-tx
_cathedral

Key management

The cathedral requires an ambry bundle for each flock that it should serve, refer to the reliquary its key management guide on how to do this.

Cathedral configuration

You use the sanctum binary in cathedral mode to run a cathedral.

Here is a minimal example configuration:

mode cathedral
instance cathedral-test

pidfile /tmp/cathedral-test.pid
local 1.2.3.4:4500
secret /home/cathedral/sync.secret
secretdir /home/cathedral/shared/identities
settings /home/cathedral/shared/settings.conf

run control as root
run cathedral as _cathedral
run purgatory-rx as _purgatory-rx
run purgatory-tx as _purgatory-tx

Besides the paths that are explicitly mentioned in the configuration file, in the secretdir you have to create a directory for each flock that you will serve with the cathedral.

$ mkdir -p /home/cathedral/shared/identities
$ mkdir /home/cathedral/shared/identities/flock-<flock>

The format for their name should be flock-<flock id>. In this directory, you have to put a secret key and cosk public key per peer in this flock. These are used for traffic protection and offer authentication between the client and cathedral, never for normal traffic.

The secret key is a 256-bit symmetrical shared secret shared between the cathedral and client and is stored on the cathedral in the format of <peer id>.key. The cosk public key is an ed25519 public key generated using ambry tool on the client side. The cosk public key is stored in this directory as <peer id>.pub. The peer id is the id you will define later in settings.conf

These keys have to be distributed to the peers that will use the cathedral in a secure way, but that is left as an excercise to the reader because that is out of scope for this guide.

Local has to be an IP and port that is reachable by all the peers that will be using the cathedral.

Secret is a key that should be 32 bytes of random data. An example of how to generate this key is by using the following command:

$ dd if=/dev/urandom of=/home/cathedral/sync.secret bs=32 count=1

While the cosk can be generated as follows:

$ ambry cosk-pair /tmp/cosk-private /tmp/cosk-public

Flock configuration

To create one or several flocks, they have to be defined in the file pointed to by the settings option in the cathedral configuration.

In our example configuration, that path is /home/cathedral/shared/settings.conf. The format for the configuration of a flock is:

flock <flock id> {
    allow <peer id 1> spi <kek id 1> <bandwidth limit>
    allow <peer id 2> spi <kek id 2> <bandwidth limit>

    ambry /path/to/ambry.bundle
}

The format for the flock name is a 64-bit hex value. The peer ids are 32-bit hex strings. The kek ids is the hex number of the kek associated with the peer, prefixed with 0x.

The bandwidth limit is a unsigned integer that represents the limit in Mbit per second that the peer will be limited to. A value of 0 is the equivalent to unlimited bandwidth.

An example flock, with two peers defined could look like this:

flock cafebabe {
    allow deadbeef spi 0x1 0
    allow deadbabe spi 0x2 0

    ambry /home/cathedral/shared/ambries/ambry-cafebabe
}

Using an ambry bundle is not strictly required for a flock, if there already is a shared secret distributed to all the peers. The cathedral would then only be used to relay the traffic but this is left as an excercise to the reader.

Peer configuration

The easiest way to configure a tunnel that uses the cathedral and flock that just was setup is to use the hymn tool.

If we were configuring the peer deadbeef, with kek id 1 to use the cathedral with hymn:

$ hymn add cafebabe-01-02 tunnel 172.31.253.1/24 mtu 1422 \
    kek /tmp/kek-0x01 cathedral 1.2.3.4:4500 \
    identity deadbeef:/tmp/cs-deadbeef cosk /tmp/cosk-private

Note that you need /tmp/kek-0x01, /tmp/cs-deadbeef and /tmp/cosk-private on that system.

Likewise, configuring the peer deadbabe to connect to deadbeef with hymn looks similar:

$ hymn add cafebabe-02-01 tunnel 172.31.253.2/24 mtu 1422 \
    kek /tmp/kek-0x02 cathedral 1.2.3.4:4500 \
    identity deadbabe:/tmp/cs-deadbabe cosk /tmp/cosk-private

Note that you need /tmp/kek-0x02, /tmp/cs-deadbabe and /tmp/cosk-private on that system.

NAT discovery and hole punching

The cathedral mode has the possibility of detecting if its possible for peers to connect directly to eachother to get a peer-to-peer connection.

To setup the cathedral to listen for traffic from the client to determine its NAT type, specify the following configuration:

cathedral_p2p_sync yes
cathedral_nat_port 4501

The option cathedral_p2p_sync enables the cathedral to sync p2p states with other cathedrals. This allows p2p discovery even if each client is connected to a different cathedral in the setup.

The option cathedral_nat_port configures on which port the cathedral will listen to additional peer traffic to determine its NAT type.

If the cathedral learns that both peers are behind a NAT type that allows hole punching it will inform both sides with the relevant information to do this.

Federation

The cathedral mode also has the possibilty of federating with other cathedrals which allows you to truly build distributed infrastructure.

For that to work all cathedrals that want to federate with each other need the same federation secret. How this federation secret is distributed between the cathedrals is left as an excercise to the reader.

To enable federation for a cathedral, use the following configuration in the cathedral configuration:

secret /path/to/federation.secret

Then add the federated cathedrals in the settings.conf file:

federate <ip address> <port>

You can federate to up to 32 other cathedrals.