Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Welcome to the documentation for Hammerhead. This book will cover (almost) everything you may need to know about Hammerhead, including history, configuration, and tips & tricks.

Some quick links:

About

Hammerhead is a nimble Matrix homeserver written in Golang, utilising the mautrix-go SDK. It is being built from the ground up - not a fork of an existing project. It is being built with being a power-user tool in mind, however the end goal is a usable implementation that can comfortably be used day-to-day.

If you want a daily-driver homeserver that is ready to go right now, I recommend continuwuity, a project from the same maintainers, written in Rust.

Demo instance

There is a demo instance available at hammerhead.nexy7574.co.uk. Please note that this instance is not suitable for casual or personal use - there may be extreme restrictions on available resources, and it is often running unstable versions of hammerhead, and the database is frequently wiped without notice.

If you want to jump on for a quick test drive, you can use a client like Element Web, Cinny, Sable, or Commet, although you can likely use whatever your existing favourite client is.

The registration token is the Codeberg repo’s HTTPS clone URL. No, you can not have admin.

Getting Started

Installing Hammerhead is relatively simple, but you need to be confident with managing a server to get the most out of it. This guide will walk you through the minimum steps required to get a working installation of Hammerhead.

This part of the guide is very in depth - consider using the TOC in the sidebar to help you navigate.

Prerequisites

To compile the server (for when you aren’t using a pre-built binary), you will need:

  • The latest version of Go.
  • A compatible Linux operating system: Debian-based, Arch, Fedora. Others may work but are not tested.
  • git and bash in your $PATH.
  • gcc is NOT currently required.
  • mdbook must be in your $PATH to compile hammerhead with embedded documentation.

To run the server, you will need:

  • PostgreSQL v14 or newer (check for issues regarding newly released PG versions).

To run the server, you will also want:

  • A reverse proxy, such as Caddy or Nginx.
  • A domain name (especially if you want to use federation).

Important

Once you have set your “server name”, it cannot be changed. This means, if you start off without a domain, and later want to switch to using one, you will have to delete your database and start fresh.

Compiling

If you wish to compile Hammerhead (instead of using the binaries created by CI or attached to releases), you can do so with the handy build.sh script located at the root of this repository. While using this script in particular is not required (compiling with plain go build is sufficient), the build script conveniently injects build metadata into the binary for accurate version reporting, and offers shorthands to compile static and release-optimised builds.

Note

You do not need to compile Hammerhead to run it. CI produces static binaries for AMD64 and ARM64 on each push to dev, and static binaries are also attached to each release.

Unless you need a dynamic binary, or are planning on hacking on Hammerhead, you likely do not need to compile.

Please skip to Installing if you just want to install Hammerhead.

With the build script

To get started, make sure you have the prerequisites detailed above. Then, you can clone the repository:

git clone https://codeberg.org/timedout/hammerhead.git && cd hammerhead

And then run the build script to produce a binary at ./bin/hammerhead:

./build.sh
#- will build dynamically-linked binary
#- will build debug binary (without optimizations)
#- Tag associated with the latest commit: N/A
#- Latest tag: N/A
#- Latest commit hash: ec0f1d0
#- Dirty? yes
#- Build date: 2026.03.09T19.48.00Z
#- Golang version: go1.26.0
#- OS/Arch: linux/amd64
#
#Compiling hammerhead
#...
#codeberg.org/timedout/hammerhead/cmd/hammerhead
#
#real    0m1.941s
#user    0m1.779s
#sys     0m0.785s

Notice how the first two lines say will build dynamically-linked binary and will build debug binary (without optimizations)? This is because by default, the script will create a debug-oriented build. While this is still plenty fast, it is dynamically linked, and retains debug symbols, meaning it’s a few mebibytes larger than necessary for most people. If you aren’t planning on running through Hammerhead with a debugger, you may wish to instead build a high-performance and/or static binary.

To compile a static binary, pass -static to build.sh:

./build.sh -static && file ./bin/hammerhead
#- will build static-linked binary without CGO (experimental!)
#- will build debug binary (without optimizations)
# ...
#./bin/hammerhead: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=..., BuildID[sha1]=..., with debug_info, not stripped

To compile a “release” binary (one without debug symbols), pass -release:

./build.sh -release && file ./bin/hammerhead
#- will build dynamically-linked binary
#- will build release binary (with optimizations)
#...
#./bin/hammerhead: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=..., BuildID[sha1]=..., stripped

You can even combine these flags (in any order) to compile a release+static binary:

./build.sh -static -release && file ./bin/hammerhead
#- will build static-linked binary without CGO (experimental!)
#- will build release binary (with optimizations)
#./bin/hammerhead: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=..., BuildID[sha1]=..., stripped

Without the build script

If for some reason you are unable to use the build script (consider opening an issue!), you can still compile without:

go build -o ./bin/ ./cmd/hammerhead

Note that this will produce a debug dynamic binary without any metadata. You will likely be unable to report bugs found when running this binary as it will not contain version data required to accurately troubleshoot problems.

Installing

In order to install Hammerhead, you will first need a binary. These can be acquired from:

Please note that CI artefacts are zipped when downloaded - you will first need to unzip them before you can get the binary within.

Make sure you select the correct binary by checking the output of uname -m:

OutputVersion
x86_64AMD64
aarch64ARM64

Other architectures may work, but are not built in CI, nor tested. Downloading the wrong binary will result in an error message like exec format error: ./hammerhead-arm64 when you try to run it.

Configuring

Important

The rest of this guide assumes your binary (from compiling or installing) is located at /usr/local/bin/hammerhead, and that /usr/local/bin is in your $PATH. It also assumes you have read+write+execute access to /etc/hammerhead.

Before you daemonise Hammerhead, you’re going to want to create the template configuration file. When Hammerhead is unable to load a configuration file, it will create one with some default values, so we’re going to use this to create a sparse config file that you can modify with your values.

First, create the configuration file by running hammerhead -config /etc/hammerhead/config.yaml. Upon success, this will create a YAML file at that location with some example values. You must edit at least:

  • server_name - set this to your server name (the part after the : in your user IDs).
  • database.url - set this to the connection URI of your postgres server. Must contain the username, password, host, and database name for the connection.

Caution

After you start the server for the first time the server_name cannot be changed. Make sure you’re certain before you hit run!

You may also wish to modify the other options, such as the registration.token (to change it to one of your choosing), changing the password requirements, increasing the max_request_bytes (to allow larger file uploads), or modifying the listeners.

A configuration reference does not yet exist but will be added at a later date.

Importing a previous installation’s signing key

It is possible to import another installation’s server signing key, provided you have it in a file with the format of ed25519 KeyID PrivateKeyPart, for example: ed25519 a_bcde hnBjtF9w9AQAWfnhAHIV3fdu9QH0YX1xWlb0qEPjE4w

You can then import this key via hammerhead -import-signing-key path/to/file. Once you confirm you want to import it, it will be imported as an “expired” key, meaning it can no longer be used to sign new events. It can, however, still be used to verify events sent from before you set up Hammerhead.

Tip

If, for some reason, you wish to continue using the imported key to sign new events, you will need to manually do so via postgres: UPDATE owned_signing_keys SET expired_ts = NULL;.

While technically supported, you should not have multiple active signing keys - if Hammerhead automatically generated one, you should invalidate it before continuing with hammerhead -invalidate-signing-key 'KEY_ID'. Again, once a key is expired, it cannot be used ever again.

Exporting your current key

If you wish to migrate your deployment to another homeserver implementation, you can export your active signing key into the synapse format with hammerhead -export-signing-key. This does not expire the key, so it can be re-used as an active key in another deployment (as long as it has the same server name).

Starting the server

To start Hammerhead after configuring it, you can just run the binary like you did the first time:

hammerhead -config /etc/hammerhead/config.yaml
#=== Hammerhead v0.0.1-dev ===
#Parsing command line flags...
#Loading configuration from  /etc/hammerhead/config.yaml...
#Initialising logging...
#Dropping output into logging system...
#2026-03-09T19:84:00Z DBG Validating configuration
#2026-03-09T19:84:00Z INF Initialising server
#2026-03-09T19:84:00Z INF running database migrations
#2026-03-09T19:84:00Z INF finished running database migrations elapsed_ms=181
#2026-03-09T19:84:00Z WRN no signing key found, generating new one. If this was not expected, I hope you have a backup of the one you expected.
#2026-03-09T19:84:00Z INF generated new signing key key_id=ed25519:vgJGqQ
#2026-03-09T19:84:00Z INF loaded signing key key_id=ed25519:vgJGqQ
#2026-03-09T19:84:00Z INF initialising media repository
#2026-03-09T19:84:00Z INF media repository initialised elapsed_ms=2
#2026-03-09T19:84:00Z INF Passing off server startup to server instance
#2026-03-09T19:84:00Z INF Starting server on
#2026-03-09T19:84:00Z INF server started. ^C to shut down.
#2026-03-09T19:84:00Z INF starting listener address=:8008 tls=false

If your configuration is malformed in such a way that Hammerhead would not be able to operate, starting the server will fail, and it will tell you what you need to change.

As soon as you see server started. and starting listener, your server is ready to go! Keep it running like this for the next step.

Creating your first user

After you’ve started the server, there will be no users. Open a client such as Element Web, Cinny, Sable, or Commet, and plug in your server name. From there, go to “register”, and put in the username and password you desire.

You will likely be challenged to supply the token that is under the registration section of your configuration. This is to prevent automated scripts or abusive users from finding your server, and registering accounts without your supervision.

Upon registering the first user, you will be granted admin, which allows you to control things that happen on your deployment via the admin API. TODO: Admin API reference.

Any users registered after the first will simply be standard users who have no special control or rights on the server.

Tip

You can mark anyone as an admin by updating their admin status via postgres:

UPDATE accounts SET admin = true WHERE localpart = 'username_here'; (NOTE: username, not user ID)

Daemonising

In order to run Hammerhead 24/7, you will want to first interrupt the running server by hitting CTRL+C (running concurrent instances of Hammerhead is not safe), and create a systemd unit file like below:

# /etc/systemd/system/hammerhead.service
[Unit]
Description=Hammerhead Matrix homeserver
Documentation=https://timedout.codeberg.page/hammerhead
Requires=network.target
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=hammerhead
Group=hammerhead

ExecStart=/usr/local/bin/hammerhead
WorkingDirectory=/etc/hammerhead
Environment=HAMMERHEAD_CONFIG=/etc/hammerhead/config.yaml

Restart=on-failure
RestartSec=2
RestartSteps=5
RestartMaxDelaySec=1m
StartLimitIntervalSec=1m
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

You will also need to create a user and group for Hammerhead to use this systemd unit, which can be achieved on most Linux distributions like so: adduser --system --group --home /etc/hammerhead hammerhead. Make sure you chown -R hammerhead:hammerhead /etc/hammerhead and chmod 755 /usr/local/bin/hammerhead so that Hammerhead can read & write in its site directory, and execute the binary.

You can then systemctl daemon-reload to init the unit file, and then use systemctl enable --now hammerhead.service to both enable the service at startup, and also start it immediately. Once start returns, you can check the status and most recent log lines of Hammerhead by running systemctl status hammerhead.service:

systemctl status hammerhead.service
#● hammerhead.service - hammerhead
#     Loaded: loaded (/etc/systemd/system/hammerhead.service; enabled; preset: enabled)
#     Active: active (running) since Mon 2026-03-09 19:84:00 GMT; 0s ago
#   Main PID: 129208 (hammerhead)
#      Tasks: 9 (limit: 57189)
#     Memory: 8.6M (peak: 10.6M)
#        CPU: 23ms
#     CGroup: /system.slice/hammerhead.service
#             └─129208 /usr/local/bin/hammerhead
#
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF Initialising server
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF running database migrations
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF finished running database migrations elapsed_ms=31
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF loaded signing key key_id=ed25519:lZ2Vwg
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF initialising media repository
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF media repository initialised elapsed_ms=3
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF Passing off server startup to server instance
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF Starting server on
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF server started. ^C to shut down.
#Mar 09 19:84:00 hammerhead-staging hammerhead[129208]: 2026-03-09T19:84:0Z INF starting listener address=0.0.0.0:8008 tls=false

Configuration Reference

Hammerhead uses YAML v1.2.2 for configuration. While not the prettiest option for configuration, other languages have been tried before and found to be even worse, so you’ll live (and likely also already know the syntax anyway).

The default configuration is generated at ./config.yaml (i.e. in the directory hammerhead is executed in). The default configuration does not contain all possible values, but will contain all required values with at least placeholder value.

This reference will list every config option available, with a description, and an example.


Required keys:


Environment Variables

There are a number of environment variables that you can set to have even more granular control over some aspects of Hammerhead, detailed below:

GOMAXPROCS

GOMAXPROCS (integer, optional): Overrides the maximum number of threads available to the program.

This is a very low-level tuning parameter you likely do not need to touch.

From the Golang documentation:

The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit.

If not provided, the value is determined via an appropriate default value from a combination of (source)

  • the number of logical CPUs on the machine,
  • the process’s CPU affinity mask,
  • and, on Linux, the process’s average CPU throughput limit based on cgroup CPU quota, if any.

Hammerhead uses this value a lot to determine the maximum number of concurrent threads that a routine can spawn for operations that may spawn an unbounded number of threads (such as global profile updates). In cases where this limit is explicitly checked for, as many threads as required will be created, but only up to N will be executed - the rest will be waiting on a semaphore.

Keep in mind not all operations that split into concurrent routines respect this limit, only those that expect to spawn a lot.

If you are on a single or even dual-core machine, you may wish to raise this. You can see the calculated value at runtime by running hammerhead -version, or by checking POST /_hammerhead/v0/admin/metrics for max_threads.

HAMMERHEAD_TEST_DB_URI

HAMMERHEAD_TEST_DB_URI (string, optional): The fully qualified postgres:// connection URI for a test database.

Used for integration testing - set this if you are running the test suite with go test. Otherwise, database tests will be skipped.

HAMMERHEAD_COMPLEMENT

HAMMERHEAD_COMPLEMENT (literal 1, optional): When value equals 1, enables complement mode.

Enables “complement mode” - changes some internal settings to better suit the complement test suite. Currently, does nothing.

Caution

Enabling this in production will result in an INSECURE server!

HAMMERHEAD_ALLOW_MULTIPLE_SIGNING_KEYS

HAMMERHEAD_ALLOW_MULTIPLE_SIGNING_KEYS (literal 1, optional): When value equals 1, permits having multiple active signing keys.

By default, when starting up, Hammerhead will immediately invalidate all but one signing key if multiple active and valid ones are found. If, for whatever reason, you do not wish for this to happen, setting this env var will skip this process. This voids your warranty.

HAMMERHEAD_CLIENT_IP_HEADERS

HAMMERHEAD_CLIENT_IP_HEADERS (string, optional): A comma separated list of header names to trust as sources of true incoming client IPs.

By default, only X-Forwarded-For and X-Real-Ip are checked for client IPs. If your reverse proxy uses another one, you will have to specify it here. Note that specifying headers here overrides the built-in ones.

Example:

HAMMERHEAD_CLIENT_IP_HEADERS=X-My-Custom-Header,X-Forwarded-For,X-Real-Ip

HAMMERHEAD_TRUSTED_PROXIES

HAMMERHEAD_TRUSTED_PROXIES (string, optional): A comma separated list of network ranges (CIDR) that will be trusted to provide proxy information (see [HAMMERHEAD_CLIENT_IP_HEADERS]).

When not provided, defaults to only loopback addresses (127.0.0.0/8 and ::1/128).

Example:

HAMMERHEAD_TRUSTED_PROXIES=172.16.0.0/12,127.0.0.0/8,::1/128

HAMMERHEAD_TRUST_CLOUDFLARE

HAMMERHEAD_TRUST_CLOUDFLARE (literal 1, optional): When enabled, enable Cloudflare direct proxy support.

It is not recommended that you do this, but if you are using Cloudflare as a proxy (i.e. have the orange cloud), and don’t have a proxy like Caddy in front of the service that handles the incoming request properly, you can set HAMMERHEAD_TRUST_CLOUDFLARE=1. This will add Cloudflare’s IP ranges to your HAMMERHEAD_TRUSTED_PROXIES, and also appends Cf-Connecting-Ip to HAMMERHEAD_CLIENT_IP_HEADERS.

It is recommended you leave this disabled unless you know you need it.


caches

caches (mapping, optional): Controls the sizes and lifetimes of some runtime caches.

Hammerhead caches some data in memory to avoid excessive round-trips to the database, especially when fetching that data may end up being expensive (e.g. fetching a lot of events in rapid succession). Generally, the size of these internal caches scales with your available resources (or, more accurately, every cache defaults to 4096*N, where N is the number of logical processors).

There are no magic numbers, if you need to tune your caches, you should do so with trial and error. What works best is different for each deployment.

Each cache map has two keys, max_entries, and max_ttl. max_entries (integer, optional) controls how many entries can be in the cache before old ones start being evicted. By default, it is 4096*N, where N is the number of logical CPUs. Set this to zero to disable limiting the size of caches (not recommended). max_ttl controls how long entries are allowed to remain in the cache before they are evicted. Set to zero to disable TTL eviction.

Warning

TTL-based cache evictions are checked on each cache operation (both read and write), which means they are inherently more computationally expensive than simple size-based limits.

On the other hand, size-based eviction is only evaluated on write operation, making them generally cheaper, but may result in Hammerhead holding on to memory purely for “stale” cache entries.

It is recommended you leave caches as their default values unless you are encountering memory constraint issues.

Examples:

caches:
  events:
    max_entries: 8192
    max_ttl: 5m

You can monitor the cache size and hit/insert/eviction rate via /metrics.

events

events (mapping, optional): Controls the event cache.

The event cache is a key-value map of {event_id: event_data}. Each value may be up to 64KiB. Generally you want quite a large event cache, and this is the first thing that will be hit during operations that involve events (so, most of them), for example: state resolution, fetching events, client sync loops, message pagination.

See caches for more details.

Example:

caches:
  events:
    max_entries: 8192

devices

devices (mapping, optional): Controls the devices cache.

There are three caches for devices which allow for faster device lookups for incoming requests. Note that the devices cache will be very hot when end-to-end encryption workloads are involved, so you should avoid lowering this one unless you know for sure you won’t be dealing with end-to-end encryption.

The default values for this cache are likely excessive when federation is not involved.

See caches for more details.

Example:

caches:
  devices:
    max_entries: 4096

accounts

accounts (mapping, optional): Controls the accounts cache.

There are two caches for accounts which enable faster account lookups. This cache will always be hot, as accounts are looked up for each incoming request. As one of these caches is an access token to Account mapping (so that Bearer tokens can be instantly related to an account), having values that do not match your devices cache may cause weird behaviour.

See caches for more details.

Example:

caches:
  accounts:
    max_entries: 4096

database

database (mapping, required): The configuration for the PostgreSQL database connection.

Examples:

database:
  url: postgresql://user:password@hostname:port/dbname?sslmode=disable
database:
  url: postgresql://user:password@hostname:port/dbname?sslmode=disable
  max_idle_connections: 2
  max_idle_lifetime: 5m
  max_open_connections: 5
  max_open_lifetime: 5m

database.url

url (string, required): The URI to connect to.

This string SHOULD be prefixed with postgresql://, but postgres:// will work for compatibility reasons.

This connection string is passed directly to the database driver, so you can configure other connection-related settings in this URL (using libpq style query args - see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS for more info).

Example:

database:
  url: postgresql://user:password@hostname:port/dbname?sslmode=disable

database.max_idle_connections

database.max_idle_connections (integer, optional): The maximum number of idle (not actively running a query) connections to keep open in the connection pool.

See: max_open_connections

database.max_open_connections

database.max_open_connections (integer, optional): The maximum number of active (running a query) connections the connection pool is allowed to have.

These options configure the maximum number of parallel connections available to Hammerhead. Generally, you shouldn’t need too many (2+2 may be a good starting point), and you should also be conscious of the additional resource usage incurred by having more postgres connections on the postgres server. More connections will allow for more concurrent operations, but you should only really be concerned about that if your server is exceptionally high traffic, and you’re seeing lots of warnings about transactions taking a long time.

By default, there is no connection pooling - this is expected to change.

Examples:

database:
  max_idle_connections: 2
database:
  max_open_connections: 2
database:
  max_idle_connections: 2
  max_open_connections: 2

database.max_idle_lifetime

database.max_idle_connections (string (duration), optional): The maximum lifetime of an idle connection.

See: max_open_lifetime

database.max_open_lifetime

database.max_open_connections (string (duration), optional): The maximum lifetime of an active connection.

Controls how long connections live for before being destroyed. You can usually leave this disabled if you aren’t running a fancy postgres server configuration, but if you are, you probably know what these values should be anyway.

Example:

database:
  max_idle_lifetime: 5m
database:
  max_open_lifetime: 5m
database:
  max_idle_lifetime: 5m
  max_open_lifetime: 5m

debug

debug (boolean, optional): Enable or disable debug mode.

While you can still get debug logs and whatnot with this option disabled, setting debug: true will enable additional runtime checks that may affect how the server operates. It is designed to be used with a step-through debugger in mind, so sometimes some conditions that would usually be handled by logging an error will instead cause actual panics and potentially crashes. An example of a side effect of enabling this option is that request handlers that don’t write a response body will cause a crash - with debug disabled, this will log an error instead.

You usually only want to enable this if you are actively debugging Hammerhead.

Example:

debug: true

default_room_version

default_room_version (string, optional): Defines the default room version for new rooms.

While clients can specify the room version they want to create when calling /createRoom, if they do not, the room version specified here will be used instead.

Note

You probably don’t need to set this - the latest version that Hammerhead fully supports is used by default, meaning generally you should just update your server if the default version is not new enough. Overriding the default value may have unintended consequences.

Example:

default_room_version: 12

dont_expose_metrics

dont_expose_metrics (boolean, optional): If true, don’t expose /metrics.

Disables the Prometheus metrics exporter route.

Example:

dont_expose_metrics: true

experiments

experiments (mapping, optional): Enabled experiments.

See: Experiments

Example:

experiments:
  msc1234:
    enabled: true

listeners

listeners (sequence mapping, required): Configures the addresses that Hammerhead will listen to.

Both TCP and unix listeners are supported. You can even mix and match them!

TCP listener:

  • host (optional): The host to listen on. Usually 127.0.0.1 or ::1, or 0.0.0.0 & :: for all addresses. Defaults to an empty string, which implicitly means all addresses, both IPv4 and IPv6.
  • port (required): The port to listen on. Must be between 1 and 65536.

Unix listener:

  • socket (required): The path to the socket file. Hammerhead must be able to create, read, write, and delete this path.

Both listeners can enable native TLS by setting tls: true.

Note

Native TLS is primarily only included for running the test suite. TLS should normally be terminated by your reverse proxy, unless you have an advanced use case.

Tip

All routes (client-to-server, appservices, key-server, server-to-server) are handled by all listeners. If you are used to the Synapse style of having to define which listeners handle which routes, you need not do that here. 127.0.0.1:8008 and 127.0.0.2:8448 both run through the same router, for example.

Example:

listeners:
  - host: 0.0.0.0  # tcp://*:8008 (any IPv4 address)
    port: 8008
  - host: "::"  # tcp://[*]:8008 (any IPv6 address)
    port: 8008
  - host: 0.0.0.0  # tcp+tls://*:8008 (any IPv4 address, with TLS)
    port: 8448
    tls: true
  - host: "::"  # tcp+tls://[*]:8448 (any IPv6 address, with TLS)
    port: 8448
    tls: true
  - port: 8998  # tcp+tls://@:8448 (any IPv4 or IPv6 address)
  - socket: /tmp/hammerhead.sock # (unix socket at /tmp/hammerhead.sock)

logging

logging (mapping, optional): The configuration for zerolog using zeroconfig.

See zeroconfig for the full schema.

If there are no loggers configured, a coloured “pretty” stdout logger will be configured for you. If you are in debug mode (or Hammerhead was built with a dirty working tree), an additional trace-level JSON logger will be configured for you too.

Example:

logging:
  writers:
    # - type: journald  # uncomment if you're using journald.
    - type: stdout
      format: pretty-colored
      min_level: info
    - type: file
      format: json
      min_level: debug
      filename: hammerhead.log  # JSON-line file
      max_size: 100
      max_age: 30
      max_backups: 3
      compress: true

max_request_bytes

max_request_bytes (integer, optional): The maximum size of a request (in bytes) to read before aborting. Defaults to 100MiB (104857600).

Since request bodies are buffered into memory (including media) before they’re worked on, it is generally necessary to limit the size of request bodies that will be read. If a request sends a body larger than this, it will only be partially read, and then rejected when the server realises it’s too large (at least one byte over the limit).

Caution

Setting this value too high will either result in the OOM reaper coming knocking, and terminating the server process, or potentially undefined behaviour ranging from catchable alloc failures, to critical panics.

Examples:

max_request_bytes: 26214400  # 25MiB
max_request_bytes: 52428800  # 50MiB
max_request_bytes: 104857600  # 100MiB (higher values are typically excessive and unsafe)
max_request_bytes: 536870912  # 512MiB
max_request_bytes: 1073741824  # 1GiB

media_repo

media_repo (mapping, optional): The configuration for the media repository.

Hammerhead’s media repository is quite a complex component that is incredibly flexible, so there are a number of configuration options to play with. Fear not, only a couple are necessary.

media_repo.root_path

root_path (string, optional): The root path to where media should be stored.

This can either be a fully qualified absolute path, or a relative path, or even point at a symlink. As long as the subdirectories local, remote, and external can be created at that location.

If the root_path does not exist, it will be created with 750 file permissions (rwxr-x---). The root path must not end in a trailing slash.

If the root path is not provided, the media repository will be disabled (but existing data will remain, if it exists).

Example:

media_repo:
  root_path: /mnt/media/hammerhead

media_repo.temp_path

temp_path (string, optional): The path to where temporary media files should be stored.

This can either be a fully qualified absolute path, or a relative path, or even point at a symlink. Temporary directories starting with the prefix hammerhead_media_ must be creatable by the server.

If the temp_path does not exist, it will be created with 750 file permissions (rwxr-x---).

“temporary media files” are typically files that are being actively uploaded (i.e. before they’re properly saved), and thumbnails that are being worked on. It is unlikely that files will remain in this directory for more than a few seconds at a time.

It is safe to put the temporary directory on an ephemeral file system.

If the path is omitted, the system temporary directory (typically /tmp on Linux) will be used.

Example:

media_repo:
  temp_path: /mnt/media/hammerhead

media_repo.max_size_bytes

max_size_bytes (unsigned 64-bit integer, optional): The maximum size (in bytes) of a single media item. Defaults to 100MiB (104857600).

Controls the maximum size of file uploads. Attempts to upload media files that are larger than this value will be rejected, even if the uploader is an administrator. The server will attempt to reject large uploads if their advertised Content-Length exceeds this value, but in cases where the Content-Length header is unavailable, the server will read up to max_size_bytes+1 bytes to determine whether the file is too large.

Important

The value of max_size_bytes MUST be less than or equal to max_request_bytes. Setting a value higher than max_request_bytes would cause the router component to reject the request for being too large before it could be passed to the media repository component for validation.

The server will refuse to start if this condition is not met.

Examples:

media_repo:
  max_size_bytes: 8388608  # 8MiB
media_repo:
  max_size_bytes: 26214400  # 25MiB
media_repo:
  max_size_bytes: 52428800  # 50MiB
media_repo:
  max_size_bytes: 104857600  # 100MiB
media_repo:
  max_size_bytes: 536870912  # 512MiB
media_repo:
  max_size_bytes: 1073741824  # 1GiB

media_repo.security

security (mapping, optional): Configures the security-related settings for the media repository.

Because the media repository is a complex component that exclusively handles potentially untrusted user input, there are several security configurations available. The configuration for this is structured in such a way that the default values are typically sufficient for most people.

media_repo.security.disable_remote_media

disable_remote_media (boolean, optional): If enabled, disables external/remote media functionality. Defaults to false.

When remote media is disabled, federated media will not be fetched, federated requests for media will be rejected, and server-side URL previews will be disabled.

Example:

media_repo:
  security:
    disable_remote_media: true

media_repo.security.disallow_mime_types

disallow_mime_types (sequence of strings, optional): Prevents files matching any of the given glob patterns from being uploaded to the media repository.

When a user attempts to upload a file, if the claimed Content-Type matches any of the given glob patterns, it will be rejected, even if they are an administrator.

Warning

The content type of encrypted files is application/octet-stream. Using a glob pattern that blocks this will effectively prevent users from uploading encrypted files.

Furthermore, Hammerhead does not currently support MIME sniffing, so malicious users can work around this restriction by lying about or omitting the relevant header.

Example:

media_repo:
  security:
    disallow_mime_types:
      - image/*  # ban all images
      - application/vnd.microsoft.portable-executable  # ban EXE files
      - application/zip
      - application/x-zip-compressed
      - application/gzip
      - application/zstd  # ban compressed types

media_repo.security.only_admins

only_admins (boolean, optional): If true, only server administrators can upload media.

When enabled, regular users are unable to upload media. This can be used to restrict media uploads to only trusted users.

Example:

media_repo:
  security:
    only_admins: true

media_repo.security.disable_checksums

disable_checksums (boolean, optional): If true, disable checksum generation and comparison.

Hammerhead makes use of SHA256 checksums to verify the integrity of media when utilising it. Typically, this means a SHA256 checksum is generated when the file has finished uploaded, and is then verified before it is transmitted to requesting clients. This prevents the file being tampered with on disk (although this is easily circumvented by just modifying the hash in the database). Over federation, Hammerhead will include this SHA256 hash in the metadata part of the download, before sending the media content itself. This means other Hammerhead servers that download media from this server will be able to verify the integrity of the downloaded file before processing it. This is not a Matrix behaviour and is currently exclusive to Hammerhead.

Turning off checksums may improve performance as it avoids an extra disk round-trip, however this opens up the potential for corrupted or tampered files to be served.

Files that fail checksum validation are not immediately deleted, however will cause an error.

Example:

media_repo:
  security:
    disable_checksums: true  # not recommended

media_repo.security.minimum_account_age

minimum_account_age (string (duration), optional): The minimum age an account must be before it can upload files.

Restricts uploading media to accounts that have existed for longer than the given duration, excluding administrators.

Example:

media_repo:
  security:
    minimum_account_age: 5m  # 5 minutes

media_repo.security.disable_server_side_thumbnails

disable_server_side_thumbnails (boolean, optional): Disables server-side thumbnail generation.

Disabling server-side thumbnails may be desirable to reduce the amount of processing done on user-generated content, which is a large attack surface. The downside of this is that thumbnails will generally be unavailable for uploaded media, such as user avatars, resulting in increased bandwidth and unhappy impatient users.

Example:

media_repo:
  security:
    disallow_server_side_thumbnails: true

media_repo.security.acl

acl (mapping, optional): An ACL event body that defines which servers are and aren’t allowed to communicate media.

Sets an access-control-list in the same way as room ACLs - servers in the allow are always allowed, unless they are denied in the deny list. You cannot create an ACL that bans the local server.

The ACL is bidirectional - forbidden servers won’t be able to download media from you, but you also won’t be able to download media from them.

Examples:

# Explicit denylist
media_repo:
  security:
    acl:
      allow: ["*"]  # Allow all servers
      deny:
        - evil.matrix.example  # Don't allow media communication with evil.matrix.example specifically.
        - "*.bad.matrix.example"  # Don't allow media communication with any server name under "bad.matrix.example".
# Explicit denylist
media_repo:
  security:
    acl:
      allow:
        - "SERVER_NAME_HERE"  # Your server name has to be explicitly listed
        - "good.matrix.example"  # Allow media communication with good.matrix.example
      # No need for an explicit `deny` here.

media_repo.security.enable_streaming

enable_streaming (boolean, optional): Allow streaming media over federation.

When enabled, media can be streamed directly to the requesting client before the server has finished downloading it over federation. This is incompatible with checksum verification, which will instead be ignored in this case. The benefit of this feature is that users can start streaming files from remote servers almost immediately, rather than having to wait for the homeserver to finish downloading it before uploading it again, which is particularly useful for videos. However, the lack of checksum verification, or preprocessing as a whole, means that other security protections may not be effective, and potentially invalid or illegal data may be sent to the client unknowingly.

You should evaluate how this fits into your threat model before changing this value.

Example:

media_repo:
  security:
    enable_streaming: true

old_verify_keys

old_verify_keys (mapping, optional): A mapping of previous signing key IDs to when they expired.

A map of old signature keys that can be used to verify events. You should prefer to import the keys via the command hammerhead -import-signing-key, but if you do not have the private key any more you can advertise it here instead.

The keys of the map are the key IDs (e.g. ed25519:foo), and the value has two required keys:

  • key: The full public signing key of this key.
  • expired_ts: The unix timestamp (milliseconds) when this key expired

Example:

old_verify_keys:
  - ed25519:auto:
      key: Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw
      expired_ts: 1576767829750

registration

registration (mapping, optional): The registration settings for this server.

Controls the registration requirements for the server. If omitted, registration is disabled.

Example:

registration:
  enabled: true
  token: ed03d0b58fa20618be08fe20c1eb05a0
  password_requirements:
    min_entropy: 70
  admin_pre_shared_secret: 38d7630f8c5bdb50df2d99e65fd0e60f

registration.enabled

enabled (boolean, optional): Whether registration is enabled at all. Defaults to false.

If registration is disabled, no new accounts can be created without the admin API, even if requirements like a token are set.

Example:

registration:
  enabled: true

registration.i_have_a_very_good_reason_or_i_am_stupid_and_want_to_allow_unsafe_open_registration

i_have_a_very_good_reason_or_i_am_stupid_and_want_to_allow_unsafe_open_registration (boolean, optional): If set to true, registration will be unsafely open.

Enables registration without any registration requirements. This is dangerous, as this means your only defence against automated bots mass-registering on your server is rate-limits, and you have no way to prevent untrusted users registering and potentially being abusive. You should never need to enable this!

Example:

registration:
  i_have_a_very_good_reason_or_i_am_stupid_and_want_to_allow_unsafe_open_registration: false

registration.token

token (string, optional): The pre-shared secret to challenge registrations with.

A pre-shared token that is required as a second step to register an account. This allows you to give out an “invite code” to people you trust, so that they can create accounts themselves. This is the recommended way to have registration enabled.

If omitted or empty, the registration token will be disabled.

Example:

registration:
  token: ed03d0b58fa20618be08fe20c1eb05a0

registration.password_requirements

password_requirements (mapping, optional): Controls the requirements for passwords on this server.

Allows you to set minimum password length and entropy requirements. Not applied to accounts created via the admin API.

Example:

registration:
  password_requirements:
    min_entropy: 50  # require at least 50 bits of entropy
    min_length: 0  # disable length requirements

registration.password_requirements.argon2id

argon2id (mapping, optional): Controls the parameters passed to the Argon2id password hashing algorithm.

Allows you to change the time cost, memory cost, and parallelism values of the Argon2id algorithm. When a field is omitted, it defaults to whatever is recommended by RFC9106 § 7.4. It is not recommended you adjust this unless you have a very good reason to.

Hammerhead does not validate that you set “secure” values, in case the definition of “secure values” differs in the future. As such, you can shoot yourself in the foot with this. Furthermore, hashes are immutable once created - if you set an insane memory cost (like the recommended 2 gigabytes), the generated hash will always use that value during verification.

Fields:

  • t (32-bit unsigned integer): the T parameter (time cost). Must be at least 1 (0 uses default value).
  • m (32-bit unsigned integer): the M parameter (memory cost, in kibibytes). Note that this is normalised to at least 8*p (per RFC9106). Consequently, cannot be lower than 8 (0 uses default value).
  • p (8-bit unsigned integer): the parallelism degree. Must be at least 1 (0 uses default value).

registration.admin_pre_shared_secret

admin_pre_shared_secret (string, optional): A pre-generated authentication token that can only be used with the admin API to create new users.

When set, POST /_hammerhead/v0/admin/users/create will accept this value as a Bearer token, rather than requiring a token for an existing admin account. This can be used to bootstrap create the first admin user, or by automated tools to create new users without necessarily needing a service account.

Example:

registration:
  admin_pre_shared_secret: 38d7630f8c5bdb50df2d99e65fd0e60f
curl -H 'Authorization: Bearer 38d7630f8c5bdb50df2d99e65fd0e60f' \
    --json '{"localpart":"admin","admin": true}' \
    http://localhost:8008/_hammerhead/v0/admin/users/create

room_directory.admin_only

admin_only (boolean, optional): If enabled, only server admins are able to publish to the room directory.

When enabled, only admin users can publish rooms to the public room directory (i.e. room list). Note that when enabled, users can still create and use aliases, they just cannot publish them.

Example:

room_directory:
  admin_only: true

server_name

server_name (string, required): The name of this server.

This is not necessarily your domain name - it is the part of the ID that appears at the end of user IDs, and room aliases. For example, @user:matrix.example would have the server name matrix.example, even if traffic was ultimately served from hammerhead.matrix.example.

The server name can be an IP address (not recommended) or DNS name, optionally with a port. Typically, you will use a DNS name without a port here (you configure the port later).

Caution

You cannot change the server name after registering the first user.

Examples:

server_name: matrix.example
server_name: matrix.example:8448  # not recommended

static_room_directory

static_room_directory (mapping, optional): Configures the “static room directory”.

Defines all the rooms and their metadata that can be used to serve room queries over federation. Allows you to define resolvable aliases for rooms the server is not yet in. See also: https://github.com/tulir/mauliasproxy.

Important

This option has no effect if experiments.federation is not enabled!

static_room_directory is a mapping of { localpart: {room_id: '...', via: ['...']} }. The localpart here is the part after #, but before : (like with user IDs). room_id is the underlying room ID, like !example:example.net, and via is a list of server names that can help prospective members join the room.

Example:

static_room_directory:
  main:
    room_id: '!hammerhead-1:nexy7574.co.uk'
    via: ["nexy7574.co.uk", "synapse.nexy7574.co.uk", "asgard.chat", "corellia.timedout.uk", "starstruck.systems"]

In this example, #main:SERVER_NAME will resolve to !hammerhead-1:nexy7574.co.uk, and will tell remote servers asking about this room to join through any of the provided servers.

tls

tls (mapping, optional): Configures TLS options for TLS listeners.

Configures the certificate file and key file for serving TLS directly from listeners. You can generate a self-signed certificate with

openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out cert.pem

Example:

tls:
  cert_file: path/to/cert.pem
  key_file: path/to/key.pem

well_known

well_known (mapping, optional): Controls the values returned to the well-known helper routes.

well_known.client

client (string, optional): The base URL for client-to-server interactions.

Example:

well_known:
  client: https://client.matrix.example

Configuration Reference - Experiments

Some features in Hammerhead are gated behind “experiments”, sometimes referred to as “labs”, “feature flags”, “unstable flags”, and other related terminologies. These configurations control potentially experimental changes to behaviours in the server, imposed by Matrix Spec Proposals (MSCs).

In order to provide a more consistent experience, all experiments are disabled by default, and must be explicitly enabled in the configuration. However, when generating a new configuration file, some experiments may be enabled or pre-configured by default, but only when they are deemed to be relatively stable (i.e. won’t change much between now and their merge time), and provide an unquestionable benefit to the daily UX.

This document will detail every experimental proposal that Hammerhead supports, and their related configuration options.

Note

As these proposals are unstable, they may drastically change over their lifetime, which means Hammerhead’s implementation may fall out of sync, potentially even to an incompatible degree.

When an MSC is merged into the spec, the experimental option will be removed from the configuration. This will not raise any alarms during init, but you should keep an eye on release notes to know if you need to change any new configuration options.


Federation

Federation allows your homeserver to communicate with other homeservers, and vice versa. In its smallest form, it can be used for (for example) server1.localhost:8008 to communicate with server2.localhost:8008, or in the greater scheme, talk to any server on the public Matrix federation network.

Warning

Federation is not itself a spec proposal, but is a complex component of Hammerhead that is still under rapid development, and may be broken in unique and wonderful ways. Please do not enable this if you do not know what you are doing, or are not aware of the potential consequences of doing so.

You must compile Hammerhead with the non-default federation build tag in addition to enabling this experiment: HAMMERHEAD_BUILD_TAGS=federation ./build.sh. See compiling for more information on the build script.

  • enabled (boolean): When true, federation is bi-directionally enabled.
  • acl (mapping, optional): A server ACL event content that can be used to denylist or allowlist federation traffic. In addition to the standard content, there is also the hide_blocks boolean option, which when enabled, will return HTTP 500 to denied origins, instead of a 403. The default is to allow all servers.

Example:

experiments:
  federation:
    enabled: true
    acl:
      hide_blocks: true
      allow: ['*']
      deny:
        - badserver1.example
        - '*.bad.domain'

If you wanted to allowlist traffic, you would do such like so:

experiments:
  federation:
    enabled: true
    acl:
      allow: ['server1', 'server2']
      # do not deny: *, as deny overrules allow.

MSC4342

MSC4342: Limiting the number of devices per user ID limits the number of sessions (devices) to 30 per user. When a user has 30 devices, they will either be unable to log in and create new devices, or their oldest device will be deleted upon login to make “room” for the new device. They will always be able to resume existing devices.

By limiting the number of devices per user, the load on the server will be reduced, end-to-end encryption will become faster and more reliable, and it slightly increases the security of an account by reducing the number of sessions that may become compromised.

Warning

Automatically logging out old devices (auto_reclaim: true) may cause minor data loss for the user. Devices accumulate encryption keys, and if they are not connected to the server-side key backup, will lose them when deleted. This means that the user may lose some encryption keys, leading to “unable to decrypt” errors.

On the other hand, in the unlikely event that the user has lost access to all of their sessions, they will not be able to log out. In this case, they will either have to contact a server admin out-of-band for assistance, or reset their password, logging out all devices in the process. The same warning w.r.t. data loss as above applies, in this case.

  • enabled (boolean): When true, enables this experiment’s behaviour. Can be safely turned off later.
  • auto_reclaim (boolean): When true, the user’s oldest device will be logged out when they log in with a new one. Otherwise, their login attempt will be refused, until they manually log out elsewhere.

Admin API Reference

Hammerhead has a few internal “admin APIs” that can be used to manage the server.

Authentication

The admin API is authenticated by supplying a bearer token like with any other Matrix request, with the additional requirement that the token points at an account that has the administrator flag. The first account created is automatically given admin at the time of registration, however you can manually give any subsequent account admin by either:

  • Running UPDATE accounts SET admin=true WHERE localpart='foo'; via psql (localpart is the bit of the user ID between @ and :).
  • With an existing admin account, call update user with {"admin": true}

Pre-shared token authentication

Some endpoints, like create user, allow you to authenticate with the static “pre-shared token” defined in the configuration file: registration.admin_pre_shared_secret. The intent is that you can create the first account without needing to enable registration, and to allow automation without needing to delegate admin to yet another account (which increases your attack surface). The PSK may do more in the future.

Not all endpoints support this method of authentication. Those that do will state so explicitly.

Versioning

The admin API is locally versioned. Any time a breaking change is made to the functionality of a route, the version is increased. This does mean, however, that there can be any number of API versions active at once, such as /_hammerhead/v0/foo, /_hammerhead/v1/bar, and /_hammerhead/v9007199254740991/hello-world.

Consumers should always use the latest version, however removals will always be included in release notes, and historical versions will be kept around for as long as it is sensible to.

Important

Until v0.1.0 is released, this versioning is not respected.

Endpoints

Get Uptime

GET /_hammerhead/v0/uptime

Authentication: None.

Fetches the epoch from when the server started. “Started” refers to the unix timestamp at when the HTTP listeners were started, a.k.a. when the server became ready, not when the process itself started.

Response body:

{
    "started_at": 1775928780089
}
KeyTypeDescription
started_atnumberThe time the server started, in unix milliseconds

Get Version

GET /_hammerhead/v0/version

Authentication: None.

Returns full version metadata, including build date, commit hash, tagged version, OS architecture, etc. This is most useful for debugging and preparing issues, as it is very comprehensive.

Response body:

{
    "build_date": 1775928757000,
    "commit_hash": "c6a5ea0",
    "dirty": true,
    "full": "v0.0.1-dev+gc6a5ea0+dirty+d2026.04.11T17.32.37Z+go1.26.0@linux/amd64",
    "go_version": "go1.26.0",
    "latest_tag": "v0.0.0",
    "os_arch": "linux/amd64",
    "short": "v0.0.1-dev+gc6a5ea0",
    "tagged_version": ""
}
KeyTypeDescription
build_datenumberThe unix timestamp (milliseconds) when the running binary was compiled
commit_hashstringThe short commit hash the running binary was compiled with
dirtybooleanWhether the working tree was dirty when the running binary was compiled (uncommitted changes)
fullstringThe full version string. Used for issue reporting, as it combines all useful info into one string
go_versionstringThe Go compiler version the running binary was compiled with
latest_tagstringThe latest git tag available at the time of compile
os_archstringThe architecture the binary was compiled for (OS/ARCH)
shortstringThe short version string. Used in the server’s User-Agents
tagged_versionstringThe tag the commit_hash points at, if available. Otherwise, an empty string

Make PDU

POST /_hammerhead/v0/admin/make-pdu

Authentication: Admin account.

Creates a PDU with the given input. You currently can’t do anything with this, as the send-pdu counterpart has been temporarily removed.

If the room_version is omitted or empty, it will be fetched from the database. If a room version cannot be found, 404 / M_NOT_FOUND is returned.

room_id, sender, and content are the only required keys in the pdu object. Anything else required will be calculated on-demand. However, anything included in the base pdu will not be modified, meaning you can specify (for example) custom auth_events, prev_events, depth, and origin_server_ts values.

The event’s calculated event ID will be included under unsigned.

Request body:

{
    "dont_hash": true,
    "dont_sign": true,
    "room_version": "11",
    "pdu": {
        "content": {

        },
        "room_id": "!...",
        "sender": "@..."
    }
}
KeyTypeDescription
dont_hashboolean (default: false)If true, hashing and signing will not be performed on the PDU
dont_signboolean (default: false)If true, hashing will be performed, but signing will not
room_versionstring (optional)If creating a PDU for an unknown room, you can manually specify the version of the room. If omitted, the room’s version will be fetched from the database.
pduobjectThe base PDU object

Response body:

201 Created:

{
    "auth_events": ["$..."],
    "content": {

    },
    "depth": 1,
    "origin_server_ts": 123456789,
    "prev_events": ["$..."],
    "room_id": "!...",
    "sender": "@...",
    "unsigned": {
        "event_id": "$..."
    }
}

Refer to the “event format” section of the specified room_version’s documentation: https://spec.matrix.org/v1.18/rooms/. This endpoint always successfully responds with 201 Created.

Errors:

  • 400 / M_BAD_JSON: The request body is missing the pdu field, or pdu is missing some required keys.
  • 403 / M_FORBIDDEN: You did not authenticate, or are not a server administrator.
  • 404 / M_NOT_FOUND: room_version, auth_events, prev_events, or depth were not supplied, and the server could not fetch required data from the database to automatically populate these fields.
  • 500 / M_UNKNOWN: Hashing, signing, or event ID calculation failed.

Reload Configuration

POST /_hammerhead/v0/admin/reload-config

Authentication: Admin account.

Immediately reloads the server configuration by re-reading the configuration path, and clears internal caches.

Warning

Configuration reloading is not comprehensive, and you should usually restart the server entirely instead.

While Hammerhead components generally all refer directly to the same configuration reference variable, some values are disassociated from the configuration while initialising (such as cache sizes), meaning reloading the configuration may not update those values.

Furthermore, not all caches are cleared. Hammerhead has a LOT of moving parts, clearing EVERYTHING is not feasible.

Request body: None (ignored).

Response body (200 OK): JSON representation of the freshly loaded configuration file.

Errors:

  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 500 / M_UNKNOWN: There was an issue reloading the configuration. Typically, the result of an invalid configuration, or when the configuration is no longer found at the initial path.

Delete Media

DELETE /_hammerhead/v0/admin/media/{origin}/{media_id}

Authentication: Admin account.

Deletes a specific piece of media. If origin is the current server, it can be shortened to _. Media that does not exist always returns a successful response (unless the database returns an error).

Request body: None (ignored)

Response body:

200 OK:

{
    "ok": 1,
    "fail": 0
}
KeyTypeDescription
oknumberThe number of media files that were successfully deleted (in this case, always 1)
failnumberThe number of media files that could not be deleted (in this case, always 0)

Errors:

  • 400 / M_MISSING_PARAM: origin or media_id were missing or empty.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 500 / M_UNKNOWN: The server was unable to delete the media (database error, or the file system returned an error other than “file does not exist”, such as a permission error).

Purge Media

POST /_hammerhead/v0/admin/media/purge

Authentication: Admin account.

Purges media from the repository matching the specified criteria. This is irreversible and uninterruptible. Make no mistakes.

When multiple criteria are specified, they create an OR condition, not an AND. For example, providing all_remote_media=true and all_caches=true will delete all media that originated on another homeserver, OR is a cache file. This is no different for origin and user_id - if you supply all_remote_media=true and origin="_", this will delete media that is from another homeserver OR belongs to this homeserver.

Definitions

Remote media: Media that was uploaded to other Matrix homeservers, and was retrieved over federation.

Local media: Media that was uploaded directly to this homeserver.

Sparse media: Local media that was asynchronously created, but never had any content uploaded to it.

External media: Media cached from non-Matrix servers, such as URL preview images. Not associated with any user.

Tip

You typically do not need to manually clean up sparse media, as there is a clean-up task that runs every few hours, and one that also runs on server startup.

Caution

Media deletion is permanent and indiscriminate. As media is not currently related to events, there is no way for the server to know if a media file is a sticker, profile picture, custom emoji, room avatar, or just a normal attachment. It also has no way of knowing if the media has ever been used. Purging media will almost always have some undesirable collateral, so you should always think thrice before deleting local media. Deleting remote media is less bad, since the media can just be fetched again over federation, but only if the original server is still online.

The user filter will only work for local media, or for remote media retrieved from another Hammerhead server. Regular remote media does not include metadata about who created it, but Hammerhead transmits that information with non-specced metadata.

This operation uses limited concurrency to delete entries in parallel. Read more about how limited concurrency works at Configuration Reference § GOMAXPROCS. This is a blocking operation, if there is a lot of work to do, your request may time out.

Request body:

{
    "all_remote_media": true,
    "all_local_media": true,
    "all_caches": true,
    "all_sparse": true,
    "origin": "_",
    "user": "@baduser:example.com"
}
KeyTypeDescription
all_remote_mediaboolean (default: false)If true, all remote media (media from other Matrix homeservers) is deleted
all_local_mediaboolean (default: false)If true, all local media is deleted
all_cachesboolean (default: false)If true, all cached media is deleted
all_sparseboolean (default: false)If true, all “sparse” media entries are deleted
originstring (default: empty)If not empty, delete all media that originates from the specified server
userstring (default: empty)If not empty, delete all media that was uploaded by the specified user

Response body:

200 OK:

{
    "ok": 420,
    "fail": 69
}
KeyTypeDescription
oknumberThe number of media files that were successfully deleted
failnumberThe number of media files that could not be deleted

Errors:

  • 400 / M_NOT_JSON: The request body was not valid JSON.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 500 / M_UNKNOWN: There was an error querying the database to fetch eligible media entries.

Delete Room

DELETE /_hammerhead/v0/admin/rooms/{room_id}

Authentication: Admin account.

Deletes a room from the database. First tries to remove all local members (leave, decline pending invites, rescind pending knocks), then tries to delete as much data associated with the room as possible.

If force is not true, and there is an error during any stage of the operation, it aborts immediately. Actual data deletion is performed in a transaction, meaning if it fails, no data is deleted. However, membership changes are not transactional, and are always committed immediately, non-atomically.

Tip

You typically do not need to delete rooms to reclaim space from abandoned rooms. When all local members leave a room, it is automatically deleted, and as such does not need to be manually “cleaned up”.

Request body:

{
    "force": true
}
KeyTypeDescription
forceboolean (default: false)If true, errors during evacuation and deletion are ignored where possible

Response body (200 OK): Empty object (literally {}).

Errors:

  • 400 / M_NOT_JSON: The request body was not valid JSON.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not an administrator.
  • 429 / M_LIMIT_EXCEEDED: There is already a room delete operation in progress (cannot have more than one at a time).
  • 500 / M_UNKNOWN: There was an unrecoverable error while evacuating or deleting the room.

Create User

POST /_hammerhead/v0/admin/users/create

Authentication: Admin account or pre-shared token.

Creates a new account with the specified pre-filled criteria.

The localpart must be a valid Matrix localpart, see the specification. On the other hand, password is not subject to the same validation that would normally be applied during registration, so low-entropy/too short passwords can be set here. It is recommended you don’t do that, though.

If password is omitted, the created user will not be able to log in (but admins can still create access tokens for them, so the account isn’t useless).

Request body:

{
    "localpart": "username.here",
    "password": "$ecureP4sswordH3re!"
}
KeyTypeDescription
localpartstringThe user’s localpart (the part of the user ID between @ and :)
lockedboolean (default: false)If true, the account will be locked upon creation
suspendedboolean (default: false)If true, the account will be suspended upon creation
passwordstring (optional)The account’s desired password

Response body: 201 Created:

{
    "account_id": 2
}
KeyTypeDescription
account_idnumberThe numberic ID of this account (in the database)

Errors:

  • 400 / M_NOT_JSON: The request body was not valid JSON.
  • 400 / M_INVALID_USERNAME: localpart is empty or invalid.
  • 400 / M_USER_IN_USE: localpart is already in use, or is reserved for another reason (e.g. service account).
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 403 / M_FORBIDDEN: You did not provide an administrator access token, and shared-secret token authentication is disabled, or the provided token did not match.

Deactivate Account

POST /_hammerhead/v0/admin/users/{user_id}/deactivate
  • user_id: Fully qualified user ID (@foo:bar.example), or localpart (foo).

Authentication: Admin account or pre-shared token.

Immediately deactivates an account. If the account is already deactivated, makes no change.

Performs the following steps:

  1. Mark account as deactivated (but not erased). Prevents further use of account immediately.
  2. Remove account’s administrator flag.
  3. If redact is true, issue redactions for every event the target ever sent.
  4. Mark account as deactivated again, this time respecting the GDPR erased flag.
  5. Deletes all profile data the target set.
  6. Deletes all account data the target set.
  7. Removes all devices (sessions) the target had.
  8. Then, with limited concurrency, for each room the target was a member of:
    • If the target was invited to the room, reject the invite
    • If the target was knocking on the room, rescind the join request
    • If the target was joined to the room, leave the room

This is a blocking operation, your request may time out.

Request body:

{
    "erase": false,
    "redact": false
}
KeyTypeDescription
eraseboolean (default: false)If true, mark the account as GDPR erased
redactboolean (default: false)If true, issue redactions for every event sent by the user.

Response body (200 OK): Empty object (literally {}).

Response body (304 Not Modified): No data.

Errors:

  • 400 / M_NOT_JSON: Request body was not valid JSON.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 403 / M_FORBIDDEN: You did not provide an administrator access token, and shared-secret token authentication is disabled, or the provided token did not match.
  • 404 / M_NOT_FOUND: The requested user does not exist or does not belong to this server.
  • 500 / M_UNKNOWN: An unrecoverable error was encountered during deactivation.

List Users

GET /_hammerhead/v0/admin/users

Authentication: Admin account.

Lists all users registered on this homeserver. Currently, filtering, sorting, and backwards pagination are not supported. Results are ordered newest → oldest.

Query Parameters:

ParameterTypeDescription
sincenumberThe end token of the previous page (from next_batch).
limitnumber (default: 1)The maximum nmber of results to return. Cannot be less than 1, cannot be more than 1000.

Response body:

200 OK:

{
    "chunk": [
        {
            "entry_id": 1,
            "localpart": "foo",
            "admin": true,
            "locked": false,
            "suspended": false,
            "deactivated": false,
            "erased": false,
            "created_at": 1775660510816
        }
    ],
    "next_batch": 1
}
KeyTypeDescription
chunkArray[Account] (optional)An array of up to {limit} accounts
next_batchnumber (optional)The next batch token to pass to since, if there are potentially more results

Account:

KeyTypeDescription
entry_idnumberThe numeric database ID of this account
localpartstringThe localpart of this account
adminbooleanThe administrator status of this account
lockedbooleanWhether this account is locked
suspendedbooleanWhether this account is suspended
deactivatedbooleanWhether his account has been deactivated
erasedbooleanIn combination with deactivated, whether this account is GDPR-erased
created_atnumberUnix timestamp (in milliseconds) when this account was registered

Errors:

  • 400 / M_INVALID_PARAM: since or limit were not numbers.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.

Update User

PATCH /_hammerhead/v0/admin/users/{user_id}
  • user_id: Fully qualified user ID (@foo:bar.example), or localpart (foo).

Authentication: Admin account or pre-shared token.

Updates any of the provided attributes on the account. If an attribute is not provided in the request body, no change is made to the relevant account attribute. As such, all request keys are optional and may be omitted.

Request body:

{
    "password": "foobar",
    "admin": true
}
KeyTypeDescription
passwordstringThe new password for this user
adminbooleanSets the administrator flag for this user

Response body (200 OK): Empty object (literally {}).

Errors:

  • 400 / M_NOT_JSON: The request body was not valid JSON.
  • 403 / M_FORBIDDEN: You forgot to authenticate, or are not a server administrator.
  • 403 / M_FORBIDDEN: You did not provide an administrator access token, and shared-secret token authentication is disabled, or the provided token did not match.
  • 404 / M_NOT_FOUND: The requested user does not exist or does not belong to this server.

Process

Anyone can hack on Hammerhead - if you want to see something implemented, get stuck in! Hammerhead is built with maintainability at the forefront of its design decisions, meaning that almost all the codebase is easy to read, understand, and intuitive to build on. You don’t need to have a decade’s experience with Golang to get something done.

In order to contribute, all you need is a code editor (JetBrains’ Goland is recommended, but Visual Studio Code with the Go extension will work fine too). You will also need git, and obviously, the latest version of Go. You will also want to have an account on Codeberg, so that you can open a pull request with your changes.

Setting up the development environment

First off, you’re going to want to acquire a copy of the Hammerhead source code. You can clone the repo however you want, however your code editor will usually have a UI option for it somewhere. Make sure you clone the HTTPS url, https://codeberg.org/timedout/hammerhead.git - cloning the SSH URL will likely give you a “forbidden” error, unless you have push access to the repository (in which case you won’t need this guide).

Tip

Codeberg can be unreliable at times, so it’s recommended you include a mirror as a secondary remote. You can do this with the below command:

#git remote add <mirror-name> <mirror-url>
git remote add nexy-forge https://git.nexy7574.co.uk/nex/hammerhead.git

Then, if Codeberg is ever unreachable, you can pull missed changes from an up-to-date mirror instead.

Do not submit issues or pull requests to mirrors - if the primary repository is unavailable, please either wait until it returns, or coordinate sharing patches with maintainers in the Matrix room.

You’ll then want to make sure you can install all the dependencies required for devel. You can do this with go mod download. This may take a while, as Hammerhead has a complex dependency tree.

Finally, to make sure you’ll be able to test your changes, run the build script with ./build.sh. Resolve any errors you encounter, or reach out for help if you can’t.

Now you’re all set to start making changes!

Pre-commit

You will want to install one of either pre-commit or prek in order to run some pre-commit hooks. These hooks will run when you run git commit (or use the associated functions in your IDE), and ensure that you don’t push code that looks/behaves badly or that fails unit tests.

Use either pre-commit install or prek install to install the hooks, and run pre-commit run -a or prek -a to run the hooks manually (otherwise they’ll be run before commit, as the name implies).

If you don’t use these hooks, CI/CD may fail on your pull request, meaning it won’t be eligible for merge.

Making your changes

First of all, you’re going to want to fork Hammerhead into your own repository, so that you can push your changes before opening a pull request. You will want to add this as a remote too, such as git remote add fork ssh://git@codeberg.org/myusername/hammerhead (you can also use HTTPS, but SSH is easier if you have MFA enabled).

Then, before you make any changes, you’ll want to switch to your own branch. Working on main will only introduce merge conflicts, and your pull request will be rejected if you try to merge your main into the upstream main. The recommended branch names are myhandle/scope/name:

  • myhandle should be your Codeberg username (but can be any handle that is unique and ideally identifiable to you).
  • scope should be one of the conventional commit scopes (such as fix, feat, docs, style, perf).
  • name should be a short, sensible self-explanatory name, such as support-msc1234.

This naming convention will allow you to avoid branch/namespace conflicts and makes it easier to find related branches later down the line.

Create a new branch from main like so: git checkout -b myhandle/scope/name origin/main.

Then, when pushing, git push -u <remote> myhandle/scope/name. You can omit all those arguments for subsequent pushes, simply git push. remote will probably be fork, as per the previous example.

Make sure that you test your changes as you go. Ideally, write unit tests to prevent regressions in the future (although this isn’t mandatory). While WIP PRs are acceptable, please don’t open PRs that aren’t near completion, or that you don’t plan on maintaining. Realistically you should only have one or two PRs awaiting review at a time, otherwise you may overburden yourself.

Also: try to break what you write. Don’t just test the best-case scenario, throw some absurd situations and inputs at it and ensure that it doesn’t behave unexpectedly. If you don’t try to break it, somebody else (potentially with less wholesome goals) will.

Documentation

Most of the time, if you’re changing something user-facing, you will need to update the documentation. For example, updating the configuration (in any form) will require changing the configuration reference documentation at the very least.

Hammerhead’s documentation uses [mdbook] (install), and compiles a bundle of Markdown (ref) files into HTML.

You can live preview your changes with mdbook serve and visiting https://localhost:3000.

When you submit a pull request that modifies anything under the docs/ directory, the Deploy documentation CD workflow will run. When running against a pull request, this workflow does not deploy the documentation anywhere (that only happens for commits to dev), but will produce an artefact containing the built documentation. You can download that artefact and unzip it to see the generated content, should you wish. Hosting documentation previews is planned.

Submitting your changes

Once you’re happy with your work, make sure you’ve pushed it to your fork, and head on over to https://codeberg.org/timedout/hammerhead/pulls. Navigate to “New pull request” in the top right, and out of the two drop-down boxes presented under the title, select the right-hand one, and plug in your branch’s name (you may have to hunt for it).

You should get the chance to preview the diff that will be presented to reviewers before continuing. Make sure that it looks like the changes you expect it to, and then activate “new pull request”.

The pull request title should be short (<50 words) and explains what your pull request does at a glance. If you cannot fit the title in under 50 or so words, chances are your scope is too large, and your pull request should be broken down in to several, smaller pull requests.

In the pull request’s description, describe the changes you made in more detail, including what you changed and why you decided to change it. If possible, also link to any open issues that relate to your pull request, and include any other context that will help others in the future determine why you made any changes.

Once happy, hit “create pull request”. The Matrix room will be notified that you have opened a pull request, so sit back and relax - a maintainer will triage and review your pull request when they get time.

Tip

Maintainers may wish to slightly tweak your pull request before merging it, typically to fix minor issues or rectify code style problems. While allowing maintainer edits is enabled by default, you have the option to turn it off. It is advised you leave it enabled if you would like your pull request to be merged faster.

On the contrary, please don’t expect maintainers to do your PR for you. If changes are requested, it is your responsibility to act on that request. PRs that go untouched for a long time may be closed due to inactivity at the maintainers’ discretion.

How will my PR be merged?

If your pull request receives an approval, it will likely be merged shortly after. If CI is still running/pending, the PR will be set to auto-merge once that process completes.

Merges in Hammerhead are typically done via rebase + fast-forward, since this retains the exact commit order specified in the pull request, which makes it easier to track down previous changes atomically. However, pull requests that have a lot of commits, or poor commit messages, will instead be squashed. Squashes lose the history and some metadata, but allow your PR to be merged into the main branch in one single, monolithic commit instead (you still get attribution). In some special cases, a maintainer may manually merge your pull request. How that happens is situation-dependent.

If you want your pull request to be merged in a specific way, please denote that in your pull request description. The only merge method that is forbidden is merge commits.

What are all these checks?

When you open a pull request, you may notice a few checks (also referred to as workflows, CI/CD, jobs) are pending. If you have had a pull request of yours merged before, these will likely start running straight away (assuming there’s an available runner). Otherwise, you will need to wait for a maintainer to hit approve, which typically happens after triage and after a maintainer has made sure you haven’t requested malicious changes.

If you made changes to anything under the cmd/ or pkg/ directories, the “build” checks will be invoked. These checks involve compiling, testing, and linting your changes to make sure that they are functional and roughly meet the code quality standards. Unless otherwise stated, these checks must pass in order for your PR to be eligible for merge. The Build / binaries check will also produce AMD64 and ARM64 binaries as artefacts upon success - you can download these (zipped) by clicking on “Details”.

If you made changes to the docs/ directory, the “deploy” checks will be invoked. These checks will handle incoming documentation changes for you. Deploy / docs specifically will also produce an artefact containing the built documentation (again, zipped) for the associated commit. See also: Documentation.

Conclusion

If you have any questions not answered in this document, please ask them in our Matrix room!

Code style

Hammerhead utilises a free-form yet still clear style for its source code. As a rule of thumb when writing code, you should ask yourself “if this was my first time seeing Golang, would I be able to understand what is going on here”. If the answer’s no, you can probably clean it up. This document will give you some guiding advice on how to write acceptable, clean code for Hammerhead.

Formatting & linting

Hammerhead uses golangci-lint with a number of formatters and linters configured (see: .golangci.yaml). This is extensively configured to enforce specific code idioms, and catch signs of bad code patterns early.

While writing code, it is recommended you run golangci-lint run and golangci-lint fmt occasionally. pre-commit (covered in TODO) will also run these before letting you commit.

If you are not familiar with writing high-quality/production-ready Golang, consider giving Effective Go - The Go Programming Language (go.dev) a read. It’s very long, but sets a very good baseline and will help you understand the differences in style between Golang and other languages you may be used to writing.

Configuring your IDE

If your IDE supports it, you should change your formatter & linter to golangci-lint:

GoLand

JetBrains’ GoLand supports this via Settings (ctrl+alt+s) -> Go -> Linters - enable both Execute golangci-lint run and Execute golangci-lint fmt (you may also want to increase the parallelism). Do not enable “Only fast linters”.

Visual Studio Code

Visual Studio code supports this via Settings (ctrl+alt+s) -> Extensions -> Go. Set Lint Tool to golangci-lint-v2.

In order to configure golangci-lint as a formatter, you will need to edit settings.json and use the below config:

{
  "go.alternateTools": {
    "customFormatter": "/path/to/golangci-lint run"
  }
}

And then set Settings / Extensions / Go / Format Tool to custom.

General rules

When writing code for Hammerhead, keep in mind:

  • Names (functions, variables, etc.) should be CamelCase (public) or lowerCamelCase (private).
  • Comments should be concise but descriptive. Don’t write essays, but explain your decisions.
  • Code should be “self documenting” to the greatest extent it realistically can be.
  • Use constant values over literal values where possible. For example, http.MethodGet over "GET". This reduces the chance for unnoticed typos to slip into the code.
  • Hammerhead uses the British English locale where possible (en_GB et al.).

Readable declarations

Declaring multiple, related variables should be done in a single var/const statement:

func main() {
 // Bad!
 x := 0
 y := "awawawa"
 z := false
 // Good!
 var (
  a int
  b = "awawawa"  // no explicit type needed for non-zero inits
  c bool
    )
}

Furthermore, zero-value initialisations should be done with var, not type{}:

func main() {
 // Bad!
 x := myThing{}
 // Good!
 var x myThing
}

References and values

Hammerhead is expected to operate in a thin environment, where high performance/abundant resources may not be available. While the computer is smart, it still has to do what you tell it to do. If you tell it to copy values everywhere, it will. As such, you should generally pass variables around as pointers where it makes sense to do so. Usually, that’s any time you’re passing something that isn’t a primitive.

Passing values of types int, string, rune, bool, and floatX as values is generally fine since they typically only use a couple bytes of memory anyway. Never pass slices or maps as pointers - maps and slices are both “reference types”, meaning while they are not pointers themselves, they simply reference a pointer to their underlying value. x := make([]int, 3); doThing(&x) is the same as x := make([]int, 3); doThing(x).

Small structs are also usually fine to pass around by value. “Small” in this context is usually a struct with up to three flattened primitive fields. Anything more than that, or a struct instance you’re expecting to live for a while, consider it a large struct.

Structs should (excluding the above “small” exclusion definition) practically always be passed around as references. If you can, you should just initialise the struct as a reference (i.e. foo := &myType{X: Y}). The only time you shouldn’t create an immediate pointer is if you are creating a zero struct, in which case the previous declaration rule applies (zero-values should be initialised with var).

Tip

You should not need to dereference pointers when passing their data to a function to avoid mutating the data. Functions you’re calling should instead be written to avoid mutating the input unexpectedly, returning the data they want the caller to apply instead. If mutation cannot be avoided for some reason, cloning the value is acceptable.

Error Handling

Casing

Errors must be lowercase (not Uppercase or Title Case), and must not end in punctuation (like a period). Good errors look like failed to do thing: operation timed out, bad errors look like Failed to do thing: operation timed out.. This is because callers that are logging errors may end up wrapping them, which can result in log lines such as Failed to create room: Failed to create room resolver: unsupported room version "1"...

Logging before returning

Do not log errors before returning them. If you want to include more context, wrap the error:

// Bad!
func main() {
 logger := zerolog.Ctx(ctx)
 result, err := myFunction(ctx)
 if err != nil {
        logger.Err(err).Msg("Operation failed!")
        return err
    }
 // ...
}

// Good!
func main() {
    logger := zerolog.Ctx(ctx)
    result, err := myFunction(ctx)
    if err != nil {
        return fmt.Errorf("operation failed: %w", err)
    }
    // ...
}

Bubbling errors

In order to not lose context as errors propagate their way up the caller chain, you should wrap them where possible. Using the %w format specifier with fmt.Errorf allows you to wrap the underlying error value, allowing receivers to then use errors.Unwrap to get the underlying error, and correctly detect certain error subtypes with errors.Is.

If you want to return a root-level error that does not propagate another one, errors.New should be used instead.

Tip

Any root-level errors should be publicly exported variables, so that they can be detected as types by other callers.

var OperationalError = errors.New("operation failed")
func myFunction() error {
 // do things here
 return OperationalError
}

func mySecondFunction() error {
 err := myFunction()
 if err != nil {
  return fmt.Errorf("failed to do operation: %w")
    }
 // ...
}

func main() {
 err := mySecondFunction()
 if err != nil {
  // Because mySecondFunction wrapped the error from myFunction, that can be detected here:
     if errors.Is(OperationalError, err) {
      println("Operational error! " + err.Error())
        }
    }
}

Panics

If the building is on fire, panic! In Hammerhead, panics are an expected condition on a few layers of the program, and will be caught accordingly - only the worst of the worst panics will cause the program to crash.

While you should prefer returning errors directly, sometimes a panic is also an acceptable condition. If you think the call chain cannot proceed without severe consequences (potentially even later panicking itself), or have encountered an impossible scenario, panicking is totally acceptable. Keep in mind that panics are ugly - they often produce long, unreadable stack traces, or accidentally lose their context along the way during propagation and become even harder to debug.

Panics are explicitly handled at these layers (top-down):

  1. runtime (cmd/hammerhead/hammerhead.go) - “catch-all”, ensures database can shut down cleanly before propagating to a crash.
  2. router/server - Catches any panic that is not captured during handle, or subsequent calls to write404 and the passthrough serve.
  3. router/handle - Catches any panic that is raised during the handle function’s execution, ensuring metrics are always saved, the panic is logged, and a response in some form is written to the requesting client.

That third handler will be the one that is hit most often - any panic thrown while handling a request will be caught by that panic handler, and will thus limit the “blast radius” to just that request.

Caution

Panics in goroutines that themselves do not have an explicit recover() call will cause the entire process to hard crash regardless of these above handlers. Always ensure you recover() in spawned goroutines, unless you are sure the goroutine’s calls cannot panic.

Hard crashes should be avoided at all costs, as they can leave the database and related external components in an unsafe inconsistent state, which may cause corruption or further operational errors after subsequent restarts.

Logging

Hammerhead uses zerolog for high-performance structured logging. Loggers are accessed by fetching them from the current context.Context via zerolog.Context(context.Context).

Unlike other log libraries you may be used to, there isn’t really a concept of “named loggers”, “instruments”, or generally identifiable sub-loggers. This means you need to be very careful with logging - don’t be too noisy, since you can’t be silenced, but don’t be too quiet, otherwise you might go unheard.

You should use the following levels only when:

  • Panic(): Never
  • Fatal(): Never
  • Err() / Error(): An unexpected error condition was encountered, and while the program is going to try to continue, the stability of the runtime may be in question, or the caller may produce equally unexpected behaviours. An example of this may be a database query failing or a file on disk is unexpectedly missing.
  • Warn(): An expected error (or otherwise odd) condition was encountered. While the routine is going to handle this, it may be indicative of an underlying problem, one that the operator may want to investigate. For this reason, it can also be used to draw the operator’s attention. An example of this may be attempting to handle an effect on a resource we don’t know about (such as trying to persist an event to a room we aren’t joined to).
  • Info(): An expected condition was encountered that the server operator may find interesting. Generally this is used to log when the server is doing something important that may potentially be interesting, such as: big operations like loading a room, starting an HTTP listener, or reporting the progress of a long-running/complex operation.
  • Debug(): Information that is generally uninteresting to the average person, but may be interesting to developers, or operators trying to actively troubleshoot a problem. Such logs should contain context regarding what triggered their log. An example of this could be logging that the program is going to perform an operation (like a request), or timing information regarding an execution.
  • Trace(): Information that is typically only useful while developing, to allow tracing function calls and decisions made by the program. This is the most verbose level and is expected to be such.

You should never use Msgf or related formatting functions - always put context in log fields:

  • logger.Info().Msgf("I did a thing: %s", foo)
  • logger.Info().Str("foo", foo).Msg("I did a thing")

This is because formatted logs are more difficult to use than structured logs, and ultimately devalue the log itself.

You should also ensure that log messages are always capitalised and do not end with punctuation.

When a type has a String() function (i.e. it implements the Stringer interface), you should use logger.Stringer("thing", myThing) instead of logger.Str("thing", myThing.String())

When logging an error, you should just use logger.Err(err) instead of logging.Error().Err(err), unless you want to use a log level other than Err, such as logger.Warn().Err(err).