Skip to main content

Backups

A cross-cutting concern of Security, not a maintenance process by itself — actually managing backups and recovery is Maintenance's job, but the reasoning behind it belongs here: get it wrong and every other security practice is just delaying the day something unrecoverable happens.

Concepts​

There are a few concepts worth noting when thinking about backups, each of these solves a different problem:

  • Blast radius: how broad the failure is. Is it a bad update to a service, mismanagement of the server, a hack of the entire system? Or something more disastrous, like the datacenter burning down? You can guard for all of these, to be like 99.99999% safe, for a price.
  • Consistency: if you copy a database while it is running you risk tearing up the files. Getting an atomic copy needs either the source's own mechanism for doing this cleanly, like access to the database itself, or a volume manager's snapshot tool. Otherwise you will need to freeze the source first, shutting the server down for a bit.
  • Coverage: a backup only protects what it actually captures. Backing up a volume that doesn't actually contain the data you care about gives a false sense of security. You need to know what you actually want to back up, which is a hard problem with self-hosted services.
  • Frequency and monitoring: coverage is really only about what you back up, not how often. Your recovery point objective (RPO) is another factor to consider, and it has a straight up tension with cost. This is also where databases become a real pain to do well yourself, for highly valuable services you'd need continuous backing up. You also need to monitor that the backup actually runs.
  • Space and price: storage isn't cheap. Backing up stuff that doesn't need backing up costs money and is polluting. Most of the stuff on the servers you will be running will be reproducible data, only a small bit will be meaningful data in databases.
  • Process and speed of recovery: after a failure you need to recover the service, and some ways are significantly faster than others. You have to also make a decision about how quickly you want to recover after a failure, but also be mindful that fast recovery from an unlikely event is costly.
  • Security: backups can contain all the data you care about, but can also include data you didn't want backed up, like various credentials. This way the backup itself can become an attack vector that you need to manage. Moreover, you also need to mind that if you're protecting against ransomware, the backup cannot be corrupted by the same hack.

These concepts keep returning in whatever backup solution you pick — none of them are perfect.

Practical rules​

Handy shorthand for the problems above, not separate ideas:

  1. The 3-2-1 rule is blast radius made concrete: three copies of anything that matters, on two different mechanisms, one genuinely off-site — so a single failure mode can't take all of them out together.
  2. Organize by intent: you need to actually plan for what you want to safeguard, not rely on a magic button offered by vendors. For this, you actually need to understand what you maintain.
  3. Credentials are their own concern, not a backup detail: a perfect backup of encrypted data is worthless without whatever decrypts it. Credentials also form a huge security risk if backed up unencrypted.

Recovery intents​

Infrastructure intent​

Discussion​

Cloud providers offer backup options tied to the infrastructure. This is a bit of a mixed bag. They are handy and often well-made for the price. But they come with important constraints, most importantly that they're tied to the infrastructure. Providers give a certain promise of safety from huge blast radiuses and in some cases pretty good consistency, but they always lack coverage, because you didn't plan for it. They back up only that server, and they're backing it up fully, even the reproducible bits.

There are problems with this:

  1. The most well-made backup functionalities are proprietary features of the cloud providers that lock you into that particular provider. This is a huge problem for an public stack project driven by ideals. You're sacrificing alot of ideals in provider choice and in open source infrastructure for some redundant assurances.
  2. The server for the most part does not hold meaningful information and should ideally be stateless. It holds highly reproducible stuff that needs to be there for an app to work, but has no uniqueness to it. You're paying for — and polluting by — backing up stuff that doesn't need a backup. This has a huge scale to it, most likely you'll be storing lots of redundant docker images. In my last test I was at 50gb for precisely 0 bytes of valuable data.
  3. The things you do want to back up, like the databases, are also hardest to back up with consistency. Only if you have access to the block storage service provider can you do this well without shutting the server down temporarily for backup. Otherwise you simply need to be able to ask the database for a data dump, needing much more specific knowledge of what's running on the server.
  4. It's questionable how much of the data is (or should be) living on block storage. It's the most expensive type of storage. It should only be used for data that needs to be highly available. The rest you put in object storage, outside of the volume, leading to coverage problems. You will not get a backup of object storage included in an infrastructure backup, because you lack intent.

Many solutions offer a false sense of security, often from a huge blast radius like nuclear bomb levels, for a price, while at the same time not backing up the files you wanted covered. Other than spending a lot on cloud services, which we try to avoid, there is no real way out of planning for all of these problems.

Infrastructure backup also does not solve a whole lot of problems:

  • The use case is pretty specific: you need to have a failure at the server level and need to recover from it quickly. This is probably the most reliable bit of the entire architecture, since we're buying infrastructure from a professional datacenter. In fact, relying on this form of backups introduces the problem in that you need to recover all services at once, rather than individually.
  • Booting up new infrastructure after a failure is not hard, especially since we're using infrastructure as code. This trains reproducibility. You can quickly start up an entire new stack and configure it quickly if you have the service data safeguarded. It makes more sense to optimise this even more than to safeguard redundant data that is possibly in an inconsistent state.

In fact, it might create a whole new bunch of risks:

  • It all but secure. Infrastructure backups contain not only the actual data, but most likely also security credentials, because you back up at a level that needs to be able to decrypt the data. This means that the backup itself needs to be secured heavily. You can avoid this, but that adds an additional layer of complexity in credential management.
  • It still needs planning, maintenance, monitoring and a recovery procedure. You need to retain awareness that these backups are effective, else you're lulled into false security once more. Yet now you're now maintaining a backup solution that you know isn't particularly fit for the job and paying for it too.

Solutions​

The backup solutions at the infrastructural level all come with coverage problems, some add consistency problems. None of these can be run frequently, without incurring prohibitively high costs. They differ a bit in blast radius, recovery and security.

At the cloud level: from the storage backend's own block storage provider, that takes the snapshot without needing anything to freeze first:

  • Provider-managed: like Hetzner's automated backups — a snapshot-based solution operated for you. It has a real cost: a flat 20% on top of the server price. Moreover, you are now locked to Hetzner.
  • Self-managed: orchestrating volume snapshots directly — it isn't too much work to write a few commands to schedule and prune OpenStack snapshots. Far cheaper, and also serves the unusual case of providing quick server recovery.

From within the server: a filesystem-walk tool like restic reading a live, changing filesystem. This works well for flat files, but you need to shut down databases to get a genuinely consistent result:

  • Provider-managed: like Infomaniak's Swiss Backup — an agent installed on the server itself, not a block storage provider mechanism. Covers a huge blast radius, but it's also expensive.
  • Self-managed: like Backrest to object storage — the same shape, just self-hosted instead of a vendor's own agent. The blast radius you protect against is smaller, but it does the same job.

Note that there is a third option of infrastructure backup, declarative configuration, which is already partially implemented. If the infrastructure is presumed to be stateless beyond the service backups, infrastructure as code serves as a declarative way of backing it up. The Docker Compose files already declaratively define the service infrastructure. This could be further expanded by making sure that the basic steps within the deployment platform are described in code as well. The documentation itself already serves this purpose quite closely, with a bit of LLM help; even though there is some degree of drift. In the current state, the degree of declarative configuration and documentation yields a pretty determinative result.

Service intent​

Service recovery is much more specific and much more structured by intent than infrastructure recovery. You have to plan for what you wish to safeguard and from what failure. This is much more fine-grained and specific, you choose which databases and files you care about. This also allows you to run much more frequent backups for a lower cost. You can also safeguard from a much larger blast radius because you control the destination of the backup.

At the deployment platform level: Coolify is an interesting tool in this regard. They've just launched pretty cool backup features. It also knows which resources are databases, and can ask databases for a dump to ensure consistency. It does not solve the coverage problem fully though, if data lives in object storage rather than on the volume, you still need a secondary backup solution.

From within the service itself: a service backing up itself understands its own data model better than anything watching from outside it. WordPress's native backup tools are the clear case: they grab both the database and whatever's referenced in S3-managed media in one step, resolving exactly the kind of external reference a database-only backup would miss entirely, and solve both the consistency and coverage problem that way.

Personally, I think this is the way to go. You don't want to back up containers with apps that are fully reproducible. Nor do you want to back up all years of logs you've accumulated. You want the actual meaningful data.

Credential intent​

The backup data is useless without the credentials that unlock it. This is the credentials rule above as its own layer rather than a footnote: a perfect Infrastructure or Project recovery is worthless without whatever decrypts what it restores, so this layer's job isn't copying data at all, it's making sure there's still a way in once the primary path to a credential disappears.

Coolify itself holds most of these credentials live — every service's environment variables, in its own database, encrypted under APP_KEY. Password managers like Proton Pass play a narrower role: a manual backup of the small set of keys (APP_KEY, SSH keys, database passwords) needed to actually decrypt that database if the server is lost, not the primary store itself. Deeper integration with something like Infisical could centralise credentials properly and make their usage auditable.

Evaluating a new service​

Not a one-off analysis — the same questions apply every time a new service gets added to this stack:

  1. Does it hold state at all? A stateless service needs nothing here — there's nothing to lose that redeploying doesn't already restore. Do we care about that state?
  2. Where does its data actually live? On the volume, or referenced from object storage it doesn't own — a coverage problem? If the latter, confirm what actually gets captured before assuming coverage.
  3. Does it have its own native backup solution? One that understands its data model — a database dump, an app-level export — beats a blind copy every time it exists.
  4. If not, does Coolify's own per-resource schedule cover it? That's the default before reaching for anything heavier.

Chances are there is not much left after this that is meaningful. The rest is just data we don't care about, lots of logs, and reproducible data.