It’s fairly obvious why stopping a service while backing it up makes sense. Imagine backing up Immich while it’s running. You start the backup, db is backed up, now image assets are being copied. That could take an hour. While the assets are being backed up, a new image is uploaded. The live database knows about it but the one you’ve backed up doesn’t. Then your backup process reaches the new image asset and it copies it. If you restore this backup, Immich will contain an asset that isn’t known by the database. In order to avoid scenarios like this, you’d stop Immich while the backup is running.
Now consider a system that can do instant snapshots like ZFS or LVM. Immich is running, you stop it, take a snapshot, then restart it. Then you backup Immich from the snapshot while Immich is running. This should reduce the downtime needed to the time it takes to do the snapshot. The state of Immich data in the snapshot should be equivalent to backing up a stopped Immich instance.
Now consider a case like above without stopping Immich while taking the snapshot. In theory the data you’re backing up should represent the complete state of Immich at a point in time eliminating the possibility of divergent data between databases and assets. It would however represent the state of a live Immich instance. E.g. lock files, etc. Wouldn’t restoring from such a backup be equivalent to kill -9 or pulling the cable and restarting the service? If a service can recover from a cable pull, is it reasonable to consider it should recover from restoring from a snapshot taken while live? If so, is there much point to stopping services during snapshots?
You start the backup, db is backed up, now image assets are being copied. That could take an hour.
For the initial backup maybe, but subsequent incrementals should only take a minute or two.
I don’t bother stopping services, it’s too time intensive to deal with setting that up.
I’ve yet to meet any service that can’t recover smoothly from a kill -9 equivalent, any that did sure wouldn’t be in my list of stuff I run anymore.
It depends on the dataset. If the dataset itself is very large, just walking it to figure out what the incremental part is can take a while on spinning disks. Concrete example - Immich instance with 600GB of data, hundreds of thousands of files, sitting on a 5-disk RAIDz2 of 7200RPM disks. Just walking the directory structure and getting the ctimes takes over an hour. Suboptimal hardware, suboptimal workload. The only way I could think of speeding it up is using ZFS itself to do the backups with send/recv, thus avoiding the file operations altogether. But if I do that, I must use ZFS on the backup machine too.
I’ve yet to meet any service that can’t recover smoothly from a kill -9 equivalent, any that did sure wouldn’t be in my list of stuff I run anymore.
My thoughts precisely.
Oooh yeah I can imagine RAIDz2 on top of using spinning disks would be very slow, especially with access times enabled on ZFS.
What backup software are you using? I’ve found restic to be reasonably fast.
Currently duplicity but rsync took similar amount of time. The incremental change is typically tens or hundreds of files, hundreds of megabytes total. They take very little to transfer.
If I can keep the service up while it’s backing up, I don’t care much how long it takes. Snapshots really solve this well. Even if I stop the service while creating the snapshot, it’s only down for a few seconds. I might even get rid of the stopping altogether but there’s probably little point to that given how short the downtime is. I don’t have to fulfill an SLA. 😂
Yeah sounds like snapshots is the way to go!
Stop the whole VM during snapshots.
Not a VM. Consider the service just a service running on the host OS where either the whole OS or just the service data are sitting on ZFS or LVM.
This is one of the reasons Docker exists.
And I’m using Docker, but Docker isn’t helping with the stopping/running during backup conundrum.
Why not?
Docker doesn’t change the relationship between a running process and its data. At the end of the day you have a process running in memory that opens, reads, writes and closes files that reside on some filesystem. The process must be presented with a valid POSIX environment (or equivalent). What happens with the files when the process is killed instantly and what happens when it’s started afterwards and it re-reads the files doesn’t change based on where the files reside or where the process runs. You could run it in docker, in a VM, on Linux, on Unix, or even Windows. You could store the files in a docker volume, you could mount them in, have them on NFS, in the end they’re available to the process via filesystem calls. In the end the effects are limited to the interactions between the process and its data. Docker cannot remove this interaction. If it did, the software would break.
docker stop containerMake your snapshot
docker start containerWhat am I missing?
That’s the trivial scenario that we know won’t fail - stopping the service during snapshot. The scenario that I was asking people’s opinions on is not stopping the service during snapshot and what restoring from such backup would mean.
Let me contrast the two by completing your example:
docker start container- Time passes
- Time to backup
docker stop container- Make your snapshot
docker start container- Time passes
- Shit happens and restore from backup is needed
docker stop container- Restore from snapshot
docker start container
Now here’s the interesting scenario:
docker start container- Time passes
- Time to backup
- Make your snapshot
- Time passes
- Shit happens and restore from backup is needed
docker stop container- Restore from snapshot
docker start container
Notice that in the second scenario we are not stopping the container. The snapshot is taken while it’s live. This means databases and other files are open, likely actively being written to. Some files are likely only partially written. There are also likely various temporary lock files present. All of that is stored in the snapshot. When we restore from this snapshot and start the service it will see all of that. Contrast this with the trivial scenario when the service is stopped. Upon stopping it, all data is synced to disk, inflight database operations are completed or canceled, partial writes are completed or discarded, lock files are cleaned up. When we restore from such a snapshot and start the service, it will “think” it just starts from a clean stop, nothing extra to do. In the live snapshot scenario the service will have to do cleanup. For example it will have to decide what to do with existing lock files. Are they there because there’s another instance of the service that is running and writing to the database or did someone kill its process before it had the chance to go through its shutdown procedure. In the former case it might have to log an error and quit. In the other it would have to remove the lock files. And so on and so forth.
As for th effect of docker on any of this, whether you have
docker stop containerorsystemctl stop serviceorpkill servicethe effects on the process and its data is all the same. In fact the docker and systemctl commands will result in a kill signal being sent to the process of the service anyway.
I don’t bother stopping services during backup, each service is contained to a single LVM volume, so snapshotting is exactly the same as yanking the plug. I haven’t had any issues yet, either with actual power failures or data restores.
And this implies you have actually tested those backups right?
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters LVM (Linux) Logical Volume Manager for filesystem mapping NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency SMTP Simple Mail Transfer Protocol ZFS Solaris/Linux filesystem focusing on data integrity
[Thread #902 for this sub, first seen 1st Aug 2024, 21:25] [FAQ] [Full list] [Contact] [Source code]
Check “green blue” deployment strategy. This is done by many businesses, where an interrupted service might mean losing a sale, or a client forever… I tried it sometime witj Nginx but it was more pain than gain (for my personal use)
Modern image snapshot backups stop the service for av instant, creates a local snapshot to backup while the service runs a Delta then you apply the Delta to the running image
When you say stopping the service for an instant you must mean pausing its execution or at least its IO. Actually stopping the service can’t be guaranteed to take an instant. It can’t be guaranteed to start in an instant. Worst of all, it can’t even be guaranteed that it’ll be able to start again. When I say stopping I mean
sysemctl stopordocker stoporpkilletc. In other words delivering an orderly, graceful kill signal and waiting for the process/es to stop execution.
Wouldn’t restoring from such a backup be equivalent to kill -9 or pulling the cable and restarting the service?
AFAIK, effectively yes. The only thing you might lose is anything in memory that hasn’t been written to disk at the time the snapshot was taken.
Thanks for validating my reasoning.
What i have seen for corporate server is when backup is started the database goes into a different mode, and a temp writable partition is used while readonly database is backed up, at end of backup that blob created is also stored.
Yeah if you’re making a backup using the database system itself, then it would make sense for it do something like that if it stays live while backing up. If you think about it, it’s kinda similar to taking a snapshot of the volume where an app’s data files are while it still runs. It keeps writing as normally while you copy the data from the snapshot, which is read-only. Of course there’s no built-in way to get the newly written data without stopping the process. But you could get the downtime to a small number. 😄
The other thing to watch out for is if you’re splitting state between volumes, but i think you’ve already ruled that out.
Oh yeah, that would be a disaster.
I’d be cautious about the “kill -9” reasoning. It isn’t necessarily equivalent to yanking power.
Contents of application memory lost, yes. Contents of unflushed OS buffers, no. Your db will be fsyncing (or moral equivalent thereof) if it’s worth the name.
This is an aside; backing up from a volume snapshot is half a reasonable idea. (The other half is ensuring that you can restore from the backup, regularly, automatically, and the third half is ensuring that your automated validation can be relied on.)
Contents of application memory lost, yes. Contents of unflushed OS buffers, no. Your db will be fsyncing (or moral equivalent thereof) if it’s worth the name.
Good point. I guess
kill -9is somewhat less catastrophic than a power-yank. If a service is written well enough to handle the latter it should be able to handle the former. Should, subject to very interesting bugs that can hide in the difference.This is an aside; backing up from a volume snapshot is half a reasonable idea. (The other half is ensuring that you can restore from the backup, regularly, automatically, and the third half is ensuring that your automated validation can be relied on.)
I’m currently thinking of setting up automatic restore of these backups on the off-site backup machine. That is the backups are transferred to the off-site machine, restored to the dirs of the services, then the services are started. This should cover the second half I think. Of course those services can’t be used to store new data because they’ll be regularly overwritten with every backup. In the even of a hard snafu where the main machine disappears, I could stop the auto restore on the off-site machine and start using the services from it, effectively making it the main machine. If this is reasonable and it works, I might trash all of the file-based backup and transfer and switch to ZFS send/recv. That should allow to shrink the data delta between main and off-site to minutes instead of hours or days. Does any of this make sense?
