General

Backups are Important

2023-11-17

About 3 weeks ago I decided that it was a good idea to start running nightly backups of my server data. If it wasn't for this decision, I may have lost all of my email, contacts, and calendar data.

What happened?

While experimenting with another VM to test out Immich as a photos backup alternative to Google Photos, I accidentally "reinstalled" my brandontoner.ca server rather than the experimental deployment... whoops! My face went pale as I realized my mistake and discovered that all of my server data and configurations had vanished at the click of a button.

How it resolved

Fortunately, I remembered that I had been running nightly backups of the user directories for my main accounts on my server. Which, hopefully, would allow me to restore everything to the pre-error state.

To get things back up and running, I reconfigured the server with the assistance of Derek's new ti.sh script for installing the basic setup of the server. Once the basic programs and configurations were reinstated, I could then restore my data from my backup drive to the user directories via rsync commands.

During the process, I documented any additional steps and configurations I had conducted in a note in Obsidian for future reference.

One lingering problem is that with the new ti.sh script, Mailjet is used to relay outgoing mail. With this, I can no longer successfully send to gmail accounts... surely this is an easy fix.

What I learned

• The 3-2-1 Backup Strategy
• Backblaze as a reasonably priced and reliable MacOS backup service (for an "offsite" component)
• Some stuff about CNAME records with the Gmail deliverability issue. Including DNS propagation and the fact that it takes a while.
• Most importantly: backups are essential. If you don't have a backup process you can trust and understand, make it a priority. You'll be glad you did.

I've always considered backups to be "a good idea", but a bit complex to figure out how to do it properly and reliably. I still largely feel that way — but at least now know more than I did and have a renewed interest in learning more to make my systems more robust.

It sure feels good to have things back up and running relatively unscathed. Phew.