About 3 weeks ago I decided that it was a good idea to start running nightly backups of my server data. If it wasn't for this decision, I may have lost all of my email, contacts, and calendar data.
While experimenting with another VM to test out Immich as a photos backup alternative to Google Photos, I accidentally "reinstalled" my brandontoner.ca server rather than the experimental deployment... whoops! My face went pale as I realized my mistake and discovered that all of my server data and configurations had vanished at the click of a button.
How it resolved
Fortunately, I remembered that I had been running nightly backups of the user directories for my main accounts on my server. Which, hopefully, would allow me to restore everything to the pre-error state.
To get things back up and running, I reconfigured the server with the assistance of Derek's new ti.sh script for installing the basic setup of the server. Once the basic programs and configurations were reinstated, I could then restore my data from my backup drive to the user directories via rsync commands.
During the process, I documented any additional steps and configurations I had conducted in a note in Obsidian for future reference.
One lingering problem is that with the new ti.sh script, Mailjet is used to relay outgoing mail. With this, I can no longer successfully send to gmail accounts... surely this is an easy fix.
What I learned
- The 3-2-1 Backup Strategy
- Backblaze as a reasonably priced and reliable MacOS backup service (for an "offsite" component)
- Some stuff about CNAME records with the Gmail deliverability issue. Including DNS propagation and the fact that it takes a while.
- Most importantly: backups are essential. If you don't have a backup process you can trust and understand, make it a priority. You'll be glad you did.
I've always considered backups to be "a good idea", but a bit complex to figure out how to do it properly and reliably. I still largely feel that way — but at least now know more than I did and have a renewed interest in learning more to make my systems more robust.
It sure feels good to have things back up and running relatively unscathed. Phew.