MeatballWiki

MeatballBackup

This page describes the MeatballBoard efforts to create a resilient BackupAndRestore system for Meatball and its related projects.

Committee Members

CategoryMeatball CategoryMeatballWiki

How does it work?

After creating an S3 account, and a S3 bucket was created using the /home/backup/s3-create.pl script (which uses SOAP::Amazon::S3 from CPAN).

Every night, /etc/cron.daily/backup calls /home/backup/backup which dumps the MeatballWiki PostgresQL database (add more if you need to), rotates the BibWiki logs, cleans out the mailboxes, and uses duplicity with BitBucket to incrementally backup the system. Once a month, /etc/cron.monthly/backup calls /home/backup/backup full to do a full snapshot (key frame) backup.

Edit the BACKUP-MANIFEST to include more in the backup. Currently, that file looks like:

/usr/lib/perl5/site_perl/5.8.7
/svn
/home

To restore, use the /home/backup/restore ''<time interval>'' script. By default, it restores the last revision. You can restore a specific file in the archive using the /home/backup/restore-file <absolute-path> script. It will put the restored files in the /home/backup/restored directory.

You can get a full list of files in the archive using /home/backup/list-current-files.

  • [May 1, 2007] I've implemented the nightly backup! C'est fini!
  • [April 30, 2007] I have duplicity + Amazon S3 working. I just need to get it into the cron tab and pick the files / systems to back up.
  • [April 28, 2007] Well, proving that if you want something done you have to do it yourself, I started implementing a Duplicity + Amazon S3 solution. -- SunirShah
  • [January 7, 2007] Meatball experienced a catastrophic system crash in July 2006. Now it is January 2007, but there is still no backup system in place. Work has been done in the intervening time to clean up the MeatballServerLayout--i.e. delete a lot of useless files, reorganize the directories, get Subversion running--as well as to ensure that PostgresQL is working more smoothly. The next steps are:
    • Find an offsite data repository where we can throw several gigabytes of data without concern. (A valid and cheap option might be Amazon's SimpleStorageService ([S3]).
    • Make a list of files, databases, systems to backup
    • Find or write and test a differential backup script
    • Find or write and test a restore script

Technically speaking, it may not be necessary to actually backup individual files if we can backup the entire Linux system.

The major issue right now is a lack of developer muscle to get this done. If anyone has done this before and has some tips, tricks, shortcuts, pitfalls they would like to relate, that would be most handy. -- SunirShah

Notes

Refactored into BackupAndRestore.

Assuming [MeatballDatabaseRelations], could use the revision column to determine whats changed. SELECT * FROM revisions WHERE revision > //previousBackUpRevision//. If previousBackUpRevision == 0 then get a full backup of the [VersionHistory]. -- JaredWilliams

Currently, the entire compressed binary pg_dump is around 25MB. Even if the backup diff algorithm cannot do anything with it, and assuming no more growth of the database, it will take 41 weeks before it costs me 15 cents USD. I don't think it is worth it to optimize it. -- SunirShah

Edit this page | History