April 17, 2015

Backup to cloud when there is no space left on device


Imagine that you are planing to move some data from one server to another. To do so, you backup your data by creating a simple compressed tar archive with:

$ tar cjf /tmp/backup.tar.bz2 $HOME

And, for your surprise you end up with a no space left on device error from tar. You need to backup and have no more space: what do you do? Simple, use some Unix Pipes to the job!

gsutil

In my case, my source box already had the Google Cloud SDK configured, and a good network connection to upload my backup. Instead of doublig the space locally and uploading later, what I did was to pipe the backup generated from tar directly to gsutil:

$ tar cjf - . | gsutil cp - gs://mybucket/backup.tar.bz2

Piping is, basically, connecting the standard output of a process into the standard input of another one. This is, simply put, a way to take the output generated from tar and sending it directly to gsutil, which in turn will do a stream upload to the cloud.

The same piping technique can be used with other tools as well. If you are copying from one server to another via SSH, you can also pipe it over with:

$ tar cjf - . | ssh user@backupserver 'cat > backup.tar.gz'

You can also use the similar technique to restore without the need to download everything.

Happy hacking!

--
References
[1] http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/
[2] https://cloud.google.com/storage/docs/gsutil/commands/cp#streaming-transfers
[3] http://en.wikipedia.org/wiki/Pipeline_%28Unix%29