April 17, 2015

Backup to cloud when there is no space left on device

Imagine that you are planing to move some data from one server to another. To do so, you backup your data by creating a simple compressed tar archive with:

$ tar cjf /tmp/backup.tar.bz2 $HOME

And, for your surprise you end up with a no space left on device error from tar. You need to backup and have no more space: what do you do? Simple, use some Unix Pipes to the job!


In my case, my source box already had the Google Cloud SDK configured, and a good network connection to upload my backup. Instead of doublig the space locally and uploading later, what I did was to pipe the backup generated from tar directly to gsutil:

$ tar cjf - . | gsutil cp - gs://mybucket/backup.tar.bz2

Piping is, basically, connecting the standard output of a process into the standard input of another one. This is, simply put, a way to take the output generated from tar and sending it directly to gsutil, which in turn will do a stream upload to the cloud.

The same piping technique can be used with other tools as well. If you are copying from one server to another via SSH, you can also pipe it over with:

$ tar cjf - . | ssh user@backupserver 'cat > backup.tar.gz'

You can also use the similar technique to restore without the need to download everything.

Happy hacking!

[1] http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/
[2] https://cloud.google.com/storage/docs/gsutil/commands/cp#streaming-transfers
[3] http://en.wikipedia.org/wiki/Pipeline_%28Unix%29

April 5, 2015

Setting up a Moodle instance with Compute Engine and Cloud SQL

Things are getting cloudy these days. And that's good! We have a lot of options available now, ranging from bare virtual machines, to fully managed platforms, and with all kinds of containerized environments in between.

In this blog post, we are going to explore the Google Cloud Platform to host a Moodle site, using a distributed setup that's easy to build and scale as we need to upgrade. The setup we are going to build looks like this:
  • One Compute Engine Instance, where we will install Moodle software
  • One Persistent Disk attached to the instance in Read/Write mode, that we will use to store Moodle Data
  • One Cloud SQL Instance, to store the Moodle Database
This setup has some advantages over putting all into a single instance. First, it uses more than one node to host the system, distributing the load between the web-server and the database. This improves performance, and allows you to scale vertically, either by increasing the Cloud SQL instance tier, or by upgrading your VM to a more performant one. Secondly, we split software from data using both a separated database and a separated persistent disk. This aproach makes sofware updates easy to manage because the software is isolated from the data disk. Making backups with this setup is also very easy: we can enable the Cloud SQL backups to keep up to 7 backups, and also create a very simple script to backup our Moodle Data regularly using Diferential Snapshots on the Persistent Disk.

Getting Started

The first thing to do is to sign up to Google Cloud Platform, using a Google Account. You can use the USD $300 free-trial credit to get started! The free trial credit is valid for 60 days, and it is only valid for new accounts.

Sign up at https://cloud.google.com/free-trial/Note: in order to use the free-trial, or to use the Compute Engine virtual machines, you need to setup a valid payment method, and may need to put in your credit card. To use the free-trial, you will create a new project; if you already use Cloud Platform, I also recomend you to create a new project to host only your Moodle system, unless you plan to share other resources with it.

With your account setup, you can start by creating a new Compute Engine VM. Compute Engine is billed by the minute, after the first 15 minutes, making it easy to setup, stop then resume the work if you need to. Let's begin with a g1-small (1 vCPU, 1.7 GB memory) instance, that can handle a small to moderated traffic very well, and is cost effective.

Launch and configure the virtual machine

To launch the instance from the Developers Console, click on your project, then navigate to Compute -> Compute Engine -> VM Instances. In the Instances panel, click on the New Instance button.

In the new instance screen, let's create our instance with an extra disk to store Moodle data. To do so, click on the Show advanced options link, and use the following values:
  • Instance name: moodle-server
  • Firewall: Allow HTTP traffic
  • Zone: choose one that apply for your location
  • Machine Type: g1-small (you can choose a higher one if you expect more traffic)
  • Boot Disk: new from image, and Debian wheezy backports as boot disk image option
  • Under additional disks, create a new one by clicking on the plus sign, and choose create a new disk.
    • Disk name: moodle-data
    • Disk type: regular persistent disk should go just fine, but if you expect a high load, I suggest SSD as they are more performant
    • Source type: none
    • Size: 200G. You can choose lower/higher values here, but keep in mind that this will be used by Moodle data, so it requires a reasonable ammount of space, and that the higher your disk, more performance it has
    • Click on create (the disk is created immediately)
  • In networking, I recommend you to reserve a static IP: just click on New static IP and give it a name (moodle-ip). The IP is reserved as you click on this option
  • For the authentication options, I suggest you choosing Read/Write to Compute and Storage, and Enable to Cloud SQL. This allows you to run scripts to backup data from within your instance
  • Finally, click on the Create button
Your instance will be started, and we are going to be able to work with it as soon as it boots up. This process is usually very fast, and in one or two minutes the instance will show up in the VM Instances screen. To manage the instance, we need to login using SSH. You can use the SSH button from the Instances page:
The web-based shell will open in a new window, and a new user account will be added to your instance. Note that the web-based shell can get disconnected and to avoid loosing your work or corrupting your instance, let's use a terminal multiplexer that will keep the shell running if the SSH session is droped. I suggest tmux, as it is very easy and straghtforward to install/run:

sudo bash
apt-get update && apt-get upgrade --yes
apt-get install tmux --yes

Once running, tmux will show a green bar at the bottom of the screen:

To leave tmux, use CTRL+D, to detach it and keep it running, or exit to close the shell session. If your connection is dropped, reopen the ssh window and reattach to tmux using:

sudo tmux attach-session -t 0

Format and mount the disk

Remember that we have attached a empty disk to the VM? Since it is empty, we need to format it with a filesystem, and mount it on a directory. To do so, we use a program shipped by Google called safe_format_and_mount, that does just that: format the disk only if it is not formated and mount it for you.

First, let's create a mount point:

mkdir -p /mnt/moodledata

Then let's format the disk (line breaks added for clarity, this is a single command without the back slashes):

/usr/share/google/safe_format_and_mount \
  -m "mkfs.ext4 -F" \
  /dev/disk/by-id/google-moodle-data /mnt/moodledata

Check that the device is mounted listing the contets:

ls -l /mnt/moodledata

You should see a lost+found entry, for an empty Ext4 filesystem. To make this mount point be remounted during boot, add one entry to the /etc/fstab file:

cat >> /etc/fstab
/dev/disk/by-id/google-moodle-data /mnt/moodledata defaults 1 1

Important! Use double grather than signs (>>), otherwise you will loose the root mount point from the fstab file!

Launch and configure the Database

Now, let's launch the database using Cloud SQL. To do that, navigate to Storage -> Cloud SQL menu, then click on the New instance button:

Fill the form with the appropriate values:
  • Instance name: moodle-db
  • Region: Choose the same geographic region of you Compute Engine VM
  • Choose the D0 tier. You can safely start with D0, and upgrade later if you see high usage on the database
  • Click on the Show advanced options link
  • Choose the MySQL 5.5 version
  • Choose a billing plan. In this case, as Moodle will constantly talk to your database, and since you will need an static IP address, the per use plan may not make much difference. I recommend choosing the daily package plan
  • Select the same region of you compute engine VM to the location option
  • Let Enable backups option checked and choose a more suitable time frame for backups; this can be changed later
  • Check the assign an IPv4 address to the instance. Altought a free IPv6 address is available, we can't use this one from within Compute Engine yet
  • Under allowed networks, click on the + sign, and put the IP address of your compute engine VM; you can put the same name of your vm: moodle-server
  • Click on the Create button to launch the instance
The server instance starts very fast. We need to create a new MySQL user and a new database for Moodle. To do so, click on the instance name in the list, and then on the Databases tab, then on the New database button:

In the new database form:
  • Database name: moodle
  • Character set: utf-8
  • Collation: default collation
  • Click on the Add button.
To create the user, click the Access Control tab, then on the Users tab, and finally click on the New User button.

In the new user form:
  • Username: moodle
  • Password: create a strong password, and take note of it to use on the Moodle setup
  • Click on the Add button
Now we are ready to install and configure our Moodle server! Let's get back to our compute engine VM terminal window.

Installing and configuring Moodle software

Moodle requires a web server and a few PHP modules. You can choose the webserver you know best, and for this tutorial I'll use the Apache webserver. Since this is a Debian server, let's use apt to download and install everything we can, so we benefit from the Debian Security upgrades:

apt-get install apache2 php5 php5-gd php5-mysql php5-curl php5-xmlrpc php5-intl

In order to get the Moodle software, let's choose the latest stable version from the download site: https://download.moodle.org/releases/latest/. Click on the Download tgz button, and dismiss the automatic download window: we need the direct download link. Right click on the click here for manual download link and copy the link address:

On the terminal window, we need to download into our instance using curl. We can type curl and then paste the copied link with CTRL+V, then finish the command with > moodle.tar.gz, resulting in a command like this one:

curl https://download.moodle.org/download.php/direct/stable28/moodle-latest-28.tgz > moodle.tar.gz

Important: do not forget the "> moodle.tar.gz" part of the command at the end, otherwise your file will be printed on the screen! Before unpacking the moodle software, remove the default index.html page from Apache:

rm /var/www/index.html

Then, use the tar command to unpack the software in the Apache directory:

tar xf moodle.tar.gz -C /var/www --strip-components=1

The tar command bellow extracts the file contents into /var/www, skipping the creation of the moodle sub-directory. Now we need to configure Moodle to use all the resources we launched.

Moodle will need read/write permissions on the software and data directories. To fix permissions on these places, use the chown command:

chwon www-data:www-data -R /var/www
chwon www-data:www-data /mnt/moodledata

To configure Moodle, launch the setup wizard by navigating to your instance IP, or by clicking on the compute engine VM IP link on the VM instances page:
You should see the Moodle setup page:

Select yor language, then click on the Next button. Moodle requires you to set some important options. The first one is the visible website address; let's keep the instance IP for now, and we change this value later when the setup is finished. You also need to tell Moodle the software directory and the data directory. The software directory is where we extracted Moodle, /var/www, and the data directory will be the mount point we have created with the 200G persistent disk: /mnt/moodledata. Put these values and click next.

In the next page, we need to setup the database connection. First, select the mysqli database driver, then click next. In the database connection settings fill the values:
  • Database host: the IP address of your Cloud SQL instance
  • Database name: moodle
  • Database username: moodle
  • Database password: the password you choose previously
  • Tables prefix: optional, just use the default
  • Leave empty database port and unix socket, as we are going to use the defaults
When you click next, Moodle will connect to your instance, and it will start in response for the first connection. Also, Moodle will present the software license; read it, and if you agree, click Continue to install.

In the next screen, if some PHP modules that are required, they will be listed in red. You can install them and hit reload to let the installer check again. If all is OK, just click on the Continue button to proceed with the installation.

The next step will take a reasonable time: Moodle installer will configure all database tables, populate some initial data, prepare the moodledata directory with some cache values, and get all things in place. Wait until the page finish loading, and you see a Continue button.

After all setup is completed, you can create the Admin user. The admin user is the one with access to all setup pages, and you should choose a strong password for it. After you create the admin user, you will be propted to put some basic site information. Fill the form and click Save changes.

Now you should see your Moodle home page, and you can start playing around, creating users and courses, installing plugins and so forth.

Enable the cron job

Moodle itself requires you to configure a job that runs periodically, called a cron job. The easier way to setup a Moodle cron job is by running it using curl. Let's enable the moodle cron with a simple shell script and the Debian run-parts directory for cron jobs. First, create a simple shell script that fetches the cron url, and logs the output for debugging:

cat > /etc/cron.hourly/moodle-cron
curl http://localhost/admin/cron.php 2>&1 > /var/log/moodle-cron.log

To finish the file, type CTRL+D and check the contents with the cat command:

cat /etc/cron.hourly/moodle-cron

Now, let's make it executable, and run a test cron:

chmod +x /etc/cron.hourly/moodle-cron

The cron job will run and you can check it's output at the log file.

Sending e-mail

Compute Engine VMs are not allowed to send e-mail. However, you can configure Moodle to use an SMTP server hosted on another service. Google partned with Sendgrid to deliver your compute engine emails, so you can follow this page to signup to a free account that can send 25000 emails from your Google projects. Once signed up, grab the Sendgrid credentials and configure the hostname, port 2525, username and password using the Moodle e-mail setup instructions. Alternatively, you can setup Mandrill, that also has a free quota for emails per month, and provides an SMTP server connection just like Sendgrid.


Some improvements that can be done:
  • Add another daily cron script that backup the moodle-data disk, as well as the moodle-server disk (the boot disk). See how to take disk snapshots for backups, and take care to make your script remove old backps at the end.
  • Change the site URL in the config.php. Moodle saves some data in the database with the full URL, so you should use the database search and replace tool to change data to the new URL.


In this post, we learned how to setup a simple, efficient and cost-effective Moodle site, that can handle a mid-size Moodle traffic, and provides a good infra-structure that can be vertically upgraded in the future.

Important! You will be billed by the resources used, so if you did this as a test-only, remember to remove the compute engine VM, the persistent disks, and the Cloud SQL instance.