Installing ClusterControl with an existing cluster and Nginx

ClusterControl is a comprehensive tool for managing and monitoring MySQL and MongoDb clusters, more information can be found here: http://www.severalnines.com/clustercontrol

ClusterControl comes with some great scripts for automating deployment, however there are a number of hard coded dependencies (like Apache) which make it a rather more painful process to integrate with your existing systems.

Background info:

There are 3 components that make up ClusterControl:

ClusterControl UI: A CakePHP based GUI
CMON: A daemon process which monitors servers and writes to a MySQL database
CMON API: A PHP based API which reads data from the CMON database

ClusterControl UI and CMON API can be on the same or different hosts and the UI can talk to multiple instances of the CMON API (hence multiple CMON daemons).

The UI also makes AJAX calls to the API via a script in the path /access which simply sends the same request to the API with the addition of the API token.

There are several .htaccess files that rewrite requests to the UI, the access script and the API.

Installation process:

CMON can be installed by following this guide: http://support.severalnines.com/entries/20613923-installation-on-an-existing-cluster

The guide contains a section at the end to install the UI/API, this does install and configure Apache, but you can simply remove it again.

Making it work with Nginx:

After a bit of tinkering I’ve put together a config which works nicely for both the ClusterControl UI and CMON API:

server {
        listen 1.2.3.4:80;
        server_name mydomain.com;

        access_log /var/log/nginx/mydomain.com-access.log;
        error_log /var/log/nginx/mydomain.com-error.log;

        root /var/www;
        index index.php;

        location ~ \.htaccess {
                deny all;
        }

        location ~ \.php$ {
                fastcgi_pass 127.0.0.1:9000;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include /etc/nginx/fastcgi_params;
        }

        # Handle requests to /clustercontrol
        location /clustercontrol {
                alias /var/www/clustercontrol/app/webroot;
                try_files $uri $uri/ /clustercontrol/app/webroot/index.php;
        }

        # Equivalent of $is_args but adds an & character
        set $is_args_amp "";
        if ($is_args != "") {
                set $is_args_amp "&";
        }

        # Handle requests to /clustercontrol/accesss
        location ~ "^/clustercontrol/access/(.*)$" {
                try_files $uri $uri/ /clustercontrol/app/webroot/access/index.php?url=$1$is_args_amp$args;
        }

        # Handle requests to /cmonapi
        location ~ "^/cmonapi/(.*)$" {
                try_files $uri $uri/ /cmonapi/index.php?url=$1$is_args_amp$args;
        }

}

Making the UI work is straight forward, however the access script and API require the partial URL and the equivalent of the mod_rewrite [QSA] flag as the ? in the try_files directives stops Nginx from auto-appending the query string from the request, hence the need for the regex and appended args.

A small code change is also required to the library used by the access script which is located in /clustercontrol/app/Lib/Access.php as the API token header uses an underscore rather than a dash which results in Nginx ignoring it (which I think is correct behaviour, probably an RFC floating around somewhere which will clarify this).

Replace both instances of:

$headers[0] = "CMON_TOKEN: {$token}";

With:

$headers[0] = "CMON-TOKEN: {$token}";

Everything should now work!

Share

Some observations while upgrading Percona XtraDB Cluster from 5.5 to 5.6

I followed the Percona guide to upgrade my XtraDB Cluster (MySQL + Galera) from 5.5 to 5.6, details can be found here: http://www.percona.com/doc/percona-xtradb-cluster/5.6/upgrading_guide_55_56.html

The upgrade process

I included putting the upgraded nodes into read only mode and I performed an additional step to reconfigure the xinetd process used to run clustercheck after installing the new binaries:

server_args = clustercheck <password> 0 /var/log/mysqlchk.log 0

This sets available when donor and available when read only to off, by default with 5.6 the SST method is Xtrabackup, but my 5.5 cluster had been setup with rsync, more on this later.

The first 2 nodes upgraded just fine, however when I restarted the 3rd node the entire cluster crashed. Restarting MySQL on all nodes got things back up and running again.

After upgrading

Everything was initially working fine, but I noticed some odd behaviour on one node, it was repeatedly crashing when I was purging old data from one of the databases. The node would fail to restart with an rsync error, only resolved when I performed a rolling restart of the other 2 nodes.

Switching from rsync to Xtrabackup

Rysnc clearly is not the best option for a 5.6 based cluster, it worked fine for the last 18 months with the 5.5 cluster but there are good reasons why Xtrabackup is the default option with 5.6. The main benefit that I can see is that Xtrabackup is non-blocking so the node can continue to accept connections during SST.

Xtrabackup requires a username/password to login to another host to perform the transfer, hence you have to create an account with permission to access all databases from each server in the cluster. I simply ran CREATE USER and GRANT statements on a single node and allowed the cluster to replicate the new accounts to the other nodes.

It’s important that the datadir variable is defined in my.cnf, or Xtrabackup doesn’t work.

A test can be carried out from the shell before updating my.cnf:

innobackupex --host=<another node> --user=<user> --pass=<password> /tmp/

Finally my.cnf can be updated:

wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=user:password

Checking the MySQL log should show SST now using Xtrabackup.

Share