I just finished a couple day's work, documenting and adapting the 'rcall' (Remote Call) tool, and added it to the Mycat project on sourceforge. The readme is available here.
'rcall' is designed to run in non-homogeneous *nix clusters, to ease use of these clusters by creating a single place from which to securely run commands across logical groups of servers.
I'll give some examples.... The past two companies I have worked for both had separate web (apache) servers, mailing (MTA) servers, and database (MySQL) servers. Using rcall (its previous incarnation, that is), I was able to easily track down an error that was causing seemingly random entries in the databases to have wrong timestamps; `rcall -on web -C 'date'` would print the current date on each web server, making it easy to see which one's clock was wrong. After making a change to apache's configuration, it was easy to restart all the webservers without any visible interruption in service; `rcall -on web -delay 15 -C '/usr/local/apache/bin/apachectl restart'` - the delay helps our loadbalancer handle the sudden loss of service from each individual server, but collectively, there is no downtime. Checking on disk utilization across all database servers is as easy as `rcall -on data -C 'df -h | grep -P "data|backup"'`.
`rcall` does not directly relate to MySQL, but it has helped me in my work with MySQL Cluster, and with clusters of MySQL servers (using the term loosely there). Of course, the other tool already in the mycat project is `rep_mon`, which IS directly relevant to MySQL.
Over the coming weeks, I will be adding another tool to this project, namely, `conftool` which creates a single location from which to sync configuration files (and back them up, and restore from backups, and so on) throughout the same groups/clusters of servers that `rcall` and `rep_mon` operate on. After that, I hope to add my database backup system, but I have yet to find a way to "generalize" it so that it might work on someone else's systems without hassle.
I've found these tools indispensable in my own work; I hope that others can find at least some use in them!