Thursday 11 December 2008

How to avoid overlapping rsync instances

Running rsync as a cron job is probably as popular as sliced bread. For small intervals, however, you risk having a second instance start before the first one has finished, beginning a downward spiral of bandwidth and processes.

To avoid multiple instances getting in each other's way there's a simple solution using flock. Example without lock (may cause overlapping instances):
*/5 * * * * /usr/bin/rsync --delete -a source_server:/source/path/ /dst/path/
Example using flock:
*/5 * * * * flock -xn /tmp/example.lock -c '/usr/bin/rsync --delete -a source_server:/source/path/ /dst/path/'
In the second example, if a rsync instance runs for more than 5 minutes, flock will fail and thus not execute a second rsync process.

This can be used for pretty much any shell problem where locking helps. Have a read on the flock man page for other examples.

1 comment:

Mats said...

Thanks for the example :-)