Replicator – Rough Project Draft #1
Replicator – Rough Project Draft #1 - 2007-09-03
This tool is trying to automate the process of maintaining duplicate files across two or more machines. It has a control file that it uses to generate SVN commands to retrieve or store files.
Ideas of use:
e-Replicator - /etc files:
Replication of configuration files. With support for machine specific location and name of files.
h-Replicator: Home Directory.
Replicating home directory files on multiple machines. With support for .ssh, shell init files and other data that needs to be customized depending on machine.
g-Replicator: aka Geplicator
System to maintain /etc/portage replication, using /etc/portage/package.* directories.
|
filename |
description |
|
*.all |
Files for all servers. |
|
*.<hostname> |
Only host get this file. |
|
*.<group> |
Setup group identities, servers, workstations, mta, dns. |
d-Replicator: The dumb replicator, it just copy files with no intelligence.
For copying say shared /usr/local/bin files, website, /home/xxx/bin and the like.
svn-Replicator: SVN backup support.
This tool provides the storage handling for all the other replicators and also support adding files to svn based on time since last modification. Say someone modifies /etc/make.conf and forgets to check it in to subversion, then svn-Replicator can monitor the file and push it to svn if last modification time is over 60 minutes (user configurable).
All files will be stored in subversion and this central svn server will be the backbone of the replication. All the servers will talk to this server, preferably with webdav and https for better security.
The SVN directory structure is setup to allow the repository to work as a normal svn for backup or version control. The /etc SVN could contain all of /etc, but only a few files would be handled by e-Replicator.
|
Component |
human/machine SVN |
Component storage |
|
e-Replicator |
/etc/<machine> |
/e-Replicator |
|
h-Replicator |
/home/<username>/<machine> |
/h-Replicator |
|
g-Replicator |
/etc/portage |
/g-Replicator |
|
d-Replicator |
/usr/local/bin/<machine> |
/d-Replicator |
slot = 3780496060
width = 728
height = 90
g-Replicator
I've managed to make a working g-Replicator, it's very minimal and it needs a big guide for configuring the SVN directory structure.
In my setup I have:
SVN Server, currently on my old Ubuntu server.
Home Server - Going to replace the Ubuntu one when it works.
Workstation.
Guest vserver.
coLinux One and Two under Windows.
Server - Hosted server.
I can now login to any of the Gentoo machines, edit something in /etc/portage and run g-update to submit it to the SVN server and mark it, so the servers next check will replicate the changes. As long as the same file isn't edited on more than one machine at the same time, the script will gracefully handle multiple submissions.
It lacks the ability to erase files when you decide you no longer need something., but it's just a matter of time before I decide on a solution for this and implement it.
While this early version wasn't planned to be anything but a test case to build on, it's actually working so well that I'm considering releasing it as 0.1.0. I need to write documentation, a SVN initialization script and decide on a delete solution. Then it will be ready for others to use.
I need to decide on a license, at this time GPLv3 seems the most likely choice. Even before I knew FSF my licenses for software I released was GPLish. My only problem is the make source available requirement, except Replicator is scripts, so it doesn't really matter as source is always included. I'm not against someone releasing binary only, as long as the source for the binary exist somewhere on the net and the binary releaser takes the time to include a how to get source document. Though if I go with GPLv3 it will be unmodified.
Status Update 2007-09-05
Project is slowly taking shape.
version 1.0.0 will be a bash script based version, so I get lots of room to experiment and use svn directly.
version 2.0.0 The applications will be done in Python, I've not done much bash code yet, but I can already see where this is going to improve things.
I've created the SVN server and begun experimenting with layout. SVN can almost do everything it self, it's some times a bit annoying that if X feature just was implemented to work on Y too, then SVN wouldn't need Replicator in many cases.
The fact that svn works so well for this is an unexpected bonus and makes the bash scripts I have to make really simple, this has allowed me to start on implementing a very rough g-Replicator. I've everything setup for the commiting of data from each machine and merging it. Working on the cron jobs to do it automatically now.
You edit some file in /etc/portage/package.* and when your done, you simply type g-update and the script will do the svn commit and create the file that the cron jobs on the other machines uses to detect they need updating.
Thats all for now, I'm back to finishing creating the first working g-Replicator implementation, for fully automatic /etc/portage Replication between my six Gentoo's.
Your always welcome to contact me by email with questions or suggestions or look me up on irc.freenode.net #Replicator, though as always with IRC you never know when someone is actually there and I'm worse than most, since I usually follow some weird 36 hours day pattern, so it keeps rotating when I'm around for 24 hour based people.