This how to shows how to make a simple load balanced ( and highly available ) cluster of machines for a TCP service.
A quick cluster has been setup for testing svn clustering ( virtual machines )
svncluster - router - ssh on port 222
This machine runs a tcp load balancer ( called pen, http://pen.siag.nu ).
[root@svncluster ~]# ps -ef | grep pen
root 143 2 0 04:45 ? 00:00:00 [ksuspend_usbd]
root 7535 1 0 05:12 ? 00:00:00 pen 22 -C localhost:19000 svnc1:22 svnc2:22
root 7999 7960 0 23:16 pts/0 00:00:00 grep pen
You can see the command line arguments in the code box above.
Pen listens for connections on port 22 and forwards them (by default in a sticky manner) , so if you connect and get assigned svnc2 as your server, pen will prefer this when its available and is in balance.
svn cluster nodes - svnc1 + svnc2 - ssh on port 22
These machines are identical apart from the host names and merely act as clients that mount the filesystem and having sshd running.
The passwd and associated files are identical so the user is simply balance to the correct available machine, according to how the pen load balancer daemon is configured.
The test
Pausing in vmware - causing the VM to be in a state like "switched off" , un-pausing a machine reanimates it quite quickly
In order to test the failover aspect, each cluster node was paused and un-paused in opposite order. That is, while svnc1 was paused, svnc2 was alive and vice versa . Finally to prove that performance was quite normal when there was no failure test commits were entered that went through near instantly as per normal operation. This is because the client gets to use its established / preferred path, if failover occurs around 5 seconds of negotiation is introduced , this is tuneable to some degree.
[user@desktop svn]$ svn co svn+ssh://svncluster/shared/svn/repo/svncluster/
user@svncluster's password:
user@svncluster's password:
A svncluster/testfile
A svncluster/file1
A svncluster/file2
A svncluster/file3
A svncluster/file4
A svncluster/file5
Checked out revision 6.
[user@desktop svn]$ cd svncluster/
[user@desktop svncluster]$ touch TEST
[user@desktop svncluster]$ svn add TEST
A TEST
[user@desktop svncluster]$ svn commit -m "added file TEST"
user@svncluster's password:
Adding TEST
Transmitting file data .
Committed revision 7.
[user@desktop svncluster]$ svn log svn+ssh://svncluster/shared/svn/repo/svncluster/
user@svncluster's password:
------------------------------------------------------------------------
r7 | user | 2009-06-14 23:52:00 +0200 (Sun, 14 Jun 2009) | 1 line
added file TEST
------------------------------------------------------------------------
r6 | user | 2009-06-14 23:12:25 +0200 (Sun, 14 Jun 2009) | 2 lines
this commit should be faster as no switching is taking place. both servers are up and the client will continue without renegotiation... hopefully
------------------------------------------------------------------------
r5 | user | 2009-06-14 23:03:16 +0200 (Sun, 14 Jun 2009) | 2 lines
added file4 while svnc1 is suspended and svnc2 is live, you get the idea ...
------------------------------------------------------------------------
r4 | user | 2009-06-14 22:54:12 +0200 (Sun, 14 Jun 2009) | 2 lines
adding file3 with svnc2 suspened svnc1 is back
------------------------------------------------------------------------
r3 | user | 2009-06-14 22:58:15 +0200 (Sun, 14 Jun 2009) | 2 lines
adding file2 while svnc2 is back and svnc1 is suspended
------------------------------------------------------------------------
r2 | user | 2009-06-14 22:51:45 +0200 (Sun, 14 Jun 2009) | 2 lines
adding file1 while svnc2 is suspended
------------------------------------------------------------------------
r1 | user | 2009-06-14 22:50:07 +0200 (Sun, 14 Jun 2009) | 2 lines
test
------------------------------------------------------------------------
Conclusion
While performance is indeed slowed down by about 5 seconds during a failover, pen does provide a seamless failover to the next svn cluster node without a failure on the client end. The test was performed using an nfs mount from an office dell machine running a Linux desktop, so it wasn't a performance solution from the start, but it was not under load either. The test files checked in were blank and merely touched files.
This solution would need to be scaled and tested with more simultaneous users and a realistic workload.
[user@desktop svncluster]$ dd if=/dev/zero of=10mb-file.bin bs=1024 count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB) copied, 0.0506667 s, 202 MB/s
[user@desktop svncluster]$ svn add 10mb-file.bin
A (bin) 10mb-file.bin
[user@desktop svncluster]$ time svn commit -m "timed commit for a 10mb file"
user@svncluster's password:
Adding (bin) 10mb-file.bin
Transmitting file data .
Committed revision 8.
real 0m3.609s
user 0m0.125s
sys 0m0.041s
[user@desktop svncluster]$ dd if=/dev/zero of=100mb-file.bin bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.641002 s, 160 MB/s
[nuseremo@desktop svncluster]$ svn add 100mb-file.bin
A (bin) 100mb-file.bin
[user@desktop svncluster]$ time svn commit -m "timed commit for a 100mb file"
user@svncluster's password:
Adding (bin) 100mb-file.bin
Transmitting file data .
Committed revision 9.
real 0m7.596s
user 0m1.020s
sys 0m0.406s
The above test commits demonstrate that even on low powered desktop computers commit speeds can be quite good.
3 seconds for a single 10mb file and 7 seconds for a single 100mb file incl password typing.
In fact it does scale out with ease , configuring the pen daemon to spread the load over more servers means less users per cluster node.
A more performance solution than using the using the dell desk / nfs as a shared filesystem is also a good idea. Think SAN / rawdisk.
It would be preferable to employ more cluster nodes to a busy svn project, perhaps a sub cluster of 2 or 3 VMs for the busiest and a single VM for the occasionally used svn repos. Dedicated storage that will support the correct amount of IO is also advisable.
Pen its self is able to failer over as it uses VRRP and cluster nodes ( such as svnc1+2 ) can also be configured with pen to be used as failover routers.
http://pen.siag.nu/vrrpd-linux.shtml
