JRRZZ Blog

For the final stages of my Bachelors degree I’m building a cluster again, just like last year. Clusters are fun. “Imagine a Beowulf cluster of those!” is a fairly common expression amongst the slashdot crowd. This particular cluster is being built to offer non-stop streaming multimedia content. The stream simply can’t fail. Ever. Since I’m doing this for a company, I’m not sure how much of this can become public knowledge. I won’t get into the nifty technical details, but I’ll describe the system globally.

I thought I could just take my old Debian Linux-HA cluster and implement multimedia streaming on top of that. For research’s sake, and a year’s worth of development had just passed, I decided to do some more research first. I’ve been looking into both high availability and manageability. The state of today’s data center management solutions is dreadful at best. Red Hat offers some nice stuff, but makes the huge mistake of requiring an Oracle DBMS. I have been ordered to stay away from any non-free software. Linux-HA is still great, but it’s GUI is broken. The command line interface is ok, but the GUI made the whole thing so much easier to set up. I’m afraid I have too cook up my own solution to get some manageability into the system.

And then there’s latency. In order to maintain a 1080p h264 video stream the connection simply can’t drop for any longer than a second or so. Linux-HA failed miserably during my tests with 5-10 seconds of fail over time. That’s quite obvious, since it’s implementation is entirely in user space. So I needed something faster, something at the routing level. I can’t use any non-free software so Cisco stuff is out of the question. A friend of mine told me about CARP, a free alternative for the patent ridden VRRP. Yes, VRRP should be free, a standard and so on. Think again.

So that brings me to BSD again. I’ve been playing around with BSD for a few years now, but I’ve never really used it for anything beyond my IRC and seed boxes. Since that post I’ve switched over to OpenBSD from NetBSD. Since I’m familiar with OpenBSD and CARP originates from this project, the operating system of choice became obvious.

Setting up this cluster was extremely easy and a lot of fun. Everything is nicely integrated, works as expected and is rock solid. OpenBSD’s packet filter is an extremely powerful tool which I’ve underestimated. It’s easy, fast, feature rich and secure. I can set up my entire network in a single human readable file. From routing and packet filtering to load balancing, it’s all there. Actually, all of the system management tasks are quite easy. I’m sure I’ll be reusing many of OpenBSD’s strengths in the future.

Meet the collective.borg.local test cluster. The hardware is kindly donated by my other employer. The big gray boxes are old AMD Athlon 32-bit machines with MSI VIA based motherboards and quite some RAM. I’ve called them cube-alpha, cube-beta and cube-gamma. Alpha and beta are routing firewalls in fail-over mode. That’s why they’ve got an extra Gig/E NIC connected to the DMZ. I can yank one of those machines out without disturbing the functionality of the network. The third, gamma, is an X11 box used for displaying log output of the other boxes in XTerms. The four black boxes are node1-4. Simple Dell/Intel boxes with a gig of RAM and Core2 or Celeron processors. They do have a nice Intel Gig/E NIC on board, though. These are the application servers. The incoming network connections are balanced equally between the four by the active router. I can unplug three of those and no one will notice. The one remaining will be handling all of the incoming connections, but the cluster won’t go down. All of this is connected using a rather nice 3com switch. All of these machines are running OpenBSD 4.4.

Here’s my work area. On the left is the X11 OpenBSD desktop tailing some log files from the application nodes. On the right my trusty Sun workstation, running Debian, decoding the 1080p h264 stream from the cluster. The movie I’m using is Big Buck Bunny, a movie entirely made using Open Source software. At this point I had killed two nodes and one router. The movie didn’t even hiccup.

There’s lots of work to be done. It’s working technically, but I’ve got to get VLC involved in this. At this moment VLC isn’t able to properly decode h264 over HTTP, so I’m using MPlayer right now. Also, Apache is serving the multimedia. This should become VLC too, using RTSP instead of HTTP. The state of VLC at this moment is very unstable, so I guess I’ve got to fix some things first. I’ve also got to write some custom code to detect errors in the streaming server, which should cause an immediate fail over event. Last but not least, manageability. I’ve got to find a way to both administrate and deploy OpenBSD streaming servers from a central console. That’s going to be a challenge.