How to Migrate from Apache to NGINX

Friday, 2009-06-26 00:10, 1245967812 seconds since Unix epoch

Just imagine you’ve got a few web sites to look after, like me, have a tendency to over-engineer things, like me, want the best out of your hardware, like me, and have some free time on your hands, unlike me. What do you do? Right! Migrate Apache to NGINX. “Why NGINX?” you might ask. Well, Apache eats too much RAM for starters. If a website is lagging because of a faulty database, Apache will prefork itself to death and claiming all of your precious RAM in the process. NGINX is fast. Really fast. It uses less resources while doing way more useful work. It has it’s shortcomings too of course. It can’t handle that many configuration options. You can throw anything HTTP related to Apache and it’ll have some kind of module that understands it. NGINX can understand HTTP. That’s about it. But on the other hand, that’s all I want a web server to do. And finally, the logo. Whereas Apache has a purple feather, NGINX is the People’s Server of the Great Soviet Union. I mean, how cool is that? In Soviet Russia, NGINX serves you!

So first, we’ll need NGINX. My machines are running Debian GNU/Linux amd64, so it should just be an apt-get install. The latest stable NGINX is stuck in experimental, and because I didn’t want to be working with the legacy version, I rolled my own package. It’s available from the WasdaPuntEnEl apt repository if you’re too lazy to build your own. I’ve also uploaded the source, so you’re welcome to port it to your own platform.

The first thing you’ll need for a proper migration is a second IP address. You should know how to configure your OS to get that effect. If you can’t get a second external IP address, I’d suggest reading SSH or OpenVPN documentation. With this dual-IP method you can test your web sites on both Apache and NGINX, and verify the absence of difference. Well, except for the obviously enhanced speed that is. Reconfigure your Apache to only listen at the first IP address, the one you’re already using to serve from. Check for Listen 80 and friends. On Debian you can usually find these directives in /etc/apache2/ports.conf. Change it to only listen on your primary IP address like this: Listen 1.2.3.4:80.

The Debian package I’ve forked shipped with a decent default configuration. It has, just like Debian’s Apache, the sites-available and sites-enabled directories, a conf.d and sane defaults. The /etc/nginx/nginx.conf file is quite understandable.


user www-data;
worker_processes  2;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;

    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  off;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

We only need two worker processes, this is really enough to completely saturate the 100Mbit/s pipe this particular 2-core server is connected to. It’s basically a rule of thumb to take a worker process for each available CPU, with a limit of six or so. Unless you’re serving 1×1 gifs, which NGINX can do insanely fast from RAM by the way, you’ll be fine. You can increase the worker_connections value to compensate for increased traffic. So this config can serve up to 2084 parallel connections, which is enough for this particular server.

Next up: virtual hosts. Just like Apache, you can set up a virtual host in a file located in /etc/nginx/sites-availabale and symlink it into /etc/nginx/sites-enabled to enable that virtual host. The configuration of the virtual host is almost like Apache’s, with a few gotchas.


server {
        listen   87.253.149.111:80;
        server_name jbisc.org www.jbisc.org;

        access_log  /var/log/nginx/jbisc.access.log;

        if ($http_host !~ www.jbisc.org) {
                rewrite ^(.*)$ http://www.jbisc.org$1;
                break;
        }

        set $webroot /var/www/jbisc;
        location / {
                root   $webroot;
                index  index.html;
        }
}

This little blurp configures NGINX to serve JBISC’s site from /var/www/jbisc/, the same place Apache reads it from. The rewrite syntax is more readable than Apache’s. Any code monkey can read, and understand, the www-forcing code. You can also use variables in configuration files, something that you can’t do without once you’re used to it. I’ve set $webroot just for good practice, because you’ll need it’s location on more places than one in complex configurations.

Most of the sites hosted on my machines are written in PHP. Including this blog. So we’d need some kind of PHP support. Apache has mod_php, which includes the PHP interpreter into the Apache process. It’s the fastest way to communicate to PHP from the web server, but it just doesn’t scale that well. Every Apache process will have this interpreter on board, even if it’s sending you a 1×1 gif image. The best way, that I know of, to serve PHP from NGINX is to use FastCGI. Unlike CGI, which starts an interpreter for every request, FastCGI has a number of interpreter processes running, listening on a socket. Luckily PHP has a built-in FastCGI method. First, you’ll need PHP listening on a socket. To get this thing running we’ll use spawn-fcgi from Lighty. It’s a nice little wrapper program making it easier to manage the PHP processes. I’ve modified an init-script that I found somewhere on the web to start a few PHP processes, which will listen on a Unix socket for incoming FastCGI requests. It will also drop PHP’s privileges to Debian’s www-data user. Copy it to /etc/init.d, make it executable and (if you wish) add it to your boot using update-rc.d. Don’t forget to start it before adding the following config to NGINX.


location ~ \.php$ {
        fastcgi_pass   unix:/var/run/php-fastcgi.sock;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME   $webroot$fastcgi_script_name;
        include fastcgi_params;
}

This bit of configuration should go into your existing server { } section. It will match the URL for the PHP extension, and send the request to the eagerly waiting PHP FastCGI socket. The fastcgi_params include is another configuration file listing all of the default fastcgi_param directives. It’s just a shorthand. Don’t forget to add index.php to your index directive. Otherwise you’ll end up with a 403.

You won’t miss mod_rewrite at all. I’ve easily migrated all of the mod_rewrite configuration to NGINX rewrites. They both share basically the same regular expression syntax. You should get rid of, or block, the .htaccess files. NGINX doesn’t support those.


if (-f $request_filename) {
        break;
}

if (!-e $request_filename) {
        rewrite ^(.+)$ /index.php?q=$1 last;
}

These two should go in the location / { } part of you virtual host. These rewrites are used to serve Wordpress for instance. Everything that can be translated to a file name gets hosted directly, otherwise it’s the q argument for index.php.

There’s one final little piece of config I’d like to share. I love mod_userdir. It’s just a nice way of putting stuff online without difficult configuration. The following should go into a file, which can be included in the server { } part of your virtual host. It realizes the same behavior, including PHP support. This won’t work server-wide, just for the domains you include this file for.


location ~ /~([^/]+)(.*\.php)$ {
        alias /home/$1/public_html$2;
        fastcgi_pass   unix:/var/run/php-fastcgi.sock;
        fastcgi_param  SCRIPT_FILENAME   $request_filename;
        include fastcgi_params;
}
location ~ /~([^/]+)(.*)$ {
        autoindex on;
        index index.html index.htm index.php;
        alias /home/$1/public_html$2;
}

I’ve also managed to get both Rails and Django running, using mostly the same technique. The only difference between those frameworks and PHP scrips is that the frameworks ship with either their own web server you can proxy (like Thin for Rails) or a FastCGI server (like manage.py runfcgi for Django).

You can test your migrated sites by editing your local DNS resolving to resolve the hostname to the second IP address. You can either change you local hosts-file, or make your local DNS caching server apply the changes. If everything checks out you can reconfigure both NGINX and Apache to respectively listen on the primary IP address and stop listening on it. A quick restart of both of them finishes your migration.

This entire server has been migrated in little under four hours, with some relatively complex web sites. It’s also still running Apache to host the Subversion and Trac sites. Subversion over HTTP is one of those things NGINX just isn’t designed for (yet).

Blog Fixed

Sunday, 2009-06-21 22:35, 1245616552 seconds since Unix epoch

As probably nobody noticed, heck, I didn’t even notice it until recently, this site was severely broken. The theme didn’t survive the WordPress update. I’ve got a new (customized) theme now, which I like. You should like it too.

I’ll add some real content some time soon. I promise.

Tweet

Friday, 2009-06-12 16:22, 1244816541 seconds since Unix epoch

Okay, I’ll join the hype.

I’ve written a little Ruby script that “tweets” my whereabouts automatically every morning. It even detects who I’m with and what I’m doing. Damn, I like Ruby.

Bye MailScanner

Friday, 2009-06-05 15:36, 1244209015 seconds since Unix epoch

Because of this:

Mail Queue

I’ve replaced MailScanner with a plain ol’ spamassassin postfix content filter. And believe it or not, spam is caught way more efficiently now. No more delays, double file extension nags and 2k line config files.

PHP APD Completely Useless

Saturday, 2009-05-23 14:25, 1243081539 seconds since Unix epoch

We all know PHP isn’t a very well thought out programming language. It tries to do a lot of things, but fails to do most of those correctly.

PHP has a nice little extension, the Advanced PHP Debugger. APD for short. It allows you to analyze and alter PHP’s internals. It also supplies the rename_function function. Apart from it’s reversed name (it should be function_rename, like function_exists etc), it’s quite useful. It gives PHP some aspect oriented features, like allowing you to rename mysql_query to add transparent logging.

But PHP wouldn’t be PHP if they didn’t fuck this up. It’s okay to rename PHP’s own functions, but don’t try to rename your own.


jorrizza@shoebox:/tmp$ cat > balls.php
function foo() {
return 0;
}
rename_function('foo', 'original_foo');
var_dump(function_exists('original_foo'));
original_foo();
?>
^D
jorrizza@shoebox:/tmp$ php balls.php
bool(true)
Segmentation fault
jorrizza@shoebox:/tmp$

Kapow! Segfault! It’s even better when you’re using Apache mod_php. An apache child will segfault, making other mod_php processes behave quite strangely all of a sudden. This bug has been known since 2007, but nobody seems to care. It’s a Zend extension, we can’t support that, oh no.

OpenBSD GCC Fun

Monday, 2009-05-18 16:08, 1242655693 seconds since Unix epoch

First some good news. The cluster I’m building is finally able to stream H264/AVC using RTSP. It’s also able to chain RTSP links in between nodes in order to support tree-shape content distribution within the cluster. All of this functionality is easily accessible from a neat little Ruby API.


require 'vlm'
vlm = VLM.new('127.0.0.1', 4212, 'admin')
vlm.broadcasts.each do |broadcast|
vlm.play(broadcast)
end

I had to fix some things in VLC to make it at least workable for my setup. Some really weird assumptions and race conditions still plague the code base. The weirdest error was the following, while dynamically linking Live555.

undefined symbol ‘__gxx_personality_v0′

This symbol is used by GCC in some Java related internal stuff. What the hell is this error doing in VLC’s output?

Because VLC can’t be built (anymore) using the OpenBSD standard GCC 3 compiler, I had to use GCC 4 from the ports collection. Apparently, when dynamically linking GCC 3 compiled binaries with a GCC 4 product, some symbols are lost in translation. The solution was shipping my own GCC 4 compiled Live555 library.

Screw Gravity

Tuesday, 2009-05-05 23:50, 1241560227 seconds since Unix epoch

I’ve got some really neat stuff planned for my next visit to New Zealand. The epitome of neatness will be to check whether gravity still works. I kind of hope it still does, actually. Anyway, to make this trip a bit more interesting a total of four jumps have been confirmed thanks to my new Kiwi friends. A jump a day, taking it easy.

The first jump will be a 43m jump off of the Kawarau bridge. It’s the place where bungy jumping was born. It’s just a jump I have to make for history’s sake. Here’s a video of a fellow Dutch tourist dedicating her jump to her grandparents. I know, don’t mind the music.

The second jump will be a bit cooler than the first. It’s a structure suspended 400m over Queenstown with a runway inside. You can run out of it with the cord attached to your waist. And best of all, you can even jump in the dark. I’m still not sure what I’m going to do, but it’ll be fun. This guy does a flip of some sort.

The third jump is even better. It’s a 134m drop from a pod above the Nevis river, hanging in mid air, attached to some cables. They say it’s a 8.5s ground rush. I can’t wait. I just had to share this video. Now I can’t possibly chicken out. They have to do something about the awful music though.

The fourth and final jump will be 4572m deep, or 15.000ft if you prefer. The 65s ground rush will be like no other. My bucket list is getting shorter and shorter. No, of course it won’t be a bungy jump. It’ll be a sky dive from an awesome airplane with teeth. Really!


Teeth

Re: 10 Questions for the Atheist

Wednesday, 2009-04-22 00:16, 1240352190 seconds since Unix epoch

Following up on the recent discussions around the post from Scott Pruett on LifeWay and the Antichristian Phenomenon’s rebuttal, here’s my input. I’m very sorry for the late reaction, I have been busy lately.

The overwhelming consensus of science is that the entire cosmos (including space and time) came into existence at a finite point in the past. All of our observations, equations, and physical laws testify to a point of origin for this universe.
In light of the troubling evidence for a beginning, and that we may not even be able to find a natural cause in principle, what explanation is given to the questions, “Why is there something rather than nothing?” and “Where did it all come from?”

I’m sorry, but I can’t match your findings with those of the scientific communities. The only thing most of the scientists agree on is that we can’t determine what has happened before the big bang, based on our current evidence. The conclusion that this must be the beginning of time and space is premature. The only conclusion valid at this point might be that the big bang is the beginning of time and space that we know of. As long as there is no insurmountable evidence in support of the claim, there is no consensus.

The past several decades have added profoundly to our knowledge of chemistry, physics, and cosmology. It has become increasingly clear that we live in a universe finely tuned for the support of complex life. This fact is so universally acknowledged that even secular scientists have coined the term “Anthropic Principle” to describe it.
How is it that we live in such an exquisitely fine-tuned universe? Even assuming that the universe could have popped out of nothingness, why should it have been such an orderly and hospitable one? Is there a scientific, testable answer for this question that does not simply appeal to imagination?

No, this is not universally acknowledged at all. The very assumption that this universe exists to support human life is ridiculous. You are confusing cause and effect. Human life is merely a link in the vast universal process of causality. Carbon based life forms have been adapting to their surroundings for billions of years on this planet. The process of evolution simply made them more adept to their surroundings. Now, after all these years, mankind is the result. We’re not perfect just yet. There are many potentially lethal things we encounter every day. People die every day during these encounters. This universe isn’t finely tuned, we are in the process of tuning ourselves to better fit this universe.

The problem of abiogenesis (the origin of the first lifeform) is one of the thorniest and most intractable issues in chemistry. Our increasing knowledge of microbiology and earth history has only added to the complexity of what needs to be explained. The simplest life is equivalent to modern bacteria, which is loaded with complex activity, information, and molecular “machines.” The fossil record does not give evidence that there was a “prebiotic soup,” or that there were any biological precursors to the first organisms, or that the atmosphere was the ideal mix to yield the necessary molecules, or that there was the expected long period of time between when the Earth could support life and when it actually appeared. Evolutionists regularly segregate the abiogenesis problem from the issue of evolution because (1) it is a challenge they’d rather not be saddled with, or (2) it is the most logical point for possible divine intervention. However, for the atheist there is no escaping this issue; they are obliged to seek out some purely natural explanation.
What hope for an explanation do you have? Are you satisfied to have problems like this that are unanswered, or even unanswerable?
In telling the tale of life on earth science writers often unconsciously use the word “miracle” for the appearance of the first organisms.

Indeed, this is a hard one. At this moment, everything is possible. Something isn’t immediately true when someone says so. Until there’s proof of what happened at the very first stages of life, all we have is conjecture. Every theory, including divine intervention, is still in the competition. Most theories are way more plausible than an improvable event, but until there’s real proof everyone gets the benefit of the doubt. Nobody gets to claim the truth just yet. So the theistic stance of “I’m right until proven wrong” doesn’t have any merit in this intellectual arena.

This is one of many riddles we still have to solve. Riddles like these make life interesting, don’t you think?

What kind of evidence is needed before we are to actually accept that something like this really is a miracle?

Come on, please re-read your question. A miracle is an event without a scientific explanation, without real evidence. A scientist would never accept such nonsense.

Logic and mathematics are abstract principles that have been discovered rather than invented. We cannot do science, communicate, or navigate this world without them. They appear to stand outside of nature to describe and measure it. As Albert Einstein said, “The most incomprehensible thing about the world is that it is comprehensible.”
What is the source of math and logic? The existence of this remarkably fine-tuned universe aside, how is it that we have these “languages of reality” to so elegantly describe and interact with it?

This question is exactly the same as your first question. These methods for describing our surroundings are merely abstractions from our highly developed pattern recognition skills. We can observe, discover patterns and reproduce these discoveries. Even ants can do this, for example.

Another transcendent entity that is a problem for atheism is morality. With no divine author or judge there is no reason to think that there should be any moral laws that we are obliged to recognize and keep, except for self-serving reasons. And yet, morality aligns with our deepest intuitions: we expect others to recognize it; we urge our kids to exercise it; therapists get rich repairing the effects of its abuse; we judge criminals insane if they do not recognize it; and all cultures affirm it in common principle if not in individual application.
Do you deny objective morality; that the difference between Mother Teresa and Hitler is not just a matter of preference, like chocolate vs. vanilla ice cream? If not, then how do you ground morality; how do you explain where it came from and why we ought to be moral tomorrow?
Skeptics often bring up the “problem of evil” as evidence against God, i.e., if there is a good and all-powerful God, then why is there evil in the world.
Do you think that this is a valid objection? If so, are you admitting that there is evil in the world? What is “evil,” and do you not admit its opposite: “good?”
The problem of evil objection only makes sense if such things as good and evil are objectively real, not just preference statements.

Good and evil are purely subjective. So yes, Mother Teresa and Adolf Hitler can bot serve as moral guidelines. Morality is a complex matter, constructed out of our primal instincts, subconscious and higher cognitive skills. During the course of our evolution, species that grouped together had a better chance of survival. Groups needed extra skills to be effective, from which social interaction is a crucial one. Of course groups operate better when the group members agree not to kill each other. This is also exactly the difference between your theistic explanation and that from evolution. If objective morality is indeed divine, why is it limited to one’s group?

But I said morality is purely subjective. The conscious mind eventually gets to decide the meaning of good and evil. In other words, you decide. Most people adhere to group instincts with little real individuality. Some people are exactly the opposite and naturally become outcasts. The balance of influences on you morality eventually defines your personality.

In the atheist worldview we are products of time, chance, and blind forces – there is no objective meaning and value to our human existence. Yet our deepest longing is for our lives to count for something. We intuitively know that humans have rights and dignity.
Does life really have no point other than what you pretend for your own sake? Will you say, like atheist philosopher Albert Camus, that the only serious question is “suicide?” What values and purpose will you instill in your children? Will you be honest with them, or will you borrow ideas from some non-atheistic belief system so as not to disappoint?

This is not something I can answer for all atheists. It’s a personal thing, this meaning of life. I live by the words of two bright people, Anton Szandor LaVey and Arjen A. Lucassen. Arjen is a musician and a fellow Dutchman. He once wrote in one of his songs “the meaning of life is to give life meaning”. It’s not the spoils that count, it’s the hunt I’m living for. Seeking for the right thing to do for me is the right thing to do. When I’m (old and) dying, I’d like to look back at my life with a smile. That’s where what LaVey has taught me comes in handy. He wrote about gaining immortality in the memories of others. In other words, if I make a positive attribution to people’s lives and when I’m remembered for it, my life has had meaning.

In the world of atheism, where there is no soul or transcendent “self,” humans are simply biological machines, and our minds are just computers made out of meat. With this in view there is really no room for something like freewill, since we are all just operating according to our “programming” and our environmental influences. And there are great difficulties in conceding that chemistry can produce something as abstract as “consciousness,” or at least anything qualitatively different from what we ourselves might ultimately produce using computer technology.
Are you prepared to accept the idea that no one is really morally responsible for their bad behavior and, conversely, that virtuous behavior is not commendable? In what way will you seek to convince me that I am really not a conscious and self-aware being; that it is just a complex biochemical illusion? Can you accept that computer programs may one day be just as much “persons” as you, yourself?

I believe I’ve answered your question in this post and this post on the ACP website.

Every known time and culture is rich with stories of near death experiences, ghosts, angels, demons, prophetic dreams and visions, and miraculous healings. While some of these are certainly spurious or not well documented, others have reasonable experimental support. In addition to this, humans seem to be incurably religious; the idea of God and the spiritual is deeply entrenched in the human psyche, if not in its actual experience.
What are we to make of all this? If man is simply an adapted biological organism, then how is it that we did not manage to adapt to our natural environment in this area – why are we not “naturalists” rather than theists? Can’t any of this be a hint toward reality, or must we think that the bulk of humanity flirts with insanity?

I honestly don’t know why people believe such irrational an illogical things. I’m still puzzled how seemingly perfectly sane people can find intellectual comfort in fairytales and conjecture. And frankly, yes, I believe this is a mental illness. The vaccine is readily available too. It’s called education. Maybe you’ve heard about it. It isn’t that popular in the theistic nations around the world.

The case for the Jesus of Scripture is extremely compelling. There is good evidence that the New Testament was written in the generation of the Apostles. We have thousands of copies of these documents in their source language, some of which go back inside of 100 years after Jesus’ death. There is no evidence of significant corruption in the known manuscripts. There is no motivation and evidence for fraud among the apostles and church fathers – most died martyr’s deaths. The trend of archaeology is toward validation, not denial, of what it is possible to confirm in Scripture. Even non-biblical manuscripts support various key details of Christian theology.
The burden of proof is generally on the one seeking to deny historical records.
What alternative explanation do you offer to the New Testament documentation and the tradition of the church, and what support do you have for your theory?
Is it because of the miracles that you doubt the Scriptures? If Jesus really were God in the flesh, how would you expect Him to confirm that fact?

The bible is a collection of fairytales, nothing more. The Jesus character in the new testament is a copy of Horus, an Egyptian fairytale’s character. The biblical texts are not historical records. The amount of contradictions and lies is simply too staggering to deny. There’s no evidence of corruption, but there’s no evidence of the opposite either. The literal texts have never been found, only fragments that vaguely resemble some of the stories. There’s just no hard evidence of Jesus ever being alive, only of some so-called saviors through the years. There are two people in Amsterdam I know of who both claim to be the savior, but that doesn’t prove the second coming of Christ. The burden of proof is still with the Christians, since there’s just no hard evidence to refute. If that Jesus character really is a god, he can strike me down right now. The fact that you’re reading this proves that he isn’t.

Christians are often accused of being simple-minded, superstitious, or irrational.
Is it so unreasonable for us to believe that the universe had a beginning because it actually was created; the laws of physics are so fine-tuned because it had a designer; people are preoccupied with good and evil because they are real things; we long for purpose and meaning because they exist to be had; life from non-life really is miraculous; consciousness and freewill seem real because they are; people are incurably religious because there is actually something real in religion; and the historical case for Jesus is so tenacious because it is actually true?
If there really is no meaning or purpose to life, no objective good or evil, and the existence of “truth” itself is open to debate, by what standard will you condemn the beliefs of Christians?

By my own standards. They’re the only standards I know and the only standards I can judge by. In this last question you’ve simply stated that you’re right because you think your right, and that you question my right to refute that. Yes, I’d call that simple-minded, superstitious and irrational.

Twitter

Sunday, 2009-04-19 19:49, 1240163341 seconds since Unix epoch

No, you don’t want a cookie. If mommy says you can have a cookie, you will consume the aforementioned cookie. Have a bullet for now.

Progress on the OpenBSD Cluster

Tuesday, 2009-04-07 14:26, 1239107203 seconds since Unix epoch

As requested by some people, another update. The project is progressing nicely. There have been a few minor setbacks and a few more lie ahead. But things are going quite well nonetheless.

Since the last post a lot of timing and connection tests have been thrown at the cluster. I’ve bombarded the poor collection of boxes with ICMP floods, gigabytes of random data and h264 media while torturing it by pulling out random cables and hard disks. The setup withstood everything without real problems. I’d run a nuclear submarine on this system if I had one.

There are a few practical problems though. The firewalls use pfsync to share connections between the machines. This enables them to keep every connection alive when a fail over occurs. The entire IP state table is essentially duplicated constantly between the routing firewalls. If an application node fails though, it’s virtual MAC address (and associated IP addresses) will be transferred to another host, but the active connections cannot. So if you’re streaming some high definition pornography and the application node providing your audiovisual entertainment fails, you’re screwed. Well, that might be a bad choice of words, because the screwing would stop. But anyway, what happens is that the connection will survive from your home box to the cluster. Inside the cluster however, the connection won’t be re-established to the new node.

Right now I’m trying to get VLC to stream over RTP instead of HTTP. RTP (usually) uses UDP, so in theory it should work. I’m still working on understanding the nature of RTP streams, especially now I’m using RTSP. I haven’t got it running just yet, I guess it’s a firewall problem or something. Also, I suspect the RT(S)P states to be stored in a node’s VLC process, which would render this entire idea useless once again. I sure hope I don’t have to write an RTSP sync protocol.

As I’ve written in my previous post about this project, the hardware was kindly donated by my employer. Since the hardware can’t leave the building, I’ve been spending most of my time at my employer’s office instead of where I should be, at M2X. Luckily the hands-on part of building this cluster has been finished, so working remotely would suffice. I’ve been playing around with NX technology lately, so I thought it to be a great idea to put that knowledge to good use. The collective.borg.local cluster is running an X11 box, so that wouldn’t be that much of a problem, right? Since I’m an Open Source kinda guy, I’m using the FreeNX variant. It works, but it’s quite limited. The biggest problem is it’s lack of XDMCP support. With NoMachine’s NX server I could easily select the cluster’s XDM server and set up a compressed X11 session. FreeNX only wants to run local X sessions using XSession or KDE/Gnome scripts. I’ve solved this problem with a nice hack, using OpenSSH’s X11 forwarding. First, I made sure the server running NX could login on the cluster’s X11 box using public key authentication. After enabling X11 forwarding on that machine, an X session running a remote openbox would enable me to mimic XDMCP behaviour. Because the DISPLAY environment variable is set for every child process in the SSH session, every application I start using the window manager will be sent over the same SSH tunnel. The only thing the NX server’s .xsession file contains is exec ssh -X collective openbox. So when I login on the NX server, it will request the cluster to start an openbox process. That process will send it’s X11 data to the NX server, which compresses it and send it to my NX client. Anywhere on the internet. Yes, I could have just used SSH, but X11 is way cooler.

NX Technology

For the time I’ve got left a few really nice projects lie ahead. I’ve got to make this cluster nicely manageable with a web interface and stuff. I think I’m going to use Puppet and Ruby for that purpose. After that’s done some QA software has to be made to ensure the cluster’s streaming quality.

I hope the links are useful enough for you to understand what I’m doing. It’s not that hard, really ;)