JRRZZ Blog

So it’s finished! Spot the penguin…

What’s a geek without his box? My box made Linux go like “this is not a software problem”, so I thought it might be time to replace my hardware. Plus, I was running all of my stuff on a single 7.200RPM S-ATA disk, including my jbisc project files. No, I know what you’re thinking. I hate waiting for rsync at shutdown. Not that this box is old or slow, oh this X2 4400+ still packs quite a punch and 2GB of RAM ought to be enough for everyone (pun intended).

So I went out shopping. Normally I’m an AMD guy but this time Intel’s performance advantage couldn’t be ignored. I’ve got the following setup, I hope you like it as much as I do. The motherboard is an Asus P5Q Premium, with an Intel P45+ICH10R chipset. It’s got lot of nice stuff on board, like that live Linux image boot splash top thingy. Of course the standard stuff like software S-ATA RAID, 4x Gigabit LAN and some Realtek audio chip. This board should support an 1600MHz FSB and it gives me the tools to OC the hell out of the Intel CPU on top. Not that I’m into OC’ing anymore. You know, stability uber alles. Speaking of which, the CPU is an Intel Core2 Quad 9550 running at 2833MHz. They say it’s fast, but I haven’t tested it yet. To add some spice to the mix I’ve added two 2GB DDR2-1066 GEIL memory modules, with those nifty mini heat pipes and LEDs. The memory timings on these modules are quite OK at 5-5-5-15. A Club3d HD4870 OC is responsible for drawing nice pictures on my screen. This is my first ATI since a long time, I love their renewed F/LOSS commitment. To make sure I can get to my files fast without losing them I’ve added a Promise SuperTrak EX8650 SAS RAID controller. It should be connected to two Seagate Cheetah 15K.5 300GB SAS disks at RAID level 1, both spinning at 15.000RPM. This whole lot is powered by an Antec TruePower Quattro 1000W, which should keep it all from powering down while decoding my National Geographic alternatives. To keep this pile of PCBs together I’ve bought a Lian Li PC-V1000z, and like Borat would tell you, it’s niiice.

Earlier today the lot was delivered to my residence. This is the result. You might think “but dude, where’s the case?”. Some guy somewhere along the supply chain thought it was OK to remove the HDD and PSU brackets out of the Lian Li box. So now I’m stuck with a useless, but still gorgeous, case. Some of you, but not many, might say “but dude, where are the hard disks?”. Well, somehow Promise thought I needed multi-lane cables, while they damn well know nobody needs multi-lane cables. So I’ve ordered a couple of multi-to-single-lane cables to get those hard disks to work.

So today I’ve only PXE booted the thing into memtest86+, which is passed flawlessly. I can’t call this my new rig just yet, it’s not even close. I could call this my new boards ‘n cables, but that doesn’t sound tough enough.

As you might (or might not) have noticed, this blog is a little faster than it used to be. That’s because it has been moved to a new machine, called gescheurd. The old machine, grebbehout, has reached it’s expiration date. I’ve bought a brand new Sun Fire X2100 M2 packed with the whole shebang. You know, Opteron 1222, 4GB RAM, 300GB 15K SAS RAID 1, 4x Gigabit, 24/7 4h hardware support and so on. This is the fourth M2 in our network and it has two identical twins. This triplet is the new wasda.nl web front end. The other M2 is slightly slower but boasts a whopping terabyte of storage, so it’s used to store spam and some email.

Yep, that’s right. Web front end. The databases are now all stored on the database cluster. This 32-core monster is still able to fit all the databases in RAM with room to spare. Also, most of the databases have been migrated to InnoDB. I should have done this way sooner.

This server eats bandwidth from ViaViela’s rack and happily shares three full address ranges with the company’s internet backbone. The network is powered by 3com switches by the way. Great no-nonsense hardware, and in my opinion superior to Cisco or HP alternatives.

This is a response to my previous post about OOP, over a year ago.

As I’m developing my own programming language, I’ve read a lot of documentation about procedural, relational and object oriented programming. Because of this newly acquired knowledge I’ve got to tune down last year’s attack on OOP. Lots of arguments are still valid, though.

Object oriented programming, as I’ve stated last year, has it’s uses. It’s not the holy grail. Not even close. OOP only works well when structures within the software are static in the real world. Following many of the examples in OOP books for instance, inheritance is explained by a car analogy. Of course car parts don’t change their order of, or dependance upon each other. But in the real world this does happen quite a lot. It’s hard and sometimes even impossible to restructure your code to reflect real-world structural changes when using the OOP model. It often requires rewriting much of the OOP wrapping code.

My previous ranting crusade against all that was OOP was a little mislead. All the OOP code I’ve worked with was primarily PHP code. PHP doesn’t support OOP. It tries to, just like it tries everything else, but fails. Just like at everything else PHP tries. Most of the C++ code I’ve worked with has been developed by fellow students. Not that I want them to take the blame for my anti-OOP stance of yesteryear, but their code has contributed to the negativity. I won’t blame them, it’s the teachers who don’t understand a single OOP concept. They only know how the language works grammatically but fail to translate the grammar to real world problems. So students just end up recreating all the stupid examples in their software the teachers conjured up.

The worst thing is, these students end up writing code in big code generating companies like Logica or Atos Origin. The cubicle philosophy is worshipped like burgers at Mc Donald’s. At these companies the code is just a byproduct. It’s the entire package they’re interested in. The V-model is followed by the letter and engineers are happy they can go home to their normal life, wife and kids at 5pm. This byproduct of theirs is sent into space, embedded into medical devices or used at your bank. Most of the IT flaws seen today can be blamed upon nobody telling these people how to write proper OOP, let alone proper code.

I know what you must be thinking. What the f*ck made this guy change his mind? Qt. If you want to know how a proper C++ OOP model looks like, look no further. Qt makes programs less complicated, it just does everything right. Alright, some aspects (like the dreadful Qt4 CSS) need some polishing, but overall Qt is a prime example. It’s documented properly, has a sane object structure and actually makes sense.

I don’t like languages that make everything behave like an object. It’s not easier for the developer, it introduces unnecessary overhead and often makes the program overly complicated when writing binary handling software like, say, file or socket streams. A file handle is an object, but fully privatised. Just remember what you’re abstracting. I also still hate database object mapping. For example Ruby on Rails rapes the database that way. Why would you need to convert table data to a language specific storage format just to convert it again to database objects? It takes at least twice the amount of data processing and, depending on your language of choice, sometimes twice as much memory. There’s no real good way to really tune your software for a certain DBMS. Database abstraction is evil, just learn proper SQL and procedural data type abstraction.

Finally I’d like to address the OOP in PHP topic. Of course you can write OOP using PHP. You’ve got two choices, really. You can use PHP objects and risk to stumble upon one of the many pitfalls it brings to the table. Or do it the Linux way. Much of the Linux kernel, and especially the API, has many OOP features but doesn’t use real objects. Of course these handles behave like objects, and in essence they are, but they’re implemented using C. You can use this technique to your advantage when designing PHP software. Use array-based handles and use the global scope in a civilised manner.

With some help from the DRBD and Heartbeat communities I hope to get our new cluster up and running within a week. With this new cluster, creatively named “The A-team”, the Wasda.nl fleet gets some serious oumpf. To give you an idea, here are some rough specs. Exact specs will follow when the cluster’s fully up and running.

• SunFire x86 machines
• 2-2 mirrored fail-over setup
• 36 64-bit processor cores
• 80GB of RAM
• 2.3TB of redundant storage
• Full gigabit backbone

According to my first measurements this cluster outperforms our current setup by 240%. When we look at processing power alone the A-team outperforms it by a whopping 1530%. Plus, our uptime guarantee will match that of our hosting provider now. Last year they had 100% uptime, while we only managed to get an uptime of roughly 97%. Server upgrades won’t mean downtime anymore, we can now update nodes safely without compromising the hosted services for any longer than 15 seconds. If this cluster can keep running and we have our weekly updates as usual, we can achieve a five-9 uptime guarantee.

A word of thanks goes out to ViaViela, the company this cluster is being built for, for supporting this great project. A list of all the people and companies will follow in this post’s follow up.

vielapeople:/home/jorrizza# uptime
 11:01:09 up 49 days, 23:49,  1 user,  load average: 183.19, 143.92, 87.82
vielapeople:/home/jorrizza#

This weekend we’ve moved all of our Sun machines over to our new data center provider, NXS. We’ve moved from Redbus Interhouse to the brand new DCG, which is quite an improvement. I’m glad TransIP has made the decision not to stay at Redbus. Not that Redbus was bad, but DCG is just a bit more modern and secure. It can also handle a lot more servers than Redbus. There have also been some major problems at Redbus lately concerning their flaky power supply. We’ve exploited the downtime to update the servers to the latest Linux kernel. We hope to get the same level of stability from this kernel as the 2.6.18-cks1 we’ve used before.

I’m sure you guys want some specs and pictures of our new home, so here they are. Geek pr0n, if you wish.

Data Center:

• Building: 1200m2 at 750kg/m2
• Floor: Raised 0.4m
• Racks: 1.2m deep 46u Minkels
• Power supply: 2 MVA redundant via ring
• Transformers: 2x1000KVA
• Power distribution: 230V AC, optional dual independent (2N) circuits
• Power: Average of 16A per rack, with a maximum of 32A per rack
• UPS: 2 x 600 KVA static UPS (upgradeable)
• Diesel Gensets: 2 x 1400 KVA diesel gensets
• Generator Supply: 48 hours onsite diesel with 24 hours refill
• Air Conditioning: ACU’s for data rooms. Free cooling concept
• Capacity: Average of 3.7kw/rack, maximum of 8kw/rack
• Chilled water generation: DX Cooling, 2N pumps, N+1 ACU’s
• Heat disposal: N+1 redundant dry chillers on roof
• Temperature: 22°c in cooling corridor
• Relative Humidity: 50%
• Rack Cooling: Cooling corridors
• High density: Up to 8 kW
• Fire Prevention: VESDA, Automatic Aron gas system
• Security: 24/7/365 on-site. Biometrics at all doors

Network:

• Routing: Redundant Juniper MX-960 routers
• Switching: Foundry FastIron SX 1600 switches
• Network: AS20857 (TransIP)
• Peering:
  • AMS-IX (10Gb/s)
  • Open Peering (10Gb/s)
  • AboveNet (2x1Gb/s)
  • Cogent (2x1Gb/s)
  • FLAG (1Gb/s)
• Edge Capacity: 960Gb/s

Wasda.nl Fleet – Amsterdam

• Edge Capacity: 400Mb/s
• Hardware: Sun Fire x64
• Machines: x2100, x2100 M2, x4100
• Effective Storage Capacity: 2.4TB
• Effective CPU Capacity: 29298 bogomips
• OS: Debian GNU/Linux
• Backup: Active redundant off site rsync

There you go, now go clean up the mess you’ve made.

I thought I’d seen most of it by now. I guess I was wrong.

The new file server, the X2100M2, has problems keeping lockd alive. The process keeps running, but somehow the connection dies on the gigabit ethernet link in between vielapeople and this machine, vielawarez. The only solution until now seems bumping the nfs-kernel-server every five minutes or so. I’ve talked to several people on IRC but the problem doesn’t sound familiar to them. I can think of two possible causes. The kernel version on vielapeople (the client) is 2.6.18 and on vielawarez (the server) 2.6.22. I’ve seen a lot of fixes in the nfs and sunrpc 2.6.22 source tree. Maybe this problem resides there. The other possibility is the second link, the one over the internet. Because both boxes have one NIC connected straight into the internet their hostnames obviously are available to each other. Maybe some weird RPC call uses the actual hostname of the machine and creates an RPC or data connection over the internet, where it doesn’t belong. The showmount utility clearly shows vielawarez accepting the vielapeople.wasda.nl hostname. This setup needs some work.

My new IBM system turned out to be a Pentium D Dual Core 2.8Ghz. It’s still not bad and still way too poweful for the task at hand. But you know, it’s not the Xeon. This isn’t really a problem. The real problem is IBM’s hardware vendor lockin scheme. I didn’t realise this until I tried mounting my brand new Seagate Barracuda 750GB SATA disks. They wouldn’t fit. This system needs special guiding rails for every drive inserted. The disk assembly itself looks really solid (of course I’ve already disected the entire machine) but has no room for standard 3.5″ screws. Normally the rails would be shipped together with the casing, or in a bad scenario, ordered seperately. But IBM doesn’t want you to use custom hardware. I’ve just ordered a 80GB IBM drive just for the rails. The bastards actually managed to put a price of 140 euros on an 80 GB drive. You don’t want to know what they dare to ask for a 750GB drive. And these drives aren’t special at all. Just standard Hitachi bulk drives. After I ripped the rails from the 160GB IBM drive and inserted the 750GB disk it just worked. So it’s just childish business keeping people from using their own parts. I was surprised the thing booted when I inserted custom DDR2 memory. Not everything on this system is bad news. On the contrary. It’s a rock-solid heavy duty case with top notch Intel hardware. Nothing exotic (except the ATI VGA). The cooling is surprisingly silent. During the POST process it does tend to lift a few feet above the ground when it tries to take off. But after it realises it’s at room temperature it turns silent again and lands safely. I wish I had the resources to upgrade it with the optional SAS controller. Now I’ve got to bootstrap a Debian install on software RAID-1. This is going to be a challenge since I’ve been spoiled with hardware RAID all the time.

Finally, the postal service (TNT) managed to loose my laptop twice in a row. I hope I’ll get the thing tomorow. I’ve already got the extended life battery. It looks a bit sad without the x61s sitting on top. The postal service also managed to loose the case badges I’ve ordered from Munnikes. Now the IBM machines have to go without Tux graffiti for a while. At times like these the socialist in me kicks in screaming “The service was so much better when it was owned by the government back in the day!”. Oh well, if they keep screwing things up like this they will be, some day.

Update (18:00):
I just got off the phone with TNT. Now it’s official. They actually have lost my thinkpad. They say they’ll try to ship it again tomorrow. That’s if they can find it. If not, they will have a problem with me. Some of my work isn’t done because of them, so they will have to pay for their stupidity. Also, the bogus hard drive I’ve ordered somehow isn’t where the hardware supplier thought it was. They said they had a few in stock, but it turns out they didn’t. So none of my new toys work right now. This really pisses me off.

I’ve got some new toys to play around with, and some toys are going to be here very soon.

At work, we’ve got a brand new Sun Fire X2100M2 with two 750GB disks, two gigs of RAM and a dual core AMD AM2 processor. It’s happily running Debian GNU/Linux and is going to be shipped to Amsterdam this week. It’s still a joy to work with these Sun machines. This will be our fourth Sun machine in almost two years. I think we’re going to place another one later this year, to get the load off of grebbehout.

Also at work, our high-end desktops are complete with the arriving of the monitor stands for the dual 22″ LCD setups. It just looks like CSI at out office right now, but without the Mac OS X. These desktops pack quite a puch, with 4 gigs of DDR2 and a speedy AMD X2 5600+ processor. Did you know Linux loves big pipes? I do now! Our software still manages to get these behemoths down to a crawl, though.

This week my new laptop will be shipped. After a collegue of mine bought an IBM Thinkpad x60s I had to have one too. It’s a great and at the same time tiny little laptop. I wouldn’t be me if I didn’t want to beat the guy. So I’ve asked to ship an IBM Thinkpad x61s with 12 cells of battery power in total. With the latest tickless patches in the -mm Linux kernel tree Tux will be able to operate for almost 12 hours(!).

And oh no, it doesn’t stop there! Danoontje, my dear ol’ file server, is getting a little hammered lately by the 24/7 OGG encode needed for radio.jrrzz.net. Constant loads of 1.00 – 1.20 on a uniprocessor VIA chip are outside it’s design parameters. Whenever I browse for files on it’s terabyte of storage LVM makes my radio stream hickup. Also panzertux, a ten year old Compaq Deskpro 3000 routing all my traffic, needs to be replaced. I’m sure it can run for another decade without servicing but I want to be sure my network keeps it’s uptime. So I’ve called in the help from the big guys at IBM. Sure, why not? They brought a nice IBM System x3200 for me to fuck around with. Two 750GB barracuda drives (what’s up with the 750GB disks lately?) in RAID-1, a dual core Intel Xeon and (only) 1GB of RAM. I’m going to put the two 500GB ATA-133 drives in there too, just for the fun of it. This new machine will be replacing both panzertux and danoontje and will be running Debian GNU/Linux, of course. With the demise of panzertux the last LFS based machine disappears from my network to leave a 100% Debian GNU/Linux powered network. To make up for the loss I’m going to use what’s left of danoontje to create a NetBSD workstation. It does have the VIA padlock, a cryptographic coprocessor. It’s a shame not to use it.

This summer it’s my very own summer of code. How geekish. Let’s see what I’m going to (try to) write this summer:

Work. Imagine having to rewrite the entire financial, social and administrative information system of a big company in eight weeks. You can’t can you? We expect writing down all the specifications takes at least two weeks. Well, that’s just for the basics. Yes, we. Luckily I’m not alone with this. We’ve got one highly skilled PHP programmer and two people who just can’t write proper code. This is going to be interesting. By the way, we even had to order a big-ass three meter white board just to be able to draw some schematics. My second employer also needs some major modifications in zjop, my e-commerce application. That’s going to be hard to fit in, but I think I can make it nonetheless.

But where would a guy be without a hobby? I’m not just going to write code in my spare time. I’m going to write coding tools in my spare time. Right now I’m working on PHP Reformatter. It can already understand most PHP code samples I’ve fed it. It’s written in ANSI-C and doesn’t use any non-standard libs. I need this piece of software because people tend to ignore coding standards. A perfectionist like me wants every byte of code exactly according to spec. Of course it’s just nonsense to edit all the files by hand.

I’m also officially announcing the birth of a new programming language. JBISC. I’m not sure wether or not I’ll be able to pull this off. But when this language eventually hits the public it’s going to be quite something. It’ll be a language designed for functional C programmers who need a fast and controllable scripting language. There’s just nothing simple out there. It’s all bloated and slow. It’s a hell to hack the languages themselves and it’s too complicated to write modules. This new language is going to look like C a lot but in the same time look and feel very different. I want it to be as easy like PHP and strict like C. No obligated memory management but still a small memory footprint. The syntax is going to be a lot like C with some of my ideas thrown in. I know this sounds too good to be true but I seriously think I’m able to create it.