Daily Archives: 09/08/2005

On a Crash Course

Posted on by

Ok I got back to work after What The Hack and I am now the senior sysadmin as my boss has now left. I’m a reasonable sysadmin but I’m still learning the ropes with regard to what I have actually taken over, so guess what happens.

First morning back and I’m in need of a gentle day browsing the net, writing email and reading my planets of choice, the main monitoring machine, running Nagios, goes down. Hard disk completely fucked, isn’t even recognised by the BIOS. Is it backed up? Is it my arse. So I spend the rest of the week flying blind and not knowing when things go down while I try to get my machine back up. This is not as easy as it sounds, I don’t know Nagios so it’s a crash course. I’m nearly there now, I just have to finish getting gnokii to send me sms messages, separating the config out into different files and adding the rest of the machines and services to the monitoring. The only problem is that we had a script and a database that created our DNS config files which was also not backed up as far as I can tell and like Nagios, it went down with the disk, so next time I have to add a new domain I’m going to have to go on a BIND crach course. Arse.

I begin to devise my own very comprehensive backup and mirroring scheme. All of our clients data is safe as houses of course, but I fear for the configuration of the infrastructure machines that I have inherited and I don’t want to have to rewrite it from scratch. I don’t want my BIND crash course to be a live crash course. Paranoia is a good trait in a sysadmin. Lesson 1: legislate for absolute catastrophe.

So, while I’m getting there with that, one of my tape backup drives dies and none of the spares work. So I’m waiting for the new drive to arrive before I start my Amanda crash course.

I was expecting the new drive today, but there was an error with the retailer’s stock levels and they don’t have the number we ordered and didn’t ship it. So maybe tomorrow.

Anyway, I’ve stepped ahead of myself a little here. That was sometime this afternoon. Last night I was helping my friend Dan move some stuff out of his old house as he has to clear the place before the landlord goes in. As gnokii isn’t texting me yet, I was unaware until this morning that one of our servers went down and wouldn’t come back up. When I found out this morning, I tried to do what I could but there was no way in remotely so, as is my responsibility, I raced into work early to sort it out. Disk errors. Lots of ’em. I fixed them but there were some complicated replication services on other machines that were dependent on this machine being available, so this problem made a mess of quite a few things. My boss managed to sort that mess out and we then spent the rest of the day building a replacement machine to take the weight of the one that went down. Hmmm crash course in installing a binary kernel module on the only supported Linux version with a huge list of kernel boot parameters on an unwilling RAID system. Took all day to get right, including trying out all of the available options that meant avoiding the binary only kernel module. Why these people don’t open source their driver I don’t know. It would mean someone could maintain the driver and bring it up to date with current mainline kernels and maybe get in the kernel proper. It would also add a sales point to their device that it is supported by the Linux kernel proper and free them in some sense of developer responsibility.

Anyway, I have to spend tomorrow installing all of the necessary, moving over the necessary from the other machine and then replacing the unnecessary, which will be a 5am start on Thursday morning :s

Just a hint to the readers: this is not indicative of the stability or quality of my employers servers, of the quality of service my company provides or the ability of the former or current sysadmins. You just don’t hear sysadmins whining out loud like this that often 😉

That was What The Hack

Posted on by

Got back from What the Hack late last Sunday in time to visit my girlfriend, whose birthday it was and who was gracious enough to not dump me for going away over her birthday.

Man, what a time. To be honest I didn’t do anything or see anything. We got wasted every night. Jono, Aq, Bill, Matt, Garp and Graham who were my travelling companions and myself spent the 4 days or so sitting around hacking on a few things in the non-security sense. Jono played with gstreamer 0.9, while I tried not to die of a hangover.

Aq, Jono and Bill also spent a good amount of time laughing at the chinny raccoon and gimping up suitable mockups of all manner of chin related articles, culminating in the Little Book of Chin (link sadly dead). For anyone who hasn’t got a clue what this is all about, it is embedded in British schoolboy humour and is to do with the fact that when someone is lying you say they have a Jimmy Hill chin (Jimmy Hill is an English ex-footballer with an infamously large chin who became renowned as a pundit for talking crap). Therefore you might start stroking your chin is you are implying someone is bullshitting. Furthermore you might say that you, “Chinny reckon” in so much as you reckon (not) with added chin, hence you chinny reckon. Jono and Aq employ this mightily on LUG Radio.

Anyway, in true Planet Gnome style, here is the 5 things I learned at WTH:

  1. Geeks can’t dance for shit. I’ve never seen anything so funny, it’s like an American high-school movie with all of the ‘cool’ kids taken out and all the dorks taking their chance to shine. Awful.
  2. Having a gay, Germanic looking geek head bull-like towards you, stop dead in front of you and stare at you for 5 seconds longer than is comfortable with their head at a coquettish angle, is scary enough to make you clench so tight that your bumhole turns into an impenetrable wall of steel.
  3. Night-time musical entertainment sure wasn’t any higher than an after thought on the agenda, if the quality of the output was anything to go by.
  4. Eindhoven is a quiet, unpopulated city, where there are very few people walking about. We concluded this to be because everyone was at work and not hanging around like jobless, dole scrounging wasters like in England.
  5. There are a lot of dull, braying, over-confident arseholes making a fuss of themselves in front of everyone, unaware of the fact that they are short, fat, squat looking and lacking in any kind of social skills. Sadly most of these people were British.

And there you have it. That was What the Hack.