Category Archives: Work

It’s Been a Long Time

Posted on by

I haven’t kept you up to date very well. It’s been 4 months since I last said anything of any note here. 4 months. Some might argue that I’ve not said anything of note here ever, but I’d like to think otherwise and in any case, that’s not the point.

Where on earth have I been for the last 4 months? I’ll try to blast it out in some kind of chronological order.

I’ve:

  • Had a new BT Business telephone line installed which resolved the problems experienced here at the cost of around 100 GBP. My ISP waived the migration fees and put me on a new, cheaper tariff, which was nice.
  • Felt proud as my team, Wolverhampton Wanderers, return to the Premier League as champions.
  • Left my previous job and took 2 very much needed weeks off before starting my new job. It was good to decompress for the first time in over a year.
  • Been to V Festival 2009 at Weston Park, Staffordshire and the most fun I’ve had in years. Really, I had a blast.
  • Earned the nickname of ‘Inappropriate Boy’ for many of the reasons I enjoyed myself at V so much, mostly to do with lacking any kind of social acceptability filter between the things which pop into my head and them pouring out of my mouth to enormous personal comedic satisfaction. Nobody was offended thankfully and it was taken mostly as intended – a bit of maladjusted cheekiness.
  • Moved house, down to Rugby in Warwickshire. BT Business contracts are 1 or 2 years. Umm. Not sure how to solve that problem without paying an enormous settlement fee. Apart from the people I work with, I know one guy in the area. Not sure how I’m going to avoid being single for the rest of my life in a strange town where I don’t know anybody.
  • Started a new job as an Open Source Consultant.
  • Been to Mönchengladbach. Met some dazzlingly intelligent, friendly people. Very impressed.
  • Organised LugRadio Live 2009, the last ever one. An awful, arduous process beset by impending disaster at every turn. As if every other year is any different. We seem to have the hang of this now, which is a bit late 😉
  • Been ill, just a cold.
  • Moved back to Wolverhampton to bridge a gap between houses. My house is currently being renovated so it wasn’t comfortable, but I was only there for a couple of days at a time.
  • Been to Yorkshire for a week.
  • Been ill again, unknown cause. Suspected stomach bug, felt like death for 4 days then miraculously started to feel better. Relieved.
  • Ran and attended LugRadio Live 2009, which despite all my complaining about the planning and organisation, actually turned out very well. It was great to see everyone there again. Unfortunately, I still haven’t worked out how not to get completely destroyed beyond all reasonable recognition on Friday night and then spend the rest of the weekend feeling so ill as to be barely human and even less functional as one. Bruno Bord is one of the funniest, most intelligent human beings alive.
  • Watched the finished version of Ubuntu UK Podcast presenter Tony Whitmore‘s LugRadio documentary – Don’t Listen Alone: A documentary about LugRadio, for the first time. Utter, utter brilliance. I’d seen a few draft versions before and they were fantastic, but the finished version was incredible and was liked universally. It took a total of 2 years to complete end to end. I can’t think of any other superlatives to heap on him, so, Tony, thank you. It’s fantastic. You’re a legend.
  • Accepted that there may well be another LugRadio Live show, not an event, just a live show, provided somebody else, such as you, organises the event.
  • Went to OggCamp. An excellent event and great fun. The combined Linux Outlaws and Ubuntu UK Podcast live recording was really good fun, very engaging. It was strange being in the audience watching other people do what I’d been doing the day before. I look forward to it next year 😉
  • Turned 33 🙁
  • Went to Dublin for a week and got very little sleep, not as a result of having fun.
  • Moved to a new house in Rugby. Nice place, shame about the heating.

So, you see, it’s not that I don’t love you or anything. I’ve just been a little pre-occupied with changing jobs, moving house repeatedly, travelling and organising the best Open Source conference which will never happen again. Things are about to get back to normal though, but not until after I:

  • Go to Devon for the weekend for my best friend’s birthday celebrations and probably make a fool of myself again.
  • Go back to Dublin for another week.
  • Go to Mönchengladbach for a long weekend to celebrate my company’s 10th anniversary.
  • Move house again probably.

Once all of those things are done, I will probably start to pay you the attention you deserve. In all seriousness though, the last 3 months have been absolutely crazy. I’ve never been so consistently occupied by real life stuff going on and changing under my feet than the last few months, so bear with me. Thankfully, I don’t feel unduly unsettled, I’ve just rolled with it and got on with it to be honest. Maybe the maturity which is lacking in my sense of humour is there in other areas.

Maybe catch up again soon?

Digging My Way Out

Posted on by

The catastrophes at work seem to be edging out of the way 🙂

I have a new backup machine, a grip on the server monitoring, text messages when something dies (though strangely nothing has since), a hand on the DNS system, a plan to make things recoverable within minutes and a replacement for the flakiest server in the world.

At last I can move on to doing things the way I want them instead of fire fighting all the time. It’s been a hairy few weeks and I’ve learned a lot. In fact when I review what I’ve done in the last few days, I’ve learned so much since I started work here.

Top of my personal list of priorities now are learning to sed, awk and shell script properly. I could do with a firmer hold on DNS too. I’d like to learn PyGTK also, but thats a little further away, the others are more of an immediate need. On a work note, I need to work out how to record bandwidth usage on a per machine basis. SNMP has been mentioned, but it looks *hard* and I’m not convinced it’s do-able that way. Suggestions?

See you at the SysAdmin of the Year awards 😉 Not that I’d win even if such a thing existed…

On a Crash Course

Posted on by

Ok I got back to work after What The Hack and I am now the senior sysadmin as my boss has now left. I’m a reasonable sysadmin but I’m still learning the ropes with regard to what I have actually taken over, so guess what happens.

First morning back and I’m in need of a gentle day browsing the net, writing email and reading my planets of choice, the main monitoring machine, running Nagios, goes down. Hard disk completely fucked, isn’t even recognised by the BIOS. Is it backed up? Is it my arse. So I spend the rest of the week flying blind and not knowing when things go down while I try to get my machine back up. This is not as easy as it sounds, I don’t know Nagios so it’s a crash course. I’m nearly there now, I just have to finish getting gnokii to send me sms messages, separating the config out into different files and adding the rest of the machines and services to the monitoring. The only problem is that we had a script and a database that created our DNS config files which was also not backed up as far as I can tell and like Nagios, it went down with the disk, so next time I have to add a new domain I’m going to have to go on a BIND crach course. Arse.

I begin to devise my own very comprehensive backup and mirroring scheme. All of our clients data is safe as houses of course, but I fear for the configuration of the infrastructure machines that I have inherited and I don’t want to have to rewrite it from scratch. I don’t want my BIND crash course to be a live crash course. Paranoia is a good trait in a sysadmin. Lesson 1: legislate for absolute catastrophe.

So, while I’m getting there with that, one of my tape backup drives dies and none of the spares work. So I’m waiting for the new drive to arrive before I start my Amanda crash course.

I was expecting the new drive today, but there was an error with the retailer’s stock levels and they don’t have the number we ordered and didn’t ship it. So maybe tomorrow.

Anyway, I’ve stepped ahead of myself a little here. That was sometime this afternoon. Last night I was helping my friend Dan move some stuff out of his old house as he has to clear the place before the landlord goes in. As gnokii isn’t texting me yet, I was unaware until this morning that one of our servers went down and wouldn’t come back up. When I found out this morning, I tried to do what I could but there was no way in remotely so, as is my responsibility, I raced into work early to sort it out. Disk errors. Lots of ’em. I fixed them but there were some complicated replication services on other machines that were dependent on this machine being available, so this problem made a mess of quite a few things. My boss managed to sort that mess out and we then spent the rest of the day building a replacement machine to take the weight of the one that went down. Hmmm crash course in installing a binary kernel module on the only supported Linux version with a huge list of kernel boot parameters on an unwilling RAID system. Took all day to get right, including trying out all of the available options that meant avoiding the binary only kernel module. Why these people don’t open source their driver I don’t know. It would mean someone could maintain the driver and bring it up to date with current mainline kernels and maybe get in the kernel proper. It would also add a sales point to their device that it is supported by the Linux kernel proper and free them in some sense of developer responsibility.

Anyway, I have to spend tomorrow installing all of the necessary, moving over the necessary from the other machine and then replacing the unnecessary, which will be a 5am start on Thursday morning :s

Just a hint to the readers: this is not indicative of the stability or quality of my employers servers, of the quality of service my company provides or the ability of the former or current sysadmins. You just don’t hear sysadmins whining out loud like this that often 😉