February 2008 Archives

Anyone who has ever worked in the world of IT operations knows how hard it is to stay afloat. We are fighting entropy. We have a seemingly never ending todo list and people never stop interrupting. Ever. It's the nature of this business, and we obviously like to hurt... err... like a good challange.

So how do you get things done? Everyone has their own system, and I'd like to share mine.

First, Come Up With an Organization System

I use a text file with vim. Seriously. It is located on a remote server that I can access from anywhere, and it has a very obvious format:

Apply patches to front-end IIS servers:

x verify that a policy exists

x schedule change

- apply patch on 2/20/2008 @ 00:01

It's a no brainer, really. The first line denotes the goal, and the following lines are the tasks. After each task is complete, I prepend an 'x' and immediately write down the next action. If the goal is realized, then I cut and paste this to the todo.archive file. Simple as that. In the morning I scan the list and pick my target. Every Monday I go through and make sure that everything still fits with reality.

Eventually, I want to add some vim fanciness so that I can navigate the file easier or at least get some nice syntax highlighting. After that, maybe I'll roll a simple Python cgi that can serve up the text file as an alternate interface. Hell, maybe it'll even have a RESTful API of some sorts so that you can have all sorts of interface. Ahhh... to dream... to dream...

I use the unix utility remind to keep up with calendar items. I find the format clear and it is something I can bend to meet my needs without too much hassle. Google Calendar may be a better choice, though. I haven't decided.

All of this data is backed up on a daily basis to Amazon S3 so I can rest easy.

Second, Keep Your Inbox Empty

Nothing raises anxiety like clutter. When I open my email inbox and see a bajillion read items, my mind races. Rather than do that, make your inbox serve your organization system. Every email is treated in one of three ways:

1. immediately answered
2. the goal is extracted and added to my todo list, and the email is moved to the 'archive' folder
3. the email is deleted

By doing this, I keep the stress level down and don't have to dig through piles or email to figure out what I still have to do. That alone has made a big difference.

Wrapping Up

So, that's how I do it. I'm no guru, and I'm sure it can be improved.

If you'd like to share your own system and you have a blog, I encourage you to write about it and trackback to this post.

If your IT operations team does not have a document library, I suggest you create one right now.  As mentioned earlier, it defines everything you do and takes very little investment to get started.  Install a wiki, make sure it is being backed up, and you are ready to start.

So how do you organize everything?

I'm one who hates complication, and try to stamp it out whenever I confront (or create) it.  Your front page of your library should contain links to all of your documents.  Don't spread them out and hide them on separate pages - it just makes it harder to find the right one.  If you make it difficult, people stop using it.  We all know this from experience.

I recommend creating a heading for each service that you provide (Active Directory, DHCP, DNS, FogBugz On Demand, etc, etc), and then bullet point your policies and procedures below.  The first item under each heading can be the general policy for that service, and then order the other documents alphabetically.

Do not worry about getting it right the first time.  You probably won't.  Since you're storing this all in a wiki, you can easily refactor as things change.

All of this may sound simple and trite, but again, just make sure you have it in place.  If you aren't writing it down, then you're not doing a good job.

...are emails like this one:

Hi,

I look after about a dozen Unix boxes (mainly Fedora Linux, but also
Solaris, HP-UX and Digital Unix). I came to the conclusion after reading
your article that I need to formalize updates and document everything on
the wiki a bit more comprehensively. I also came to the conclusion that
anytime I update any system configuration files on any of the servers I
should check them in to the corporate CVS.

You can't have too much documentation :-)

Thanks again for a useful resource.

Cheers,
Jan.

You can't beat that! It is fantastic encouragement, and really makes me feel like I may be able to do some good here.

Wanting to do things right around here, I have decided to summarize why I started this weblog and what I really want it to become. I have run a number of sites in the past, but have had a very hard time keeping them on track. This one is different, as the domain name itself doesn't leave a whole lot of room for variance. ;-)

Every day reminds me that no matter how many years I can claim to have worked in the IT industry, I'm still a beginner. Twelve years ago I was a beginner to Microsoft Windows NT 4.0 and to the still young GNU Linux. Four years later, I was a learning how to keeping my head in a large environment. Five years from then, I had to figure out how to lead a large team of systems folk in an even larger organization and keep my soul. I'm still not sure I made it all the way out of that one.

Today, I'm the sole systems guy in a thriving software development company. I have my hands full with keeping up with day-to-day issues, managing risk, planning for the future, being continuously on call, and trying to build a capable operations team from the ground up. I am regularly amazed that a company so young and so small can challenge me like this - but it is doing a damned good job.

One of the most frustrating realizations is that there isn't a guidebook to lend a hand. There are shelves full of operations management books out there, but none of them give you any idea as to what the hell you're going to do come Monday morning. The same can be said about a lot of the technical books out there: it is easy to find a book on your favorite Linux distro or a guide to implement Active Directory, but that doesn't solve the core problem of how you keep ahead.

Sometimes I think that the technical books on the wall behind me are just pleasant distractions from the mess I live in. It's far easier for me to tinker with something new or arcane than it is to face up to the more immanent and difficult problems.

So this is what is going on: I am going to build a guidebook of sorts on how one can truly take charge of their environment. It will be concise, and it will be living. I'm going to use my posts to cover the topics, and I will write revisions as necessary. I expect to make mistakes. In the end, I'll have a process that I trust and can use on a daily basis to not only keep things running in a safe and sane manner, but to keep making improvements.

Now, I will drift from time to time to keep things fresh and interesting. I'll ramble about code and and related news here or there. As an example, I have a small series of articles detailing typical stack smashing techniques that will be fun to polish and publish. This is a blog, after all - it is allowed to wear different hats.

There you have it, folks. Now you know who I am and what I am up to here. Time to get back to writing.