Look Ma, No Laddershakers

Back in the 1990s, I ran a technical team supporting the core computer systems for a major multinational supporting its critical business processes across the world. There were several hundred systems supporting sales, invoicing, finance, supply and logistics. Technology has moved ahead since then, but this organisation and culture can still work today. I’m not bigging myself up here, in truth the team ran itself, but this team is a great example of how it is possible to manage in a critical global environment without many of the pitfalls that are common in modern organisations.

The team was laddershaker-free. No problem managers, incident managers, release managers or change managers. That doesn’t mean we didn’t do these things, we just didn’t have dedicated roles for them, so the people doing them were well acquainted with the subject matter, not asking questions of others and influencing decisions where they had no expertise. Everyone in the team was an incident manager and a problem manager. We dealt with incident management, problem management, change management and pretty much all the ITIL processes, but we did it because it made sense, not because we found it in some IT Bible. We did these things as practical steps to keep systems running and frankly make our lives easier, not as slaves to some imposed doctrine. We didn’t have people called Incident Managers, Problem Managers and so on; people just did it as part of their job. That meant that the people doing them were technical experts, not process jockeys, and it kept the number of people involved to the minimum necessary, which cuts out excess emails, phone calls, forms and meetings.

When a system failed, or more often was at risk of failing, a pager went off and whatever the time of day or night the person paged got onto the system, they worked out what was wrong and fixed it. If the system was down and needed to be restarted, they did it right away. No reference to an incident manager, no looking up the rules in a book in case they got criticised later, no audio conferences with fifty people taking two hours to decide to reboot. If things got hairy they called me or one of the senior members of the team for help or advice, or called in someone from another team if the problem hit their patch. There is a key issue here of trust in technicians to do their job and make sensible decisions which has now commonly been removed by the laddershakers.

Every week, the whole team reviewed every incident, even if it caused no business disruption. The root cause was identified and remedial work was added to a prioritised list of fixes which the team worked through (using a role rotating through the team that we called Odd Job Man, there being no women in the team at the time). For less critical issues, this work list was used to fill troughs in demand, smoothing the workload of the team. This was managed with a spreadsheet, not a fancy incident and problem management system. We didn’t think of this as problem management, it was just common sense. No one had the job title Problem Manager – everyone was a problem manager. The culture of the team was robust to put it mildly. There was a strong professional pride in the team and intolerance of sloppy work. Stand-up arguments in the office were not uncommon where a team member was felt to have been slack, not least because it could result in another team member being woken at 5am to the resulting failure. We had no physical fights, but sometimes it came close.

We did not farm work out to low-cost locations, but as the system became global, we expanded our London base to Melbourne and Chicago. We didn’t do this to save money, but to give us local coverage in three continents and provide follow-the-sun support. We did not have an offshore (low cost) team because there was no routine work to offshore. We adopted ruthless standardisation which allowed automation. Automation in turn supported the standardisation because automatic processes left no room for people to set things up their own way (techies always think their special way is best). The automation was defined and created within the team, not through books written by people disconnected from reality. I was shocked later in my career to find how little was automated because offshore work is regarded as cheap so there was no perceived incentive to automate it.

We didn’t buy flashy and expensive tools. These were typically sold with impressive-looking dashboards with coloured lights and dials that didn’t achieve what we wanted but looked pretty to management. We built our own simple programs to check everything was running OK; when something wasn’t then if it was critical it set off a pager. Less critical problems were sent to an email account and dealt with as a background activity according to their urgency.

Nothing was outsourced. When we wanted to install a new system, we ordered hardware from the supplier, spoke directly to the in-house electrician for power requirements, the network engineers for connections and so on. Sometimes in the smaller offices we had the security staff change over the backup tapes. OK, that’s old technology but the point is that we took a pragmatic approach.

So if you have an organisation full of process jockeys, you really can remove them; just train your technicians to run the processes first.