Look Ma, No Laddershakers
Back in the 1990s, I ran a technical team supporting the core computer systems for a major multinational supporting its critical business processes across the world. There were several hundred systems supporting sales, invoicing, finance, supply and logistics. Technology has moved ahead since then, but this organisation and culture can still work today. I’m not bigging myself up here, in truth the team ran itself, but this team is a great example of how it is possible to manage in a critical global environment without many of the pitfalls that are common in modern organisations.
The team was laddershaker-free. No problem managers, incident managers, release managers or change managers. That doesn’t mean we didn’t do these things, we just didn’t have dedicated roles for them, so the people doing them were well acquainted with the subject matter, not asking questions of others and influencing decisions where they had no expertise. Everyone in the team was an incident manager and a problem manager. We dealt with incident management, problem management, change management and pretty much all the ITIL processes, but we did it because it made sense, not because we found it in some IT Bible. We did these things as practical steps to keep systems running and frankly make our lives easier, not as slaves to some imposed doctrine. We didn’t have people called Incident Managers, Problem Managers and so on; people just did it as part of their job. That meant that the people doing them were technical experts, not process jockeys, and it kept the number of people involved to the minimum necessary, which cuts out excess emails, phone calls, forms and meetings.
When a system failed, or more often was at risk of failing, a pager went off and whatever the time of day or night the person paged got onto the system, they worked out what was wrong and fixed it. If the system was down and needed to be restarted, they did it right away. No reference to an incident manager, no looking up the rules in a book in case they got criticised later, no audio conferences with fifty people taking two hours to decide to reboot. If things got hairy they called me or one of the senior members of the team for help or advice, or called in someone from another team if the problem hit their patch. There is a key issue here of trust in technicians to do their job and make sensible decisions which has now commonly been removed by the laddershakers.
Every week, the whole team reviewed every incident, even if it caused no business disruption. The root cause was identified and remedial work was added to a prioritised list of fixes which the team worked through (using a role rotating through the team that we called Odd Job Man, there being no women in the team at the time). For less critical issues, this work list was used to fill troughs in demand, smoothing the workload of the team. This was managed with a spreadsheet, not a fancy incident and problem management system. We didn’t think of this as problem management, it was just common sense. No one had the job title Problem Manager – everyone was a problem manager. The culture of the team was robust to put it mildly. There was a strong professional pride in the team and intolerance of sloppy work. Stand-up arguments in the office were not uncommon where a team member was felt to have been slack, not least because it could result in another team member being woken at 5am to the resulting failure. We had no physical fights, but sometimes it came close.
We did not farm work out to low-cost locations, but as the system became global, we expanded our London base to Melbourne and Chicago. We didn’t do this to save money, but to give us local coverage in three continents and provide follow-the-sun support. We did not have an offshore (low cost) team because there was no routine work to offshore. We adopted ruthless standardisation which allowed automation. Automation in turn supported the standardisation because automatic processes left no room for people to set things up their own way (techies always think their special way is best). The automation was defined and created within the team, not through books written by people disconnected from reality. I was shocked later in my career to find how little was automated because offshore work is regarded as cheap so there was no perceived incentive to automate it.
We didn’t buy flashy and expensive tools. These were typically sold with impressive-looking dashboards with coloured lights and dials that didn’t achieve what we wanted but looked pretty to management. We built our own simple programs to check everything was running OK; when something wasn’t then if it was critical it set off a pager. Less critical problems were sent to an email account and dealt with as a background activity according to their urgency.
Nothing was outsourced. When we wanted to install a new system, we ordered hardware from the supplier, spoke directly to the in-house electrician for power requirements, the network engineers for connections and so on. Sometimes in the smaller offices we had the security staff change over the backup tapes. OK, that’s old technology but the point is that we took a pragmatic approach.
So if you have an organisation full of process jockeys, you really can remove them; just train your technicians to run the processes first.
I remember when I could run a script each morning which logged itself on to each database looking for problems. When it found one I sent a fix script before the problem was reported. I did this as I was fed up getting urgent problems reported via the incident management system at 4pm (problems that had occured hours earlier) keeping me at work late when I wanted to play cricket or go to the pub. I also remember being able to call the customer directly. There are problems with that, but you could build up a rapport with people and get problems fixed quickly. Much lost over the years.
We certainly fought the good fight! You’ll have to email me the name privately.
If you think back ladder shakers did exist, we just didn’t let them get involved, partially because people could see the results we got and partially because they were scared of us! I remember one meeting where some “expert manager” from the USA was being put over us and he held a meeting with us to explain how things were going to work.. end result was me losing it to the extent I told him I was leaving if he went his way and walked out… not exactly professional but demonstrating the passion we all felt about how things should be done. Can’t remember what happened to him but I don’t remember him darkening our doors again!