Blog

Disabling Thousands of Production Desktops in a Retail Call Center

Written by Bob Fischer | Nov 21, 2022 5:00:00 AM

DevOps is often misunderstood as simply tools and process, and that's part of the story but misses the mark. DevOps is really about building greater cross-organizational teamwork. Teamwork that ultimately enables speedier time-to-market, higher quality, and more rapid learning. Traditionally, this isn't how people work. It wasn't how I worked when I was a functional manager. I was focused on my individual part of the puzzle being great, not on what really mattered to our customers or the organization. A big production outage provided a great catalyst for me to rethink how I worked. It wasn’t pretty, but it was very instructive.

It started with a group call with a Senior Vice President at the large financial services company where I worked. We'd recently had a serious production outage that affected thousands of desktops in our call center. He wanted to understand what happened. Each manager involved, me included, described how our group had done the right thing. We implied, without saying directly, that one of the other groups must have messed up. We had great alibis, but no sense of a collaborative interest in really understanding the gaps in our process. We were missing one of the key ingredients of what is now considered the "DevOps Mindset" - systems thinking.1 This event, and the follow-on process work, radically altered how I think and behave. It was a key moment for me adopting a "DevOps mindset,” years before the term DevOps had been coined.

After this incident, we asked two consultants to help us improve. The consultants brought us together to diagram the end-to-end process. Each functional group told their part of the story and added to the diagram. As our drawing expanded, we began to see the systemic challenges. Regular actions of my group were causing problems for other groups, and they in turn were causing issues for us. We began developing empathy for the other people in the process. We began to see what was hard about their job and the things we could do to help. In the end, we realized that the only thing that mattered was the overall experience of our customers. We realized that our processes must be open to change if we were to service the goals of the whole.

Immediately after this session, we began to operate as a team.

  • We established a cross-functional team made up of people from each of the functional groups and elected one person to be the Product Owner. The Product Owner was responsible for our backlog of improvements.
  • We identified success measurements from the customer's perspective. That was our collective scorecard.
  • We agreed that blaming anyone for faults in the process was counterproductive. What mattered was finding issues and working collectively to eliminate them.
  • We came up with a backlog of items to improve and started implementing those improvements.
  • We got together monthly to review progress and choose the next place to improve.

The result was not only greater success of the process, but greater teamwork. We gave up the blaming and focused instead on the customer experience. Like any transformational journey, it starts with the people, all aligned on a vision for the future. Then obstacles can be overcome one at a time to really make a difference.

  1. Kim, Gene. "The Three Ways: The Principles Underpinning DevOps." IT Revolution. N.p., 22 Aug. 2012. Web. 19 Feb. 2017.