The Three Ways

These are the three ways of creating an effective DevOps environment.

The First Way: Create a Fast Flow of Work Through Development and IT Operations

• “…helps us understand how to create fast flow of work as it moves from Development into IT Operations, because that’s what’s between the business and the customer.”
• The flow within your team needs to be good. The flow to other teams needs to be good. (Think about the entire system, not just your piece of it)
• Use tools like a Kanban Board to see work going through the system (TODO → WIP → DONE)
• Eliminate bottlenecks that prevent work from being done (see below)
• Prioritize the work that is actually important for the business

The Second Way: Create Constant Feedback Loops from IT Operations to Development

• “…shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework.”
• Work to eliminate unplanned work and manage technical debt. These prevent you from working on things that can give Development teams feedback
• Create faster deployment cycles – deploy 10 times per day
• Advice: Automate the build and deployment process. Ensure environments are available. Use Infrastructure as Code. Create Deployment Pipelines. Get everything into version control. Create monitoring systems.

The Third Way: Create a Culture for Improvement and Learning

• “…shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.”
• “Improving daily work is even more important than doing daily work.”
• Make improvements to the system a habit. Engineers should do something to improve the system everyday
• “…if you are not improving, entropy guarantees that you are actually getting worse…”
• Outage Drills – simulating problems to make sure people remember how to fix it
• Make sure your systems are resilient to faults – do chaos testing

The Four Types of Work

These are the types of work that IT Operations is responsible for.

1. Business Project Work
2. Internal IT Project Work
3. Changes to Systems Work
4. Unplanned Work

Bottlenecks (Constraints)

• Constraints: There are often a small number of resources that dictate most of the output for the entire system
• If they aren’t fast then everything else is slow
• Make sure the constraint is never idle – it should never be waiting for resources. It should always be working on high priority tasks

“Work in Process” (The Silent Killer)

• Theory of Constraints: Anything done that doesn’t resolve the bottleneck is useless. Whatever you did will never be reached because it’ll be waiting for the bottleneck.
• You need to keep the flow moving. Eliminate constraints. Ensure fast and uninterrupted flow of work that delivers business value
• Minimize the disruption from unplanned work

Unplanned Work

• This is the most destructive type of work
• This work prevents you from working on the things you committed to doing
• This is commonly referred to as “fire fighting”. You could also call it “anti-work”
• Figure out where you’re unplanned work is coming from and determine how to prevent it
• Technical debt is a big factor in causing unplanned work

Brent Character Analysis

Brent is an example of when only a single developer on a team knows how everything works. He has all the knowledge. He also hasn’t had the time to socialize that information among all the other developers. Therefore he becomes a bottleneck. Everyone needs him to do everything and nobody else can get anything done.

Brent is a bottleneck.

Always Be Learning

• “Technology keeps changing faster and faster, and it’s nearly impossible to keep up anymore.”
• ““It’s crazy what programmers, and even managers like me, have to learn every couple of years. Sometimes it’s a totally new database technology, a new programming or project management method, or a new technology delivery model, like cloud computing. “Just how many times can you throw out everything you know to keep up with the latest new-fangled trend?”

Miscellaneous Notes

• Training people is something that should be prioritized. You need to prepare for your companies future
• Documentation is very important (If Brent leaves the company then it would collapse)
  • Document all of the tribal knowledge
• Results are the only thing that matters
  • You need to know what is important for the business
  • The processes and tools don’t matter – what matters is what you get done
• IT is critical – it’s in the middle of every company effort
  • Business and IT can’t make decisions without each other
  • Every COO needs to have a deep understanding of IT Operations
• A great team doesn’t always mean that they have the smartest people. A great team means they can work well together and trust each other
• As an engineer you are being paid to think, not just to implement code. Your job is to make decisions and tell people when you don’t agree with an approach.
• “This should be your guiding principle: You win when you protect the organization without putting meaningless work into the it system. And you win even more when you can take meaningless work out of the it system.””
• Dev and QA environments need to match the production environment