Chapter 5 selecting the right value stream as the entry point
- Our first goal was to help all the teams measure the cycle, visualize it, and try to shorten the delivery cycle, and then iterate.”
5.1 Greenbelt projects and brownfield projects
- Greenfield projects are brand new software projects
- Brownfield projects are products or services that have served customers for years or even decades. Such projects often carry a lot of technical debt
- Many people think that
DevOps
Mainly for green space projects, but successfully appliedDevOps
Brownfield projects undergoing transformation abound
5.2 Both recording system and interactive system should be considered
- Traditional recording systems are similar to
ERP
Systems, recording systems are usually slower to change - Interactive systems often change quickly and require rapid feedback
5.3 Start with the most innovative team
- Our goal is to find those who believe
DevOps
A team with principles and practices and the willingness and ability to innovate and improve existing processes - We’re not going to spend a lot of time trying to change conservative groups, especially in the early stages. Instead, focus on teams that can create success and are willing to take risks, and slowly expand from there
5.4 enlargeDevOps
The scope of the
-
No matter how you choose your entry point, show your results early and actively advertise them, right
-
(1) Identify innovators and early adopters: In the beginning, focus on teams that are truly willing to improve
-
(2) Win the silent majority: In the next phase, strive to expand the DevOps practice to more teams and value streams with the goal of building a more solid base of people
-
(3) Identify “nail households” : The so-called “nail households” refer to those high-profile and influential opponents
Chapter 6. Understand, visualize, and apply the value stream
- An effective way to start optimizing the value stream, based on years of experience, is to practice value stream mapping with all core stakeholders to help the team tease out all the steps to create value
6.1 Identify the team needed to create customer value
- Regardless of the complexity of the value stream, no one of them knows everything that needs to be done to create value for customers, especially when the work is done by multiple teams. These teams are often in different departments, not even in the same office, and are evaluated differently
6.2 Draw a value stream map for team work
-
The goal of a value Stream map is not to document all the steps and details, but rather to identify the areas that prevent the value stream from moving quickly, thereby reducing lead times and improving reliability. Ideally, participants in a workshop should have the power to make changes to the parts they are responsible for
-
In my experience, such value stream mapping exercises are always eye-opening. Many people are often seeing for the first time just how much work needs to be done to deliver value to the customer
-
Based on all the information provided by the value stream participation team, the following two types of work should be analyzed and optimized
-
Work that can take weeks or even months, such as preparing a production environment, a change approval process, or a safety review process
-
Initiate or handle significant rework
- concept
- Lead time of work items:
LT
- Processing time (or increment time) :
VA
(Value Added Time
) - % measured by downstream consumers
C/A
6.3 Establish a dedicated transition team
-
Dedicated transformation teams should be set up and kept separate from the departments that run the day-to-day operations (they call the former “dedicated teams” and the latter “performance engines”)
-
Most importantly, the team should be responsible for achieving clearly defined, measurable, system-level goals (for example, reducing the lead time of the process “from submitting code to deploying to production” by 50%). In order to achieve this, the following measures should be taken:
-
Members of the transition team are dedicated to DevOps transformation (rather than having them continue their previous work but spend 20% of their time on DevOps transformation)
-
Select a generalist who is familiar with multiple areas as a team member
-
Select as team members people who have long maintained good relationships with other departments
-
If possible, find a separate office area for the team so that members can interact with each other as much as possible and keep a reasonable distance from other departments
6.3.1 Have common goals
-
The first part of any improvement plan is to set measurable goals for the next six months to two years. The team needs to put in considerable effort to achieve the goal, and the achievement of the goal should create significant value for the organization and the customer
-
Once the overall goal is identified, the team should develop a detailed plan and pace for improvement. Like product development work, transformation work should be done in an iterative, incremental manner. Generally, each iteration is completed within 2-4 weeks. For each iteration, the team should set a set of small goals that will generate value and work toward the long-term goals. At the end of each iteration, the team should review progress and set new goals for the next iteration
6.3.2 Keep the improvement plan of small span
- Ability and flexibility to replan and change priorities
- Strengthening feedback loops by reducing the delay between implementation and implementation will be more likely to reinforce desired behavior — initial success is good for increased investment
- Learn more quickly from an iteration and apply it to the next iteration
- Can be improved with less effort
- Achieve meaningful and differentiated improvements faster in daily work
- Reduce the risk of projects being terminated before they come to fruition
6.3.3 Set aside 20% of development time for non-functional requirements and reduce technical debt
- Debt in order to actively manage technology, development and operations to ensure that at least 20% of the time into the reconstruction work, automation, architecture, optimization and non-functional requirements (sometimes referred to as the “quality attributes”), such as maintainability, manageability, scalability, reliability, testability, deployable and security (see below)
- The collaboration between the product owner and the engineer goes something like this: The product owner needs to consider allocating 20% of the team’s resources to engineering-related activities to rewrite or refactor problematic parts of the code base, which could be 30% or more if things are already bad enough
- If the organization is not willing to pay the “20 percent tax,” then the technical debt will eventually deteriorate to the point where all available resources are exhausted. There will come a time when services will become untenable, feature delivery will stagnate, and all engineers will be working on reliability issues or improvising solutions
Case StudiesLinkedIn
“Reverse action” (2011)
LinkedIn's Operation InVersion is an interesting case study of why paying off technology debt is part of your day job. LinkedIn had a successful IPO in 2011. But half a year later, the company is still struggling with deployment. Within two months, they stopped all feature development and implemented a complete optimization of the computing environment, deployment, and architecture. By 2010, most of the newly developed features were deployed as new services, with nearly a hundred services running independently of Leo. But Leo itself is still only deployed once every two weeks. Today, LinkedIn engineers make three major updates to the site every day. In 2010, we had over 150 separate services. Today, we have over 750 servicesCopy the code
6.3.4 Improve the visualization of work
- Everyone in the organization needs to be aware of the current state of work. There are many ways to visualize a state, but the most important is to effectively present the latest state, and to revise it constantly to ensure that the team is up to date
6.4 Reinforce expected behavior with tools
- Another benefit of shared queues is that they unify the task list so that everyone can think globally about the highest priority and choose the work that is most valuable to the organization or that pays off the technical debt the most. When a technical debt is found, it can be added to the task list if it cannot be resolved immediately. For problems to be resolved, use the 20% time reserved for non-functional requirements to fix them
Chapter 7 refers to Conway’s law to design the organizational structure
-
Conway’s Law: “System design is limited by the organization’s own communication structure. The bigger the organisation, the less flexible it is, and the more obvious it is.”
-
Conway’s Law: “The architecture of the software is the same as the structure of the software team, which basically says, ‘If you have four teams working on the same compiler, the compiler will end up with four stages of execution.'”
-
The structure of a software development team has a huge impact on the architecture and results of a software product
7.1 Organization Prototype
-
There are three main types of organizational structure
-
Functional organizational structure: focus on improving professional skills, optimizing division of labor or reducing costs
-
Matrix organizational structure: attempts to combine functional and marketing. However, as many people who manage or work in matrix organizations can see, such organizational structures are often very complex
-
Marketable organizational structure: focus on quick response to customer needs. Such organizations tend to have a flat structure, consisting of multiple cross-functional departments (such as Marketing Department and engineering and technology department), and the whole organization may often have redundancy. This structure has been adopted by many prominent organizations implementing DevOps
7.2 Harm of over-functional orientation (” Cost optimization “)
- traditional
IT
O&m organizations tend to have a functional structure, with database administrators grouped in one group and network administrators in another… This approach obviously leads to longer lead times, especially in complex activities such as large-scale deployments, where multiple teams have to issue a stack of work orders and coordinate work transitions, resulting in long waits at each step
7.3 Building a market-oriented team (” Speed Optimization “)
- Integrate engineers and their professional skills (e.g., operations and maintenance,
QA
And information security) are embedded in each service team, or the team provides a self-service platform that can configure a production-like environment, perform automated tests, or deploy
7.4 Make functional orientation effective
- On the left is the functional orientation: all work flows through the centralized
IT
Operations team - Market orientation on the right: All product teams can deploy loosely-coupled components to production in a self-service manner
7.5 Integrate test, operation and information security into daily work
-
Common pain points can reinforce a team’s common goals. A classic example is Facebook. In 2009, Facebook was experiencing rapid growth and significant issues with code deployment. While not all issues directly affect the customer, the team is in a perpetual fire-fighting mode
-
One of the most effective steps Facebook has taken to increase deployment efficiency is to rotate all of its engineers, engineering managers, and architects to operate and maintain the services they build themselves. By doing so, everyone building the service has a hands-on feel for the architecture and code they are responsible for upstream, which has a huge positive impact on downstream work
7.6 Make all team members generalists
- Silos are created when departments become too specialized
- Any complex operations and maintenance activity requires multiple handoffs and queues between different parts of the infrastructure, resulting in delayed delivery times (for example, each network change must be implemented by someone in the network department)
- One response is to make every team member a generalist (see chart)
- E-type talents refer to those who excel in experience, professionalism, exploration ability and executive ability
7.7 Invest in services and products, not projects
7.8 Set team boundaries according to Conway’s law
-
A side effect of Conway’s Law is that improper organization of teams can have undesirable consequences. These include dividing teams by function (for example, locating developers and testers in different offices, or outsourcing testers entirely), and splitting teams by architectural levels (e.g. application layer, database layer, etc.)
-
Poor organization requires a lot of communication and coordination between teams, but can still lead to a lot of rework, conflicting requirements definitions, inefficient work handovers, and people sitting idle waiting for upstream people to finish
7.9 Creating loose-coupled Architectures to improve productivity and security
-
An architecture that enables small teams to develop, test, and deploy independently, safely, and quickly increases and maintains developer productivity and improves deployment quality. Service-oriented Architecture (SOA) has this characteristic
-
It is an architectural approach that supports independent testing and deployment of services, typically consisting of loosely coupled services with bounded contexts
-
Bounded context is a concept proposed by Eric Evans in Domain-Driven Design 9. The idea is that developers should be able to understand and update the code of the service without knowing the internal logic of its peer service
Keep it small (the “Two Pizzas rule”)
-
In 2002, Amazon used the “two pizzas principle” to keep teams small as it tried to move away from a single code base — two pizzas were enough for everyone on the team, which typically had 5-10 people
-
This size limit has four important effects
-
(1) It ensures that team members have a clear and consistent understanding of the system. As teams get larger, the amount of information that needs to be communicated multiplies if everyone is to understand the state of the system
-
(2) It limits the growth rate of the product or service being developed. By limiting the size of the team, the speed at which the system can evolve is also limited, which helps ensure that team members have the same understanding of the system
-
(3) It decentralizes power and achieves autonomy. Each double Pizza team worked autonomously as much as possible. Reporting directly to management, the team leader determines the key business metrics (also known as fitness functions) for which the team is responsible and uses them as an overall evaluation of the team’s practices
-
(4) Leading a “double pizza” team is a way for employees to gain leadership experience. In such an environment, failure is not catastrophic. A key element of Amazon’s strategy is the link between the organizational structure of the “double Pizza” team and its service-oriented architecture
-
When conway’s Law is applied well, teams can safely and independently develop, test, and deliver value to customers; When used poorly, security and agility can be compromised
Chapter 8 integrates operation and maintenance into daily development work
-
How do centralized operations teams achieve market-driven results
-
Three common strategies:
-
Build self-service capabilities to help developers become more productive
-
Integrate operation and maintenance engineers into the service team
-
If the number of O&M engineers is limited, you can use the o&M contact mode
8.1 Creating shared Services to improve development productivity
- Operations to market-oriented achievements, one way of doing this is to create a centralized service platform and the tools set, make all the development team to be able to through the use of this platform and services to improve productivity, for example, set up a production environment, the deployment of assembly line, automated testing tools, such as production environment remote console
8.2 Integrate O&M engineers into the service team
-
The integration of operations engineers enables the product team to become self-sufficient, reducing the reliance on centralized operations. These product teams can be fully responsible for service delivery and support
-
Product teams often have a dedicated budget to hire these operations engineers, but interviewing and hiring decisions may be done by a centralized operations team to ensure consistency and quality
-
This paradigm has an important advantage: close coordination and collaboration between the development team and operations engineers is an extremely effective way to cross-train operations knowledge into the service team and gradually transform operations knowledge into automated code, making it more reliable and widely reusable
8.3 Assign o&M contacts to each service team
- For various reasons (such as cost or resource constraints), it may not be possible to assign an operations engineer to each product team, but similar benefits can be achieved by assigning an operations contact person to each product team
8.4 Invite O&M engineers to a development team meeting
- Invite them to various development team meetings. Our goal is to help operations engineers and other non-developers better understand the culture of the current development team and actively participate in planning and day-to-day work so that operations teams can better integrate operations capabilities into product teams and implement them before they go live
8.4.1 Invite O&M engineers to the daily station meeting
- Some information is scattered across the development team, which is a common problem. Through the o&M engineers who attend the meeting, the o&M department can fully understand the activities of the development team, thus better planning and preparation
8.4.2 Inviting O&M engineers to a review meeting
-
The operation and maintenance engineers who attend the retrospective can also learn and benefit from it. Also, when a deployment or release occurs during the review period, the operations engineer should report the results and provide feedback to the product team. This will improve the way future work is planned and executed and the quality of work
-
We need to remind everyone that improving the daily work is more important than the daily work itself, and that all teams must set aside time for this (for example, allocate 20% of your time to improving work per cycle, schedule one day per week or one week per month, etc.). If you don’t, your team’s productivity will surely suffer under the pressure of paying off technical debt
8.4.3 Use Kanban diagrams to display O&M work
-
It is rare to use kanban diagrams to show relevant operations. However, these operations are necessary if the application is to run successfully in a production environment, where customer value is actually generated
-
Because operations work is part of the value stream, it should be presented on a Kanban chart along with other work related to product delivery
Part II Summary
- We thought about it and talked about it
DevOps
Many aspects of transformation, including the selection of entry points, understanding the relationship between architecture and organization, and building a transformation team, also discussed how to integrate operations into the daily work of the development team
books
- Crossing the Chasm
- Lean Enterprise
- Apocalypse: Building Products that People Love
- The Cathedral and the Fair
- Domain-driven Design