Scrum provides a solid, proven foundation for project execution. However, in every project, Scrum must be tailored to the specific needs and environment, which can be a decisive factor in the success or failure of the project. In this article, I’ll show you how to successfully complete a large (20 person-years, over 100,000 lines of code), distributed (developers in India and the Netherlands) Scrum project that was once abandoned under traditional development. In order to help readers run large-scale projects smoothly, I will also cover the lessons we learned from starting a project, finding the right product owner, the importance of estimating, effective communication, testing, and documentation.

background

The Dutch railways are among the world’s most used rail systems, carrying 1.2 million passengers a day. The department has created a new information system to provide passengers with more accurate train information and reduce human intervention. As part of this system, we made the PUB publishing system, which centralized control of information display and audio broadcast in all stations.

Someone had tried this PUB system before, but they used the traditional waterfall method. The customer hands the developer a detailed specification of the requirements document, and then waits for the complete system to take shape and deliver. Three years later, the project was canceled because the developer failed to develop a working system. Then the client hired us to start from scratch, and we introduced agile development, using Scrum, working closely with the client, communicating openly, taking small steps.

start

When the project started, we scheduled a launch phase before the first sprint, which took three weeks to get everything ready for the sprint. This startup phase is completed by a project manager, an architect, and a Scrum___ master.

Choosing a product owner can be difficult, and we can’t find one person who has the time, domain knowledge, and authority to set requirements priorities. We nominated two business analysts to share the responsibility of product leader. They have the time and they have been involved in building the PUB before, so they have enough business knowledge to take on the role of product owner and act as a good agent for multiple groups of customers. Senior decisions about priorities and scope are made by a project manager appointed by the customer, who has limited time and understanding of the requirements. Usually cooperation is fine, but occasionally the project manager will change the priorities set in the planning meeting and the meeting will have to be revisited. Ideally, the person with the final decision on priorities should attend the Sprint planning meeting every time.

Because previous attempts to build a PUB system have been made, detailed requirements documentation for some parts is already available. They comply with the MIL standard [1], but are not in a form suitable for Agile planning and estimation [2], because in agile development requirements should be organized into small sections, each of which can be implemented, tested, and demonstrated in a sprint, but the existing documentation does not meet this requirement. Product owners also didn’t have much experience writing user stories. To solve this problem, Scrum___ Master helped them come up with the original product Backlog, which contained fine-grained, estimated user stories for use in the first few iterations.

The software we build is part of a larger software system that includes many related software systems that display information and install display equipment in the station. We had to make sure everything was done on time to straighten out the complex system. So there needs to be an overall plan. After several iterations, we were able to estimate each function of the system to the best of our ability, and the problem was solved and we had a reasonable productivity rate. Release burn out charts can then be used to track and communicate progress. The lesson here is that even with very little information, an estimate is better than no estimate.

Extend to distributed teams

When the project started, it started with seven people and iterated every two weeks. The project was planned from the beginning to use some Indian manpower, so two Indian developers joined the team from the first sprint. They came to the customer site to participate in development and spent six weeks familiarizing themselves with domain knowledge, user representatives, and other team members.

From the start of building a team, decide how you will work together. Together with all of our team members — including our Indian colleagues — we organized a “norms and rules” event. We decided on practices such as how to pair, which tools to use, quality goals, core work hours per day, and so on. Then record it on a Wiki. When the whole team is on the same page, things are much easier. Whenever the consensus needs to be changed, such as to suggest improvements at a retrospective meeting, the practices are updated on the wiki so that when new people join, they always see the latest content.

In the first few iterations, the team successfully built, tested, and validated the user stories that make up the core of the system. This pleased the client, especially since we were moving faster than in the past and the client had control over the direction of the project.

After a few iterations, we expanded the project: the Indian developers returned home, and then we added resources in both India and the Netherlands, resulting in two Scrum teams of five developers each, sharing the same tester. Then there were three teams, three testers. Each team has a mix of Indian and Dutch employees. This approach keeps the project productive and quality of work high.

So how do we work together? First, we use Skype a lot. We all have webcams, headphones, microphones, and a big screen. So we can have one-on-one meetings as well as full-body meetings. All using off-the-shelf products and free software. It doesn’t cost much more, just UPS to keep Skype meetings going even when the power goes out, which improves connectivity in India. Second, only people in the same place pair off. So people in India pair up with people in India, people in Holland pair up with people in Holland. Experience has taught us that no matter what tools are available today, the interaction required for pair programming still requires two people sitting together.

Finally, we use ScrumWorks to keep track of who is doing what, keeping track of the progress of the Sprint. Because we’re a distributed team, this is much better than a whiteboard. ScrumWorks is also useful when discussing the product backlog with the product owner.

We have overcome many difficulties in implementing this distributed model. For example, the product owner is not used to speaking English. According to Scrum, the planning meeting is divided into two parts, and in the first part, the product owner tells the team the user story and sets priorities. Because of the language barrier, only the Dutch team participated in this part of the meeting. The second part is usually about talking about specific tasks and making estimates. This part was done by Skype with the Indian team, in English, without the product owner. We’ll spend a little more time communicating about the first part of the meeting. The Sprint demo was also only in the Netherlands. When it was done, the Dutch team wrote an internal briefing to inform the Indian team — and the rest of the company — of the results of the presentation. The in-house briefing proved popular.

Break down a team that only focuses on architecture

Our project is just one part of an application chain that must be seamlessly connected to the customer’s existing IT infrastructure. Although our product owners were familiar with the core functional requirements, they knew very little about security, metrics, usability, performance, etc. It can be difficult to understand these needs from the customer’s organization because it involves talking to many people in different departments. This kind of research is a drag on Scrum’s iterative pace. To solve this problem, we created an independent team that focused only on the architectural aspects. Their job is to figure out non-functional requirements so we can translate them into user stories in the backlog. We use “Scrum of Scrum” meetings to communicate with feature teams. We all like this approach because feature teams can go full speed ahead. And some employees like to work on an “architecture team.”

The document

The client requires extensive documentation and compliance with the MIL documentation specification. And write it in Dutch. It was clear that the job could only be done by the Dutch. Developers and testers were not familiar with the MIL specification, and writing user manuals was not their forte. So we decided to hire a technical copywriter who had written MIL documentation before. Development and testing can continue to focus on their respective responsibilities. This was also successful, but we found that it also required a lot of communication between the technical writer and the rest of the team, which was important to note because the developers just wanted to “do their thing.”

Demand management

Our product owners are business analysts who are used to writing extensive requirements documents in Dutch. In our process, as long as the stories are in the backlog and the product owner can explain them in planning meetings, that’s enough. But the client wants lots of documentation. So we decided to work with the product owner to translate the requirements into user stories. As a result, requirements are placed in two places: the requirements document and the backlog. This can cause problems when requirements change. We did a lot of coaching to make sure the product owner wasn’t just looking at the requirements document, but also the backlog.

With a one-line user story and an explanation from the product owner, our Scrum team was ready to build and test the software. Still, the requirements document was valuable for external testing teams, even though it was difficult to “map” the implemented user story to parts of the requirements document in several iterations.

Looking back, we never really had an ideal requirements management process. We just did our best to deal with these conflicting requirements: we needed user stories, and customers needed detailed requirements documentation.

test

We did automated testing on the project to ensure that the tested software was delivered at the end of each Sprint, with no regressive bugs. Even as the system expanded, we managed to have only one tester on an eight-person Scrum team, and the quality was high: the external test team found at most one defect per thousand lines of code.

Our automated testing consists of two parts: unit testing and acceptance testing. In the former case, we used JUnit and Clover to measure test coverage, aiming for 80% test coverage of server-side code. Acceptance tests are automated with FitNesse. For each completed user story, there is a set of acceptance tests on FitNesse. With a large test suite, regression bugs can be found and fixed during sprints. Another benefit of this approach is that testers can be actively involved from the start, writing test cases before user stories are implemented.

There is one place that has been troubling us for a long time. Part of the system is a rich client with a very complex user interface. It’s much harder to do automated testing on this stuff than it is on the server. So most of the UI functions are still manual. As the system grows, retrospective testing takes longer and longer, and worse, it is only in this part of the system that external teams find regression bugs. This can be prevented with automated testing. The lesson here is that even if automated testing is difficult, putting some effort into it will pay off sooner or later, especially late in the project.

Output results

The client is satisfied with our work. As an afterthought, like most projects, features, time, and budget change as the project progresses, so “on time and on budget” is just a vague definition of completion. More importantly, we often discuss how to do the project well with our clients during the project process, and they are very satisfied. Unfortunately, because of problems with other systems, the product had problems when it was deployed nationwide.

The customer got an external audit firm to review the software. Their conclusion:

1. The maintainability of the system is excellent.

2. Source code quality is very high.

In the review company’s report, they said they had never given so many positive reviews to a project.

conclusion

Here are the most important things we learned from the project:

1. It’s hard to find a product owner who has both extensive requirements knowledge and the authority to set priorities. So people often have several people in the product owner role, especially on large projects.

2. If deadlines are a must, keep the product backlog complete and estimate. An estimate is better than no estimate for requirements, even if the information is small. When the estimates are combined with team productivity, the release plan has the necessary information.

3. Scrum works well with multiple distributed teams. Each of our Scrum teams is both Dutch and Indian, which brings out the team spirit well and focuses us on effective communication. In communication, using off-the-shelf hardware and free software can save money.

4. When you start a distributed project, it’s good to get everyone together in the same place to agree on what the team is doing.

5. For tasks that do not fit into a Scrum Sprint (such as finding key personnel, communicating with other customer departments), a separate team can do them more efficiently. Feature teams can focus on developing software. It’s also good to have a dedicated technical copywriter, even if it adds to communication costs.

6. Although the software development process does not require extensive requirements documentation, customers may. In a Scrum project, however, requirements documents are no substitute for user stories. If you have both requirements documents and user stories, you need to plan for the additional overhead of reconciling requirements in both places.

7. Automated testing plays a key role in incremental software delivery by eliminating regression bugs. By the end of the project, the return on investment will exceed the cost.