Scrum : How Empirical Process Control Helps Manage the Complexity of Software Development

We often say that the devil hides in the details. From a project perspective, the devil is simply another name for what we call complexity. When problems are complex, they behave unpredictably, and not only are these problems unpredictable but even how they will prove unpredictable is hard or impossible to predict.

In organizations developing software, chaos often occurs when a project’s complexity is greater than the ability of project managers to direct meaningful progress toward a goal. Progress might be made in fits and starts, but it is often indiscernible and unsatisfactory.

Over the year, we learned that the more complex the project, the more necessary it becomes to delegate decision making to independent agents who are close to the work.

Where traditional project management models fail

Waterfall: A relic from 1970

On too many occasions, defined process control has been unsuccessfully applied to complex IT projects. Waterfall is such a traditional model. The first formal description of the waterfall model is often cited as a 1970 article by Winston W. Royce.

The waterfall development model originated in the manufacturing and construction industries, where the highly structured physical environments meant that design changes became prohibitively expensive much sooner in the development process. When first adopted for software development, there were no recognized alternatives for knowledge-based creative work.

In Royce’s original waterfall model, the following phases are followed in order:

• System and software requirements: captured in a product requirements document
• Analysis: resulting in models, schema, and business rules
• Design: resulting in the software architecture
• Coding: the development, proving, and integration of software
• Testing: the systematic discovery and debugging of defects
• Operations: the installation, migration, support, and maintenance of complete systems

Where Waterfall fails?

The difficulty with Waterfall model roots at our attempts of trying to figure out everything in advance in a complex project:

• We broke down the project’s activities into linear sequential phases, where each phase depended on the deliverables of the previous one and corresponded to a specialization of tasks.
• After each release was planned, the work was broken down into tasks and organized on PERT charts.
• These tasks were then divided among the various analysts, designers, developers, testers, and documenters.

As the complexity of the releases increased and the customer base expanded, the release planning phase grew until it was simply unacceptable. The results of each release planning phase were also unsatisfactory, and we can list several drawbacks to using waterfall for a complex software project:

• Clients may not know exactly what their requirements are before they see working software and so change their requirements, leading to redesign, redevelopment, and retesting, and increased costs.

• Designers may not be aware of future difficulties when designing a new software product or feature, in which case it is better to revise the design than persist in a design that does not account for any newly discovered constraints, requirements, or problems.

• The sequential nature of the tasks divided the team. The people who analyzed the situation and set requirements were not the same people who would design the solutions that met these requirements. The people who designed these solutions were not the same people who would code the solutions. To communicate and collaborate, the divided team members had to produce after each task had been completed, a document detailing the work they had done.

• Organisations may attempt to deal with a lack of concrete requirements from clients by employing systems analysts to examine existing manual systems and analyze what they do and how they might be replaced. However, in practice, it is difficult to sustain a strict separation between systems analysis and programming. This is because implementing any non-trivial system will almost inevitably expose issues and edge cases that the systems analyst did not consider.

Empirical Process Control : A better alternative, Scrum

Scrum is based on empirical process control theory and accepts complexity, and uncertainty as necessary evils. Even better, it cuts through complexity and wrests order out of chaos, by enabling a team to organize itself, which allows a particularly productive order to emerge.

In a Scrum project, decisions are made based on observation and experimentation rather than on detailed upfront planning. Scrum relies on individual and team commitments rather than on top-down control through planning. Self-organization and human commitment are far more powerful mechanisms than imposed controls, plans, and even loyalty.

Process control: Evolutionary product planning vs defined product planning.

Scrum addresses the complexity of software development projects by implementing the visibility, inspection, and adaptation requirements of empirical process control with a set of simple practices and rules.

Out-of-the-box Scrum doesn’t have practices that address the complexities of every project. However, it’s enough to refer to the Scrum theory to find Scrum practices that can be readily adapted to handle even the most complex projects. For example, as the degree of complexity rises, the number of inspections must be increased. The increased frequency of inspections offers more opportunities for adaptation.

It is typical to adopt the defined (theoretical) modeling approach when the underlying mechanisms by which a process operates are reasonably well understood. When the process is too complicated for the defined approach, the empirical approach is the appropriate choice. —B. A. Ogunnaike and W. H. Ray

Traditionally, the project manager tells the team what to do and manages its work. In a huge reversal of ordinary management practices, Scrum makes the team responsible for managing development activities. The team selects the work that it will do during each Sprint, and after that initial selection is made, it is up to the team to figure out how to do the work at hand. The team decides how to turn the selected requirements into an increment of potentially shippable product functionality.

In analogy to Sashimi, the Japanese delicacy made of thin slices of raw fish, complete and with similar taste as every other slice, Scrum requires that every slice of functionality created by the developers be complete. It means that all of the requirements gathering and analysis, design work, coding, testing, and documentation that constitute a complete product are required to be completed in every Sprint and demonstrated in the Sprint increment of functionality.

By focusing on increments of functionality, the team makes orderly progress toward completing the release. Since each increment is tested as it is coded, the number of bugs never overwhelms the project. Scrum’s iterative incremental practices provide the team with a sense of accomplishment and an awareness of where it is in the release cycle. Scrum’s requirement that each increment of code is potentially shippable requires the incremental removal of defects and minimizes ongoing bugs.

With sprints, Scrum offers many opportunities to make changes and to respond to new opportunities. Scrum also keeps everything highly visible. The Product Backlog and its prioritization are open to everyone so that they can discuss them and come to the best way to optimize ROI. The Daily Scrum keeps all team activities visible so that the ScrumMaster can enforce the rules and help the team stay on track.

Sprints are kept short enough that the stakeholders don’t lose interest in the project before the Sprints are completed. And stakeholders can see that they have an opportunity to redirect the project at the start of every Sprint to optimize the value they derive from the project.

The sales and marketing departments are often at odds with the development organization. Sales and marketing want quick responses to every opportunity that comes knocking. Developers need to focus on producing the product. The chaos in a company is often caused by an inability to resolve these conflicting needs. Scrum strikes a balance between the two through the use of 30-day sprints. The development organization will do whatever is needed every Sprint, addressing whatever is deemed the top priority for that Sprint. However, everyone else must leave the developers alone to work.

By encouraging transparency, and keeping everything in full view, the type of “backroom politicking” and “influence swapping” that is common and normal in most organizations is minimized. These mechanisms are useful in bureaucratic organizations as a way to get particular things done. But when Scrum is already getting things done, these behind-the-scenes pressures are counterproductive.

Yet, and unfortunately, too many teams tend to interpret Scrum within the context of their current project management methodologies. They apply Scrum rules and practices without fully understanding the underlying principles of self-organization, emergence, and visibility and the inspection/adaptation cycle. They don’t understand that Scrum involves a paradigm shift from control to empowerment, from contracts to collaboration, and from documentation to code.

Scrum is not a process, technique, or definitive method. It’s a framework within which you can employ various processes and techniques.

Main roles in scrum and responsibilities

Clearly defined roles and expectations help individuals perform their tasks efficiently.

In Scrum, there are three roles: Product Owner, Development Team, and Scrum Master. Together these are known as the Scrum Team.

Product Owner, focusing on maximal ROI

Traditionally, customers get to state the requirements that optimize their ROI at the start of the project, but they don’t get to assess the accuracy of their predictions until the project is completed. Scrum lets the Product Owner adjust the ROI much more frequently.

In Scrum, the Product Owner is responsible for those funding the project for delivering the vision in a manner that maximizes their ROI. She formulates a plan for doing so that includes a Product Backlog.

The Product Backlog is a list of functional and nonfunctional requirements that, when turned into functionality, will deliver this vision. The Product Backlog provides the Product Owner with a powerful tool for directing the project, Sprint by Sprint, to provide the greatest value and ROI to the organization.

The Product Owner uses the Product Backlog to give the highest priority to the requirements that are of highest value to the business, to insert nonfunctional requirements that lead to opportunistic releases and implementations of functionality, and to constantly adjust the product in response to changing business conditions, including new competitive offerings.

The Development Team, delivering “done” increments

The Development Team is responsible for building an increment of product functionality every Sprint. This increment must be potentially shippable because the Product Owner might choose to immediately implement the functionality. This requires that the increment consist of thoroughly tested, well-structured, and well-written code that has been built into an executable and that the user operation of the functionality is documented. This is the definition of a “done” increment.

Development Teams are self-managing, self-organizing, and cross-functional, and they are responsible for figuring out how to turn Product Backlog into an increment of functionality within an iteration and managing their work to do so. Members are collectively responsible for the success of each iteration and of the project as a whole.

Optimally, a team should include seven people. During the Sprint planning meeting of a team of seven people, the team members lean forward, interact, collaborate, look at each other in the eye, and form a plan of action.

The Scrum Master, making Scrum work!

The ScrumMaster fills the position normally occupied by the project manager. While the traditional project manager is responsible for defining and managing the work, the ScrumMaster is responsible for managing the Scrum process, success, and fit within the organization’s culture, so it delivers the expected benefits.

As the Scrum process defines practices, meetings, artifacts, and terminology, the Scrum Master is responsible for knowing these and knowing how to apply them correctly, to protect the team from impediments during the Sprint.

With a skillful Scrum Master, the probability of success increases as she helps the Product Owner select the most valuable Product Backlog and also helps the Development Team turn that backlog into functionality.

The scrum master achieves this goal with a set of activities:

• She teaches Scrum to everyone involved in the project and guides the implementation of Scrum by making sure everyone follows Scrum rules and practices.
• She removes the barriers between development and the Product Owner so that the Product Owner directly drives development. Often, it means she has to teach the Product Owner how to maximize ROI and meet his or her objectives through Scrum.
• She improves the lives of the development team by facilitating creativity and empowerment.
• She improves the productivity of the development team in any way possible, fostering the engineering best practices and tools so that each increment of functionality is potentially shippable.
• She keeps information about the team’s progress up-to-date and visible to all parties.

The behavior of the ScrumMaster is dramatically different from that of people staffing a formal project management office that assigns work and controls its completion.

The ScrumMaster is a leader and a facilitator, not a manager. Her authority is largely indirect and springs mainly from the Scrum Master’s knowledge of Scrum rules and practices and his or her work to ensure that they are followed. The ScrumMaster earns the team’s respect because he or she fulfills the duties of the role.

The shift from delegating to being personally responsible is difficult for some people to accept. The “hands-on” aspect of Scrum scares some people.

Armed with a clear understanding of how Scrum works and experience applying it, the Scrum Master guides the Scrum project through the shoals of complexity. Like sheep in an open field, individuals in a project tend to stray.

The Scrum Master’s job is to keep the flock together. I often compare a ScrumMaster to a sheepdog, responsible for keeping the flock together and the wolves away. - Ken Schwaber

Can the Scrum Master also be the Product Owner?

The Scrum Master is responsible for ensuring that the team follows the Scrum process, while the Product Owner ensures that the team spends time building things that bring the most value to the organization. These roles sound so deceivingly simple that sometimes a single person will try tackling both.

There are exceptions to every rule, and one person can serve both roles successfully. However, this isn’t the norm, nor should it be a long-term solution.

To give your organization the best chance for success when using Scrum, allow two distinct individuals to serve these roles. Even if you’re struggling to find those with the right aptitude, you’ll be better off with two people who have a passion to grow into their respective roles, rather than one exceptional individual struggling to balance both.

Committed or Involved?

The three described Roles have committed to the project. Others might be just interested (involved) in the project, but they are not on the hook.

• Committed: The Roles who are responsible for the project have the authority to do what is necessary for its success.

• Involved: The other Stakeholders who aren’t responsible for the direct success can’t interfere unnecessarily.

It should always be clear who for example is responsible for the ROI, and who has a stake in the ROI but isn’t accountable.

Scrum makes a clear parting between these two groups and ensures that those who are responsible for the project have the authority to do what is necessary for its success and that those who aren’t responsible can’t interfere unnecessarily.

“A chicken and a pig are walking down the road. The chicken says to the pig, “Do you want to open a restaurant with me?” The pig considers the question and replies, “Yes, I’d like that. What do you want to call the restaurant?” The chicken replies, “Ham and Eggs!” The pig stops, pauses, and replies, “On second thought, I don’t think I want to open a restaurant with you. I’d be committed, but you’d only be involved.”

Scrum Artifacts

In Scrum, artifacts are “information radiators” and serve to capture the shared understanding of the Scrum team at any particular point in time. They are specifically designed to maximize the transparency of key information so that everybody has the same understanding of the artifact.

Artifacts play a vital role for the team to reflect on how they are doing with the sprint goal.

In the Scrum framework, the following artifacts are deemed essential:

Product Increment

The most important Scrum artifact is the Product Increment.

The Product Increment is a piece of working software that creates transparency for all the stakeholders.

It is the sum of all the items of the Product Backlog completed during the current sprint and adds value to the increments produced during all of the previous sprints.

Each Sprint the development team produces a potentially shippable product increment. This Product Increment must align to the development team’s “Definition of Done” and this increment must be accepted by the Product Owner.

The Product Increment must be in a usable condition regardless of when the Product Owner decides to release it.

Product Backlog

The Product Backlog is a list of the requirements for the product that is dynamically prioritized by Business Value.

The requirements are broken down into User Stories by the Product Owner, and the Product Owner is responsible for the content and validity of the Product Backlog, as the Product Backlog emerges and evolves.

The requirements are always changing and evolving over the successive Sprints and as the team gathers new learnings, therefore the Product Backlog is a living artifact. It may be subjected to change when there is a change in the external business environment, market conditions, regulatory changes or technology changes.

Sprint Backlog

The Sprint Backlog contains all user stories and tasks the Development Team has committed to, for the current Sprint.

Only the Development Team can change its Sprint Backlog during a Sprint. The Sprint Backlog is a highly visible, real-time picture of the work that the Development Team plans to accomplish during the Sprint, and it belongs solely to the Development Team.

To fulfill the commitment, all items of the Sprint Backlog should be “done”. Most often, that means “developed, tested, documented and integrated”.

The Sprint Backlog formation is usually guided by the Sprint Goal. It is a forecast by the development team on what functionality the team will have to work and deliver. The Product Owner helps the team to come up with the sprint goal during the sprint planning meeting.

The Sprint Backlog can be changed by the Scrum team as it evolves. The development team may discuss the work in progress during the Daily Scrum and modify the Sprint Backlog throughout the Sprint, as the Sprint Backlog emerges during the Sprint.

Burndown Chart

The Burndown chart shows the amount of work remaining per Sprint. It is a very useful way of visualizing the correlation between work remaining at any point in time and the progress of the Team. It will serve as a basis to measure sprint velocity and make any necessary adjustments.

The theory state that each member of the Development Team can update the Burndown chart because the team is self-organized. However, the Scrum Master is often the one who updates the Burndown chart for the Scrum Team, so that the team can fully focus on software development for the ongoing sprint.

Overview of the Scrum Flow

Together, the Sprint planning meeting, the Daily Scrum, the Sprint review, and the Sprint retrospective constitute the empirical inspection and adaptation practices of Scrum. Below is an overview of how each ritual is orchestrated over the length of a sprint.

1. A Scrum project starts with a vision of the system to be developed. The vision might be vague at first perhaps stated in market terms rather than system terms, but it will become clearer as the project moves forward.

2. The Product Owner formulates a plan for doing so that includes a Product Backlog listing all the functional and nonfunctional requirements that, when turned into functionality, will deliver the vision.

3. The product Owner transforms it into a prioritized Product Backlog where the items most likely to generate value are top priority and are divided into proposed releases. The prioritized Product Backlog is a starting point, and the contents, priorities, and grouping of the Product Backlog into releases usually changes the moment the project starts—as should be expected. Changes in the Product Backlog reflect changing business requirements and how quickly or slowly the Team can transform Product Backlog into functionality.

4. All work is done in Sprints where each sprint is an iteration of 30 consecutive calendar days. Each Sprint is initiated with a Sprint planning meeting, where the Product Owner and Team get together to collaborate about what will be done for the next Sprint. Selecting from the highest priority Product Backlog, the Product Owner tells the Team what is desired, and the Team tells the Product Owner how much of that it believes they can turn into functionality over the next Sprint. The goal is to get to work, not to think about working. Sprint planning meetings are time-boxed to avoid too much hand-wringing about what is possible, and limited to eight hours.

   1. The first four hours are spent with the Product Owner presenting the highest priority Product Backlog to the Team. The Team questions him or her about the content, purpose, meaning, and intentions of the Product Backlog. When the Team knows enough, but before the first four hours elapses, the Team selects the items from the Product Backlog that it believes it can turn into a completed increment of potentially shippable product functionality by the end of the Sprint.

   2. During the second four hours of the Sprint planning meeting, the Team, which is responsible for managing its work, prepares a tentative plans to start the Sprint. The tasks that compose this plan are placed in a Sprint Backlog. The tasks in the Sprint Backlog emerge as the Sprint evolves. At the start of the second four-hour period of the Sprint planning meeting, the Sprint has started and the clock is ticking toward the 30-day Sprint time-box.

5. Every day, the team gets together for a 15-minute meeting called a Daily Scrum. The purpose of the meeting is to synchronize the work of all Team members daily and to schedule any meetings that the Team needs to forward its progress. Each Team member answers three questions:

   • What have you done on this project since the last Daily Scrum meeting?
   • What do you plan on doing on this project between now and the next Daily Scrum meeting?
   • What impediments stand in the way of you meeting your commitments to this Sprint and this project?

6. At the end of the Sprint, a Sprint Review meeting is held. For a 30 days sprint, this is a four-hour, time-boxed meeting at which the Team presents what was developed during the Sprint to the Product Owner and any other stakeholders who want to attend. The product increment is presented during this informal meeting, intended to bring people together to collaboratively determine what the Team should do next.

7. After the Sprint Review and before the next Sprint planning meeting, the Scrum Master holds a Sprint retrospective meeting with the Team. At this three-hour, time-boxed meeting, the Scrum Master encourages the Team to revise, within the Scrum process framework and practices, its development process to make it more effective and enjoyable for the next Sprint.