Where Product Analytics Fit in your Scrum Rituals

Successful product teams combine quantitative and qualitative information regularly to inform decisions, course-correct and make impactful changes.

They know we should not add features based only on our intuition and expertise, but, instead, based on customer response: most used features, user requirements, problems generated in the call center, etc.

Product analytics, supported by specific and technical tools such as A/B Testing or MVT (Multi-Variable Test), plays a major role in this process of qualification of each functionality, as it is the only truthful and objective way of knowing whether or not a business idea works in the market, as well as analyzing user behavior in interaction with this idea.

In a Scrum context, attention to data comes through incorporating analytics activities into your regular rituals. Note that it is possible to apply the key concepts described here to other flavors of agile, such as Kanban.

How to bring product analytics into your Scrum rituals

The Scrum methodology includes a few recurring rituals that can be “hijacked” to include more attention and focus on product analytics. Taking systematically on these opportunities to leverage data will help position analytics as a solid part of the team’s routine.

Sprint Backlog Refinement

Backlog refinement is when the product owner and some, or all, of the rest of the team, review the items on the backlog to ensure the backlog contains the appropriate items, that they are prioritized, and that the items at the top of the backlog are ready for delivery.

This activity occurs regularly and maybe an officially scheduled meeting or an ongoing activity.

Generally, the activities that occur during this refinement of the backlog include:

• Removing user stories that no longer appear relevant
• Creating new user stories in response to newly discovered needs
• Re-assessing the relative priority of stories
• Assigning estimates to stories which have yet to receive one
• Correcting estimates in light of newly discovered information
• Splitting user stories which are high priority but too coarse-grained to fit in an upcoming iteration

You can make the product analytics part of this ritual by starting your refinement session with a review of your product analytics.

Doing just that might :

• Create immediately obvious work to do,
• Create research tasks to dig into.
• Highlight glaring oversights in what you are tracking (what do you mean we put that feature live and have no idea if anyone uses it?).

By starting your backlog refinement with data, your team can now make informed decisions about prioritization.

Sprint Planning Meeting:

Sprint planning is an event in the Scrum framework where the team determines the product backlog items they will work on during that sprint and discusses their initial plan for completing those product backlog items.

Teams may find it helpful to establish a sprint goal and use that as the basis by which they determine which product backlog items they work on during that sprint.

Sprint Planning is where you decide on what you are going to do in the next sprint, work out who will do it and make concrete plans for getting it done.

Analytics has a limited role to play here if you’ve incorporated product analytics into your Sprint Backlog Refinement.

Anyways, the ways product analytics can play a part in sprint planning are:

• Giving you a last-minute sanity check on what you’re going to build. (Does the most recent data confirm we still need this?)
• Reviewing the performance of features/tasks completed in the previous sprint. (Did that work the way we expected, or do we need to make adjustments?)
• Ensuring that each task you will work on in a sprint includes details on how analytics will be developed and used for the work. (How will we track the success of this feature?)

Daily (Stand-up) meeting

Each day at the same time, the team meets to bring everyone up to date on the information that is vital for coordination. Each team member briefly describes any completed contributions and any obstacles that stand in their way. Usually, to structure the discussion, the meeting is normally held in front of the task board and each member answers Scrum’s Three Questions during the daily meeting:

• What have you completed since the last meeting?
• What do you plan to complete by the next meeting?
• What is getting in your way?

This meeting is normally timeboxed to a maximum duration of 15 minutes, and attendants stand up to make it as uncomfortable as possible to make it last. To keep the meeting short, any topic that starts a discussion is cut short and discussed in greater depth after the meeting, between the people affected by the issue.

If you don’t have a daily analytics review, then you need to figure whether you need it or whether it is overkill for your context. With today’s technology, daily analytics is easy-to-have, and not a superior effort as weekly analytics, and can only lead to better outcomes.

Things can shift overnight. The daily meeting offers an opportunity to quickly review the key metrics for your team or product as well as any focus metrics if you’re focusing on a specific user journey.

Additionally, the more regularly you’re talking about analytics the more knowledgeable each team member becomes about product usage, and the team simply gets more data-driven.

To facilitate this, it helps if someone on the team (generally the product manager, product owner or analytics lead) has reviewed before attending and brings any important insights to the team in a synthetic and story-telling fashion, to not make it boring.

Sprint Review Meeting

At the end of each sprint, a sprint review meeting is held. During this meeting, the Scrum team shows what they accomplished during the sprint. Typically this takes the form of a demo of the new shippable product increment, feature, etc.

As a natural result of the sprint. it is kept very informal, and little to no preparation is expected. A sprint review meeting should not become a distraction or significant detour for the team.

Including the product owner, the Scrum team, the ScrumMaster, management, customers, Anyone is welcome to attend. Feedbacks are welcome.

During the sprint review, the project is assessed against the sprint goal determined during the sprint planning meeting. Ideally, the team has completed each product backlog item brought into the sprint, but it’s more important that they achieve the overall goal of the sprint.

Incorporating product analytics into the Sprint Review Meeting involves:

• If you are continuously deploying, you can review the initial analytics results of any work that was deployed earlier in the sprint.
• Ensuring completed features include analytics capabilities so you can measure whether they are successful or not.

Sprint Retrospective Meeting

The team meets regularly, usually adhering to the rhythm of its iterations, to explicitly reflect on the most significant events to have occurred since the previous such meeting, and take decisions aiming at remediation or improvement.

This is often a facilitated meeting following a set format. Several distinct formats have been described, depending in large part on the time set aside for the meeting, typically between one and three hours. One important reason to use a facilitated format is to allow all team members to speak up.

You can ensure a standing agenda item in the retrospective is focused on reviewing your product analytics. To start the conversation you can ask questions like:

• How could we improve the tools and technology we use to obtain and analyze analytics?
• How could we improve the way we track and capture insights that come from analytics?
• How could we increase the speed at which we action analytics insights?
• What insights from analytics did we miss? How can we ensure we don’t miss them again?
• Are we happy with the way analytics is part of our sprint and rituals? How can we improve this? Do we need more or less focus on analytics?

Product Analytics Roles and Responsibilities in Scrum

In becoming a data-driven team you will need people on the team to take on some product analytics-related roles.

The roles you need to consider are:

• Product Analytics Lead: who will be responsible for bringing analytics insights to the team and championing analytics. You do want everyone to share some responsibility but having a lead will help drive improvements.
• Data Engineer: you may need to consider adding an engineer or making one of the engineers on the team responsible for data and analytics because sometimes getting analytics in the form you need requires engineering effort.
• Database Administrator: in some instances, product analytics requires database experts to help build queries and transformations with existing data to get data into the format or database you need to do analytics.
• Product Analyst: someone responsible for interpreting product analytics in the context of your commercial objectives, user needs, and technology. This might be the Product Manager or Product Owner but it can also be a specialist.
• Data Scientist: someone with mathematics and machine learning understanding that can apply more advanced algorithms to your data to perform the research you need to understand your customers and users. The need for a data scientist when it comes to product analytics is rare. You only need one if you’ve maxed out your options with the other roles.

How to know if it’s working

The journey to becoming more data-driven can be a long one and it requires taking people on the journey with you and coaching them.

As your team becomes more data-driven:

• You will start to add more and more research-style tasks to your backlog to further investigate points of interest and possible solutions to impact your data.
• You’ll start to find what you were thinking you would do next sprint completely changes based on the data you are seeing. This can be a very good thing.
• You will pay less attention to the loudest customer and reflect on what your broader, satisfied customer base is doing.
• Every team member will start contributing ideas of what could be measured and what could be improved based on their analysis of the data.
• Some team members become obsessed with data – “hey did we manage to improve the conversion rate on that form I was working on last week?”
• Expect push back on your decisions from the team if you haven’t justified it with data.

Wrapping up

Incorporating product strategy and analytics into Scrum is not complicated and under no circumstances contradicts its principles. It solves two errors that occur very often, in Scrum projects:

• Each Product Backlog item should have a business value associated, the value that this functionality adds to a product, preferably in incremental revenues. Understandably, it is a guiding value that is hard to calculate individually for each functionality, but fundamental together with the cost estimate when prioritizing the next functionality to be addressed. Real datapoints help determine an estimate, which is always better than solely relying on a “guesstimate”.
• It is common to convert the Sprint Review to a misnamed “demo” and forget that we should choose what we will do in the next sprint as a function of market acceptance of the features that have already been delivered. Product analytics tell us just that if properly used.