A simple process for getting data insights


By Tom Gilbert 

Senior Data Scientist at Atlassian, Hercules Konstantopoulos, shares with us some steps on how to apply a more agile mindset to your roadmap and how to become a data informed team.

We’ve all heard of teams that transformed their products with data-informed decision making, leading to happy customers and improving conversion rates in the process. Sounds amazing, right? But how do you become one of these teams? Where do you get started? Who should be involved? What data do you need? And how on earth does this fit onto your roadmap?

Uncertainty around how to become a data-informed team can make it all seem too hard.

What if I told you that it doesn’t have to be that way? I’m going to share how we applied a well-known practice to data science…and how it worked!

The secret? It turns out that data science isn’t rocket science — it can be run like any other Agile project. And that’s what we did! We chose to run a data and analysis program the way we’d build a feature.

By the end of this blog, you’ll have 6 steps to apply a more Agile mindset to data and analysis. Steps that include and motivate your team, whilst not railroading your roadmap. My team has gone from trailing to trailblazing because of these steps.

So let’s get started!

1. Set up your core team

Form a team of five or six people who represent the core disciplines for your area. For our team, these are product management, engineering, design, and data science. This will be the core team that drives your data and analysis program forward, so choose wisely!

While you’re at it, make it official by giving yourselves a catchy team name (come on, every team needs one!). Ours is Star Fox, which we assured each other is a very cool and not dweeby retro video game reference.

2. Define your purpose

This part is crucial. Get the core team together, break out the Sharpies and Post-it Notes, and have a kick-off meeting. You’ll want to achieve the following:

Set an overarching goal: As a team, you need to align on your goal early if you want to succeed. Ask yourselves - why did we form this team to focus on data and analysis? Our answer to this was pretty simple and formed our goal: we want to use data to build intuition and solve problems.

Establish focus areas: Define and stack-rank the top 3 - 4 focus areas that will help you achieve your goal. Ask yourselves - what can we focus on that will provide the most value? Examples might be:

  • Improving conversion rates for your user flows
  • Understanding user composition
  • Better data hygiene

The team may generate many ideas here, so use your intuition to limit it to a max of 4 focus areas and avoid spreading yourselves too thin.

Create hypotheses: For each focus area, determine a hypothesis (or two) that you want to test. Setting hypotheses will help you hone in on the data you need to instrument and interpret. An example hypothesis could be: We can reduce login friction by allowing users to log in with Google.

Define a cadence for reviewing your purpose: While you should continuously consider your purpose, set gates to ensure you do so. We review our goal, focus areas, and hypotheses quarterly.

Set up your comms channels: To build momentum, you must keep talking to each other. You don’t have to meet in-person every time you want to talk. Working asynchronously is good too. Create a page with your goal, focus areas, and hypotheses on your staff wiki (ours is Confluence of course!), then get the team to add their comments. Set up a Slack channel to chat when it suits throughout the week. Use whatever works for your team.

3. Get instrumenting

With your goal, focus areas, and hypotheses set, you need to get some data pumping through.

Define the analytics events you need: Map out your user flows as a team so you can document the events you need. As you do this, question the value of each event. Is it a key step in the funnel? Will it inform your hypotheses? If you can’t define any value, you don’t need to instrument it. And spend time ensuring that your events are consistently and intuitively named. This takes time but it will pay off massively in future, allowing anyone in your team to spin-up your analytics tool and easily find the data they need.

Begin with an MVP of just enough events to start drawing insights: this way you’ll get the data quicker. It’s better to get some data flowing through than trying to instrument every little, last event. Shipping a small but representative MVP set of events and seeing data flow through early has helped us:

  • Find time in our dev teams sprints without the need to deprioritize other in-flight projects
  • Cut down the time it takes to get from in development to actionable insights
  • Spot gaps in our instrumentation plans early, so we can resolve them in the next release

Plan multiple iterations of event instrumentation: It’s unlikely that you’ll nail it on your first release. There will be a key piece of data that you realize you’re missing, or a bug with one of the events. Plan time in your upcoming dev team’s sprints to ship fixes and more events.

4. Notch up some early wins

Now that you’ve got data flowing through, it’s time to strut your stuff! You need to show the value of your work early. This will build momentum for the team and get the rest of the company on bard. Look at your data, find an insight, decide how you’re going to act on it, then do it and share the results with everyone.

Our data immediately showed us the scale of an issue that was affecting WAY more people than we thought. So we put immediate priority on identifying a fix. We identifed and released a fix within 2 weeks. Then we went straight back to the data and, to our delight, saw a big increase in conversion rates for these users. We made sure to share the outcome with the rest of the company through an internal blog. Using data to fix user pain + boost conversion rates = win. Big win!

5. Settle into a routine

Look at you! You’ve set up the foundations for a great data and analysis program. You’ve sparked the fire, now you need to keep stoking it.

To continue with your success, you need to make data and analysis business as usual. To do this, you have to ensure there’s a structure and regularity. Here are some tips to achieive this:

Find a tool to track your data needs and progress: as with any Agile project, you need to document what you’re doing, who’s doing it and its status. Like your dev teams track their work through cards on a wall or project tracking software, you should do this for your data and analysis program too.

We use 2 simple Trello boards — one for instrumentation and one for analysis. They look like this:

Instrumentation board (sample content used)

Analysis board (sample content used)

Use sprints as gateways for maintaining momentum: set up bi-weekly meetings with your core team and treat the period in-between as a sprint. In your meetings:

  • Determine the value you want to achieve in two weeks time: This could be instrumenting new data, a dashboard to send to the execs, or some completed analysis
  • Set and assign the tasks that will get your there: A data scientist and PM to pair on analysis? A dev team lead to get their team instrumenting some events? Make sure the tasks are clear and that you’ve assigned the right people
  • Continue, pause, or abandon: Look back on the tasks you set in the last sprint and review what you achieved. Pat yourselves on the back for the tasks you achieved. And for the ones you didn’t? Make a call on whether you continue, pause, or abandon them. The key to running an Agile data and analysis program is being flexible and scrutinous over what you focus on. Know when to pause or abandon something. The last thing you want to do is burn precious time for little benefit.

If this sounds like how your dev teams operate, you’re right! Using sprints as gateways is a brilliant way to maintain momentum and focus.

Set up core team planning and review meetings: We have these quarterly:

  • Retro: We do a retro on how our data and analysis program has performed over the quarter. We align on what’s worked and we should keep doing, as well as what’s not working and we should stop. These meetings are also a great way to remind ourselves of the wins we’ve had along the way.
  • Quartley planning session: We set our plans for the next quarter. Remember the goal, focus areas, and hypotheses you set at the start? It’s time to revist them. Are they still relevant? Could you tweak them based on what you now know? Have you completed some of them? Are there new priorities? Treat this meeting like a mini version of the kick-off you had at the start but time-boxed for the next quarter.

6. Get the entire team involved

Everyone in your team or department is responsibile for drawing insights from data. Teams love seeing data and discussing insights from the features they’ve built. This creates a shared understanding of your customers, their painpoints, and how to provide the best experiences for them. And with a diverse range of perspectives, the more people across the data, the better the insights you’ll get. Many eyes make light analysis!

As the core team, it’s your responsibility to get the wider team involved. We have a strong internal blogging culture at Atlassian, so we blog our insights to the rest of the company. This gives everyone in the company a chance to learn and provide input. Some of our teams also have a rolling insight champion, who’s responsible for sharing an insight they found from our data. Everyone takes a turn — developers, designers, product managers, program managers, everyone! The team then discusses the insight and whether there’s any action we can take. Other suggestions for insight sharing are team demos, show and tells, stakeholder presentations, and basically any project meeting.

That’s a wrap

So there you have it. A simple, six-step process for running your data and analysis program like an Agile project. One that bands your team around insights, keeps you focused whilst making time for exploration, and doesn’t railroad your roadmap.