Planning and executing larger engineering epics
How does a mid-sized development team execute large technical epics that consist of dozens of features? Senior developer, Chris Kelly, describes his approach.
These are my thoughts on the major steps to planning and executing larger engineering epics using Agile project management methods.
To define some Agile terms for you:
Stories are short requirements or requests written from the perspective of an end user.
Story points are units of measure for expressing an estimate of overall effort relative to work complexity, the amount of work, and risk or uncertainty.
Epics are large bodies of work that can be broken down into a number of stories.
1 – Take a step back
There are a large number of moving parts when rolling out improvements to a complex web application, for example, work we did on the myki dashboard for Public Transport Victoria (PTV).
The myki dashboard is a hub for travellers needing easy access to do things like check their account balance, report a lost myki card and update account information.
On this type of large epic it can take time to ‘grok’ everything in play.
I find that taking a step back from the screen helps me immensely when it comes to large scale projects. It helps my brain digest high level requirements without the distractions of notifications and, more importantly, reduces the temptation to enter "implementation mode" too early.
The goal of this step is to see the big picture as much as possible. I find it helpful to avoid focusing on any one story, and see the epic as a whole.
2 – Whiteboard
Pick a diagramming medium – this can be a physical whiteboard, Miro, or even pen and paper.
Whiteboards allow you to ruthlessly dispose of ideas that aren't working. At any time, you should feel comfortable wiping the board clean and starting fresh. It's much cheaper to dispose of ideas at this stage, before developers start coding.
Use your diagram to rapidly plot out the building blocks, data flows and state transitions. Aim to avoid implementation – rather, focus on how the building blocks piece together.
Tip: Don't spend time making these diagrams pretty, an especially real temptation when using digital whiteboards. The prettier they are, the less likely we are to dispose of them if they aren't working.
By the end of the whiteboard stage of the myki project, we’d mapped out the key data flows and state transitions, and had a solid high-level understanding of how the new myki functionality would be integrated into the existing project.
3 – Talk, continually
Communication. Hey, who would've thought that was key to success?
This might seem obvious, but ask questions of key stakeholders, such as project owners, and continue to confirm your understanding of the requirements. This is especially important during the early stages, as it's likely that all the stories within the epic are not complete, fully understood, or even completely explored.
4 – Take stock before abstracting
One of my favourite micro-patterns is the rule of three.
A quick sidestep: the rule of three is about avoiding refactoring reused code too early, instead, refactoring it once you have sufficient information to create the right abstraction. I find this pattern useful because it balances out the more commonly applied DRY principle (Don’t Repeat Yourself), which leads into …
Devs hate repetition, but often aim to remove duplication too early. This leads to rigid architecture due to the wrong pieces of code being refactored out.
It is difficult to create the correct abstractions on stories that are within a larger, in-development epic because all the code hasn’t been written. You might feel you need a crystal ball.
However, you can still take stock of the upcoming work by looking through the requirements of upcoming stories within the epic, even if you won’t be the developer for that story.
When making these abstractions ahead of time, make sure to keep your team in the loop. A note in Slack can be useful, or even better an example branch in Git:
Hey @chris, looks like you'll be working on ticket003. That shares some behaviour with ticket001 I'm working on, so I'm going to pull that behaviour into a service. I'll push up an example branch of that service for when you pick up ticket003.
5 – Iterate and be flexible
On larger epics, despite all the best planning, established solutions often still need adjusting. Some issues and gaps only become clear once further features are added or pathways are explored. Sometimes certain stories are dependent on others and time is needed to apply the ‘glue’.
When estimating individual stories, factor the above into your budget. When a story is part of a much larger deliverable, it’s very likely you’ll need to allow extra time.
The key to remember here is that a large epic is made up of multiple features. They’re built in parallel, so it helps to maintain a flexible mindset during development as the pieces fall into place.
In the myki project, this often meant stubbing particular components in one story – a stub being a snippet that shims in placeholder functionality – while another developer worked on the stubbed functionality in another. We allowed time to glue those pieces together.
6 – Document, constantly
Try to avoid leaving the writing of your documentation to the end. Details may be lost over the course of a large project, so it's best to document as you go. Iterate on your documentation as you iterate on code.
With larger projects, timely documentation is important because you have multiple developers working in parallel, on interlocking features. They need to understand how your parts work to avoid being blocked or approaching it from another angle.
As a developer, if you aren’t seeing documentation from others in your team, don’t be afraid to ask them to add it, or mention it as part of reviewing a pull request.
Good luck as you successfully navigate complexity
Every team has their own techniques for navigating complexity found in large scale projects. I hope you find some of these ideas useful for your upcoming projects.
