Agile Metrics Resources

in Resources by

My last post talked about using Agile Metrics. I reflected mostly on my experiences at nuBridges and the metrics we put into place. These metrics were good at establishishing heartbeats and, in general, monitoring of the process. There are a bunch more practical metrics that help organizations assess value, identify and measure quality, and understand better where things can improve. When writing the last post, I found some great websites and resources on agile metrics that are worth sharing:

  • Earned Value for Agile Development by John Rusk of Optimization, Ltd. This document was found on This is a PDF that contains a solid explanation of how earned value can be applied to any agile project. The document is well written and easy to understand, even if you don’t have a practical project management background.
  • Agile Metrics page on Dave Nicollete’s wiki/blog site ( Dave assembles some of his own presentations as well as some links to other valuable sites. His presentations are excellent and make for great training materials.

Another, Another Post About Agile Metrics

in Agile Engineering by

Even if you are on the right track, you will get run over if you just sit there.
– Will Rogers

As you can see by the title of this blog (or shall we get fancy and call it an article), it is about Agile Metrics. And as you can also glean from the title, this is yet another blog about this topic. However, I plan to be different – I’ll come after this topic from purely personal experience and reflection. Now I’m not saying the other writers do not reflect upon their personal experiences, but over the last eighteen months the teams I worked with have done an excellent job at forming out and tracking metrics. These metrics do two things:

  • Provide the stakeholders with insight regarding to what the teams are doing, how well they are doing it, and what is coming when.
  • Provide the team with measurements that can be monitored to identify and understand how the process of software development and deployment is working.

These both lead to consistency, trust, and continuous process improvement. Now don’t get me wrong, metrics alone don’t provide these things – but, capturing metrics, measuring, and then taking stock in the measurements do help establish consistency, foster trust, and drive continuous improvement.

We chose six metrics to capture:

  • Roadmap Effort Achieved. This equates to enhancements, new features, or business value activities performed during the iteration versus defects and other technical tasks which although are required for sustaining the product, they don’t have the agreed upon value with the Product Owner.
  • Utilization. This is broken out into two metrics: (1) Ideal Utilization which considers actual hours booked against all project activities divided by the team’s ideal hour capacity; and (2) Planned Utilization which considers actual hours booked divided by the team’s planned capacity. The first measure is generally mid 60% to low 70% depending on the roles and the level of the team.
  • Defects Raised. If this isn’t clear, then nothing is – just kidding, this metric tracks those defects that are born out of all touch points including level 3 support, testing, and those raised by developers that should have been caught during continuous integration cycles.
  • Sprint Task Complete. This metric is similar to a sprint task burn down; however, it is a measure of how close was the team at completing tasks they signed up for. This measure did not include tasks introduced during the sprint – only those they signed up for. The challenge we find with this measure is that if a team is small, this number could be skewed. However, this metric clearly indicates either too much cross-winds hitting the team, lack of focus, or simply too large of stories/tasks.
  • Sprint Effort Complete. Effort relates to hours, so this gives a pretty good barometer of how close the team came to finishing the tasks. We measure this for two reasons – (1) smaller teams that may not complete but 1 of 3 tasks looked bad, but the reality was they finished 95% of the work and just need another day or two to get it across the finish line and (2) this allows for us to understand impact of estimates and cross-winds (a.k.a. support). The team still got recognized for the effort, but there was recognition that either through more focus or better break down of tasks may help complete what was committed.
  • Estimate Accuracy. This type of metric is not unusual in most traditional PMOs; however, the goal here was to drive the behavior of breaking down tasks more and not estimating something you didn’t understand. The result of this awareness helped the organization to build features consistently and follow best practices surrounding analysis and design. We also found it valuable with prototyping in that the team was able to break down the tasks in a fairly repeatable way and the estimates only got better during these periods.

How have these metrics worked out?

These metrics have served us well; however, some of these metrics can be frowned upon since they appear to be too focused on performance. But what we’ve found is that you adapt the metrics as the teams mature and the inevitable organizational change occurs. So for instance, Utilization was always a product of ideal hours – therefore, you generally saw 60-68% utilization for teams that performed project tasks and played a role in level 3 support. By plotting the utilization over time, we arrived at a median utilization rate of 67%. Once we felt this stayed level, we turned our focus to planned utilization – meaning are we planning our sprint capacity accurately and therefore filling up the buckets properly. We are an organization that uses story points; however, we’ve found velocity to be cyclical based on the phase of the project and possibly the opportunities brought forth by the business (a.k.a. several large deals sign and we find ourselves involved in implementations that are not managed via Scrum, thus velocity collapses). Nonetheless, now that we’ve moved to looking at planned utilization, we are able to establish a level of consistency and trust with the business that when we say we have X capacity and we can deliver by Y – the business is lock-step saying the same thing because they too see the metrics.

Lessons Learned

Well, we’ve learned that the data wasn’t useful in short cycles – the data was useful once you had five or six sprints worth of data points, then adjustments to process could be made. Also, by this time, the reasons you would expect things to go sideways were very clear there on the charts (e.g. defect rate went up prior to the release cycle – as airspeed is slowed on the feature development and the focus turned to technical debt). We also learned that tracking estimate accuracy and utilization altered the teams’ behaviors – and not in a good way. The team started finding ways to work the metrics in the favor. They did so even after being told that we weren’t measuring their performance based on these numbers, but using them for identifying opportunities for improvement and demonstrating successful delivery. We had folks that would sandbag their estimates, or underestimate and work late then log work for the exact hours they estimated.

We also learned the obvious and, frankly what we wanted to see, and that is that once a team started to understand the metrics – they matured as a team. Part of this was the equation of time and the improvement of the Scrum/Agile processes. The other part was the fact that we chose specific metrics to focus on – especially those surrounding productivity and quality. We started conducting group code review meetings and pairing up more on tasks that we deemed to have higher risk.

What to Look at Next

We need to start look at tracking business value better – looking at using earned value. The challenge we have is that we generally don’t put a cost on projects – we are a small software firm that generally runs with department budgets only and assess costs along the line of business only. It seems that we would make better decisions along the way if we kept cost and revenues in plain view during the development process.

Also, need to look at measurements surrounding quality automation (unit test coverage and LOC test coverage).

I’m sure there are other areas, but now that we have a good handle on consistency and trust in delivering what we said we would, it is time to focus on business value and look deeper into quality. Both of these will help make the products and the business more successful.

Self-Organizing and Strengths Finder

in Agile Engineering/Product Management/Teams by

Team WorkA key part of any good team, especially an agile team, is teamwork and more importantly a team of hardworking individuals that both care about their teammates and they understand the strengths of their fellow teammates.  In the sporting world you will often hear of situations where one player will attempt to play outside his strengths.  Or what is more prevalent, a player has an outstanding year due to either scheme or competition or simply a well formed team that emphasizes the strengths of the players.  Then that player is a free-agent and goes to another team, and the player’s career sags.  This is primarily due to familiarity, chemistry, and scheme that doesn’t fit the strengths of the player.

There is a concept that surrounds Scrum teams and that is the concept of “self-organizing” team.  Now this concept is often not clearly defined and depending on the Agile coach and or trainer, a team member might view this differently (see Does Self Organizing Team Imply Self Assembly?) .  Nonetheless, if we define self-organizing in the simplest form of a team that has the ability to take inputs and based on knowing certain attributes of their fellow team members, work is doled out, people are paired up, issues are worked out, team members are brought in and pushed out, and, in summary, things just start getting done and done effectively.

The “certain attributes” that a team uses for team self-organization often includes:

  • Team member workload
  • Technical background
  • Domain expertise
  • Team member availability
  • Understanding of the tasks-at-hand
  • Priority of tasks
  • Team member “likes”
  • Team member weaknesses
  • Team member strengths

Of course, all of these are important to consider; however, one would argue that the strengths are where all self-organizing should be focused.  As you guessed it, I’m the one to make this argument – and I encourage everyone who runs any type of team to take advantage of Clifton StrengthsFinder from the Gallup News organization (yes, that’s the same Gallup that does many politico and social polls).   StrengthsFinder came about from Dr. David Clifton – he is known as the father of Strengths Psychology.  The focus of StrengthsFinder is to help people identify their top five strenghts where they work best, understand these strengths, and see how focusing on using these strengths versus working on improving areas of weaknesses is much more productive, self-rewarding, and ultimately successful.

Some have grouped StrengthsFinder into the area of self-help or lumped it with personality tests such as Meyers-Briggs Type Indicator.  It is definitely not a personality test and if you did want to categorize it as self-help, you will find it to be much, much better and more effective than any other self-help book or tool.  When I was approached by a colleague that went through the StrengthsFinder exercise and he express the passion that this is for real and fulfilling, I was convinced I had to give it a go.  So after I took the 30 minute survey and got my results, I quickly learned and understood better why I say, do, and act certain ways.  It helped me rationalize and focus my energies within the areas of my strengths so that I can help my team and the company become more successful.  This didn’t mean I wouldn’t ever work on improving my weaknesses, but it did help me recognize where I can use my strengths to overcome or disguise my weaknesses.

So, you are now asking, why does this have anything to do with Self-Organizing agile teams.  Well, soon after going through the exercise myself, Dave (my colleague mentioned above) implored me to have our whole product development team go through the survey – we did.  After everyone had their top five strengths, we compiled the list and posted it on our team wiki.  We all sat down and discussed what we learned about ourselves and reflected on the strengths of others.

At the time when we did this exercise, we were about ten months into using a Scrum agile methodology.   We had earlier delivered a huge product release and we were coming off a really tough period where everyone was getting burned out by the product launch, supporting new customers, and having some frustrating issues largely due to incomplete quality processes.   Soon after experiencing StrengthsFinder, we saw that the team which would normally “mouse-up” during difficult team discussions, would start leaning on each other strengths.  This helped me as the team lead, to simply sit back and let the team work through challenges that when previously encountered, the team would simply brush under the rug.  Going through StrengthsFinder seemed to give the entire team more confidence and it helped those team members that would often simply focus on trying to improve their weaknesses to instead, embrace their strengths and take on the tasks that best suit them.   Not only would they have more self-confidence, but they also embraced the strengths of their peers.  When we would start new feature or enhancement development, the team members often signed quickly and they would ask a fellow team member that may have been stronger in another area to assist (e.g. using a person that has the Relator strength to interview subject matter experts).

Within two months after working through StregnthsFinders, we experienced measurable quality improvements and we moved closer to a self-organizing team where anyone on the team could lead within their strengths.

So if you are working within an Agile team and you are struggling with the “self-organizing” aspects or your team has hit a continuous improvement plateau, give Clifton StrengthsFinder 2.0 a try.  The books required can be obtained for less than $13 at Amazon or Borders.

Five Ways to Establish Trust

in Agile Engineering/Product Management/Teams by

As everyone knows, a key success factor of an agile process implementation, specifically Scrum, is establishing trust between the team delivering and those waiting on the delivery (basically between the product development team and the business).  In order to establish trust, a development team should be meeting their commitments made during the planning meetings; thus, delivering what they said they would.  Sounds like a pretty simple formula and for the most part — it is.

Never trust the teller. Trust the tale.
– DH Lawrence

Outside of simply delivering what you said you would in a sprint, here are five ideas to establish trust:

  1. Always act with integrity. It goes without saying, truth sets you free.  But what is more than simply being honest while communicating with someone, but it is more important to perform your job duties with integrity.  This means, no cutting corners, maintain the utmost quality, and always keeping the team informed when it comes to matters that may impact the core deliveries.
  2. Ask questions and share information. Make sure that if an idea is not understood, be sure to not be afraid to ask a question (sometimes, the first thing that comes out the mouth is “I don’t know”). Be sure not to assume you are all-knowing about an issue, so share what you know — this is an excellent way to facilitate conversation and potentially discover what you don’t know.
  3. Reasonably accurate expectations (estimates to complete). In the agile world, estimation can be challenging.  This is primarily due to the lack of data present at the beginning of tasks.  The easy way to overcome this and become more accurate with your estimation is to be sure to break down stories into common development tasks.  This may include an analysis, design, build out, unit test, etc type task.  I also like the proto-type, stub, or even the sub-functional task where a small piece of functionality is implemented.  Estimates generally get better the smaller the task and they are of course better when more than one person sets the estimate (leveraging the experiences of others will help).  Finally, always accurately update estimates to complete daily.
  4. Deliver within the boundries of the commitment. Simply stated, don’t always try to be a hero and deliver beyond the expectations of the original commitment. This can be viewed by management (be it product or development) as sand bagging estimates, or simply going rogue and “doing as you want to”. Not to say that finding a great way to do something that was much more productive is not a good thing or seeing that you can do a lot more within the context of the product functional area is not appreciated. But the right way to handle this is to disclose that you found a faster way to do something and your estimate-to-complete is much less than orginally anticipated, so you would like to work on another key capability that is similar to the functionality being implemented. Negotiating this within the context of the daily standup (and possibly within a follow up meeting) would be the correct course of action.
  5. Consistency is key.  Having one good sprint is not enough, a team needs to establish a trend of delivering on sprint tasks on a consistent basis.  This applies to the team as well as the individual.  Once a team is able to demonstrate a consistent trend surrounding capacity, story points delivered, hours worked, etc — the product stakeholders can start to reasonably expect that what the team says they will finish will in fact be delivered.

Remember the idea of trust should also exist within the scrum team and between scrum teams where there is a need to share deliverables or even if you all work within the same development organization. The five ideas above should be applicable to ensuring that all scrum teams work well together and within the overall scope of an organization.

What techniques or ideas do you have about how to build trust with the product management or project stakeholders?  What have you’ve done that works?

AgilePalooza in Atlanta

in Randomness/Resources by

NOTE:  This blog was originally posted on March 17, 2010

Several weeks ago, I attended AgilePalooza in Atlanta with some of my co-workers.  It was a well put together conference and, I must say, a very cost-friendly conference.  This was  a one day event, that was primarily put together by VersionOne.  I attended three sessions, and learned:

  1. It’s good to get away from the office to simply press the reset button.  Even if it is for an afternoon.
  2. COMMON SENSE RULES!  The basic principals of do what you say, treat others as you wish to be treated, enjoy life (which includes work since we spend a lot of time at it), and remember that tomorrow brings another opportunity and today is a blessing.
  3. Some people need to practice their speeches and make darn sure they are relevant to the topic as promised.

I have both Mike Cottmeyer and Lee Hensen (a.k.a. Agile Dad) in my “Like Minded” links.  Both of these guys gave advanced topic presentations and they were very good.

Mike’s focused on scaling agile across the enterprise.  The interesting thing is that at my work, we are doing this fairly well — specifically, buy in to the process and seeing value of implementing in other areas.  But even more interesting, it seems the adoption of agile in big organizations seems very hit-or-miss, and primarily miss.  I’ve worked in mostly large organizations until now, and I can tell you — I have felt the frustration and dismay that some of the folks in the room have.  But you can make the case, and prove it can work — but you have to continue forging forward and not give up.  Ensure you have all folks seeing the value and establishing the trust.

Lee’s discussion was a re-hash of one I’ve seen him present at the agile conference in Orlando last year.  He updated, expanded, and expounded the content.  It was a good refresher and a solid reminder that ownership and accountability must be at a team and individual level, not one or the other – but both.  I know common sense, but repeated the need to establish trust amongst all of the parties involved and ensure communication occurs constantly.  And importantly, keep focus on the value of the project or work product.  One other thing he demonstrated is his method for estimating, very good and worth putting to practice.  I’ll write more about that later and reach out to him to comment.

Here’s the links to their presentations at the AgilePalooza website:

BTW – This ‘palooza was nothing like my first Lollapalooza experience in ’94 – George Clinton, Beastie Boys, Smashing Pumpkins, L7, Cyprus Hill, Black Crowes and Green Day – just to name a few.  That was a great time!

Go to Top