Testivus for the Rest of Us: A Whole Team Testing Experience


The Story

I want to share a true story of a relatively newly formed agile team. We were made up of ten engineers writing and testing code in C++, Java, and fully embracing the emerging javascript frameworks. The application we were building was both complex and complicated. We were learning how to launch a product, trying to embrace user-centric design, and honestly, I often screwed up as a lean agile leader. All that said, we were a pretty great team and we were constantly learning.

We were in the midst of a new product release that was careening towards a GA date. The testers on our team — who were pretty great at what they did — were getting pressed against that GA date with the build-up of feature inventory. As a team, we were doing an okay job at establishing automation, but we could do more — a lot more. So with the testers in the needs of some aid and the need to shift left more testing, we arrived at Testivus for the Rest of Us.

How It Started

As a team, we often themed our sprints. Sometimes these sprint themes were merely broad brushed sprint goals. We already had the “Unleash the fury, Mitch” sprint in which we literally sprinted to the finish line several key product features. In need of raising our game around test automation and in general, switch the team from a “test right” to “test left” where quality was truly the starting point of all work — we decided to theme a sprint “Testivus.” What better theme than Testivus — time to air our grievances and demonstrate feats of strength.

[su_note note_color=”#fdfdce”]NOTE: In case you’re not a Seinfeld fan, Testivus is a play on Festivus. See Festivus – Wikipedia for more information.[/su_note]

In preparation, we built the sprint backlog around those items that needed the most attention (we looked at a heat-map of components based on bugs previously found, code complexity, and skeletal impact to the product). This backlog wasn’t just a bunch of technical stories, we focused on the functional areas and created functional validation stories that restated the user outcomes, and we refined the acceptance criteria based on usage validation (as well as early learning from the field – e.g., performance needs, essential data needs, etc.). Some may call this just a regression backlog, and that’s fair; however, our mindset was different — it was all about outcome validation and, to some extent, validation that the product we were making was innovative and would kick-ass in the minds of our prospective users. Also, to note, this wasn’t about a “Hardening” sprint. You could argue this; however, we weren’t at the end of the release, and our core goal was to bring testing left in the cycle.

Once we built the stories and the resulting tasks included fleshing out a test harness, more in-depth unit testing practices and practicing TDD, functional UI test automation, and refining our CI process to extend into CD including bootstrapping test and demo data creation.

The Results of the First Testivus

The tangible outcomes were:

  • Stronger installation and deployment automation including packaging.
  • A robust test data generator and demo data bootstrap.
  • A new & improved test harness that handled both functional aspects as well as continuous performance benchmark validation. The harness covered over 1,800 scenarios nightly.
  • Increased code coverage from single digits to a substantial double-digit percentage.

The most impactful outcome was that team’s mindset which was already pretty good, became hyper-focused on how quality should be built-in. From this point forward, during refinement, the first thing we would talk about is attributes of testing.

The Longer Term

Testivus became a regular occurrence in the context that we called for a “Testivus” day or two every 5-6 sprints. Sometimes the need surfaced when we noticed a trend of bugs in a particular area, field feedback, and a couple of times we “Testivus” as part of a quality hack day.

Reflecting on Testivus, there was a need and the theme was awesome. It changed our mindsets, we had some focused energy on quality, and we had some fun — even though it was intense at times.

The great thing is — this concept wasn’t unique to us. Sometimes teams need a cause and a theme to try something new, and this helped us gel around some learning of good practices.

A couple of years later I joined VersionOne where the development team adopted “meatfest” which was a bug bashing event that had the team grilling out meat and vegetables, and drinking some beer throughout the day — all the while just fixing a bunch of stuff (I would point you to the awesome blog on this topic, but someone took it down).

Please let me know if you have experienced the same or would like to discuss further. Thanks for reading!

Update: We definitely weren’t the only one’s going down the Testivus path — just Google “Testivus.”

It’s Not a Bug, It’s a Feature


featureI know everyone who has worked in or on or amongst a development team has heard the saying, “it’s not a bug, it’s a feature.” When said, it is usually followed by some laughter, sometimes remorseful laughter. Heck, you’ll even hear the users beat you to the punch when they say, “that’s a feature, not a bug – right?”

Ding. Ding. Ding. Mr. User, you are absolutely correct!

Anything that goes out the door is a feature of our product (or anything we release to our users is a feature). Now whether the feature improves the value of our product or service, or if it detracts from the value — they are all features. We chose to release our software — we may have known about the problem or we may have known that we did not test enough or we may not have had the awareness of a unique usage scenario – no matter what we chose to release and everything in the product is a feature. Couple the concept of releasing bugs in our code with the premises around technical and design debt, well now our heads are exploding as producers of software.

Now, you are probably saying, “yes Matt, we get that … but what’s your point?”

Well a lot of organizations struggle with the concept of balancing the amount of bugs (and technical debt) in the software that we address versus the amount of new features and enhancements that we add. We often find that stakeholder who is all about the sexy or the cool — looking for that differentiator that users want.  As a delivery team, we are quick to say to the stakeholder, “you don’t get it, users aren’t going to like this and it’s going to costs us more to fix later.” This statement is 100% correct. The stakeholder, who may have a case of “rabbit ears” responding to a big prospect need and he or she may not have visibility or understanding about the extent of the technical debt within the software.

So what do we do about it and how do we find balance?  There are four things that I’ve seen work really well:

1. Make all bugs and technical debt part of the backlog. I run into folks that will manage bugs in one system and stories (requirements) in another. They need to be together — remember a bug is a feature, it’s just a negative one. You can write the bug as a story. The bugs should be ranked and treated like any other member of the backlog.

qualitychart2. Create Metrics and make them BIG AND VISIBLE. A chart that demonstrates number of bugs by severity on a specific day is not too useful to communicate this problem. But, a chart that shows a trend over the last six months the # of bugs found with a corresponding chart series that shows # of support calls for the same period and with another line that shows % of maintenance renewals — well, now you have a powerful story. How about a chart for measuring quality that is based on the number of known bugs released / number of support calls? Or, similar to this the number of new features released / number of support calls. And, how about a simple number — the percentage of our total backlog that is associated with bugs and technical debt. I guarantee a high number will get someone’s attention. Make these charts part of your team’s information radiator, post them on the wall in the break room, or start making them part of that weekly project report.

3. Conduct frequent Extermination Hack Fests. Establish one day an iteration as a bug extermination day or maybe ever three iterations, use multiple days for this activity. This is not just about having an hardening sprint, it is about allowing the engineers (architects, testers, and developers) to team up and knock out bugs with the only consideration that once you start working on a bug, it must be fixed or an approach designed to fix, by the end of the Extermination period.

4. Align resolving bugs and technical debt with strategic goals. Almost all companies have goals associated with customer satisfaction, or growing our footprint within our existing customer base, or establishing market leadership. These goals are not achieved by putting out crappy product. By leveraging the metrics above, value can be placed and impact measured of resolving bugs and paying back debt to achieve strategic goals. Be sure to measure improvements and investment against the goals as it surrounds bugs, and make it known when the team has an Extermination Hack Fest that resolves 15 customer found bugs which will improve our positioning on customers paying ongoing maintenance.

These are the four I’ve seen.  What have you used to get over the tug-of-war between new stuff vs. bugs?

Top 10 Top 10 Agile Lists

Top10Top10A while back I stumbled upon a clever blog post that showed the Top 10 of Top 10 lists and given my lack of creativeness today, I thought I would put together a Top 10 of Top 10 Agile Lists. I took this on and felt it would be an easy task given the amount that folks are talking and writing about agile. Well, it really wasn’t that easy, but I did find 10 Agile Top 10’s that are worth a read. Check them out, and please share your thoughts and links to other agile blogs that you feel are good ones:

#10 10 Misunderstandings About Agile Development. Written by the folks that maintain the Scrum Project Management blog, this is a clever and funny blog that tries to dispel some of the misgivings of agile. My favorites are “We do not want testers in Agile Development” and “Agile will likely be too chaotic to track progress.”

#9 Top Ten Agile Analogies. Damon Poole (founder and CTO of Accurev) is a funny guy without writing this blog, but he of course delivers with this one on Do it Yourself Agile blog. My favorite from this one is an anonymous quote, “Agile is like being pregnant, either you are or you aren’t.”

#8 10 Contracts for Your Next Agile Project. This is not really a top 10 list; however, it is a solid list of 10 contracting methods you can use if you are starting/contracting on a project that will be an agile project. It’s a good read and a constant question I hear while working with teams.

#7 Top Ten Organizational Impediments. This blog was penned by Craig Larman and Bas Vodde, this top ten list lays out some of the issues that slow or prevent organizations from successfully using agile as a software development process. Much of the list centers around the organizations that say they are agile but they are not serious about it. What I mean is that they are not aware of the commitment they must make.

#6 Top 10 Estimation Best Practices. I like to say that we make estimating way too complex; however, we all do it and we do make it too complex. This blog from Leading Answers (Mike Griffiths) takes a fairly broad brush and paints out 10 guidelines surrounding estimation. The best point made is “Agree on what ‘It’ and ‘Done’ Means”. This is where the conversations for agile requirements should be centered and estimation will be natural and easy.

#5 Top 10 Essential Product Owner Characteristics. Peter Saddington does a great job maintaining AgileScout, and he has done a good job helping to hone the skills of the ever important Product Owner role. This top 10 list does a great job spelling out those things to look for within a good product owner — the best one here is “Collaborative by choice.” To me, this characteristic is make or break when it comes to a good Product Owner.

#4 Top Ten Reasons to Love Agile Testing. This is a favorite list of mine. One reason is that a team that has embraced Agile Testing best practices is usually a successful agile team and the second reason is that it is a fun list.

#3 Top 10 Agile Development Traps. Found on BrainsLink.com by Vin D’Amico, this blog posting should be shared with all developers as part of their training and indoctrination to agile development. I’ve seen multiple teams fall into these traps.

#2 10 Agile Strengths. John Goodpasture writes a great positive blog about why Agile has worked for him. He stated that his first post about agile weaknesses went negative and I must say, you are not alone. A lot of folks write negative slanted blogs about agile and it was bought for me to find many positive top ten lists. I don’t get this because I see that when Agile principles are applied and used, the long-term results are positive. Nonetheless, thanks John for writing a positive top 10!

#1 Top Ten Things Programmers Hate About Agile. And to wrap up this top 10 of top 10’s, Damon Poole provides us the perfect satirical top ten to make folks reflect on why they think they don’t like agile. It’s a great list in that it pokes good fun at some behavioral aspects of developers/engineers which often lead us to dislike the fun change that Agile introduces. Nice job Damon 🙂

If you have a top 10 list to share, please post a comment and a link.