“Everything is so chaotic”
“Seems like we are constantly in a state of chaos.”
“Is agile always crazy and chaotic?”
Chaos is a word I hear a lot lately while working with software development teams that are either initially adopting agile software development or possibly undergoing a Lean/Agile reshaping to improve their existing agile approaches. I am often asked if the chaos will subside, and — the good news, it will — to a certain extent. But to best understand when things will slow down (or even if they’ll slow down), it’s good to explore some of the causes that make things chaotic with software development.
And in my world, there’s no better way to assess than to make a top 10 list:
1. New Teams Forming.
This one is obvious — as people come together to work with one another there’s a feeling out period. Usually during this period, teams are learning how to collaborate and work through conflicts. In addition to the people learning to work with one another, there are plenty of established organizational structure and culture barriers that slow the progress of a team forming.
2. Process Learning.
Another obvious one. Although most agile process frameworks (e.g. Scrum or Extreme Programming) are fairly easy and well documented, it takes time and practice to learn the mechanics. Couple this with #1, and well — there can be some real challenges to getting things done.
3. HEAVY Learning Mode.
This may seem redundant, but it really cannot go under emphacized. In addition to learning about each other and learning the process frameworks, as engineers — we are constantly learning about technologies, learning about our products, learning about our customers, well — just getting smarter. Needless to say, this all adds up.
If you ever have watched the Deadliest Catch on Discovery Channel, you get to see the struggles and pains of the first time deckhands – called Greenhorns. They are often way over their heads, running into challenges around every corner, and are just flat out exhausted. In addition to physically and mentally killing themselves, they are trying to prove themselves. Well, this is true with just about every new team member. Not only are they dealing with items 1-3 above, the intensity of the learning is magnified and until they have some wins and time under their belt, chaos exists.
5. When Quality is NOT Built-In.
It is my opinion that in software development, over the years we’ve invented this approach to quality that is “risky and invites failure.”  Yes, I stole that — in most software development shops, quality is something that is done by different people to ensure “separation of duties” and because of the mentality that developers make bad testers. Couple that with the attitude that QA Engineers can’t or don’t necessarily need to know how to code, we have what we do today — a ton of end-of-stream testing, out-of-band automation efforts, and honestly, even the staunches of agile shops struggling with testing and ensuring quality. Without quality weaved into the Design>Build>Test cycle, we tend to see a lot more of these noisy things called Defects. Defects take a ton more time and energy than building it right in the first place.
6. Quality Automation Doesn’t Exist.
Without automation your going to find it almost impossible or at least extremely constraining to effectively and efficiently deliver with quality in a predictable manner. Similar to the “build quality in” challenges, teams often struggle because of their estimation processes calls quality out separately (which makes it a target for evil doers) and often it doesn’t account for the work around automation. Many organizations adopting agile software development don’t have an automation strategy for their legacy code. Therefore, they tend to have bouts of analysis paralysis around the problem space or they simply say, “our product or software is too hard to automate the testing” — so they won’t try. The other challenge around automation is some see it solely as an end-of-line UI automation thing where a couple engineers work on it — test automation is a holistic challenge and needs to be treated as such.
7. Lack of Cadence.
8. Unpredictable Release Cycles.
This one is an enigma, because there are teams I run into that say, “our product is to large and complex to release in short cycles” and then there are those that say, “we just release when we have to, it could be twice today or two weeks from now.” Looking at these two statements, they appear to be opposite in cause — however, it all comes down to #7 above and understanding what is the ideal batch size that reduces thrashing, allows for tighter alignment amongst teams, reduces “Integration Hell”, and prevents the amoeba style releases that never seem to end.
9. Deadline Rich Environment.
Projects are the problem — or at least the traditional sense and meaning of a project drives the idea of fixed scope and fixed schedule. Let’s look at the PMI definition of what is a project?
A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources.
At the end of the day, we drive our business off the idea of push for a date — we get emails from colleagues asking “when?”, we get emails from tools telling us “now!”, and we have other teams telling us “yesterday!” Ultimately, projects drive expectations which are generally dates, we can’t seem to get away from them until everyone realizes, we should design and define scope to fit the schedule, not the schedule to fix the scope — because the schedule usually flexes in these situations not the scope.
10. Estimation (and for that matter, Capacity) is Not Understood.
We often see teams measuring productivity units of time versus being measured as units of value. This is the case even in mature agile shops. Everyone is so focused on trying to come up with a voodoo formula to determine capacity of a team or organization and another voodoo formula to normalize story points across teams in order to build a long-term plan based on the cooked up numbers. The funny thing is that in many cases the approach used for estimation doesn’t really changed once an organization starts using agile — everyone continues to predictively plan what we really don’t know. Agile Software Development is an adaptive approach to estimation and capacity. We work with what we know, we measure value, we assess complexity, and we often simply size efforts based on relative uncertainty. If people would just keep it simple, try to get all stories to the same level or near the same level of certainty, and do the same with the to-do’s (a.k.a. tasks and tests) — then in a couple iterations, teams should be able to just count the stories and count the to-dos accomplished within a time box and use that for planning. Oh, if only it could be that simple … it is.
I recently got the priviledge to listen to Alistair Cockburn talk to a small group at the VersionOne offices. He told the story about Kent Beck and how the idea of how the user story on notecards was born (I’ve read about this before, but it was great hearing it from Alistair). He talked about how the card was just that — a card with a title that reminded the developer and the target customer/user what they were building, maybe there were a few notes jotted down on the card. The narrative about the story wasn’t captured on the card — the details were verbally communicated because the developer and customer were all in close proximity of one another — they did this crazy thing, they simply walked down the hallway and talked.
In addition to this story, Alistair spoke about his experiences with a team that had development in Singapore and the business folks and architecture worked in the states. The company was having challenges with the team in Singapore at getting things done efficiently and, in some cases, getting things done right. It seemed like there was an inordinate amount of time that had to be spent on discussing requirements and getting questions worked out. The company decided to send some of the business and architect team members to Singapore to work out some improvements. When they did — the quality and productivity went up. When they returned, it took just a couple weeks before productivity slipped and all gains were lost. The reason for these results seems obvious — timezones, communication barriers, and not being able to have impromptu conversations can have a great impact on how well teams understand the requirements especially since a lot of conversations capture the specifics.
Hearing these two stories triggered some ideas in my head, and I thought — why can’t there be a simple formula we can use to gauge the amount of detail (words) we capture in a story to make it work. Through some hard work — I’ve now have that formula.
Here are some factors that I think go into figuring this out:
- Complexity – just how well does the team understand the story and how hard the team thinks it will be, either from experience or flat out guessing. Usually captured as story points.
- Difference in Timezones – the number of hours that separate us greatly impacts how fast communications can happen, especially if the communications occur near the bookend points of the day.
- Auditing Requirements – audits are generally costly and require an additional amount of documentation overhead.
- Multiple Offices – when people are in different buildings, it raises the amount of details that have to be captured.
- Offices on Different Continents – although this somewhat relates to timezones, there is also the fact that people speak different languages. Even if they all agree to use english as the primary business language, there’s still slang, accents, and even cultural differences that impact communications.
How Long the Team has Worked Together – knowing each other is key, we figure out accents, we are able to exchange less verbal words, and we learn how to work with one another — what level of information is required to work effectively as a team.
So I reached out to some friends and the folks at Tagri Tech, and with their help — here’s the formula we came up with:
Amount of Words in Story = (Complexity * (1+ max absolute difference between timezones) * (1 + # of annual audits) * (Total # of locations * # of continents)) / (# of quarters the team has worked together)
And here’s the formula in action:
Multiple simulations have been ran and I found that the formula works in less than 0.01% of the situations. The people I’ve been working with at Tagri Tech’s statistics analysis department have ascertained that every team is different, and every product is different — this means that no matter how much documentation you have — the people involved in creating the documentation or the target consumer of the documentation really are not going to read it. They found that while some documentation like compliance reporting and end user documentation are necessary, much of the collaboration level docs are only consumed by the person that created it.
Ultimately, Teams will generally work together to figure out how much details are truly required to facilitate working effectively. If your teams are having challenges with how much documentation is required within Agile Software Development projects, then please come to Agile 2014 and check out my session on Friday at 9AM in the Sanibel room, we’ll talk about some ways teams can workout their own formula.
Hey, this post was originally published by the good folks at SD Times as an opinion piece. I appreciate the honor as well as being asked to contribute. I also appreciate those who helped a ton to edit including Maureen Upchurch and the marketing team at VersionOne. They helped me get it down to a more concise form. Below is the unedited version of the opinion piece, it has a few more links and some of my bad english and ramblings in all it’s glory. Please give it a read and let me know what you think. – Matt
Unless you’ve been under rock, you might have noticed a heightened level of discourse abound in the agile software development community. There is always a healthy amount of naysayer banter surrounding Agile coming from those that have tried and failed or those that have been forced to “be agile” and fail or those who use the phrase “we’re Agile” and don’t know what the hell it means. But, if you’ve been paying attention over the last several months, you will have noticed more and more harsh words and general discomfort about the state of Agile coming from the pioneers and visionaries (early adopters) of agile software development.
A couple examples are Agile is Dead (Long Live Agility) by Dave Thomas and Tim Ottinger’s post I Want My Agile Back which has spawned it’s own Google+ page. Oh yeah, and I should shouldn’t forget, there’s The Anti Agile Manifesto which is a whole site unto itself. Now don’t get me wrong, these posts are not all negative, in fact I find them inspiring. Dave and Tim provide constructive facts and ideas to move forward. But just the fact that we are having this discussion shines a light on the state of Agile today.
So does the elevated level of scrutiny and general unhappiness in the software development community mean certain doom for Agile Software Development?
The most prevalent driver of angst that is causing intense, partisan politic like arguments are the concepts around Enterprise Agile or scaling agile. It is well known, that organizations need to address their Enterprise Agility — which means organizations can adapt rapidly and efficiently in response to business and or market changes. In fulfilling the need of Enterprise Agility — large organizations are working to adopt agile processes within software development and other departments. A number of frameworks and approaches have emerged to address the Enterprise Agile needs and as a part of this movement, a consultancy-economy has formed to help implement the frameworks. This makes agile software development purists uncomfortable. The fear is that the only thing the consultancies and frameworks are doing is providing middle management and PMOs what they always wanted — a heavy-weight, control-oriented, certified process that has the word “Agile” in it. This is exactly what the original anarchists who wrote the Agile Manifesto were trying to change.
As a coach in the field, I’ve seen reasons to be concerned. A couple examples I’ve seen are where some software development teams scoffing at the ideas of craftsmanship, or the inspect-and-adapt loop was being used as a management tool to force their views of improvement upon the teams. Even with my concerns, I don’t view the frameworks or consultancies as the core challenge around the state of Agile.
Instead, I think the concepts of Agile are at a point of innovation discontinuity. In 2006, Diana Larson wrote that Agile has crossed “The Chasm” that exists in the Early Adopters phase of Everett Rogers’ Innovation Adoption Lifecycle. I would definitely agree and I believe that that we are on the downward side of Late Majority curve. If someone simply looked at the Innovation Adoption Lifecycle, they might assume that this means Agile will simply die-off and yield to the next best thing. The problem with this belief is that it has been demonstrated that as a core technological innovation reaches the crest of the Early Majority curve — a whole new round of innovations emerge that improve and or change the original innovation; thus, continuously extending the lifecycle. At this point in the curve, a bubble forms — this bubble is called innovation discontinuity. It’s a bubble where the original innovators demonstrate dissatisfaction to changes of their invention, while the new innovators morph the technology to meet the current and, hopefully, future needs of a changing market.
We can debate as to whether or not the move by the large organizations to adopt agile processes and practices have pushed us into this round of innovation and subsequent discontinuity. Or, was agile software development already in a period of innovation discontinuity and the ideas of Enterprise Agile have just exasperated it. At the end of the day, I’m not sure it matters — what does matter is that the agile community has grown substantially, and we need to base ourselves in the values and principles of the Agile Manifesto that have served us well over the past thirteen years.
At this point of innovation discontinuity, we need to focus our efforts and learning energies on several fronts. We need to renew the emphasis on agile engineering good practices and embrace the ideas of craftsmanship — without this, agility does not happen. We must practice relentless transparency — trust does not exist without it. We must get our executive and management ranks engaged — education is not enough. We must always put people first in that people are how things get done, not the processes. And finally, as Dave Thomas’ blog points out, we have to get back to the core of Agility — assess where you are, take a small step toward your goal, adjust based on learning, and repeat — all the while, making change easy with all decisions made.
I firmly believe Agile Software Development is not doomed. And it is imperative that if you are one of those who believe ardently in the founding values, principles, and practices of Agile Software Development — that now, more than ever — get involved in the community. Some discourse is valuable; however, we need to make sure the resulting conversations drive innovation not decay. At the end of the day, I think all we want to do is make better software and solutions that customers want to buy and we enjoy making. Agile Alliance Survey: Are We There Yet?, InfoQ, Diana Larsen, http://www.infoq.com/articles/agile-alliance-survey-2006  Diffusion of Innovations, Wikipedia, http://en.wikipedia.org/wiki/Diffusion_of_innovations  Technology Innovation S-curve, just Google it :). Great post about personal growth/innovation s-curve – http://blogs.hbr.org/2012/09/throw-your-life-a-curve/  A reference to Bob Martin’s (a.k.a. Uncle Bob) “mushy” statement made in Snow Bird to close out the two day meeting that birthed the Agile Manifesto http://www.pmp-projects.org/Agile-Manifesto.pdf