15
Jul
2014
0

10 Causes of Chaos in Agile Software Development

“Everything is so chaotic”
“Seems like we are constantly in a state of chaos.”
“Is agile always crazy and chaotic?”

Chaos is a word I hear a lot lately while working with software development teams that are either initially adopting agile software development or possibly undergoing a Lean/Agile reshaping to improve their existing agile approaches. I am often asked if the chaos will subside, and — the good news, it will — to a certain extent. But to best understand when things will slow down (or even if they’ll slow down), it’s good to explore some of the causes that make things chaotic with software development.

And in my world, there’s no better way to assess than to make a top 10 list:

1. New Teams Forming.

New Team

This one is obvious — as people come together to work with one another there’s a feeling out period. Usually during this period, teams are learning how to collaborate and work through conflicts. In addition to the people learning to work with one another, there are plenty of established organizational structure and culture barriers that slow the progress of a team forming.

2. Process Learning.

Complex Process

Another obvious one. Although most agile process frameworks (e.g. Scrum or Extreme Programming) are fairly easy and well documented, it takes time and practice to learn the mechanics. Couple this with #1, and well — there can be some real challenges to getting things done.

3. HEAVY Learning Mode.

Tired Student

This may seem redundant, but it really cannot go under emphacized. In addition to learning about each other and learning the process frameworks, as engineers — we are constantly learning about technologies, learning about our products, learning about our customers, well — just getting smarter. Needless to say, this all adds up.

4. Greenhorns.

Crab Fishing Greenhorn

If you ever have watched the Deadliest Catch on Discovery Channel, you get to see the struggles and pains of the first time deckhands – called Greenhorns. They are often way over their heads, running into challenges around every corner, and are just flat out exhausted. In addition to physically and mentally killing themselves, they are trying to prove themselves. Well, this is true with just about every new team member. Not only are they dealing with items 1-3 above, the intensity of the learning is magnified and until they have some wins and time under their belt, chaos exists.

5. When Quality is NOT Built-In.

Rolex Quality Built In

It is my opinion that in software development, over the years we’ve invented this approach to quality that is “risky and invites failure.” [1] Yes, I stole that — in most software development shops, quality is something that is done by different people to ensure “separation of duties” and because of the mentality that developers make bad testers. Couple that with the attitude that QA Engineers can’t or don’t necessarily need to know how to code, we have what we do today — a ton of end-of-stream testing, out-of-band automation efforts, and honestly, even the staunches of agile shops struggling with testing and ensuring quality. Without quality weaved into the Design>Build>Test cycle, we tend to see a lot more of these noisy things called Defects. Defects take a ton more time and energy than building it right in the first place.

6. Quality Automation Doesn’t Exist.

Sonar Dashboard

Without automation your going to find it almost impossible or at least extremely constraining to effectively and efficiently deliver with quality in a predictable manner. Similar to the “build quality in” challenges, teams often struggle because of their estimation processes calls quality out separately (which makes it a target for evil doers) and often it doesn’t account for the work around automation. Many organizations adopting agile software development don’t have an automation strategy for their legacy code. Therefore, they tend to have bouts of analysis paralysis around the problem space or they simply say, “our product or software is too hard to automate the testing” — so they won’t try. The other challenge around automation is some see it solely as an end-of-line UI automation thing where a couple engineers work on it — test automation is a holistic challenge and needs to be treated as such.

7. Lack of Cadence.

Perpetual Motion - Agile CadenceWhen teams are starting out, they don’t realize that the first thing to establish is their cadence — get their schedule in place and time box everything. The cadence surrounds the process touch points that require formal communication which help us to build habits; thus, making the process aspects of agile software development more muscle memory. If you feel like you are always in meetings or your iteration meetings are not occurring at the same Bat-Time and same Bat-Place, it might be time to reset — your cadence is lost.

8. Unpredictable Release Cycles.

Roll Dice

This one is an enigma, because there are teams I run into that say, “our product is to large and complex to release in short cycles” and then there are those that say, “we just release when we have to, it could be twice today or two weeks from now.” Looking at these two statements, they appear to be opposite in cause — however, it all comes down to #7 above and understanding what is the ideal batch size that reduces thrashing, allows for tighter alignment amongst teams, reduces “Integration Hell”, and prevents the amoeba style releases that never seem to end.

9. Deadline Rich Environment.

Deadline

Projects are the problem — or at least the traditional sense and meaning of a project drives the idea of fixed scope and fixed schedule. Let’s look at the PMI definition of what is a project?

It’s a temporary group activity designed to produce a unique product, service or result.

A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources.

At the end of the day, we drive our business off the idea of push for a date — we get emails from colleagues asking “when?”, we get emails from tools telling us “now!”, and we have other teams telling us “yesterday!” Ultimately, projects drive expectations which are generally dates, we can’t seem to get away from them until everyone realizes, we should design and define scope to fit the schedule, not the schedule to fix the scope — because the schedule usually flexes in these situations not the scope.

10. Estimation (and for that matter, Capacity) is Not Understood.

Voodoo-doll

We often see teams measuring productivity units of time versus being measured as units of value. This is the case even in mature agile shops. Everyone is so focused on trying to come up with a voodoo formula to determine capacity of a team or organization and another voodoo formula to normalize story points across teams in order to build a long-term plan based on the cooked up numbers. The funny thing is that in many cases the approach used for estimation doesn’t really changed once an organization starts using agile — everyone continues to predictively plan what we really don’t know. Agile Software Development is an adaptive approach to estimation and capacity. We work with what we know, we measure value, we assess complexity, and we often simply size efforts based on relative uncertainty. If people would just keep it simple, try to get all stories to the same level or near the same level of certainty, and do the same with the to-do’s (a.k.a. tasks and tests) — then in a couple iterations, teams should be able to just count the stories and count the to-dos accomplished within a time box and use that for planning. Oh, if only it could be that simple … it is.

Okay, this was just my brainstorming or brain dump (literally) of ten things that cause chaos in software development, in particular in the situations where an agile adoption or reshaping is underway. Just keep in mind, change is constant in business — now, more so than ever. Software development is complex, so are people. The great thing about tomorrow is, we don’t know what is going to happen. So, just practice and keep in mind, if today is bad — then there’s tomorrow.

9
Jul
2014
0

Problem Solved: Formula to figure out How Much Detail is Required with Agile Requirements

WARNING: Shameless plug for my session at Agile2014 ahead.

I recently got the priviledge to listen to Alistair Cockburn talk to a small group at the VersionOne offices. He told the story about Kent Beck and how the idea of how the user story on notecards was born (I’ve read about this before, but it was great hearing it from Alistair). He talked about how the card was just that — a card with a title that reminded the developer and the target customer/user what they were building, maybe there were a few notes jotted down on the card. The narrative about the story wasn’t captured on the card — the details were verbally communicated because the developer and customer were all in close proximity of one another — they did this crazy thing, they simply walked down the hallway and talked.

In addition to this story, Alistair spoke about his experiences with a team that had development in Singapore and the business folks and architecture worked in the states. The company was having challenges with the team in Singapore at getting things done efficiently and, in some cases, getting things done right. It seemed like there was an inordinate amount of time that had to be spent on discussing requirements and getting questions worked out. The company decided to send some of the business and architect team members to Singapore to work out some improvements. When they did — the quality and productivity went up. When they returned, it took just a couple weeks before productivity slipped and all gains were lost. The reason for these results seems obvious — timezones, communication barriers, and not being able to have impromptu conversations can have a great impact on how well teams understand the requirements especially since a lot of conversations capture the specifics.

Hearing these two stories triggered some ideas in my head, and I thought — why can’t there be a simple formula we can use to gauge the amount of detail (words) we capture in a story to make it work. Through some hard work — I’ve now have that formula.

Here are some factors that I think go into figuring this out:

  • Complexity – just how well does the team understand the story and how hard the team thinks it will be, either from experience or flat out guessing. Usually captured as story points.
  • Difference in Timezones – the number of hours that separate us greatly impacts how fast communications can happen, especially if the communications occur near the bookend points of the day.
  • Auditing Requirements – audits are generally costly and require an additional amount of documentation overhead.
  • Multiple Offices – when people are in different buildings, it raises the amount of details that have to be captured.
  • Offices on Different Continents – although this somewhat relates to timezones, there is also the fact that people speak different languages. Even if they all agree to use english as the primary business language, there’s still slang, accents, and even cultural differences that impact communications.

How Long the Team has Worked Together – knowing each other is key, we figure out accents, we are able to exchange less verbal words, and we learn how to work with one another — what level of information is required to work effectively as a team.

So I reached out to some friends and the folks at Tagri Tech, and with their help — here’s the formula we came up with:

Amount of Words in Story = (Complexity * (1+ max absolute difference between timezones) * (1 + # of annual audits) * (Total # of locations * # of continents)) / (# of quarters the team has worked together)

And here’s the formula in action:

Documentation Formula

Multiple simulations have been ran and I found that the formula works in less than 0.01% of the situations. The people I’ve been working with at Tagri Tech’s statistics analysis department have ascertained that every team is different, and every product is different — this means that no matter how much documentation you have — the people involved in creating the documentation or the target consumer of the documentation really are not going to read it. They found that while some documentation like compliance reporting and end user documentation are necessary, much of the collaboration level docs are only consumed by the person that created it.

Ultimately, Teams will generally work together to figure out how much details are truly required to facilitate working effectively. If your teams are having challenges with how much documentation is required within Agile Software Development projects, then please come to Agile 2014 and check out my session on Friday at 9AM in the Sanibel room, we’ll talk about some ways teams can workout their own formula.

7
Jul
2014
0

The State of Agile Software Development – Are we in a phase of Innovation Discontinuity?

Hey, this post was originally published by the good folks at SD Times as an opinion piece. I appreciate the honor as well as being asked to contribute. I also appreciate those who helped a ton to edit including Maureen Upchurch and the marketing team at VersionOne. They helped me get it down to a more concise form. Below is the unedited version of the opinion piece, it has a few more links and some of my bad english and ramblings in all it’s glory. Please give it a read and let me know what you think. – Matt

Khan ... I mean ... AGILE!Unless you’ve been under rock, you might have noticed a heightened level of discourse abound in the agile software development community. There is always a healthy amount of naysayer banter surrounding Agile coming from those that have tried and failed or those that have been forced to “be agile” and fail or those who use the phrase “we’re Agile” and don’t know what the hell it means. But, if you’ve been paying attention over the last several months, you will have noticed more and more harsh words and general discomfort about the state of Agile coming from the pioneers and visionaries (early adopters) of agile software development.

A couple examples are Agile is Dead (Long Live Agility) by Dave Thomas and Tim Ottinger’s post I Want My Agile Back which has spawned it’s own Google+ page. Oh yeah, and I should shouldn’t forget, there’s The Anti Agile Manifesto which is a whole site unto itself. Now don’t get me wrong, these posts are not all negative, in fact I find them inspiring. Dave and Tim provide constructive facts and ideas to move forward. But just the fact that we are having this discussion shines a light on the state of Agile today.

So does the elevated level of scrutiny and general unhappiness in the software development community mean certain doom for Agile Software Development?

The most prevalent driver of angst that is causing intense, partisan politic like arguments are the concepts around Enterprise Agile or scaling agile. It is well known, that organizations need to address their Enterprise Agility — which means organizations can adapt rapidly and efficiently in response to business and or market changes. In fulfilling the need of Enterprise Agility — large organizations are working to adopt agile processes within software development and other departments. A number of frameworks and approaches have emerged to address the Enterprise Agile needs and as a part of this movement, a consultancy-economy has formed to help implement the frameworks. This makes agile software development purists uncomfortable. The fear is that the only thing the consultancies and frameworks are doing is providing middle management and PMOs what they always wanted — a heavy-weight, control-oriented, certified process that has the word “Agile” in it. This is exactly what the original anarchists who wrote the Agile Manifesto were trying to change.

As a coach in the field, I’ve seen reasons to be concerned. A couple examples I’ve seen are where some software development teams scoffing at the ideas of craftsmanship, or the inspect-and-adapt loop was being used as a management tool to force their views of improvement upon the teams. Even with my concerns, I don’t view the frameworks or consultancies as the core challenge around the state of Agile.

Instead, I think the concepts of Agile are at a point of innovation discontinuity. In 2006, Diana Larson wrote that Agile has crossed “The Chasm”[1] that exists in the Early Adopters phase of Everett Rogers’ Innovation Adoption Lifecycle[2]. I would definitely agree and I believe that that we are on the downward side of Late Majority curve. If someone simply looked at the Innovation Adoption Lifecycle, they might assume that this means Agile will simply die-off and yield to the next best thing. The problem with this belief is that it has been demonstrated that as a core technological innovation reaches the crest of the Early Majority curve — a whole new round of innovations emerge that improve and or change the original innovation; thus, continuously extending the lifecycle. At this point in the curve, a bubble forms — this bubble is called innovation discontinuity[3]. It’s a bubble where the original innovators demonstrate dissatisfaction to changes of their invention, while the new innovators morph the technology to meet the current and, hopefully, future needs of a changing market.

Point of Discontinuity

We can debate as to whether or not the move by the large organizations to adopt agile processes and practices have pushed us into this round of innovation and subsequent discontinuity. Or, was agile software development already in a period of innovation discontinuity and the ideas of Enterprise Agile have just exasperated it. At the end of the day, I’m not sure it matters — what does matter is that the agile community has grown substantially, and we need to base ourselves in the values and principles of the Agile Manifesto that have served us well over the past thirteen years.

At this point of innovation discontinuity, we need to focus our efforts and learning energies on several fronts. We need to renew the emphasis on agile engineering good practices and embrace the ideas of craftsmanship — without this, agility does not happen. We must practice relentless transparency — trust does not exist without it. We must get our executive and management ranks engaged — education is not enough. We must always put people first in that people are how things get done, not the processes. And finally, as Dave Thomas’ blog points out, we have to get back to the core of Agility — assess where you are, take a small step toward your goal, adjust based on learning, and repeat — all the while, making change easy with all decisions made.

I firmly believe Agile Software Development is not doomed. And it is imperative that if you are one of those who believe ardently in the founding values, principles, and practices of Agile Software Development — that now, more than ever — get involved in the community. Some discourse is valuable; however, we need to make sure the resulting conversations drive innovation not decay. At the end of the day, I think all we want to do is make better software and solutions that customers want to buy and we enjoy making[4].

[1] Agile Alliance Survey: Are We There Yet?, InfoQ, Diana Larsen, http://www.infoq.com/articles/agile-alliance-survey-2006

[2] Diffusion of Innovations, Wikipedia, http://en.wikipedia.org/wiki/Diffusion_of_innovations

[3] Technology Innovation S-curve, just Google it :). Great post about personal growth/innovation s-curve – http://blogs.hbr.org/2012/09/throw-your-life-a-curve/

[4] A reference to Bob Martin’s (a.k.a. Uncle Bob) “mushy” statement made in Snow Bird to close out the two day meeting that birthed the Agile Manifesto http://www.pmp-projects.org/Agile-Manifesto.pdf

Nothing can stop the man with the right mental attitude from achieving his goal; nothing on earth can help the man with the wrong mental attitude.Thomas Jefferson

Thomas Jefferson Statue

19
Mar
2014
0

Hey Buddy, 2001 Called and Wants Its Agile Back

In the past couple months, and really over the last few years, the frustration level of where agile software development is heading has been growing. The good thing is that the frustration is not forcing folks to give up, instead we’ve seen some great conversations form and a desire to ensure that as we move into another round of innovation with technology, processes, and practices — now more than ever we need to understand the original agreement and intent of the Agile Manifesto.

Tonight I was given the opportunity to talk to the PMI Atlanta Agile Interest group. I really enjoyed it and I appreciate the discussions.

Please take a look at the presentation, and let me know if you want to chat. There’s some great talk to be had.

14
Jan
2014
0

Best Day Ever

My dog Mocha has a great tail — her fur is fawn colored and her tail has a white tip. My wife and I call Mocha’s tail the Happiness Meter. You see, you can tell how Mocha feels based on what her tail is doing. She has multiple wagging patterns:

TailPatterns

Every morning, when I wake up — Mocha is there with her tail going in the helicopter pattern. Yes, this means that she is extremely happy to simply be waking up to another day. She does this every day — to her every day is the BEST DAY EVER. The one thing to know about Mocha is that she’s getting older, she definitely has stiffness in her joints, she has a bum knee, and she suffers from bouts of mild pancreatitis. Even if Mocha went to bed not feeling well or wakes up stiff — her outlook is always, this is going to be the BEST DAY EVER.

We’ve all heard of the power of positive thinking — heck, there’s probably a couple hundred books on Amazon written about or referring to the power of positivity. But since we are humans, it’s difficult to squelch down the trials and tribulations of day. In the software development world, we talk about “the death march.” This is that project that drags on, is considered a failure before the first user even sees it, the one where the management tells folks to work on the weekend, the one where for every bug fixed someone finds two more. If you work in information technology, then you’ve been on one of these projects.

Now since you are reading an Agile blog, you are probably waiting for the Agile Punchline or to hear about the Agile Secret Sauce to help over come the challenges of these projects. Well, the reality is that Agile generally will expose problems earlier and maybe make you realize that this project is going to be painful and heading for certain doom. We hope that we course correct or try something different, but sometimes we are simply just in a bad place and cannot get out of the rut.

So while working on these projects, people come dragging in for another day of drudgery. Another day of running on the hamster wheel and not getting anywhere. Another day that the project is dragging us down — in fact, today just may end up as the worst day ever.

Remember, today didn’t start out as the worst day ever, it started as the BEST DAY EVER.

  • Today is the day that no matter what happens we will find something positive and see how today is better than yesterday.
  • Today we will find that our small failure is just that, a small failure — and we learn from it, and have a piece of knowledge that we didn’t have yesterday.
  • Today is the day that we celebrate finding defects, because those are defects our customers did not find.
  • Today is the day that we find joy in having to refactor some old crappy code, because we know that when the next developer visits that code — they’ll get to enjoy changing it, not dreading the fact that they are likely to break something.
  • Today is the day that we move one day closer to getting the project out the door.
  • Today is a great day to laugh with a coworker.
  • Today we admire what has been accomplished versus what is remains.
  • Today is going to be the BEST DAY EVER.

And if you don’t believe me, just ask Mocha.