Agile is Wrong — Part 1: Scope and Requirements Changes

Welcome to the first installment of a multi-part series of why many of the principles of Agile process, as laid out in the original Agile Manifesto, are actually detrimental to modern development teams.

What’s Agile?

Part of the problem is that the original defining principles laid out in that first document have largely been lost or forgotten, and what we’re left with is often boiled down to the idea that a software team should be “agile” in the literal sense; that is, able to easily and quickly adjust to any change at any time with no impact to cost, schedule or the team. And as anyone who has worked on a software engineering team knows, this is an absurd fantasy. While the ability to iterate and change is essential for digital product development teams, thoughtful and thorough planning is also incredibly important. And that’s why, even if proponents of Agile actually knew what it means, it would still be a poor framework for modern development teams. Today, we’ll consider the first principle of the traditional Agile process, and why it sucks.

Principle 1: “Welcome changing requirements, even late in development”

The agile manifesto was written in a world where most software products were self contained application packages that you bought in a box and installed from a CD-ROM. If you wanted a new version, you purchased one in a store, brought it home and installed it on your PC. Even in that context, this first principle of the agile manifesto only makes sense if cost and schedule are flexible. If requirements change late in the game, releases would need to be delayed, which meant actual physical production timelines had to be altered, which was a significantly expensive prospect. But that’s not why this principle sucks.

Nowadays, with distributed applications being built by multiple teams of engineers, the problems with this principle are the same, except worse. In order to develop solutions for modern software products, multiple teams have to work together. Those teams generally include product owners, design, backend development, front-end development, and test. In order for teams to build the interlocking clockwork of a connected, distributed application, requirements have to be clearly defined and locked at an early stage, in order for teams to figure out how all the pieces need to be built in order to fit together. Depending on how dramatic it is, and how big the features of the product are that are being built, a requirements change late in the process can be incredibly costly, time consuming and detrimental to the long term health of the product. Why? Drastic changes in requirements often require a rethinking of overall solutions, and potentially, redevelopment at multiple layers. This takes time, which also means, money.

Or does it? This principle of the Agile Manifesto, both then and now, assumes that schedules and cost are negotiable, but this is almost never the case. Even if the financials and timelines of the business aren’t flexible, the cost of late-in-the-game requirements changes still has to be paid. This generally results in one of two outcomes. Either:

1. The team cuts corners to rapidly redevelop to adjust for the change, incurring piles of technical debt that will impact the maintainability of the product in the long term, OR —
2. The development team works long hours under high pressure to get everything completed and tested, with questionable methods of measuring either of those goals.

Organizations that expect engineering “agility” of this kind will have high turnover, and difficulty retaining quality talent at all levels. And THIS is why this principle falls flat. The majority of the time, it isn’t business stakeholders, product managers or the corporate pocket-book that pays the price for this type of change, it’s teams of engineers. And that sucks.

This is exacerbated by the fact that, frequently, these last-minute requirement changes result in half baked solutions which are initially riddled with defects and that eventually completely fall apart. Under the stress of increased or changed scope with no accompanying change in schedule, multiple dependent layers will be developed concurrently and testing time will be shortened. Product instability and failure due to poor project planning without ample time for the vetting and consideration of requirements will almost certainly incur the third, and perhaps most important, cost of all — loss of customer loyalty, trust and revenue.

Level of Effort

Changing requirements late in development can also have a profound impact on a team’s ability to estimate the complexity of new work and create timelines. Software problems are heuristic, so no piece of work is entirely comparable to any other. This means that the ability to estimate how long a feature is going to take to build is already an abstract, best-guess. And that guess is educated by the team’s performance over time. A Scrum team, for example, estimates in terms of relative complexity, and over time, develops an average amount of complexity that it can complete in a given time period (the “Sprint”). The development of this average can take weeks or months of estimating and executing, in order to achieve the history necessary to estimate future work. When requirements aren’t clearly defined, or change late in the schedule, it makes Sprints difficult to plan, and teams wind up trying to recover by pulling in extra work, or working extra hours, or otherwise throwing a wrench into the machinery of estimation, which makes estimates and measurements invalid. If this happens frequently, it significantly impairs the team’s ability to develop timelines for new work. An organization can accept the chaos that poor planning and changing requirements inevitably produce, OR it can have the predictability necessary to have accurate estimation. But it can’t have both.

To me, there is a direct correlation between product mismanagement, lack of planning and user retention. But proving this quantitatively is immensely difficult to do. That’s why we’re all able to milk the old out-of-date principles of Agile for all they’re worth and convince ourselves that everything is fine. But every time we do this, one of our best developers or first-line managers will quit to go to a team that knows the real truth:

“Agile” is increasingly and strategically wrong.