- The Difference between Agile and Waterfall
- The Agile Manifesto
- Misconceptions about Agile
In traditional project management, you initiate the project, plan the whole project, do most of the design, then do all the development, test it all, and finally release it and hope the customer likes it. After you’re done, you hold a “lessons learned” meeting to discuss better ways of doing things in the next project. This is often called a “waterfall” project because each step is completed before the project moves down closer to the end goal:
In Agile you do all those steps, too. But after initiating the project partly the same way, in most formal Agile approaches like Scrum you do the planning, design, development, testing, release, and lessons-learned for a tiny piece of the final functionality in days or weeks. I think of each iteration as a “mini-waterfall,” as illustrated below:
By the way, Agile is not a “method,” as detailed in a moment. It is a work philosophy, and a set of principles common to various methodologies that existed prior to any formal statement of the philosophy. Furthermore, as defined on the Philosophy page, agility does not require Agile.
Almost all traditional projects fail to deliver what was originally planned by the original planned date at the original cost. Every source I have consulted on this since becoming a project manager in the 1990s has supported this statement. The word “fail” is unfair, however. The idea that waterfall ever could do that is a myth. The simple truth is, no human can predict the future. You and I are not gods.
Consider this data quoted from a journal article about the ability of a group of experts to predict demand for electronics products:
|Future prediction||Forecast accuracy|
|1 month||± 5%|
|2 months||± 20%|
|3 months||± 50%|
|Beyond||Toss a coin|
This dates to 1994, when waterfall was the only kind of project management most people knew. In its Project Management Body of Knowledge (PMBOK), the Project Management Institute states firmly that changes to a project plan will result from execution of the project. Every project manager is taught how to “rebaseline” a plan, providing new date, scope, and/or budget expectations as new information arrives. That information ranges from revised customer requirements to better ways of accomplishing work, learned by doing that work. Other possibilities include changes in resources, changes in priorities, disruptions in supply, turnover of key individuals, and new innovations on the market. Recognizing this, I have long argued as a project manager and teamwork consultant:
Change by itself is neither good nor bad. Change is inevitable. The question is, do you manage it, or let it manage you?
Sadly, most companies have built their governance and supporting systems on the Waterfall Myth, turning inevitable change into something to fear. Ponder if you will the “Triple Constraint” in traditional project management, meaning scope, schedule, and budget. Many visuals put quality in the middle, like this:
Sometimes referred to as the “Iron Triangle,” the term has lost favor because executives misunderstood the “Iron” in the title to mean the shape of the triangle remains as-is when the project manager does their job well. This is simply untrue. The point is to explain the impact of change. In any triangle, changing the length of one leg changes the shape of the triangle.
When you increase scope, cost and/or time must go up. Cut cost, which on any project is mostly people, and scope must go down and/or time must go up. To decrease time, you must either decrease scope or increase cost to add people (directly or through outsourcing). The only other option in each case, as the figure above makes clear, is to decrease quality.
The further into the future you get, the less that future ends up looking like it did at the start. If a demigod looked down from the heavens at a new project, they could state how long it was actually going to take regardless of the project management method used. Waterfall almost always promises a date sooner than that, wastes a lot of labor hours on planning work that will never get done, and puts people through significant stress at the end due to the inevitable changes. Truly agile approaches like FuSca promise no date up front, yet please the customer when the “demigod date” is reached. In fact, the final date in any properly run Agile project will likely beat the date waterfall would deliver, because fewer labor hours are wasted on unnecessary planning, unused development, and unwieldy change control.
One driver of what we now call Agile methods was software people tired of getting pushed to meet dates that were impossible to predict. Months of planning in hopes of long-term predictability resulted instead in achingly slow releases of software that no longer met the customer’s needs, and still failed to meet the plans.
Full Scale agile™ does not predict a delivery date until that delivery is virtually certain. Because customers and the other business units that support them need some prior warning to implement a delivery, this system does predict with 80% confidence what scope will be delivered a quarter in advance. Some project managers and many executives don’t want to admit this, but that is as good or better than any waterfall project plan if quality isn’t sacrificed. The reality is, release dates are rarely firm more than a month in advance in waterfall. Per the PMBOK, a good practice for waterfall is to update the schedule every time a change is approved. Every time I saw this done in a disciplined manner in waterfall companies, this moved that release date regularly until the last 30 days or so. FuSca™ provides the same predictability with less planning/replanning time and higher quality.
“The problem with predictive measurements is that they often do not reflect reality,” the Project Management Institute says in its 2017 Agile Practice Guide. “For example, project leaders describe the project as ‘90% done.’ At that point the team tries to integrate the pieces into a product. The team discusses missing requirements or surprises, or finds that the product does not integrate the way they thought it would.” In fact, PMI continues, the project often goes from “green” status to “red” one month before the release date “with seemingly no warnings…”
In practice, most waterfall companies remain in denial about delays of major releases until then, at which point some upper manager is forced by the facts to “bite the bullet” and tell the customers the bad news. Then everyone affected by the change in the customer and supplier companies has to re-plan everything in short order—sometimes more than once. Meanwhile, workers have been burning themselves out to meet a deadline that ends up moving anyway. In doing so, they introduce more mistakes, and managers often cut testing. Thus quality is lower, and customer satisfaction takes a hit because their schedule and scope expectations were not met. None of that happens in Full Scale agile. Bad news is inevitable, but FuSca never delivers a bad surprise, because it never hides anything from the customer.
Unfortunately, most middle and upper managers came up in waterfall environments awash in the Waterfall Myth, and that is all they know. Thus upper managers in non-Agile companies often insist on delivery dates for specific scope in a waterfall fashion from all teams, Agile or not, and expect the original numbers to be met. The concepts of Agile seem like chaos to these folks. Hence methods have been proposed to create long-term release planning in Agile projects, resulting in all the same problems as waterfall. FuSca—and from what I hear about failed attempts to scale Agile in general—will not work well in companies whose executives remain married to the Waterfall Myth and insist on “commitments” to deliver by specific dates beyond the next sprint.
I prefer the compassionate view that resistance to Agile, like resistance to the unknown in general, is due to lack of information. Most executives are intelligent people trying to balance the best interests of their customers, companies, and workers. I hope to convince you in the rest of this “Agile 101” chapter that adopting agility across your enterprise is the easiest way to accomplish that balance.
You may have heard terms like “Agile” and “Scrum,” but not be sure what they meant. That’s understandable; there is a lot of confusion on the Web, and many people treat the words as synonyms. Having given a lot of “Agile 101” talks to executives and sales people and engineers, I can reassure you most did not realize one term is a subset of the other—and the subset came first!
As defined under “The Pillars of Radical agile,” agility is having human and technical systems that allow your organization to quickly adjust to changing market and customer demands. “Agile” is a philosophy, or a set of principles for putting those systems in place, not a “method” of project management. The term did not even start in software, the industry with which it is most associated. Various alternatives to waterfall that were more responsive to change had been bubbling up in manufacturing, supply chain management, and then software development since the 1960s.
In 2001, experts in some of these methods for software met in the ski town of Snowbird, Utah, USA. A common feature of their methods was delivering working software code quickly and improving it in small batches instead of trying to plan and deliver the whole product at once. They came up with four statements of priorities and 12 related principles common to their methods, and documented these as the “Manifesto for Agile Software Development,” commonly shortened to “Agile Manifesto.” There are competing manifestos for different fields at various stages of revision and acceptance, but the software Manifesto is by far the best-known:
Although the Manifesto and principles use the word “software” because they were written by software engineers, I believe you can replace that word with “product(s)” or whatever you create for customers. Agile has been used for deliverables in a wide range of disciplines, as I will discuss in a bit.
I’ve heard people say things like, “We’re not supposed to document,” or, “Agile means we don’t have to follow a process.” Not true, as the last sentence in the Manifesto shows. But Agile puts a greater emphasis on people talking to each other and dealing with change as it comes. As the American Management Association has put it, “In the agile environment, the PM emphasis is moved from planning to execution.”
The Manifesto’s authors added “Twelve Principles of Agile” that go into more detail about how their methods put the Manifesto’s priorities into practice. Again, if your team does not produce software, replace that word with your own deliverables as you read these:
- “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
Instead of waiting for months to release new functionality to end users, Agile methods can do so several times a day. Even if these deliverables are held until they can all be given to the customer in a bigger “release,” a fully tested “potentially releasable product” is produced at the end of each shorter period of time. (In Scrum, this period is called an “iteration” or “sprint”).
The emphasis on customer satisfaction comes up over and over in discussions of Agile, and this is a critical point to bear in mind. Project success depends, above all else, on making the customer happy. Meeting the Iron Triangle doesn’t matter if that doesn’t happen. And satisfaction is a strong correlate to the customer’s willingness to buy from you again.
- “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”
In traditional project management, the customer often does not see a working product until months have passed. The desire to meet the Triple Constraint is paramount, so the team will spend months just planning, if necessary, in hopes of a short and accurately predicted execution phase. Meanwhile, the customer may have changed their minds about some requirements, creating significant rework and delaying project completion. The team may have realized there are better ways of meeting the requirements. And when the customer sees the deliverable, they may realize things important to them at the start no longer are, or identify new things they want.
Because of this, waterfall project teams impose formal change management processes. These take time and effort and may result in a change being rejected as too large or too late. Meanwhile the teams have wasted time doing work the customer decides is not valuable. In Agile, a customer can see a functioning deliverable quickly, and change their mind just as quickly. This cuts way down on wasted time for the team and ensures the customer ends up with the product or service they actually wanted… no matter what they initially asked for!
- “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
I’ve already covered a lot of the benefits to shorter cycles. Also, estimation of work is much more accurate when limited to weeks instead of many months. (As explained later, some predictability beyond a few weeks is necessary for customer satisfaction in some settings.) Plus, teams can get the personal satisfaction of producing something tangible far sooner than in traditional projects.
Another advantage the Manifesto writers may not have been aware of is the power of the deadline in speeding up work. Researchers have learned that teams with deadlines months in the future tend to wander aimlessly until the halfway point, and then get their acts together. Prior to my introduction to Agile, I recommended managers of teams that did not have deadlines create artificial ones via continuous improvement projects. Iterations of a month or less keep up the sense of urgency. However, FuSca includes techniques to ensure people do not burn out in the face of repeated deadlines.
- “Business people and developers must work together daily throughout the project.”
By relying on a contract; a relatively static requirements list; and formal change requests, waterfall disconnects the project team from the humans for whom they are providing value. Agile asks the customer, or at least a representative in regular contact with the customer(s), to actively participate in the project. That way the team gets feedback regularly to ensure it is meeting the customer’s current and future needs.
This approach goes a long way toward managing customer expectations. As we will see, those expectations are critical to satisfaction. Agile customers have the same view of progress as the teams, and understand their own impact upon it.
- “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”
Academic research into persuasion shows that an individual who comes to believe in an idea no longer requires outside motivation. They will work just as hard to accomplish that idea as they would if the idea had been theirs. Research also shows that the conditions people work in, from physical environment to policy constraints, have a greater impact on their behavior in a specific situation than personality traits. They can only perform at their best when they have the resources they need, are not blocked by policies or politics, and are only being told what is needed… not how to accomplish it.
- “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”
Conversations in person allow you to read an individual’s body language and facial expressions, cutting down on misunderstandings. They are more time-efficient than e-mail or chat sessions because responses are quicker and do not require multiple interruptions. Agile, and studies on teams in general, recommend team members have direct physical access to each other for that reason.
However, the reality of the modern business world is that many companies force teams to work virtually across multiple time zones. In that case, those teams should encourage regular use of media that allow as much “nonverbal” communication as possible, such as video meetings and phone calls instead of e-mail or instant messaging. Members should also be close enough in terms of time zones to easily interact every day, as discussed further down.
- “Working software is the primary measure of progress.”
As stated in the first book on Scrum for software, “Measure work items done, not time spent per task.” Providing a usable product is the only way to add value for the customer. Everything else you track, if it does not directly contribute to this goal, wastes time on administration that could be used to provide value to the customer.
- “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”
When people are forced to work long hours over long periods, they burn out. A vast amount of research shows those people make more mistakes, get slower in hour-by-hour output, get sick more often, and are more likely to quit. Agile focuses on realistic and ethical workloads in exchange for predictable short-term deliveries at high quality.
- “Continuous attention to technical excellence and good design enhances agility.”
Being responsive to change does not mean skipping design and quality. To the contrary, because these activities occur in each iteration and resulting deliverables are viewed by the customer earlier, the team has no choice but to keep quality “top of mind.” Regularly releasing products also makes fixing old defects more of a burden, encouraging members to find ways to reduce them up front.
- “Simplicity—the art of maximizing the amount of work not done—is essential.”
When the customer’s wants are a list of line items in a spreadsheet written months ago, it is too easy for project teams to veer off course into what they think the customer wants, or worse, what they think is the better way to go as they learn more about the project. This can create an awful surprise for everyone when the customer sees the final product and doesn’t want the “other stuff,” again adding rework and a lot of wasted time.
Related to the previous principle, when a Scrum team gets customer defect (or “bug”) reports, they have to take time out of their next iteration to go back and fix them. No one likes re-covering old ground, and the time required can put the team’s current-iteration promises at risk. Agile teams have very personal motivation to do only what the customer wants, in the easiest way to maintain that.
- “The best architectures, requirements, and designs emerge from self-organizing teams.”
A consistent fact from decades of scientific research into group productivity is that “empowered” teams that make most decisions on their own outperform similar teams directed by a boss or dominated by a single expert. Based on my teamwork research, I believe this principle is the single most powerful reason for the success of properly implemented Agile methods. Agile not only allows but prescribes team empowerment.
As practiced, unfortunately, this is also the most corrupted principle. Companies often allow managers to overrule decisions made by teams. Worse, organizations that make money by selling certifications or placing people in jobs have misled companies into thinking the facilitation roles in the most popular Agile methods are full-time jobs. As a result, people in those roles, understandably trying to justify their pay, fill the vacuum by making decisions by themselves that are supposed to be team decisions. In short, they turn back into classic “team leaders.” This is a waste of payroll money and eliminates the psychological value of self-organization.
- “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
By meeting after every iteration to review what went well and what didn’t, the team catches more mistakes and improves its operations earlier than standard project teams do. This saves significant time, hassle, and therefore money over the course of the project.
Among the many myths in business is the belief that Agile is something different from project management. Let’s go to the experts to explore that myth.
The Project Management Institute is the primary certification body for project managers. I am a long-time member, and proudly hold its best known certification, mistakenly associated with waterfall only, the Project Management Professional® (PMP®). Although PMI is guilty of some of the same sins I have accused the Agile Industrial Complex of committing, the certifications are more stringent, and, critically, you have to have actually done the role—you can’t just take a short class and test and be considered “certified.”
The institute publishes its collective wisdom as the Project Management Body of Knowledge or PMBOK. I see in job ads a common misconception that the PMBOK is a “method.” Instead, Section 1.1 of the book states that it merely describes “good practices,” and continues:
“Good practice” means there is general agreement that the application of the knowledge, skills, tools, and techniques can enhance the chances of success over many projects. “Good practice” does not mean that the knowledge described should always be applied uniformly to all projects; the organization and/or project management team is responsible for determining what is appropriate for any given project.
Therefore, nothing in the PMBOK contradicts agility. Yes, you choose to leave many parts out, but the same is true for any project. The remaining parts are all used in an agile system—just in a different way than you may be used to. As an example, check out this graphic from the PMBOK:
The cost of a project needs to be justified regardless of the method used, and that is part of the “Initiating Processes.” However, the graphic clearly shows the “Planning” and “Executing Processes” recurring throughout the project in a cycle, with ongoing “Monitoring and Controlling Processes.” Some of the “Closing Processes” are done multiple times in an Agile project (with each release of a deliverable), but others are only done once, like closing out the project in the accounting system. Notice, too, that the title says this is only “One Approach” to one project.
As conclusive evidence that Agile is a form of project management, PMI has a certification in Agile, which I also hold: the PMI Agile Certified Practitioner® (PMI-ACP®). And the latest version of the PMBOK, Edition 6, is packaged with the Agile Practice Guide quoted above.
Agile is associated with software to the degree that many hardware designers don’t think it applies to them. To the contrary: The 1986 paper that first applied the term “Scrum” to business, “The New New Product Development Game,” was based on development of six physical products: copiers, cameras, a computer, and a car! The method called “Kanban” and now considered part of Agile started in Toyota manufacturing plants.
According to a Dutch researcher, “agility is the capability to rapidly reconfigure a manufacturing system for efficient production of new products as they are introduced. Agility is often equated with rapid response manufacturing and mass customization, as these concepts all aim to produce exactly what customers want… Agile manufacturing provides mechanisms to react quickly to changing markets, to produce high quality products, to reduce lead times and to provide a superior customer service, in a dexterous way.”
To feed that kind of manufacturing, supply chain experts have recognized the need for agility as well. “Companies that focus on agility are market-sensitive and will profit by exploiting their supply chains to rapidly and cost effectively respond to unpredictable changes,” wrote three professors at The Center for Engineering Logistics and Distribution in 2007.
Applying the FuSca principles to hardware design and development projects merely requires some changes of “sprintly” expectations, as detailed throughout this site. You cannot design an entire circuit in one sprint. You can determine the logical place on the board for a particular capacitor, given what you now know, and demonstrate a diagram or 3-D model to stakeholders for comment. As one researcher noted in reporting a study of Scrum hardware companies, “These product versions, or ‘protocepts,’ can be computer-generated 3-D drawings, virtual prototypes, crude models, working models, or early prototypes. The result of a done sprint, in this context, may not be a working product, but it is something physical that the customer can respond to.” The introduction of computer modeling and 3-D printing techniques mean hardware design processes can move much closer to software development cycles.
One veteran hardware engineer I worked with resisted the concept, but begrudgingly agreed to serve in an FuSca role for a team of utility meter hardware engineers. This was in part due to his new boss, who successfully used Agile in cellular equipment design in a previous job. A year later the engineer admitted that earlier he did not want to do it. “But now I wouldn’t go back,” he said.
The PMI Agile Practice Guide mentioned above says, “Examples of people encountering high-uncertainty work include software systems engineers, product designers, doctors, teachers, lawyers, and many problem-solving engineers.” Agile methods have been employed by each of these. Nontechnical organizations are gaining the benefits, too. Marketing and Human Resources professionals have promulgated their own versions of the Agile Manifesto, and Scrum has been applied to public relations teams. Nonprofits can easily apply Scrum to longer-term deliverables from grant-writing to supplying new services, and Kanban to their daily service provision. I have seen Agile methods help high-level managers complete cross-functional strategic planning and process change efforts. There are even books about using Agile to run your family! I knew a Scrum Master who found it radically effective at getting the kids engaged in household activities.
Related to that, your company already has or hires workers that use agile concepts: the housecleaning staff. These service workers do not typically dust all of the desks; then go back and empty all of the trash cans; and then go back and vacuum each room. They do all of those things in one room until it meets their quality standards, repeat those steps for the next room, and so on until the building is done. Often this is completed without direct involvement by a manager, except maybe to “accept” the work at the end of the shift (the Scrum term for “approve”).
Partially in response to the failed rollout of the HealthCare.gov site in 2013, the United States government began hiring Silicon Valley experts to improve its IT services. They have been “spreading the agile development model to far corners of the US federal government.” Having worked with government teams from the local to federal levels, I can confirm both that government is slow to adopt better practices and that it can learn them.
Alistair Cockburn, co-founder of the Crystal Clear method and a well-known author on Agile, used iterative methods to rebuild his house. He gave the Agile Advice blog an example: “Rather than excavate the whole basement, we decided to excavate only 1/3 of the basement. This gave us the basement entrance we needed, and left open the question whether we would extend the excavation or build a side wing in the next project… Three years later we still have no plans to extend the floor space, either above or below ground.”  Excavating the whole thing would not have saved money at the time, he said, and as it turns out, would have wasted a lot of money, since they decided they didn’t need the extra space. Cockburn, by the way, started his career as a hardware engineer.
The Agile Practice Guide provides a related example: “builders may want to show a finished room or floor of a building before they continue with the remainder of the building… The customer is able to see and approve the style, color, and other details, allowing adjustments to be made before further investments of time and money are made.”
I used Kanban to stain my back deck. I cleaned, treated, washed, and stained one small section at a time. As someone who does not enjoy home improvement, I found it remarkably stress-free. When other priorities or weather prevented my working on it, I didn’t have to worry about it or redo any steps.
Well, I had to once, because I failed to re-check the weather forecast, which I should have under Full Scale agile!
 Naylor, J.B., M. Naim & D. Berry 1999, quoting: G.H. Watson (1994), Business Systems Engineering: Managing Breakthrough Changes for Productivity and Profit, Wiley:New York.
 AMA 2007.
 Curtis, Abratt, Rhoades & Dion 2011.
 Wheelan 1994; Hackman & Katz 2010.
 Morgan 1995.
 Schwaber & Beedle 2002.
 Project Management Institute 2013.
 Takeuchi & Nonaka 1986.
 van Assen 2000.
 Baramichai, Zimmers & Marangos 2007.
 Cooper 2016.
 Cooper 2016.
 van Ruler 2014.
 Christy 2016.