Making the Jump: A Journey From Waterfall to Agile

I used to be part of the “non-believers” when it comes to agile software development.  I’ve since converted to an agile evangelist of sorts, and appreciate the significant benefits the agile approach has to offer.  While my own conversion was not so easy, I’ll explain the transformation, what I consider to be the key program management benefits associated with the agile approach, and a few challenges that I think still remain.

Why the change to agile?

Having just completed a very lengthy and stressful, yet highly successful waterfall development project, I turned my nose at the thought of moving to the agile approach.  Why not just capitalize on everything we learned and kick-off the next project initiation phase?  Truthfully, I was anxious to get started on the next release, and time spent changing our life cycle methodology was only going to delay our start.  Plus, I didn’t see how we would deliver the end product any sooner.  The methodology was being thrust upon us from an oversight organization, essentially calling it the “panacea” for our lengthy, high risk, waterfall development approach.

My Initial Skepticism

With the decision to move to agile essentially made for me, I attended the Scrum Master Certification class and passed the certification exam.  Taking the class certainly didn’t convert me into a believer.  In fact, I was even more skeptical after reviewing the “agile manifesto”:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

As a practicing Project Management Professional (PMP), one thing became very clear to me – the agile manifesto had to be written by a group of software developers!  My experience showed that software developers (1) hate to plan, (2) hate giving me status reports even more, (3) hate documentation, (4) hate process, and (5) steer as far away from a contract as possible.  Surely, only a group of software developers would come up with this methodology that allows change to happen repeatedly and documentation to take a back seat to working software.  I left the Certified Scrum Master (CSM) certification class feeling like the agile manifesto was a publicity stunt crafted by developers to let them focus solely on writing software.  Hmm…. can’t wait to see this actually in process.

The Transformation Begins

Looking back, I suppose the things that started making me a believer were directly related to the things I dislike about the waterfall development approach.  The agile development approach solves many of these key items:

  • Trying to plan and estimate how things will go during the latter stages of development and testing while you are in the project initiation phase is almost a complete waste of time, no matter how many times you’ve delivered projects of the same scope and size.  The Project Management Body of Knowledge teaches us that every project is unique.  Trying to estimate future development activities before the design phase is complete will always lead to inaccurate results no matter how many times you’ve done it before.  The agile development approach allows you to plan in reasonable “chunks” called sprints or iterations or increments.  This activity is much more reasonable, and incredibly more accurate.  Additionally, once you’ve done it successfully a few times and your stakeholders buy-in, you can use your own team velocity as a predictive measure.
  • Having daily, in-person interactions with the team is extremely compelling.  One of the things I hated during the waterfall cycle was requesting weekly status from developers to update my integrated master schedule.   This hatred was exacerbated when we became “behind schedule” and significant re-planning was necessary to get us back on track and remain “within baseline”.  No amount of time spent re-planning ever allowed us to make up schedule variance.  Scheduling is one area where looking at comparative project data was accurate – software development projects that are behind schedule rarely, if ever, get delivered on time.   Listening to developers give status each day, in person, benefits the entire team by avoiding multiple interpretations of a status report and confirms the entire team is on the same page.
  • Communicating status to your peers every day can be motivating.  Using the agile approach, team members are required to articulate what they did yesterday, what they plan to do today, and any impediments they have to completing assigned tasks.  This ceremony – “ the daily stand-up” is the center of the workday for the team. During a long, 6-month waterfall development cycle, slipping a week or two seems harmless (even though as noted above, we know it never gets made up).  In a typical agile two or four-week sprint, not completing your assigned tasks can be scary.  Who wants to admit to their peers on a daily basis they can’t get the job done?  Whether the challenge is related to a technical issue or a functional issue, any barriers to success (called “impediments”) are highlighted and addressed daily.  This increased accountability to your teammates drives increased productivity.
  • Frequent feedback from an end user representative is even more compelling.  User acceptance testing happens at the end of the traditional waterfall life cycle.  I recall the overwhelming stress associated with demonstrating an “end-product” all at once.  Of course this final acceptance led to significant changes at a very costly time – when development was complete.  Getting feedback from a fully empowered product owner during the course of a 2 or 4-week sprint, and an expanded set of stakeholders at the end of the sprint almost eliminates the acceptance testing risk.  The frequent and early feedback cycle allows the development team to make corrections as part of the normal sprint process – much more effectively than at the end the waterfall cycle.
  • Designing as you go is more successful than overhead-intensive quality gates.  Producing, reviewing, inspecting, and finalizing dozens of deliverables to pass a waterfall quality gate increases project stress.  Largely a paperwork drill, preliminary and critical design reviews for example allow a team to create a baseline design, which is sure to change once full-scale development is finally underway.   Building accurate requirements, design, and testing artifacts as you progress through the agile development life cycle is much more successful, and whole lot more logical.

The Agile PMO Manifesto

All of these benefits associated with moving towards an agile development methodology not only convinced me it was a much better approach, but also showed the agile methodology creates a much more enjoyable environment for all parties.   This better environment can be summarized in the “Agile PMO Manifesto” –

Two Week Sprint Plans over 1000+ line integrated master schedules

Daily lightweight status updates over monthly status reports and variance analysis 

Daily risk and issue management over risk control boards and endless risk analysis 

Frequent cycles of user acceptance and design reviews over infrequent quality gates 

This agile PMO manifesto suits a PM much better.  Even the most hardcore, “by-the-book”, PMP would have a hard time disagreeing with these principles.  If the agile manifesto has been supplemented with these items, I would have been a believer in the agile development approach since day one.

Still not a Panacea, but a Great Start

Moving to an agile approach does not solve all software development problems.  Indeed, program sponsors will always want answers to two key questions:  (1) “When will you be done?” and (2) “How much is this going to cost?”  Agile development actually makes the answers to these questions challenging.  At IBC, our collective experience with agile projects has shown us the “triple constraint” of weighing scope, schedule, and resource variables against one another is very much still a requirement.  Agile development is considered a “capacity-driven” approach, meaning the agile team can accomplish whatever the agile team can accomplish.  Until the “velocity” or throughput of an agile team is determined, the end date of a project, and the associated budget, remain largely estimates.   This scenario makes it difficult for a sponsor to fund a project if the velocity, and answers to the two key questions, will not be known for a few sprint cycles.  We’ve identified this challenge in multiple client scenarios, and came to the realization that agile is more than just a development methodology –this new approach to developing and delivering working software requires an organizational change in mindset.

At TeraThink, our agile development services and solutions help organizations achieve the multiple benefits of an agile methodology by addressing the technical, managerial, and organizational components of moving to a new approach.  If you’re interested in learning more about how to move your organization towards an agile development methodology, how to get started with a new agile team, or tips to work with your project sponsor, please contact us.  We can help your organization realize the benefits associated with the “Agile PMO Manifesto” and make software development projects rewarding for everyone – not just the folks trying to do less documentation.  In doing so, we can convert your organization into believers, just like me.

[Editor’s Note: This post was originally published on the blog of Dominion Consulting. On November 1, 2017, Dominion Consulting merged with TeraThink and are now operating jointly as TeraThink. All blog posts migrated from the Dominion Consulting website have been updated to refer to ourselves as TeraThink.]