Agile Development

"Thinking is the hardest work there is, which is the probable reason why so few people engage in it." - Henry Ford

Introduction

Before the reader goes any further, I would like to provide some context on my views here. As a reader you may come with pre-existing biases on Agile. You may have already worked on Agile projects and have your own thoughts on the methodology. I also have opinions and am biased from my experiences in various banks where I've seen Agile methods being used well and not so well. Therefore, you may not agree with everything written here, that's fine, these are my own views, not the same as yours, but read on.

My exposure to Agile has been on projects in the $600K+ (CAD 2008) budget range primarily in banking for application development.

AGILE SCRUM

My first introduction to Agile was in 2008. It was an introduction to SCRUM by Mishkin Berteig, a well known Agile trainer in Toronto. I recall that back in 2008 there was a nascent interest in Agile. The class must have had at most eight participants. There wasn't much interest yet in Agile from corporations, the people attending were there because of their own interest in the subject. For most students, their companies paid for the training. I wanted to learn more about Agile, so RBC paid for my training. There was someone also from Scotia Bank, Nortel and attendees from smaller firms.

A this time, SCRUM (Jeff Sutherland and Ken Schwaber) was the de-facto Agile philosophy. There was no Disciplined Agile Delivery (Scott Ambler and Mark Lines) or SAFe (Dean Leffingwell), or variations thereof. At the time the SCRUM Alliance provided Scrum Master and Product Owner certifications. Agile Alliance did not yet exist. Over time more organizations provided training and Agile certifications, seeing this as an opportunity to set standards and monetize on the trend. Agile education became more formalized. Ken and Jeff were two of the seventeen original signatories of the Agile manifesto.

Initially, the Project Management Institute (PMI), did not acknowledge Agile methods. Today, it publishes an addendum to the Project Book of Knowledge (PMBOK) on PMI's version of Agile. As Agile has become mainstream "corporate friendly", PMI provides Agile training/certification and publications on the methodology. In fact the PMI Agile methodology has its origins in Scott Ambler's Disciplined Agile Delivery (DAD) methodology which was acquired by PMI in 2019.

The manifesto (agilemanifesto.org) for Agile software development laid out the founding principles, much recognized today. The foundational principles are:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The four high level principles then evolved into twelve principles for Agile software development. These principles form the basis of most Agile software development methodologies.

1. Our highest priority is to satisfy the customer. through early and continuous delivery of valuable software.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Business people and developers must work together daily throughout the project.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
7. Working software is the primary measure of progress.
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity, the art of maximizing the amount of work not done, is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

EARLY AGILE PRINCIPLES

Given the attention that various Agile methodologies available today get, it would be easy to assume that Agile principles were a recent discovery. Agile principles have in fact been used in software development projects for decades. These principles have also been observed in other industries, such as aerospace. There is a general misunderstanding that projects, prior to Agile, were only managed in a waterfall, or plan-based sequential method, but that's not the case.

An excerpt from "The Starts Guide" (A guide to methods and software tools for the construction of large real-time systems, Second Edition, 1987, Volume I, ISBN 0 85012 619 3) prepared by the National Computing Centre in 1987 describes incremental software development as:

"This is an approach to system development whereby the initial system is developed through all the system life cycle stages from analysis to implementation. As it becomes clear that the system has inadequate functionality or additional desirable features are identified, the full development life cycle is re-entered and repeated. This iteration is different from the normality expected iteration within development life cycle phases, in that iterations run across the boundaries of the development cycle phases. These phases occur in their normal sequence but may occur more than once in any project. Thus products of early iterations will be retained, though modified perhaps, in later system implementations.

The effect of the incremental approach on the traditional development life cycle needs careful consideration. The separation into phases, with recognized products appearing before full entry into the next succeeding phase, is the primary means by which project control is exercised to ensure final system quality and suitability. Incremental development requires a modified life cycle and associated project control mechanisms which can deal with such matters as:

• the number of iterations before the system is finished
• controlling the increased number of deliverable items for such a project
• continual revision of estimates

It is recommended that a flexible cyclical version of the normal life cycle should be adopted, whereby modifications, possible developed by prototyping, are fed back into the life cycle at their point of initiation or at an earlier phase for revision of subsequent stages. Project management needs to decide which functions are to be included in each iteration and identify consequent effects on other parts of the system. The need for careful change control is very important to ensure the right versions are produced as deliverables."

Clearly there are elements of Agile principles insofar as introducing new requirements and breaking the development into software development iterations similar to sprints. Other incremental development approaches, such as Spiral, have also been used for many years.

In aerospace, similar Agile principles were used by Lockheed in the development of the U-2 and SR-71 aircraft by the Skunk Works division in the 1960s. Kelly Johnson was an engineer and senior manager at the Skunk Works division and developed 14 project management rules over 60 years ago that are still relevant today. These practices are the same familiar Agile principles followed today (see Appendix A below).

First there is an emphasis on small teams (rules 2 and 3) as each person added to the team means an increase in communication and team complexity. Smaller teams have faster lines of communications and can make decisions faster. Team bonding and a team's culture can be formed much quicker with small teams. The ability for small teams to meet face to face is beneficial for communication and innovation. Meetings ensure everyone understands the goals of the project. However, too many meetings can be unproductive. The same applies to reports and documentation. Reporting and documentation is time consuming and takes people away from doing work. A balance is needed to determine what is just enough; ideally a minimum should be provided that still meets the project's needs.

Finally, the manager should shield the team from decision making boards, steering committees and financial approval processes so the team can stay focused on development. Kelly also speaks about trust, a principle that is often quoted about Agile teams trusting each other.

There are likely many more examples of Agile principles being used in various industries prior to the development of the Agile manifesto. I'm guessing that the Agile principles described in the manifesto were derived from the learnings of predecessor projects that used such principles early on.

Agile versus Waterfall

There's been quite a lot of discussion in various social media forums and blogs on the demise of plan-based (waterfall) methods. Gartner Consulting proposes a bi-modal approach to software development whereby, Mode 1 refers to maintaining core systems key to an organization's survival. whereas Mode 2 pursues digital innovation by pursuing transforming business ideas into applications.

Mode 1 is a linear and predictable approach to change. The focus is to ensure reliable, accurate, and stable IT projects as mentioned by BMC Inc. What this implies is that where requirements are well known, for example, as part of application maintenance, or regulatory changes to applications where the requirements are set in advance and must be complied with, then plan-based approaches work well. As one CIBC bank executive mentioned many years ago; the (plan-based) approach has been working so why change it.

Indeed most large banking programmes have been managed using traditional plan-based methods. However, in the quest to create applications faster (cheaper, faster, better) most have adopted Agile for all application development.

One of the arguments against plan-based methods for application development is that by the time the application is delivered it no longer meets the client's needs as they would have changed. Changing client requirements is something a project has to manage. Change is especially prone on large projects where multiple stakeholders are involved in providing input to the requirements.

The Mode 2 approach is really what we refer today as Agile methods. As opposed to having all the requirements fixed and locked on day one, the approach allows for frequent feedback and inspection cycles so that the client may examine the product as it's being developed and provide input on any required changes. This approach also solves another popular requirements problem, lack of detailed or clearly understood requirements. Business Analysts (BAs), as good as they may be, can create a requirement that is not what the client intended. Natural language requirements can be mis-interpreted by the BA when gathering information from the client. Having frequent progress reviews between the client and the project team catches such mis-understandings earlier in the process. In a plan-based project, the misunderstood requirements, would only be caught at the tail end of the project during client product acceptance, by which time it's too late in the process to fix the product and will cost much more to do so. Periodic client reviews of application/product development progress would also catch such deviations in requirements, but often this is not done.

On RBC's large plan-based Basel II programme, the regulator (OSFI), who was in effect the client, demanded periodic progress reports to ensure the Basel paragraphs (requirements) were being addressed. This minimized any variance on requirements upon programme completion.

In Agile projects there is a higher requirement for the client to be involved in the various review sessions. The client involvement is more intense. Plan-based projects typically involve the client at the beginning and tail end of a project; during client requirement specification and client acceptance.

The advantage with the Agile method is that any deviations in the client's vision of the application or product is caught earlier on, thereby putting the client requirements back on track. Frequent project team and client reviews, known as retrospects, ensure the end product meets the client's requirements.

Firms have to understand the client and their requirements before deciding on a methodology. If it is understood that the client does not have clear requirements, it's a brand new application or product being developed, or is an application with many client stakeholders, then perhaps an Agile approach is best to manage the expected change storm. As well, the client has to agree to be available to the Agile project team for the duration of the project. This is one barrier most Agile teams encounter. Clients are busy and don't always have the time to participate as frequently as the project team needs. There are many other barriers to adopting Agile methods, but client participation and organizational resistance to change are two factors.

How Does it Work?

So how does Agile actually work? Agile SCRUM practices described below are derived from the twelve Agile manifesto principles. These are foundational to many other Agile methodologies.

1. User Stories - requirements are stated as user stories. The requirement is stated from the user's perspective. This is similar to collecting requirements through "Use Cases". It's no accident that Dr. Alistair Cockburn (Writing Effective Use Cases, Addison-Wesley, 2008) was one of the Agile manifesto signatories. He is also known as an authority on use cases having published several books on the subject.

   When we say the requirement is stated from the user's perspective, means that the requirement describes how the user will interact with the application, not how the application will implementation the requirement. The format of a user story is:

   As a (user role) I can (tell the story from the user perspective) so that (describe the benefit of the action).

   As an example, an application function to reset the user's password would be stated as: "As a bank client I can reset my password on-line so that I won't be locked out after the 180 day password expiry" .

   The underlined areas denote those sections of the user story that are filled in for that particular requirement. The important thing here is that we don't focus on how the ask is implemented in the application, but how we satisfy the user needs. It's the user's expected interaction with the application that counts. In this case the application must allow password changes or the client will be locked out after if the password has not been changed within 180 days.

   From an implementation perspective this simple user story drives many other decisions by the development team. For example, what is the password length, should there be a mix of symbolic, numerical and alphabetic characters, should there be a password history tracker so that the previous three passwords cannot be used? However, the discussion always starts with the needs from the user's perspective; the user story. Below is an example of a user story template used by Berteig Consulting.

   

   Often, user stories are stated a too high a level and need to be broken down further into more detailed user stories. It takes practice.

2. Story Points - user stories as with any other requirements need to be estimated as to how much effort will be needed to complete the development. This is as true with Agile as is with plan-based applications. Clients will ask how long will it take to deliver a project. The estimates then drive to budget to be allocated. Often, the budget is set and the project must live with the set budget. In traditional software development, the estimates are based on how many hours each component of a project will take. For example, a developer may estimate that developing a web screen take in three fields of character information and call a CGI back-end program to consume the data will take 15 hours to develop.

   The same estimation process is done on Agile with user stories with one exception; the estimates are not provided in hours. Estimates are often provided as T-Shirt sizes (small, medium, large), Fibonacci numbers (1, 2, 3, 5, 8, 13, 21, etc), or other numerical sequences representing the relative complexity of the effort, not the time taken to implement it. When numbers are used, we can say that a user story has this many story points as an estimate. For example, the web screen data entry development effort may be given 21 story points to represent its level of complexity. Alternatively a simple function could be allocated 3 story points due to low complexity.

   The reason that's been given for using an abstract numbering system representing the level of complexity instead of hours is to prevent management using hours as a measure of team or individual productivity. Each team in a department may use slightly different numerical scales to denote complexity when estimating. This is done to avoid politics: management comparing the productivity of their teams. Story points have been misused in organizational downsizing to target individuals who are perceived as weak contributors.

   My view on this, is that when management uses story points to compare productivity of teams or individuals for unintended purpose; it shows a lack of trust of their teams. A management culture that operates by measuring teams against each other is counter to the successful sustainment of Agile practices in their organization. It erodes trust.

   As a former developer, I'm guessing that other developers would rather provide estimates in hours. That's my preference. It's how most developers have provided estimates before Agile came along. As well, I believe it provides more realistic estimates of effort for each user story. Hours of effort as story points, as with Fibonacci numbers or other estimation methods, can also represent user story complexity.

3. Sprints - one of the principles of Agile software development is that work should be time-boxed into what is known as sprints. Whereby a plan-based project may do application development for months after the requirements are set, an Agile approach means that the development is broken into several stages; sprints. A sprint duration is typically two weeks. That seems to be the time-box that most firms follow. However, in banking I've seen sprints lasting three weeks and even one month. Once the sprint duration is set, it should not be changed for the duration of the project as per Agile principles. In practice, I've seen the sprint duration changed from 2 weeks to 3 weeks after the first few sprints as the team realizes that it's taking longer to achieve any demonstrable results, but this is not the norm.

   Sprints allow the team to work on a set of user stories, then stop after let's say two weeks, examine the progress made on the user stories for that sprint and determine what went well and what needs improving (retrospect). Perhaps the team could not finish all the user stories allocated for that sprint, perhaps there are issues with some aspects of how the team works. Perhaps Quality Assurance (QA) is taking too long to test the user stories and this is holding back the development team's progress (true story).

   During a sprint, there should be no prioritizing of tasks, interruption of the work in progress or the introduction of new user stories. The sprint should be focused development time with no breaks.

   The end of a sprint is time for reflection on what went well, what did not go well, re-calibrate and plan the next sprint. The idea with breaking the development into several stages is to elicit client feedback on the work performed to date so that the application's functionality does not stray from the client's expectations.

   The risk with long sprints is that application developers may be building incorrect application functionality. Letting these mistakes linger longer will be more costly for the team to recover thereafter. Short sprints mean any errors in functionality will be caught quicker. In Agile, the saying goes that teams should fail quicker. In effect, catch errors quicker in the software development life cycle (SDLC). This principle requires frequent client feedback. By getting client feedback quicker minimizes the chance of the application development going off-side and not meeting the client's expectations. It's an opportunity to correct course quickly before further development occurs. This is the fourth principle of the Agile manifesto, clients must work together daily throughout the project.

   Velocity

   At the end of a sprint the team's velocity can be calculated. Velocity is the number of story points that a team executed within the sprint's time frame. For example, during the sprint the team completed development for 4 user stories.

   • Story 1 = 12
   • Story 2 = 5
   • Story 3 = 16
   • Story 4 = 9

   Assuming a two week sprint, the team completed 42 story points of work. This is known as the team's velocity; the number of story points that can be accomplished in a sprint. Why is this important? When planning the next sprint we need to know how many story points the team can complete in the next two week cycle. It's a way to measure the team's capacity. Not all sprints will yield the same velocity. Perhaps, in sprint 3 or 4 the velocity is only 35. By measuring velocity after each sprint and averaging it over the last sprints will give the team an idea of their average velocity. With an increasing number of sprints the team will get a better estimate of their capacity to complete story points. As an example, consider a project with 10 sprints. By averaging the story points completed by the team over four sprints should give a good estimate of the team's capacity.

   • Sprint 1 - story points completed = 42
   • Sprint 2 - story points completed = 35
   • Sprint 3 - story points completed = 39
   • Sprint 4 - story points completed = 47

   The average story points this team can deliver is therefore (42 + 35 + 39 + 47)/4 = 41. The team's average velocity is 41 story points. The average velocity should be calculated after every sprint to better plan future sprints. Being aware that the team's velocity is 41 story points means that when planning user stories for the next sprint, the total story point sum should not exceed 41 points.

   Sprint Planning

   

   At the end of each sprint, the team plans what tasks they can execute next. Some practitioners refer to this as the sprint goal, however, in practice I've never heard it stated that way. Fundamentally, it is looking at the product backlog items and deciding which ones can be developed for the next sprint. This would take into account the team's velocity, task dependencies on other task completions (sequencing) and the number of story points required to complete each task in the product backlog. Once the tasks are selected from the larger product backlog, they become that sprint's backlog items to execute.

4. Product Backlog - the backlog represents the collection of user stories, the number of story points for each story and the priority of the story in relation to other stories. There are some features of an application that the client may deem mandatory for the project to be successful. Often, there are other features that are not so critical. These can be dropped from the application due to funding or schedule constraints. Sometimes those features that cannot be accommodated can be pushed off to a future project phase.

   In addition to user stories, the product backlog also contains bugs from previous sprints for remediation, tasks and other work items.

   Any item of work in the product backlog is referred to as a Product Backlog Item (PBI).

Sprint Planning

A quick note about sprint planning. Some organizations only plan one sprint ahead. For example, having concluded sprint 1, the team now plans the stories for sprint 2. This is all fine and well until the client asks if the project will deliver all functionality (all user stories) on time and on budget. Planning only for the next sprint does not give a level of comfort that all functionality will be delivered. This was one of the complaints from one of my managers on using Agile; "how do I know if I will get all the functionality based on the funding that's been allocated to the project". Well, if I'm planning only one sprint at a time; you don't know exactly when you will be done.

At another bank, we used an approach whereby we would estimate how many user stories we could accomplish in a sprint; a very rough estimate. We would then look at the number of user stories and the story points. We allocate stories and tasks to sprints. For example, if we estimated that the development would require 400 story points, and we assumed that using a two week sprint we could deliver 40 story points, then the development phase would need at least (400/40 = 10) 10 sprints to complete, not counting the first planning sprint.

By performing this calculation after every sprint will give a good idea of what can actually be accomplished. Perhaps the original estimate of 40 story points per sprint is inaccurate, and only 32 points is the actual velocity. Then we can forecast that we may need (400/32 = 13) 13 sprints to accomplish the work. As well, after each sprint we know the cost to implement the work within the sprint. Forecasting forward, based on observed sprint costs, we can estimate if there was sufficient funding allocated to complete all the client's functionality in the project.

So, after three sprints, we've noted that the development costs were:

• Sprint 1 - Cost = $35,000.00
• Sprint 2 - Cost = $25,000.00
• Sprint 3 - Cost = $39,000.00

The average cost per sprint is therefore $33,000.00. As a rough estimate we can expect that a project with 10 sprints will cost $330,000.00 for the development phase. However, the client's budget was $250,000.00. Here we have three options, we examine the backlog for functionality that can be removed from the product, the client allocates an additional $50,0000.00 for the project or the client decides to terminate the project before spending any more funds.

Knowing this information early on in the project is super valuable for making a decision on next steps. Funding is the lifeblood of any project, no matter if it's a plan-based or Agile. Knowing what can be accomplished within the client's budgetary constraints is key to success.

Note that these are calculations for software development costs only. There are additional project costs to consider for meetings, daily stand-ups, client reviews, documentation, an agile coach (if needed), a PM (some banks), etc.. These activities need to be funded by the project sponsor in addition to the actual development costs.

5. Estimates and Burn Rate - throughout the project the rate at which user stories are being completed versus the funding available needs to be tracked. In Agile this is managed using "burn-down" charts. The burn-down chart tracks how many of the PBIs have been completed in relation to the project time and project funding. For example, if 35% of the user stories have been completed and 55% of the development funding has been consumed, then there is a problem. Likewise, if 35% of the user stories are completed but there are only three more sprints, then it's also a problem. Tracking completion results against time and budget is needed on Agile as as with any other project methodology.

   The team must know what value has been delivered to date. As well, what actions are needed to get projects back on track when the metrics show problems ahead.

6. Reviews - these are in effect client demo sessions. This is an opportunity for the client to view what has been delivered and provide feedback to the project team. It's an opportunity for the client to examine if the product is meeting their needs, if changes are required or if there are new requirements the client wants added.

   In banking, it's difficult to have a client fully dedicated to an Agile team as the Agile manifesto purports. Clients have a day to day business to attend. I have not seen many Agile projects in banking where the client participates daily alongside the Agile project team. This is why product demos are important for clients to attend to review the application functionality and provide on-going feedback to the Agile team.

7. Retrospects - this an opportunity for the project team to reflect on what's gone well and what needs improving. This should be done at the end of each sprint to understand what's worked and what the team needs to improve upon in their development effort. This is a feedback cycle of continuous improvement. By reviewing the team's performance, any blockers any good or bad practices, is a way of improving how the team works for subsequent sprints. This continuous feedback method after every sprint optimizes the Agile team's performance.
8. Minimal Viable Product (MVP) - in Agile there is a concept of building applications incrementally in stages and making it available to the client when usable functionality is available. The reason for this is to get early feedback on the application's functionality and give the client the opportunity to request any other changes before proceeding further. It is an excellent client feedback mechanism. It also helps the development team shake out the product to address any issues that may come up; e.g. performance, interface incompatibility issues, changes needed to UX (user experience) for satisfying regulatory requirements, etc.

   For example, many web-based applications have gone into production with a base set of functions. Over time, through incremental releases, more user functionality is added. Early users of LinkedIn will be familiar with it's original limited functionality. Over time more functionality was added to give LinkedIn a richer user experience. Incremental improvements continue to this date based on user feedback.

   This is the basis for releasing functionality in stages, it relies on frequent user feedback to make the product better suited to the user/client expectations. In some cases the reason for releasing an application incrementally may be to gain a "first mover advantage" in the market. Minimal functionality is released to capture a user base and then build more functionality based on usage patterns, user feedback or new innovations, e.g. adding Artificial Intelligence responses.

9. Frequent Testing - test cases are written by QA as the product is developed through sprints. QA should be involved day one when the project starts. QA needs to understand the user stories being proposed and start thinking about the test cases for each user story. Application testing should be done in each sprint. This approach differs from a plan-based methodology where QA is performed at the end of development and is often a lengthy process.

   The need for frequent QA poses some challenges. If QA is performed during the sprint, I've seen cases where the developers are waiting for QA to complete before proceeding to the next sprint. Normally this is not an issue. However, one way to approach this, where implementation speed is needed, is to have a parallel QA environment for testing. That is, when development completes a sprint, have the code moved to the QA environment for testing so developers can continue on the next sprint. QA can then have their own copy of the code developed to test full-time in parallel with development. Any errors detected by QA will be placed into the backlog for remediation during a following sprint.

   Due to frequent testing, Agile projects have a high QA cost. The number of test cases to regression test increases with each sprint and subsequently, so does the QA effort/cost. The demands on QA resources thus increases with every sprint. One remedy has been to automate regression testing but this comes with a cost. Automation is valuable because is eases the testing of future enhancements throughout the application's life cycle. However, I have seen very few projects whereby automated testing has been used. The issue being that building automated test cases takes time and adds cost to the project. The QA team is responsible for testing and will bill out their time to the project. If automated testing is acceptable, then the cost has to be considered prior to funding the project.

   Defects

   Agile purports that defect remediation should take precedence over completing new user stories. When QA finds a test case for a user story that is deficient, that user story should be put back on the backlog for remediation on the next sprint ahead of any other user story. The rationale is that latent errors will be costlier to remediate the longer they are in the application. For example, the error may be in an API which if left uncorrected will cause code utilizing the API to operate incorrectly. Hence, in principle, fix the errors before adding new application functionality.

   In practice, I've seen defects carried over several sprints due to the project having to meet a deadline for certain functionality. We've referred to this issue as "technical debt". It's a debt that has to be eventually paid in terms of effort and cost to remediate errors that we've accepted for future. It's similar to a financial debt accumulating interest; in this case the interest is the additional cost accumulating with every sprint until the error is corrected.

10. Daily Stand-Up Meeting - this is a daily meeting usually held in the morning that gives everyone an opportunity to explain what they are working on, offer progress status or raise any obstacles they are running into (e.g. dependencies from other teams, budget, technical constraints, etc.). In effect three questions to be answered at every meeting:
    1. What was completed?
    2. What will I complete today?
    3. Do I have any obstacles impeding my progress?

    These are meant to be short meetings lasting no more than 15 minutes for a small Agile team. If there are any obstacles or bigger issues to discuss, these should be parked and addressed after the meeting. The SM is responsible for removing any obstacles the team faces; this is their role and this is the opportunity they have to hear it from the team. Because the meetings are held daily, any issues can be quickly addressed before they linger on and impact the team's success. It's called a stand-up because literally (traditionally) everyone stands during the meeting, to convey the message that the meeting should be quick; not a relaxed sit-down meeting. This speaks to Agile principle 6: the most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

    The exposure in banking I've had to Agile projects, this is one practice that prevails across all projects and banks I've worked at. The mistakes, I've seen regarding this practice is that the stand-up meeting sometimes evolves into a solution discussion session. As well, lengthy discussions on blockers should be taken off-line and addressed later. Lengthier discussions need to be taken off-line. The SM needs to manage the daily stand-up to ensure it does not devolve into a discussion session; brevity is key!

11. Definition of Done - as part of Scrum Agile project language is the "Definition of Done" (DOD). In effect, how do we know when we're done with product development. The DOD is the understanding between the client and the team of what "done" looks like. How do we know the work is done? For example, a client may define the definition of done for a software application as:
    • Full application functionality is delivered.
    • There are no known Severity 1 errors associated with the application.
    • Application passed the JAWS accessibility test.
    • Application support documentation is delivered to the client.
    • User training manuals completed and delivered to client.
    • Client acceptance sign-off of the product.

    It's the criteria, defined at project initiation, the client considers for the work to be realized as: done.

    It's possible that in reality the project runs out of money and all the functionality may not be delivered. In this case the DOD has to be re-negotiated with the client. If there is a financial constraint, then the client will have to live without some functionality, or the client may provide additional funding to get all functionality completed.

Roles

1. Product Owner (PO) - the PO represents the client. It is really the personal who is the liaison between the client and the project team. As an example, in building banking applications there could be many layers to clients. A Foreign Exchange (FX) division of the bank has been asked by their corporate clients to build an FX API so their systems could query current FX rates for several currencies. The FX banking division approaches the bank's IT group supporting their division for developing the API.

   So here, the need is really to satisfy the bank's corporate clients. An internal BA in the bank's FX division, typically acts as the client representative in getting the API built; they are effectively the PO since they understand what the client wants and represents the needs (requirements) to the Agile team. The PO will drive the requirements with the Agile team. The PO is responsible for creating the user stories and prioritizing the stories into "mandatory" and "nice to have" functionality.

   In cases of budget shortfalls or project overruns, it's necessary to know which stories can be chopped off the backlog.

2. Scrum Master (SM) - the SM's role is to support the application team. They remove any obstacles the team encounters, ensures that agile practices are followed, facilitates sprint planning meetings with the PO, facilitates daily stand-ups and ensures the team has the resources required to execute the project. The SM monitors the budget to execute the user stories and tracks the project progress.

   At one bank, the Project Manager (PM) role was retained to manage the financials and status reporting for senior management and the PO.

   The SM roles I've seen in banking have been filled by manager and senior team managers. Often, these folks have been thrust into the SM role with a bit of training. Often, an Agile coach is brought in to help the team with the bank's Agile methodology, which is most often a variation of SCRUM. The problem I've seen with contract Agile coaches is that they don't understand the bank's own approach to Agile. Their advice may counter management's expectations of how the project is managed. For example, the bank's project documentation, weekly executive status reporting, budget and project completion forecasts may seem foreign concepts to an outside consultant.

3. Application Team - the application team is a small self-managed cross-functional team. This means the team manages the work amongst themselves; who is best qualified to undertake a development task should step up and take stories/tasks from the backlog.

   If that same developer has Usability eXperience (UX) skills then that developer may also volunteer to work on UX design tasks. There are exceptions, often teams don't have the expertise on certain technologies and may not need a full-time resource on the team with a unique skill set. For example, if a database needs to be instantiated and a DBA is needed, this does not have to be a dedicated role on the team (assuming the team can manage the SQL development thereafter). Likewise if an accessibility test is needed on the user interface, likely the team will reach out to another team with expertise on JAWS testing. Expert niche skilled resources are often lacking on Agile teams but can be brought in from other areas of the bank. One project I worked on at GE, we had to bring in a contractor to teach and mentor the team on SUN Java spider classes as had never heard of these.

   Team Size

   Typical team sizes have been 7 (including PO and SM) and no more than 12 resources. The reason for small teams is to make communication more effective. Smaller teams have fewer resources to exchange communications with and certainly easier to keep everyone informed. Based on the number of people in a project, the number of communication channels (c) is defined as c=n(n-1)/2. A team of 7 will have 21 communication channels. If the team is doubled to 14, the communication channels increase to 91; a 4.3 fold increase.

   At Amazon, when the team size grows beyond a certain number the team is divided into two teams (pods). This keeps application development teams small for improved communication and efficiency.

   This practice is known as Scrum of Scrums. That is on a large development project, beyond 7 people as an example, each team (also referred to as a pod in some banks) will have multiple small pods. The goal is to facilitate coordination of work amongst multiple smaller teams for delivery. The Scrum of Scrum collection of pods acts as a virtual single Agile project team.

   Co-Location

   Co-location of teams is associated with improved communication and collaboration amongst team members. Agility principles dictate that staff proximity to each other reduces the communication distance. Some organizations have made it mandatory that Agile teams sit close to each other to improve communication and rapid decision making. This was the case at a bank I recently worked at and with technology company, Northern Telecom (NT), that my team visited while at GE; we were looking to understand how NT organized their development teams with hopes of emulating the approach.

   At Northern Telecom, when a project was started, the project team was moved to a common working area. They stayed there for the duration of the project. Project teams were formed and disbanded as per the nature of the project and the skills required. Every time a new project was created the teams were moved to a common area. This was so frequent that NT in Brampton, Ontario had a full-time Herman Miller (furniture) team member to manage furniture moves.

   In reality, close proximity can be achieved with modern telecommunications methods such as video conferencing, team sites and text messaging. Team members can be virtually present. The co-location goal is to eliminate waste associated with waiting for decisions or getting advice. If my team is co-located and I need advice on a task, I can go to the developer a cubicle away and tap him for an answer. I can't do that if he/she is not physically close-by and is not responding to e-mail, texts or telephone. I would have to wait until they get back, in the meantime, my work could be stalled until they come back. Waiting for people to respond by telephone is the biggest time waster. I've found this to be the biggest source of frustration in getting work done quickly, is waiting for a response from a team member who is not available.

   My recommendation would be for teams, whereby some members are remote, agreements need to be established at project start time on an Operational Level Agreement (OLA) defining the amount of time they will get back to a team member when asked to respond. Whenever we wait for others to respond back, we are no longer living the Agile principles of removing waste.

   At bank I worked at, co-location introduced some challenges. For one, often people in various geographical locations were part of an Agile team. The VP mandated that all Agile project teams be co-located. This meant that people in other locations would have to commute daily downtown to a common location to be on the team. Naturally, this was frustrating for those people who now had a longer daily commute away from their home base.

   Another challenge is access to the PO. An Agile team requires that the PO be part of the team, to be there on a daily basis to provide guidance to the team. In banking this role is often assigned by the business area to a BA familiar with the business and the needs. However, more often than not, they turn to their management for help with decisions. They are not the ultimate end-client. As well, they may also be allocated to other projects and don't have the dedicated time to one Agile team. In traditional organizations, these limitations contribute to decision making and communication delays (waste) but are accepted as compromises in adopting Agile practices into the organizational culture.

AGILE PHASES

Agile is known for for frequent review and remediate cycles based on team and client feedback. In the example above, the gray boxes represent the two week sprint cycles. After each sprint there is a team retrospect followed by planning for the next sprint. The blue arrows from the retrospective to the sprint planning, indicates this is a continuous improvement process the team uses to review how the work went during that sprint and make any changes needed for them to work better the following sprint.

Likewise, the yellow arrows after sprint 3 and 5 indicate that client feedback is used for planning sprints 4 and 6. These feedback loops ensures the product the client receives meets their intended needs by the time it's completed.

There are three MVP phases. At sprint 3 and 5 the team releases the product to an area that the client can review the work. This is normally a working prototype whose purpose is to get the client to see a working product and elicit feedback. At sprint 6, assuming the product is ready, it is deployed and a final product demo is given to the client before the product. If the client is satisfied with the product, provides sign-off, it then moves from development/staging to production where the client's users can access the application. The project can then be closed; artifacts saved, client satisfaction results and capture lessons learned for further application enhancements or the next project.

The section below suggests the activities within the various Agile development phases.

1. Initiation

   Summary
   Setup project support applications, resources, co-location and create the product backlog for future sprints by adding details, priority and estimates to the user stories.
   Activities
   Determine team capacity to complete the project. Ensure development environments are available for the project (Development, QA, Staging, Production). Setup Jira and project Confluence sites, or alternate tools as needed. Team co-location, if needed. Establish quantitative progress and quality metrics. Determine if an Agile coach is required. Determine if project has dependencies on other projects or Scrum teams. Create and prioritize the backlog. Estimate the effort associated with each backlog item (Fibanoci, shirt size, etc.). Split or group backlog user stories. Refine backlog items with input from SMEs. Estimate effort,acceptance criteria, dependencies and assumptions. Break work items into tasks. Client's definition of done criteria is understood by the Agile team.
   Inputs	Outputs
   PO's requirements. Funding. Agile Team Resources.	Prioritized product backlog. Team members have shared understanding of the project. Team co-location area available. Spend monitoring and burn-down tracking readied. Team ready to start sprint 1.

2. Next Sprint Planning

   Summary
   Stakeholders and the Agile team plan the work to be completed in the next sprint.
   Activities
   Define the objectives for the sprint, e.g. will there be a client review? Identify PBIs in the backlog to be completed for the sprint. Breakdown the user stories into tasks (if not done yet). Remediate any bugs from the previous sprints. Assess team capacity for completing the work items (calculate velocity). Assess if external skills are needed to support the team. Re-plan subsequent sprints. If productivity is below estimated velocity the team will need more time to complete the user stories. Calculate % work completed and % budget spent. Re-forecast project end date.
   Inputs	Outputs
   Product backlog. Financials and progress status from previous sprint.	Objectives for the sprint. User stories the team will complete within the sprint. Refined product backlog. Updated financial and progress measures. Project status report for management (optional).

3. Retrospective

   Summary
   The team evaluates the processes used, team communication, locality issues, team dynamics, resource capacity and any blockers to continuously improve the team and the work it produces.
   Activities
   Review if actions from previous sprint retrospectives were implemented. Assess if actions taken have resulted in improvements. Review what has gone well and what needs improvement for next sprint.
   Inputs	Outputs
   Action plan from last retrospective. Agile team feedback	Team learnings; good and bad. Improvement plan for next sprint.

4. Client Demo

   Summary
   Demonstrate completed work to the client. This would be the case where an MVP is demonstrated.
   Activities
   Demonstrate completed work to date. Capture client feedback and incorporate any add/changes into the product backlog and sprint planning.
   Inputs	Outputs
   Completed Work (MVP). Stakeholder Feedback.	Team learning from stakeholder feedback. Changes to work items as required by client, if needed. New user stories to incorporate into backlog, as needed.

5. Daily Stand-Up

   Summary
   Review team progress against plan, identify and escalate any blockers.
   Activities
   Everyone update the status of their work items in progress. What will the team (individually) work on next? Are there any impediments to getting the work done?
   Inputs	Outputs
   Product backlog. Team tasks and assigned resources. Team feedback.	Updated team tasks. Blockers are identified for subsequent actioning by SM.

SUSTAINING AGILE PRACTICES IN REGULATED ENVIRONMENTS

My experience has been that most banks have adopted Agile principles. In my research, I noted that although Agile principles have been adopted, each bank has adapted to fit it to their own needs; no one bank is following a purist Agile principled approach, but tailored to their ways of working and organizational structures.

Agile is also difficult to adopt into an organization that has followed plan-based software development methods. It's not only a cultural change, it's also about adopting new skills and a new way of working that will challenge middle management's power.

One banking VP mentioned that the only way an Agile project succeeded in their organization was by breaking all the PMO rules.

Another VP at a different bank, noted that adopting Agile is like turning on a wall dimmer. It's not an instant change as in turning on a switch, but rather a gradual organizational change that takes much time.

There are some banks that have attempted adopting Agile and failed. It is a massive change. Those that have been successful have approached it in an incremental fashion, almost in an experimental approach, much the same way you would run an Agile project; frequent feedback on progress, great communications and senior executive champions to remove barriers.

One of the best Agile transformations I observed was at Royal Bank of Canada (RBC). What made it successful was a well orchestrated communication plan and roll-out strategy. Training was available to everyone who was interested in RBC's Agile methodology. Upon completion of training there was a certification exam. If you passed, you would receive an "RBC Agile Practitioner" certificate.

Senior managers and project managers were encouraged to evaluate if some of their projects could be run under the Agile methodology. This was an incremental adoption approach as opposed to forcing projects to follow Agile. There were frequent seminars highlighting the benefits of Agile, how Agile was being used at RBC. These seminars often featured speakers using Agile on their projects and the benefits realized. There were even some speakers from the broader industry brought in to speak about Agile benefits.

A professional monthly newsletter gave much credence to the adoption effort. There was support from the executives for introducing Agile methods, but above all there was a dedicated, enthusiastic and knowledgeable Agile champion that drove the adoption. Attitude and perseverance make a difference beyond the executive sponsorship.

I don't believe this initial Agile adoption was part of the PMO, perhaps it was better that way so it would not be overly burdened by process.

SUMMARY

This is by no means a complete description of Agile Scrum practices. There are many books and Internet resources that describe fully the way to approach Agile projects and Agile transformations in companies. Much has been written about Agile transformations by leading consultancy firms. Scott Ambler and Mark Lines book (An Executive's Guide to Disciplined Agile) has a good description on Agile transformation approaches, although, I'm not a fan of the Disciplined Agile approach in the book. My own academic research focused on Agile transformation success in Canadian banking and there was an overwhelming amount of literature to review.

PMI's book (Agile Practice Guide) has a chapter on organizational considerations for project agility; what needs to be considered at the people level when implementing Agile transformations.

The information available from public sources on Agile is overwhelming. A book could be written about all the variations; LeSS, Scrum, SAFe, DAD, etc. How banks have implemented their own Agile transformations is less published. In most cases they have been aided by top consulting firms such as Deloitte or McKinsey. Each bank holds onto that information and processes as if it was a military secret. My own research participants remained anonymous because they were afraid of reprisals from sharing their bank's experiences with Agile transformation successes and failures. There were many failures, for many different reasons. Implementing Agile in banking is in itself an incremental slow process tailored to fit the culture and established ways of working in a regulated industry.

The consultancies are a great source of information on Agile (or digital transformations, as some firms refer to this). They will give you the seeds of information on how to successfully achieve these transformations, not the entire roadmap, in the hope they'll be chosen to lead your transformation. Their conundrum: how do you show your transformation competence without giving away the processes that drive their success.

I hope this section provided a high level explanation of Agile principles and adoption challenges.

APPENDIX A

Kelly Johnson's 14 Rules and Practices at Lockheed's Skunk Works:

1. The Skunk Works manager must be delegated practically complete control of his program in all aspects. He should report to a division president or higher.
2. Strong but small project offices must be provided both by the military and industry.
3. The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).
4. A very simple drawing and drawing release system with great flexibility for making changes must be provided.
5. There must be a minimum number of reports required, but important work must be recorded thoroughly.
6. There must be a monthly cost review covering not only what has been spent and committed but also projected costs to the conclusion of the program.
7. The contractor must be delegated and must assume more than normal responsibility to get good vendor bids for subcontract on the project. Commercial bid procedures are very often better than military ones.
8. The inspection system as currently used by the Skunk Works, which has been approved by both the Air Force and Navy, meets the intent of existing military requirements and should be used on new projects. Push more basic inspection responsibility back to subcontractors and vendors. Don’t duplicate so much inspection.
9. The contractor must be delegated the authority to test his final product in flight. He can and must test it in the initial stages. If he doesn’t, he rapidly loses his competency to design other vehicles.
10. The specifications applying to the hardware must be agreed to well in advance of contracting. The Skunk Works practice of having a specification section stating clearly which important military specification items will not knowingly be complied with and reasons therefore is highly recommended.
11. Funding a program must be timely so that the contractor doesn’t have to keep running to the bank to support government projects.
12. There must be mutual trust between the military project organization and the contractor the very close cooperation and liaison on a day-to-day basis. This cuts down misunderstanding and correspondence to an absolute minimum.
13. Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures.
14. Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay, not based on the number of personnel supervised.

Reference Links

Below is a list of links relating to the subjects discussed above.

• *click here* - User stories with examples and a template, by Max Rehkopf (Atlassian).
• *click here* - Spiral Model - Software Engineering..
• *click here* - Scrum.org list of resources; learning content.
• *click here* - Scrumalliance.org, Scrum Alliance Website.
• *click here* - Agile Alliance website.
• *click here* - SAFe, Scaled Agile Framework (Dean Leffingwell).
• *click here* - Berteig Agile Training and Consulting.
• *click here* - Kelly’s 14 Project Management Rules: Everything You Need To Know.

Compiled on 04-27-2024 22:55:50