20161121 goat2016 - ken mcmillan - bas lessons learned - all notes

These notes deliberately left free of chocolate

1

Ken McMillan | Gatineau Ottawa Agile Tour 2016 | 2016Nov21

In this culture, people say yes way too often

2



3


Tender notices always in top 10

Contract history always in top 25

4



5



6



7



8


1. Make many mistakes often – not a surprising approach to an Agilist

2. Say no often (remember Steve Jobs?)

9


--------------------

Do not allow a long-horizon project management methodology to override practical, business-driven outcomes with early, demonstrable results. If you assume you know more about the problem than you really do, you risk committing to a solution path too early.

Risks to manage (warning signs):Expending a great deal of energy, time, and money prior to seeing any tangible stakeholder-facing deliverables.Committing to highly described (function and cost) outcomes in the distant future (longer than 6 months away).Suggesting that all risks can be identified and documented prior to project commencement.Stakeholder resistance to engagement due to the long or unknown amount of time prior to a tangible result.Stakeholder resistance to engagement due to the overly-early descriptive solution provided with little input on their part.

10


What we did:We engaged key stakeholders early and often. We provided tangible, usable examples to guide discussion and build confidence.We talked to key stakeholders to discover and document their perceptions of their needs, problems, and desired outcomes.We did not necessarily talk about requirements as stakeholders are likely uncertain about end-state solutions.We assumed seeing is believing and used working proof-of-concept demonstrations and pilots to help stakeholders envision outcomes and discover needs.We learned to be more tolerant of our own mistakes and used post mortems to gain insight and discover what the problems really were.We sought incremental funding and attempted to have useful deliverables such that early termination would still have a positive outcome.

What we recommend:Adopt an open and flexible approach incorporating problem discovery in the early stages of a project. Consider early engagement of stakeholders via demonstrable and tangible outcomes.Be prepared to control the stakeholder desire to jump ahead during problem discovery and move towards solutions.Be prepared to control your own tendency to assume you understand the problem or business environment better than the stakeholder.

Anecdotal example(s):Early in the program we developed a simple concept to combine a number of Government of Canada legacy services into one single web site. We believed this new site should focus on helping business people, allowing them to easily find information to help them do business with the Government of Canada. This user-task oriented approach was in contrast with the incumbent practice of telling stakeholders what we considered important in Government of Canada terms. It was client-driven. Intuitively, we believed that changing both the what (a single website) and the how (a new way to organize and present the information) would be a difficult transition, particularly for our internal stakeholders. Our first two deliverables were a User Requirements document and a simple proof-of-concept. The User Requirements document allowed internal stakeholders to express their opinions, needs, pain points, and so on, such that they could be synthesised into a user-driven story. This was used as input for the second deliverable, a working proof-of-concept.The working proof-of-concept addressed feasibility questions around modern tools for web site management and allowed the small project team to re-engage internal stakeholders. Stakeholders were able to see and click on a working demo, building


10

confidence and providing a stepping stone for further problem discovery by all parties.


10

Stakeholder expectations must be managed throughout the project.Successful projects deliver a working service to their stakeholders. They must do what is expected and stakeholders must be willing to use them. Unsuccessful projects miss one or both of these conditions.

Risks to manage (warning signs):Stakeholders discuss business requirements too early. Business requirement discussions lead to an overly specific solution-orientation that is not achievable early in a project.Stakeholders express concern (often informally) they are committing to something they don’t fully understand in terms of the impact on their business, budget, or both.Stakeholders focus more and more on process (how) rather than outcomes (what) as their fears increase regarding project success.

What we did:As the project evolved we expanded the notion of user engagement to a broader community engagement.We designated a small, skilled team of roadmap managers that met often. They were ambassadors for the project amongst diverse communities including management,

11


end users, program managers, standards stewards, and operational groups.We presented potential changes to stakeholders and listened to their feedback on how those could affect their work.The roadmap managers interacted with focused design and development teams to incorporate environmental scans, best practices, standards, and appropriate technology into priorities.The roadmap managers were accountable for describing and prioritizing business needs.

What we recommend:Build a small team of experienced, multidisciplinary decision-makers who listen. Listening and distilling are key skills. Adjust the degree of stakeholder involvement to suit the degree of change to the stakeholder. Engage early with potentially challenging communities that may be threatened by or adverse to change.Do not overlook smaller communities that may have interesting insights or needs.

Anecdotal example(s):Government has a number of obligations that it must meet to deliver services that involve data. A key obligation is to put in place appropriate business and technological safeguards to protect Canadian’s data. We were mandated to follow the Certification and Accreditation (C&A) process to demonstrate our solutions to these responsibilities.We engaged an Information Technology Security team early in this process. Together, we devised a work plan that ensured the new service met the requirements of the C&A process in a manner that would not delay the delivery of the service or put undue pressure on the security team due to late engagement in the overall project.


11

Traditional project management systems (often modelled on construction projects) tend to deliver usable components only at the very end of a long development cycle. To have meaningful engagement, stakeholders must be able to judge tangible results on a regular basis throughout development.

Demos to user and program stakeholdersDuring the early discovery and pre-launch phases of the project, we convened a working group of program officers from within the Branch. We demonstrated our progress and asked them for feedback using demos that provided a tangible reference point for engagement. After the official launch, our engagement focus shifted to a broader external audience. We were also able to leverage analytical data from website usage logs.

Risks to manage (warning signs):Milestones are marked only by status reports rather than demos, proofs-of-concept, pilots, or high-quality betas.Waiting until demos are feature-complete before soliciting feedback.Failing to prune or defer changes that do not align with a clear guiding principle or

12


strategic intent.Failing to consider non-strategic changes that stakeholders are heavily invested in due to tactical realities.

What we did:We had regular stakeholder engagement meetings with demos of working code. These purposely only showed a limited number of features, but they were real, and they allowed the stakeholders to judge what constituted real improvement in their eyes.Our roadmap managers used the feedback from these demos to prune or defer changes that failed to align with a strategic intent or that had significant pushback from the stakeholders. Our roadmap managers evaluated the feedback with particular attention paid to non-strategic items that were obvious hot-buttons for the stakeholders.We attempted to show in subsequent demos to the same stakeholders how their input had influenced the project.

What we recommend:Demo early, demo often, and respond to feedback.Have the roadmap managers ask themselves such questions as: “Does this move us toward open, web-based apps and services, or does it keep us in the closed, proprietary world?”, “Would I be proud to share this as a best practice with my colleagues, or am I glad it’s happening where no one can see it?”, “Does this change contribute to the overall web and service development and operations community?”Be aware that taking technical shortcuts (using hacks) to meet a demo schedule can lead to compromises that may need to be fixed at a later date. The roadmap managers need to weigh the potential cost of the fix against the benefit of the confidence engendered by a smoother demo.Strategic intent may not be explicit or well defined at the beginning of a project. There may even be a tactical advantage to deferring an explicit definition. Make sure the roadmap managers are prepared to discover and refine the strategic intent themselves throughout the development process.


12

Do not attempt to deliver all of the solutions to all of the problems at the end of an extended development effort. Rather, provide usable solutions in an incremental fashion, such that stopping at any point still leaves you with measurable gains and working systems.

Risks to manage (warning signs):A long development phase with no intermediate partial solutions. (Note that a partial solution must be a working and stable deliverable - not just a demo or pilot.) Large expenditures with no viable deliverables.Expectations from stakeholders for a complete solution long before all problems have been identified.Reluctance of stakeholders to provide signoffs on partial deliverables, even when the deliverables provide viable solutions.

What we did:We managed the project in stages, where each stage comprised a viable business deliverable. Although subsequent stages often relied on previous stages, each stage worked as a deliverable unit, with no subsequent dependencies.We learned not to ask the underlying technology to do something it was not designed

13


to do. (This lesson was sometimes learned by negative example.) Because development was staged, such a lesson learned early on would benefit many subsequent stages.We learned not to attempt to fix a broken business process with a software solution. Rather, the roadmap managers were able to engage with the stakeholders to fix the processes at the business level.By delivering often, we were able to keep expenditures in line with benefits and could demonstrate to management the value being provided. This also made management much more confident about signing off on future stages.The roadmap managers were charged with managing stakeholder expectations. Although many stakeholders initially expected a complete solution, they quickly realized the benefit of having usable incremental improvements.

What we recommend:Implement early and often. It is much easier to redirect efforts when feedback is received in a timely manner. Do not allow analysis cycles to become too long. The validation provided by a working component outweighs the value of having a perfect design.Know your tools. All software has best-case design patterns. Use them to your advantage.Allow the roadmap managers to balance technical capabilities against business requirements. Do not allow business requirements to dictate technical solutions. Similarly, do not let technical concerns affect business requirements.

Anecdotal example(s):During the first two years of operation of Buyandsell.gc.ca, we delivered a variety of incremental improvements based on business drivers and stakeholder feedback. Each of these delivered a new public-facing service and allowed the team to introduce or refine processes and services. This continuous improvement allowed the delivery team to learn by doing. Each new capability built confidence within the team and demonstrated a commitment to listen and improve to the stakeholders. This eventually led to the launch of buyandsell.gc.ca/tenders which became the official Government Electronic Tenders Service in June 2013.

May 2010: Pre-Launch (gain early insight, exercise new web publishing process) Sept 2010: Public Launch (new high value content, namely: Canadian Innovation Commercialization Program (now the Build in Canada Innovation Program), For Government, For Business, Find Goods and Services)Jan 2011: News Items capability (ability to publish short timely notices)June 2011: Service Selector Tool for Professional Services Sept 2011: Standing Offer/Supply Arrangement (data available as PWGSC’s first Open Data set)


13

Oct 2011: Event Calendar (first ‘self publishing’ service)Jan 2012: ‘Buy Green’ Micro-site March 2012: Supply Manual & Standard Acquisitions Clauses and ConditionsJune 2013: Government Electronic Tendering Services


13

Example:

The Accounts module is an excellent example of a self-contained, well-defined component that interoperates with all of the other parts of the system through a clean, modern interface. It has no knowledge of the business processes inside the other services – it provides platform-level security enforcement, service and user on-boarding, authentication and role-based authorization. In practical terms, this means that even if the business rules for another module (such as the Tender Management App) need to change, it does not affect Accounts. Similarly, Tender Management does not have to know anything about Account management processes – it simply consumes the service.

--------------------

Take advantage of modular delivery to allow separation of concerns. By managing each module separately, you can use tools, products, and business knowledge specific to the module to its greatest advantage.

Risks to manage (warning signs):

14


Tightly coupled, complex communication between modules (keep it simple).Lock-step dependencies between modules, such that if development on one stops, development on all stop.Large teams trying to manage too many components.

What we did:We divided the project into self-contained modules such as Accounts, Tender Management, Publishing, and the Supply Manual. We ensured that each module had a clean, outward-facing interface that let other modules interoperate with it without being dependent on any internal details.We developed shared standards for all modules to simplify and standardize their development and interoperation.

What we recommend:Be sure to dedicate sufficient resources to standards development and enforcement.Strive to keep the system architecture loosely coupled. Swap team members from time-to-time to encourage cross-fertilization.


14

Integration, whether between modules or different systems, requires too much knowledge of the other component’s internals. Interoperation through a clean external interface allows modules to remain independent and autonomous.

Risks to manage (warning signs):Attempting to integrate with existing systems and other modules.Tightly coupled, complex communication between modules (keep it simple).Asking external systems or modules to change their interface.Pushback from external system stewards concerned about changes they might have to make.

What we did:We began interoperating with external systems at an early stage. Testing for interoperability was not left for the end of the project . It was executed incrementally.Accommodated external systems by reverse engineering their data structures.Built data adaptors and connectors to allow our modules to consume data from and supply data to external (legacy) systems.Built clean XML-based interfaces between modules.Built standards-based (Atom) data publishing streams.

15


What we recommend:Build simple, standards-based interfaces for system modules.Explain to external data suppliers and consumers that the interface requires no change to their processes or internals.Encourage owners of external systems to start testing for interoperability at an early stage.


15

Open technologies provide many significant advantages: they have no external platform dependencies, they are easy to procure, they are broadly supported by large user communities, and they do not create large sunk costs.

Risks to manage (warning signs):Attempting to choose a product with perfect fit. Requirements may change (rendering the fit imperfect) and a good fit is almost always sufficient.Using the very latest branch or version of a dynamic product. The so-called long term support (LTS) versions are a superior choice for all but the most demanding applications that require the latest features.Procurement professionals or managers unfamiliar (and uncomfortable) with the open source model and its benefits.

What we did:We actively promoted the benefits of open-source solutions to our stakeholders, particularly the absence of vendor lock in and the broad base of usage and support.We did feature versus benefit analysis even for support system software (ticketing & bug tracking system and documentation wiki). We reviewed the level of community support and adoption of the products, using this

16


data as a proxy for the product utility and stability.We sought guidance and advice from other users (both public and private) who had used the products to deliver new services and systems.

What we recommend:While it is important to match product features against your anticipated requirements, remember and use the 80/20 rule. If the product matches 80% of your known requirements, that is probably good enough.Use part of the effort that would have been expended on a costly RFP exercise to analyze several open technology options.Engage with the product’s user community. They can provide a wealth of information and a potential pool of resources.

Anecdotal example(s):The entire Buyandsell.gc.ca platform leverages widely adopted open-source software and open standards. Open-source software is software that is created by a collective of individuals (often backed by a major foundation or organization) and shared under one of a number of common open licences. Open-source software makes evident the fact that a number of problems are so well understood that it is possible to collectively create software that codify the solutions. Previously challenging problems have been elegantly solved with open-source software: operating systems, web servers, networking software, content management systems, and many others. Even companies that provide closed-source software recognize the value in open-source solutions. For example, Apple now gives away its main computer operating systems Mac OS and iOS.Open-source technologies are particularly desirable in Government. Open-source greatly simplifies the procurement process. Procurements are service-based, which is less challenging and time consuming than proprietary software procurement. Open-source software conforms to industry standards which prevent technology lock-in. Enhancements and revitalization of open-source solutions is not tied to a single vendor or technology. An active community of independent, collaborative developers keeps systems up to date.


16

Anecdotal example(s):Buyandsell.gc.ca/tenders was a new service that incorporated functionality that had been delivered by a long-standing third-party service to the Government of Canada. This third-party service was well understood by the key stakeholders (business, Government procurement officers). It had been adapted in small ways over many years to meet special quirks (usually undocumented) in the procurement process. As we explored the requirements and began implementation of Buyandsell.gc.ca/tenders we failed to refer back to the existing provider’s implementation. This was likely a result of a bias that the new solution would be more effective and efficient. While the new service did achieve its goals, there were a number of areas where we experienced issues post-launch that could have been avoided by paying more attention to what was already implemented and widely adopted.

--------------------

When replacing or improving a service, there is a tendency to start with a clean slate without regard to the utility or functionality of existing systems and procedures.

17


Risks to manage (warning signs):Assuming that since part of a process is broken, all of it is broken.Believing that stakeholders can accurately describe how they work, when, in fact, they may neglect to mention many implicit assumptions that they use.Let bad presentation blind you to the underlying utility of the information being presented.Focussing on one aspect of a process at the expense of another equally (or more) important one.

What we did (wrong):We focussed on the front end of the procurement process (gathering the data from procurement professionals) and paid less attention to the back end (presenting the data to the public). We spent a lot of time engaged with demos to government procurement professionals and relatively little with public data consumers.We did not promote the benefits of the new cost-free (and registration free) tenders system as aggressively as we should have.

What we recommend:Engage with stakeholders at both ends of your processes with equal vigour.Don’t ignore existing things that work well – even if you plan to replace them.Be humble. Your improvements may actually make things worse.


17

Avoid two common pitfalls: 1) Using your development environment as a demo environment; 2) Taking shortcuts to make demos work. Make the demos even more useful by using the collected feedback for the next demos.

Risks to manage (warning signs):No dedicated demo environment.Demo systems that appear to function, but are actually little more than scripted animations.Attempting to mask any and all problems that a demo system may experience.

What we did:We used three environments: 1) development; 2) staging/demo; 3) production. This allowed development to continue even while demos were underway (where a series of demos might take a few weeks) and it insulated the demos from development changes.We showed demos with warts and all which actually increased the confidence of the audience in the validity of the demo and allowed us to show the direction improvements were taking even if not complete.

18


What we recommend:When you use demos to solicit feedback, make them real – otherwise the feedback may be invalid and you may miss an opportunity to make small adjustments with a big payback later in the project.Do not avoid giving a demo with known flaws. Stakeholders will still see visible improvements and have more realistic expectations of what can be done in a given time frame.

Anecdotal example(s):Throughout the project we used working demonstrations to both engage stakeholders as well as to facilitate communication and understanding amongst the small project teams. Overall, this was an effective approach and it provided visible, date-driven milestones to maintain project delivery momentum.However, from time-to-time, particularly if the scheduled demonstration was to a senior executive stakeholder group, the team took shortcuts in order to meet the deadline. Typically, the shortcut was a faster, easier, but ultimately less sustainable solution to the problem at hand. This often led to later implementation issues and required re-work. In the industry this is often referred to as technical debt. At some time the project will need to repay that debt in time and resources.Earlier in the project, when computing resources were scarce, the demo effort could block or slow down efforts for the main stream of the project deliverables.Demonstrations are vital to solicit correctional feedback, but they do need to have appropriate resources and planning to be used most effectively.


18

At some point, discovery and development will give way to operations. Developers should strive to make the transition as frictionless as possible.

Risks to manage (warning signs):Imposing operational disciplines and change management too early in the discovery and development phase where it will hinder rapid progress. Attempting to retrofit operational reporting and management tools into completed components.Ignoring operational realities during development. Operations should be a vital, contributing stakeholder.Having QA focus solely on functional specs. QA should also consider operational concerns and constraints.

What we did:As part of the C&A process (see lesson 2) we created all of the required standard operating procedures (SOPs) for an operational environment.We used the SOPs to inform the efforts of developers, particularly in the areas of audit logging, system security, and change management.At a very early stage, we began using three environments – one of which was

19


production. This gave us early feedback on operational requirements.

What we recommend:Engage the operations community at the earliest stages of the project.Recognize and acknowledge the difference between development and operations culture.Ensure that operations have an opportunity to review early demos where they may be able to suggest improvements or spot problems from an operational perspective.

Anecdotal example(s):One of the benefits of delivering smaller, modular components is that they can be delivered into an operational environment with little or no disruption.

When first released in May 2010 as a pre-launch service, Buyandsell.gc.ca was a fairly straightforward content management web solution. By September 2010, the team had gained four months of operational experience in a relatively low pressure environment.

After the official launch in Sept 2010, features were incrementally added - allowing gradual and increasing opportunities to gain operational experience. By the time the service was released as the full Government Electronic Tendering Services, the team had over three years of operational experience to help manage the transition to this new and very visible environment. As a measure of complexity, Buyandsell.gc.ca went from a two virtual machines at the start to over thirty for the version released on June 1st, 2013.One significant challenge during the entire project was having limited financial resources which meant we often did not have the optimal number of people to do the work. Many individuals had to switch from design and build mode to operations mode - sometimes more than once a day. This affected the individual’s efficiency, and as a result, the delivery schedules.


19


20



21



22



23


20161121 goat2016 - ken mcmillan - bas lessons learned - all notes

Government & Nonprofit