The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.

What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.

Why is agile data modeling a good idea? To paraphrase Ron Jeffries, one of Extreme Programming’s founders, the best way to implement a DW/BI system is to implement less of it. The best way to have fewer defects in your DW/BI system is to have a smaller/simpler one. The problem with comprehensive up-front modeling is that you must design for all contingent requirements, both known and speculated. This inevitably results in an overdesigned model that costs more to implement, is costlier to maintain, is more likely to contain defects, and is more difficult to understand.

Agile data models are as simple as possible while being sufficiently detailed, accurate, and consistent. They also fulfill a well-understood purpose and provide positive value (i.e., their benefit outweighs the cost of keeping them updated).

How to Safely Evolve Data Models

These concerns about evolutionary modeling resulting unnecessary rework, unintended side effects, and design degradation are legitimate. Additionally, the prospect of making a data model change to a high-volume data warehouse in production can be scary. Agile data modeling calls for a new set of practices that enable the safe evolution of models, even those in production. I’ll summarize those practices here. Consider this list a brief introduction; each deserves a deeper study to gain proficiency.

Data Model Patterns: Data models evolve toward excellence when we take advantage of tried and proven designs. Design patterns enable us to benefit from mature solutions that have previously been developed. Effective application of patterns relies on familiarity and awareness of patterns catalogs, and the ability to use them appropriately and sparingly.

Michael Blaha’s Patterns of Data Modeling is the most recent catalog of patterns, but David Hay first started cataloging data model patterns in 1996 with his Data Model Patterns: Conventions of Thought, and followed in 2006 with Data Model Patterns: A Metadata Map. An extension of data modeling patterns is the adaptive data model (ADM), a generalized data model designed to accommodate multiple domains. The ADM has been successfully used in data warehouse design and I write about it in detail in the Cutter executive report, The Message Driven Warehouse.

Technical Debt Management: Data models evolve toward excellence when changes are easy to make due to low technical debt. Technical debt is common in data warehousing. It is the entropy that occurs in any system over time due to development shortcuts, suboptimal design choices, maintenance activities, and so on. Like financial debt, a little technical debt is acceptable as long as we monitor it and pay it back quickly. When technical debt accrues unabated, the cost of change becomes unacceptably high. Agile data modelers continuously identify, prioritize, and monitor technical debt in the data model, seeking to eliminate it to assist with fast response to new requirements.

Database Test Automation: Data models evolve toward excellence when we have continuous confidence that our ideas are working. Agile BI practitioners work in short iterations delivering business value every few weeks. We need confirmation that what we build in later iterations doesn’t break what we built in early ones. The only practical way to accomplish this is with an automated test suite. Automated database tests validate data structures, data content and quality, schema constraints and integrity, data derivations, and so on. We can run automated tests quickly and simply at any time to confirm that everything still works. Tests are added as data model changes are made, so the test suite grows alongside the model. I devote an entire chapter to this topic in my book, Agile Analytics.

Database Refactoring: Data models evolve toward excellence when we can safely make changes to the design, even if it is in production. Database refactoring is a technical discipline that enables the safe evolution of data models without breaking previously working features and components. In their book Refactoring Databases, Scott Ambler and Pramod Sadalage explain database refactoring as “… a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.” Refactoring combines automated regression testing with a change transition period to ensure that changes haven’t broken anything. The change transition period is a window of time during which the model revision lives alongside the former version to ensure that nothing breaks. Refactoring combined with test automation are the central disciplines for effectively evolving data models.

Recognizing Data Model Smells: Data models evolve toward excellence when we recognize where they need improvement. Experienced data modelers develop a nose for elegant designs … and stinky ones. Discovering smells in the model is an essential precursor to improving it. Smells may include smart keys, multipurpose columns and tables, data redundancy, very large tables, and so on. By learning to pay attention to these smells, you can focus your attention on possible problem areas and consider them as candidates for technical debt management.

Change Deployment: Data models evolve toward excellence when we can quickly deploy changes at any time without fear. Agile BI practitioners develop data model changes using techniques that safeguard production deployment. All data model changes are scripted and those scripts are kept under version control. Database schemas are versioned and scripts are developed to roll forward to the next version, and roll back to the previous version in case things don’t work as expected. Data migration scripts ensure no loss or corruption of production data. Everything is automated and tested carefully in preproduction. Automated deployment, automated testing, and database refactoring disciplines support frequent, fast and fearless deployments.

Take Small Steps: Data models evolve toward excellence as a series of small, easily understood changes. It’s easier to undo a sequence of little changes than one big, complicated one. A side benefit of agile BI is that short iterations force us to plan in small steps. Agile data modelers quickly learn to change only what is needed to support the BI features currently in development.

The Final Word

All of these techniques combined form a strong safety net for evolutionary data modeling. Moreover, these techniques aren’t exclusive to agile BI, they should be considered as modern data warehousing practices that should be among the skills of every data modeler and data warehouse practitioner — agile or otherwise.

Recommended Reading

Ambler, S. W. (2002). Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process. New York: John Wiley & Sons, Inc.

Ambler, S. W., & Sadalage, P. J. (2006). Refactoring Databases: Evolutionary Database Design. Boston: Addison Wesley.

Blaha, M. (2010). Patterns of Data Modeling. Boca Raton: CRC Press.

Collier, K. (2011). Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing. Boston: Addison-Wesley Professional.

Collier, K., & O’Leary, D. (2009). The Message Driven Warehouse. Cambridge: Cutter Consortium, Inc.

Hay, D. C. (1996). Data Model Patterns: Conventions of Thought. New York: Dorset House Publishing.

Hay, D. C. (2006). Data Model Patterns: A Metadata Map. San Francisco: Morgan Kaufman.

Longman, C. (2005, December 7). Data Warehousing Meeting – December 7, 2005. Retrieved November 16, 2008, from DAMA UK – Data Management Association:http://www.damauk.org/Building%20the%20adaptive%20data%20warehouse%20-%20Cliff%20Longman.pdf

A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.

The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:

1. Site – The location where the building sits
2. Structure – The foundation and frame
3. Skin – The outer shell of the building
4. Services – Water, electric, sewage, etc.
5. Space – The interior layout and configuration
6. Stuff – Lighting, colors, flooring, decor, etc.

The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.

While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:

• What is our design objective? (to improve understanding or to communicate the solution to others)
• Have we accomplished our objective yet? (i.e., have we done enough for now?)
• If so, what’s keeping us from getting started developing?
• If not, what is the smallest/simplest thing we can do to accomplish our objective?

Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.

Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier

One of the many things that agile leaders do is motivate teams. Dan Pink is the author of the recent best seller, Drive: The Surprising Truth About What Motivates Us. You may have seen Dan Pink’s “Motivation” presentation on Ted. This video went viral a couple of months ago. But, even more compelling is this artistic animation of Dan’s earlier talk. Give it a look, it’s well worth 10 minutes of your time.

——

It’s beginning to look like 2010 may be the year of agile business intelligence. I’ve been preaching about agile BI since I first started doing it in 2004. Although there are a few converts, the BI and data warehousing community hasn’t been fast to adopt. I’m suddenly hearing a lot more buzz in BI/DW circles about agile this year, including the agile theme of the upcoming TDWI World Conference in San Diego. I’m excited about this because I’ve personally experienced the joy of iteratively building BI systems in small increments with tons of collaboration with business experts and end users, and I want you to experience the same joy.

Since 2004 I’ve had one foot firmly planted in the agile software development community and the other in the data community. Agile seems to have “crossed the chasm” in the software community, and there is an interesting (and undesirable) tendency emerging that the BI/DW community can learn from. Some software organizations are falling into the trap of looking to agile as a prescriptive methodology that can be imposed on software engineering teams to make them faster and better. Guess what — it isn’t working!

It isn’t working for these organizations because they have fundamentally misunderstood agile. Agile is a style of working, not a methodology or process. Agile is founded on a shared set of core values including:

• Individuals and interactions
• Working software
• Customer collaboration
• Responding to change

(For more, see agilemanifesto.org.) Agile espouses a set of guiding principles including:

• The early and continuous delivery of value
• Welcoming changing requirements
• Frequent delivery of a working system
• Daily collaboration between developers and business experts
• Self-organizing teams of experts

among others. (To learn more, visit manifesto.org/principles.html).

Sure, there are methodologies such as Scrum, eXtreme Programming, and Lean that introduce practices that are consistent with these values and principles, but agile is inherently a way of “being” that involves the entire organization, not just development teams. The specific practices and processes are simply manifestations of agile behaviors. Agile’s core values and guiding principles give rise to good process, not the other way around. The great agile success stories (such as Google and Salesforce.com) reflect organizations that embrace this notion and have found agility to be very powerful. Organizations that adopt agile practices without embracing the values and principles often struggle.

As an agile consultant I get to work with many companies in various stages of agile transformation with varying corporate cultures and experiencing different challenges and successes. Through these experiences, success patterns and failure patterns emerge. I’ve begun referring to success patterns as being agile and failure patterns as doing agile. Doing agile refers to organizations that fail to move beyond the simpler trappings of agility: iterations, daily stand up meetings, and so on. Being agile refers to organizations with inherent values, behaviors, and guiding principles that exhibit the essence and spirit of agility: adaptive, evolutionary, value-driven, and quality-driven development. Organizations that are agile also do agile, but the inverse is not necessarily true. Many organizations are decidedly non-agile while still utilizing many agile practices.

Symptoms of Doing, Not Being

I was recently asked to help a struggling agile organization figure out why they weren’t seeing the benefits of their investment in agile adoption. Senior management in this company had expectations of improvements in quality, increased productivity, better customer satisfaction, and happier developers. Unfortunately, their progress was going backwards in every area. After some digging, I began to see some familiar anti-patterns emerging. These are the symptoms of an organization that is doing, but not being, agile.

Iron Triangle Planning: Agile BI projects deserve sufficient planning, but agile plans are projections, not promises. Experienced managers sometimes have difficulty breaking the habit of expecting a fixed price, fixed scope, and fixed schedule project plan. An agile plan reflects the teams’ best projections based only on the information that is currently available. As requirements change and uncertainty is uncovered, those projections are likely to become obsolete. Agile managers anticipate and adapt to this.

Management Styles Don’t Change: The best agile BI teams are self-organizing, self-managing, and self-responsible. This doesn’t mean that the role of management is subverted — instead, it changes. Managers are enablers, decision makers, and facilitators. They work with development teams to remove barriers and protect the team from unwanted outside pressures. Agile managers are effective at replacing command and control leadership styles with more collaborative ones.

Emphasis on Productivity: The promise of agile BI is the delivery of a high-value, high-quality working system, not increased productivity. Leaders who emphasize productivity are surprised when quality suffers and end users are unsatisfied with results. Yet, when developers are pressured to be more productive, they take shortcuts such as reduced testing and hurried workmanship. Conversely, when the emphasis is on high-value and high-quality BI features, user’s needs are often met after only 60 to 70 percent of the planned features are complete, effectively shortening the project cycle by shrinking the scope. Agile managers emphasize quality and value; they trust their teams to be productive.

Adapting to Change is Only Lip Service: Change is inevitable, especially in today’s climate. Unfortunately, embracing and adapting to change is not a normal part of human nature. We go to great lengths to control and limit changes. We establish change control boards and change management processes. Instead of controlling changes, these processes only made them more disruptive. Embracing change means seeking it out, inviting it in, and encouraging more of it to assure that we build the right solution for our customers. Agile managers are eager to add new requirements, eliminate unnecessary requirements, overhaul project plans, rearrange priorities, and even discard working BI features in order to respond to — and embrace — change.

Customer Collaboration is Short-Circuited: One of the four agile core values is customer collaboration. Unfortunately, effective customer collaboration is hard. Our customers are busy and hard to pin down. They sometimes tell us things that don’t make sense or won’t work. Customer collaboration cuts into development time, so we create roles such as product owner or business analyst to be the “voice of the customer.” It’s easier to do this, but it’s almost never as effective as real customer collaboration. BI customers have interesting stories to tell, and when developers get to hear these stories first-hand, they get a more holistic understanding of the BI system they are building – and they build it better. Agile leaders insist on frequent collaboration between BI development teams and end users, and they enable effective collaboration between these groups.

The Last Word

What is the lesson for us in the BI/DW community? Although agile BI developers are responsible for many agile engineering practices, the ultimate success of agility cuts across the entire organization. The management community, the customer community, the stakeholder community, and the development community are equally responsible for adopting agile attitudes and behaviors. Only then will an organization be agile rather than simply do agile.

I’ve been practicing and consulting on Agile BI since 2004 and have worked with teams that are “dislocated”. Since Agile BI values individuals and interactions as a more effective way to build the right thing, colocation is favored. However, outsourcing and geographic distribution are unavoidable realities in today’s business. Take a look at Scott Ambler’s recent blog about Agile and Geographic Distribution in which he discusses 3 key challenges caused by dislocation:

1. Communication – Just being in two separate buildings reduces the preference for face-to-face communication.
2. Temporal – The farther apart the time zones, the greater the impact on synchronous collaboration.
3. Cultural – Different work ethics, values, etc. affect the effectiveness of team interactions.

So, outsourced BI in India for example, is about an 11 hour difference from North American timezones, communication is asynchronous, and there are distinct cultural differences. All risk factors to true agility.

But, all is not lost – The best example of a high-functioning but dislocated agile team I’ve worked with was a situation that had experienced/talented people in both the US and India. The company brought 3 of the senior Indian developers to the US for 2 months to work face-to-face with the US team. Two BIG things occured during this 2 months:

First, the project envisioning, chartering, speculating and planning was a shared face-to-face process that got everyone galvanized around a shared vision and set of expectations (including senior management). Plus the first three project sprints were face-to-face, so the team established shared working agreements.

Second, The US members hosted the Indian members, including showing them the local attractions; taking them to shows; and generally hanging out with them (this was done voluntarily by the US members). The result was an invaluable building of friendships and trust between both groups.

After the Indian contingent returned home, the team figured out ways to divide the work in each sprint to minimize the dependencies between teams; and maximize the separation of concerns so that each team could work relatively autonomously. Every sprint both contingents integrated their work into one system and partnered in the feature showcase for the user community.

If you discovered a critical defect in your production BI system, how long would it take to revert to a previous version while you resolve the problem?

Like other mission critical systems, it should take no more than a few days to rebuild your BI system from scratch – including reconfiguring new servers and reloading data. The actual redeployment of your warehouse implementation (database schemata, ETL scripts, BI applications, etc.) should range from minutes to hours, not days or weeks.

Rapid deployment is not only essential due to the mission critical nature of today’s BI systems; it is also a critical aspect of agility. Remember that our highest priority is to satisfy the user community through early and continuous delivery of BI features. This means releasing new BI features into production every iteration or every few iterations. Doing so requires a highly optimized deployment process.

Optimizing data warehouse deployment time requires a combination of several good engineering and IT practices. Central to this goal is proper code management and version control, concepts with which many seasoned BI professionals remain unfamiliar. Unfortunately, it is all too easy to establish bad habits and practices in code management if we aren’t careful. Fortunately, our colleagues in the software development community have been managing code for decades, and have paved the way with effective tools and techniques that we can utilize.

Traditional BI development methods push system and acceptance testing to the end of the project cycle. This backend testing is typically manually intensive, possibly supplemented by the use of semi-automated tools. We need an entirely different testing discipline for Agile BI development.

First and foremost testing is integrated into the development process. Each development iteration must include plans for QA activities. One of the great things about this is that bugs don’t get a chance to accumulate over the project lifecycle. Quality feedback is immediate and frequent, and bugs are handled as they arise. QA specialists become integral members of the development team rather than gatekeepers at the end of the development cycle. Developers become integral to the QA process and learn sound testing practices as an extension of their technical skills. When I first introduce this notion to new agile teams I often get pushback from developers who say things like, “I don’t have time to test.” Or, “Testing is not my job.” I generally quell the urge to say something like, “If building a high quality BI system is not your job, then what exactly is your job?” Once developers establish the rhythm of making testing an integral part of development they usually love it. I’ve had a number of BI developers wonder why they didn’t learn to integrate testing and development long ago.

Essential to integrated testing is test automation. Manual testing is just not practical in a highly iterative and adaptive development environment. There are two key problems with manual testing. First, it takes too damn long and is an inhibitor to the delivery of frequent working software. Teams that rely on manual testing ultimately end up deferring testing until dedicated testing periods, which allows bugs to accumulate. Second, it is not sufficiently repeatable for regression testing. While we seek to embrace and adapt to change, we must always be confident that features that were “Done, done!” in a previous iteration retain their high quality in light of the changing system around them. Test automation requires some initial effort and ongoing diligence, but once technical teams get the hang of it, they can’t live without it.

Teams that do not practice integrated, automated testing are not really agile. It just isn’t feasible to create production quality, working features for user acceptance every 1-3 weeks without integrated and automated testing.

And by the way, quality assurance must be more comprehensive than system level testing. I’m always surprised at BI teams who treat final system testing as the only testing that is required in BI development. Agile BI developers test every unit of code, every integration point, every data structure, every user feature, and ultimately the entire working system not matter how embryonic. Unit testing involves testing the lowest level components that make up the BI system such as SQL scripts, ETL modules, stored procedures, etc. Integration testing involves testing all of the data transition points and wherever commercial tools are receiving or returning data. As data is pumped from source systems into staging databases; or from staging into multidimensional databases or OLAP engines, each data structure along the dataflow path must be tested to ensure that data integrity is preserved or enhanced. Simple mistakes like copying a VARCHAR(50) value into a VARCHAR(30) field in the staging database can wreak havoc on data integrity. Finally, each newly developed feature must be tested for acceptance and accuracy. Does it do what the user wants, needs, and expects; and does it do it correctly? While this is the ultimate acid test, we need confidence that our system is behaving well throughout the process flow.

The benefits of integrated, automated testing are even further boosting by using test-driven BI development. TDD is as much an implementation practice as it is a testing practice. In this approach test cases are written first, and then the code (or script, or configuration, etc.) is written to pass those test cases. When the system passes all of the test cases, and the BI practitioners can’t think of any new test cases, then the implementation work is “done”. That is it works as the developers think it should, and is of production quality. It is now ready for user acceptance to consider it “done, done”. While test-driven development may not be as mandatory as test automation and test integration, this is a technical practice that yields tremendous benefits since testing and development are inextricably linked. The test suite grows alongside the system, and since testing is automated the suite can be rerun frequently to maintain a high level of confidence in BI product quality.

TDD is applied most prominently at the lowest component level, or unit level, which ensures that high quality exists in the building blocks that make up the system. Story test driven development will help ensure that the user acceptance criteria are clearly defined for each user story before development begins.

Integrated automated testing in database development presents a unique set of challenges. Current automated testing tools designed for software development are not easily adaptable to database development, and large data volumes can make automated testing a daunting task. Data warehouse architectures further complicate these challenges since they involve multiple databases (staging, presentation, and sometimes even pre-staging); special code for data extraction, transformation, and loading (ETL); data cleansing code; and reporting engines and applications.

They insisted that this was the necessary precursor to building BI applications for end users. I asked them what specific business problems the data warehouse was required to solve. They speculated on a lot of possible ideas, but admitted that they had no business requirements. They explained that their project was an IT initiative, and their first job was to consolidate the data. After that, they planned to begin building BI applications against the warehouse. This was how this group had always worked. When I asked how often their customer/user community was completely delighted with the resulting BI applications, the team chuckled and reluctantly admitted that users had never been “delighted”.

This story is too often the way data warehousing projects go. They fail to focus on the early delivery of business value and lose end user trust and acceptance. We “data geeks” take a data-centric approach to building a data warehouse, and we convince ourselves that we have to solve lots of thorny data issues before we can build the applications for users. It’s all about the data! Many data experts make the wrong assumption that if they get the data right, they can meet all possible future user requirements. Oddly enough, when you talk to the business users, it’s not about the data. The data is just a means to the end goal of handling business problems and supporting business decisions. Users have stories to tell, and they are usually NOT about the data. The data merely plays a supporting role.

Agile data warehousing is a feature-driven or story-driven approach. We are eager to produce user features that enable our customers to do their jobs better or more efficiently. The data just supports this goal. And it just so happens that the best way to deliver that data is via a data warehouse.

Story-driven development is a very gratifying way to work once you get the hang of it. Agile BI developers know how to write good user stories for BI systems; how to make the stories small enough and simple enough to complete them in short iterations; how to prioritize stories; and estimate effort. By the way, Mike Cohn has written extensively about “user stories” among other agile topics. You should read Mike’s book for a deeper dive into this topic. My goal is to show how to do story-driven data warehouse development, and to help you wrestle with the common question, “How can we build anything meaningful in only two weeks?”

Two of the Agile Manifesto’s core values are focused on collaboration – one between team member and the other with customers. Another one focuses on responding to change. Fundamentally responding to change requires collaboration and the continuous realignment of expectations between developers, customers, and stakeholders. Despite the apparent focus on budgets and schedules, projects are generally declared successful when the results meet the expectations of stakeholders and customers – even if those expectations have changed dramatically during the course of the project. Organizations with agile in their DNA facilitate interactions among team members, and collaboration with customers.

I also occasionally get asked what kinds of projects are not suitable for an agile approach. After much thought, I’ve concluded that I would use Agile on every project, small or large, as long as I had the committed participation of stakeholders, customers, and developers. Not long ago I was asked to coach an agile data warehousing group in a large company. They had recently adopted an agile approach, but were struggling to be effective. During the assessment phase of my coaching process I asked the team technical leaders to describe their collaboration with end users. I learned that the users were too busy to be involved in planning, story prioritization, and new feature review and acceptance. The same was true for many of the business experts and stakeholders, so I advised the team to halt the project. A project that isn’t worth the involvement of users and stakeholders is not worth doing – time is better spent on other endeavors. It is essential to effectively involve everyone in the project community, and to ensure that their time is well-spent.

On a positive note, the most gratifying and successful agile projects I have experienced or witnessed have active project community involvement. The strong bond of trust that develops between users, team members, and stakeholders on such projects is a wonderful side effect of successful agile projects. Since success breeds more success, these bonds carry forward to other future projects.

A live variety show team consists of a diverse set of skills including studio executives, producers, writers, actors, stage hands, props, lights, camera crew, etc. After an episode airs, the execs, producers, cast and crew celebrate their success, monitor viewer ratings, and then immediately start planning for the next episode. The team must work fast and be highly collaborative to pull this off. There is absolutely no room in the schedule for superfluous meetings, ceremony, or formality. However, there must be a sufficient attention to detail and rigor to ensure that the show is highly successful every single week.

This got me thinking – what if we developed software and BI systems as if we were producing a live variety show every week? And, what if we measured success with the same ruthlessness that TV networks use viewer ratings? Agile developers work in short iterations delivering chunks of end user functionality incrementally. What if we behaved as if the project’s future were dependent upon high “viewer ratings” at the end of our current iteration? We had better not only have new features for our “viewers”, these features better be great!