In honor of the holiday season, this month’s topic is focused on an all-too-familiar problem: the common hangover. No, I’m not speaking of an “adult beverage” hangover, although if I find a cure for that I’ll be sure to share it with you. The type of hangover I’m speaking of is the routine failure of your agile delivery team to complete the user stories it commits to during sprint planning.

I’m surprised at what a common occurrence this is for many agile analytics teams. Here are a few example scenarios:

1. The team completes development but testing hangs over into the next sprint.
2. The team completes the data integration elements of a story, but the demonstrable corresponding BI features hang over to the next sprint.
3. The story is too big to be completed in a single sprint, so the team just allows it to hang over into the next sprint.

There are other examples, but you get the idea. A hangover is not the same as a BI feature that is technically done, but the customer review calls for further modification or refinement (it isn’t “Done! done!”). A hangover is simply a failure to have the completed story ready for customer review at the end of the sprint.

Back in 1975, software engineer Fred Brooks observed that, “A project gets late one day at a time.”¹Similarly, an agile project gets late one iteration at a time, when teams fail to honor their sprint commitments. During project chartering/planning the entire project community speculates on an anticipated project outcome. While this outcome is subject to the routine fluxuations of requirements uncertainty, technical uncertainty, and risk, it is hindered by fluxuations caused by the failure of the team to meet shared expectations.

Agile development works in large part because we are continuously realigning the expectations of management stakeholders, customers, and delivery team members. These three constituencies collaborate during sprint planning to ensure that everyone has a shared understanding of the expected sprint outcome. They also collaborate at the end of every sprint to validate that the results are valuable and acceptable. Many times they collaborate during midsprint to provide guidance and clarity.

Delivery teams must accept that customers and product owners know what is important and can focus the team on the next most important stories. Customers and product owners must accept that the sprint outcome is constrained by the team’s limited capacity. However, when delivery teams fail to honor their end of the bargain or when the customer team puts unreasonable pressure on the delivery team, the trust relationship between customer community and delivery team erodes. The customer community and product owner can no longer rely on the team to meet their mutual expectations, and the process loses predictability and reliability.

The root of the problem is what Jim Highsmith refers to as “wish-based planning” instead of “capacity-based planning.”² The team succumbs to wishful thinking, overly optimistic estimating, and the pressure to do more than is realistic. Moreover, they fail to base their plans on previously demonstrated actual capacity. The problem snowballs when the team does this routinely sprint after sprint after sprint. Anticipated project outcomes are no longer valid, but they cannot be replaced with any degree of certainty.

So, now to the cure for the common hangover:

Ingredients Include

• Two parts acknowledgement
• One part self-discipline
• Two parts effective user story decomposition
• Two parts shared responsibility
• One part team honor
• Two parts celebration

Recipe

1. The customer team (including product owner) and the delivery team collaboratively acknowledge the symptoms of hangover. The delivery team acknowledges the undesirable consequences of wish-based planning and routine hangover. The customer team acknowledges any undue pressure placed on the team to plan beyond its capacity. Both teams recommit to the principle of continuous delivery of business value at a sustainable pace, and within capacity.
2. The delivery team explicitly recognizes that with the freedom of self-management comes the responsibility of self-discipline. Agile team self-discipline calls for moderation and self-control. The delivery team recommits to more thoughtful estimating and more realistic sprint planning. This involves pushing back on the pressure to overcommit, while still focusing on the delivery of high-valued, production quality working BI features.
3. Both the delivery team and the customer team seek to improve the art of user story decomposition. The customer team learns to see value in small, simple, immature, early working BI features rather than always expecting fully embellished and mature BI reports. The delivery team uses story-driven development to slice the architecture as thinly as possible, doing the minimal amount of data modeling, database development, ETL development, BI development, and so on, needed to complete the story. The delivery team does no more than is necessary for each story, and learns that each incremental change in the implementation is not rework, but instead is an opportunity to evolve a high quality DW/BI system.
4. The customer team takes responsibility for helping the delivery team plan within its capacity, and the delivery team takes responsibility for making realistic sprint commitments and keeping those commitments. Both teams take responsibility for the mutual goal of establishing a steady and effective cadence of successful sprints.
5. Despite Steps 1 through 4, the delivery team may occasionally still overcommit during sprint planning. When this happens, the delivery team must make every effort to honor those commitments, even if it means working at an unsustainable pace for a short period of time. Afterwards the team must seek to rebalance its planning for the next sprint to achieve a more realistic commitment.
6. Finally, every successful sprint should end in a celebration by both the customer team and the delivery team. When both teams get a taste for celebrating success rather than bemoaning failure, the agile project takes on a positive and energetic tone that carries forward into successive iterations. This practice builds a powerful camaraderie between customers and developers and a shared sense of purpose. But don’t celebrate too hard; that can cause hangovers of a different sort.

Finally, I wish each of you a happy New Year and continued growth and improvement in your agile analytics efforts. As agile luminary Mike Cohn says, “Agile is not something you become. It is something you become more of.” Best wishes on curing your project hangovers in the process of becoming more agile. I’ll see you in 2012.

Note

¹Brooks, F.J. Mythical Man Month: Essays on Software Engineering. Addison-Wesley, 1975.

²Highsmith, Jim. “A Recipe for Success, Part 3.” Cutter Agile Product & Project Management Advisor, 8 March 2007.

1. Conventional Leadership – Agile expert Jim Highsmith says, “Conventional leaders expect projects to be on-track early and off-track at the end. Agile leaders expect projects to off-track early and on-track at the end.” Agile leaders measure, monitor, and manage in ways that are very different from conventional leaders. Successful Agile adoption involves leadership training as well as technical and project management training. Significant friction occurs when organizational leaders maintain conventional mindsets, habits, and behaviors, while expecting project teams to “go agile”.
2. Failure to Limit Work in Progress – Friction occurs when IT leaders try to execute too many simultaneous projects. People are saddled with multiple assignments and their time is split between too many activities. This causes significant “context switching” resulting in diminished effectiveness of individuals and teams. The perception that parallel projects will achieve goals faster is false. Cycle times are shorter and results are better when teams focus on one project at a time. Agile leaders prioritize program portfolios and limit the number of parallel projects based on organizational capacity.
3. Shuffling of Team Members – When IT leaders treat project staffing as a matter of shifting resources between projects based on skillset demands, every project suffers. People are much more than resources with specialized skills. When people are dedicated to a project team for the duration, they naturally develop a sense of ownership of the project. When people are constantly shifted in and out of an Agile project, it is impossible for the team to establish a steady cadence and predictable velocity that can be used for effective iteration planning. Agile Analytics projects should be staffed with a dedicated, cross-functional, self-organizing team.
4. Functional “Resource Pools” – Friction occurs when developers, architects, modelers, testers, project managers, and others are organized into functional silos or “resource pools”. People are pulled from their functional pool as-needed and returned to the pool when their skill is no longer needed. This organizational structure further contributes to the problems previously discussed. While skills specialization is important, Agile projects benefit from cross-functional teams with a lots of collaboration and interaction. Specialists assigned to dedicated, cross-functional teams naturally broaden their skillsets and become more general-purpose. This serves to boost the performance of the Agile team. Agile leaders seek to deconstruct functional silos and encourage team cross-functionality.
5. Off-shoring – Agile Analytics favors team colocation, but can still be highly effective in distributed teams under the right conditions. Inhibitors to distributed agility include: large time zone differences, language barriers, and cultural mismatches. Off-shoring in places at opposite ends of the globe introduces all of these characteristics. When considering the benefits of off-shoring, be sure to also consider the cost of rework and the difficulty in team collaboration. Since geographical distribution unavoidable in most companies, Agile IT leaders seek to maximize team interaction by investing in the appropriate technologies and techniques to boost “virtual colocation” wherever possible.
6. IT Policies and Restrictions – Friction occurs when DW/BI developers do not have unfettered control over their development environments, such as when developers must submit data model change requests rather than making such changes directly in the development sandbox. Such restrictions create undue friction for Agile teams that must deliver business value frequently. Agile leaders seek to strike the best balance between important governance policies where it matters, and the freedom from such policies where it does not.
7. Shared Services – Many IT organizations have shared services groups to handle activities like deployment, administration, maintenance, etc. These groups are most familiar with conventional phased-sequential projects rather than iterative-incremental ones. Unlike conventional methods in which development is delayed until later in the project, Agile projects call for an immediate start of development. Therefore, support services are needed earlier and in a timely fashion. To prevent the mismatch between shared services’ conventional policies and the iterative needs of Agile teams, Agile IT leaders invest in appropriate Agile training for these ancillary support groups.
8. Agile is a Project Management Approach – Friction occurs when IT leaders send project managers to Agile training but fail to invest in similar training for delivery teams and management. Suddenly project managers must be the Agile trainers as well as the “enforcers of agility”. Delivery team members lack the context and fundamentals necessary to understand agility, and often resent the disruption that this new change causes. Managers retain conventional habits, which are often at odds with Agile tenets.
9. Agile is a Technical Method – Friction occurs when IT leaders perceive Agile Analytics as a technical methodology. Some IT leaders believe that if the delivery team receives Agile training, then they will be more productive and able to do more with less. Every project involves management stakeholders, business users, and delivery team members. It is nearly impossible to align the expectations of these constituencies when Agile adoption is relegated to the delivery team only. Everyone must have a shared understanding of Agile Analytics.
10. Minimal Customer Collaboration – Friction occurs when actual end users of the DW/BI system are not actively involved throughout the Agile Analytics project. Many DW/BI programs rely too heavily on business analysts to be the “voice of the customer”. This is caused by concerns about intruding on busy end users, and users unwilling to commit to frequent review and feedback on new BI features. If a project is worth doing, it is worth doing right – which requires the active participation of actual business users.

How does your IT organization stack up with respect to these points of friction? As an IT leader can you influence chances that will help reduce or eliminate these friction points? If you could only eliminate one of these friction points, which one would produce the greatest benefits to your Agile DW/BI teams? How soon can you get started?

The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.

What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.

Why is agile data modeling a good idea? To paraphrase Ron Jeffries, one of Extreme Programming’s founders, the best way to implement a DW/BI system is to implement less of it. The best way to have fewer defects in your DW/BI system is to have a smaller/simpler one. The problem with comprehensive up-front modeling is that you must design for all contingent requirements, both known and speculated. This inevitably results in an overdesigned model that costs more to implement, is costlier to maintain, is more likely to contain defects, and is more difficult to understand.

Agile data models are as simple as possible while being sufficiently detailed, accurate, and consistent. They also fulfill a well-understood purpose and provide positive value (i.e., their benefit outweighs the cost of keeping them updated).

How to Safely Evolve Data Models

These concerns about evolutionary modeling resulting unnecessary rework, unintended side effects, and design degradation are legitimate. Additionally, the prospect of making a data model change to a high-volume data warehouse in production can be scary. Agile data modeling calls for a new set of practices that enable the safe evolution of models, even those in production. I’ll summarize those practices here. Consider this list a brief introduction; each deserves a deeper study to gain proficiency.

Data Model Patterns: Data models evolve toward excellence when we take advantage of tried and proven designs. Design patterns enable us to benefit from mature solutions that have previously been developed. Effective application of patterns relies on familiarity and awareness of patterns catalogs, and the ability to use them appropriately and sparingly.

Michael Blaha’s Patterns of Data Modeling is the most recent catalog of patterns, but David Hay first started cataloging data model patterns in 1996 with his Data Model Patterns: Conventions of Thought, and followed in 2006 with Data Model Patterns: A Metadata Map. An extension of data modeling patterns is the adaptive data model (ADM), a generalized data model designed to accommodate multiple domains. The ADM has been successfully used in data warehouse design and I write about it in detail in the Cutter executive report, The Message Driven Warehouse.

Technical Debt Management: Data models evolve toward excellence when changes are easy to make due to low technical debt. Technical debt is common in data warehousing. It is the entropy that occurs in any system over time due to development shortcuts, suboptimal design choices, maintenance activities, and so on. Like financial debt, a little technical debt is acceptable as long as we monitor it and pay it back quickly. When technical debt accrues unabated, the cost of change becomes unacceptably high. Agile data modelers continuously identify, prioritize, and monitor technical debt in the data model, seeking to eliminate it to assist with fast response to new requirements.

Database Test Automation: Data models evolve toward excellence when we have continuous confidence that our ideas are working. Agile BI practitioners work in short iterations delivering business value every few weeks. We need confirmation that what we build in later iterations doesn’t break what we built in early ones. The only practical way to accomplish this is with an automated test suite. Automated database tests validate data structures, data content and quality, schema constraints and integrity, data derivations, and so on. We can run automated tests quickly and simply at any time to confirm that everything still works. Tests are added as data model changes are made, so the test suite grows alongside the model. I devote an entire chapter to this topic in my book, Agile Analytics.

Database Refactoring: Data models evolve toward excellence when we can safely make changes to the design, even if it is in production. Database refactoring is a technical discipline that enables the safe evolution of data models without breaking previously working features and components. In their book Refactoring Databases, Scott Ambler and Pramod Sadalage explain database refactoring as “… a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.” Refactoring combines automated regression testing with a change transition period to ensure that changes haven’t broken anything. The change transition period is a window of time during which the model revision lives alongside the former version to ensure that nothing breaks. Refactoring combined with test automation are the central disciplines for effectively evolving data models.

Recognizing Data Model Smells: Data models evolve toward excellence when we recognize where they need improvement. Experienced data modelers develop a nose for elegant designs … and stinky ones. Discovering smells in the model is an essential precursor to improving it. Smells may include smart keys, multipurpose columns and tables, data redundancy, very large tables, and so on. By learning to pay attention to these smells, you can focus your attention on possible problem areas and consider them as candidates for technical debt management.

Change Deployment: Data models evolve toward excellence when we can quickly deploy changes at any time without fear. Agile BI practitioners develop data model changes using techniques that safeguard production deployment. All data model changes are scripted and those scripts are kept under version control. Database schemas are versioned and scripts are developed to roll forward to the next version, and roll back to the previous version in case things don’t work as expected. Data migration scripts ensure no loss or corruption of production data. Everything is automated and tested carefully in preproduction. Automated deployment, automated testing, and database refactoring disciplines support frequent, fast and fearless deployments.

Take Small Steps: Data models evolve toward excellence as a series of small, easily understood changes. It’s easier to undo a sequence of little changes than one big, complicated one. A side benefit of agile BI is that short iterations force us to plan in small steps. Agile data modelers quickly learn to change only what is needed to support the BI features currently in development.

The Final Word

All of these techniques combined form a strong safety net for evolutionary data modeling. Moreover, these techniques aren’t exclusive to agile BI, they should be considered as modern data warehousing practices that should be among the skills of every data modeler and data warehouse practitioner — agile or otherwise.

Recommended Reading

Ambler, S. W. (2002). Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process. New York: John Wiley & Sons, Inc.

Ambler, S. W., & Sadalage, P. J. (2006). Refactoring Databases: Evolutionary Database Design. Boston: Addison Wesley.

Blaha, M. (2010). Patterns of Data Modeling. Boca Raton: CRC Press.

Collier, K. (2011). Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing. Boston: Addison-Wesley Professional.

Collier, K., & O’Leary, D. (2009). The Message Driven Warehouse. Cambridge: Cutter Consortium, Inc.

Hay, D. C. (1996). Data Model Patterns: Conventions of Thought. New York: Dorset House Publishing.

Hay, D. C. (2006). Data Model Patterns: A Metadata Map. San Francisco: Morgan Kaufman.

Longman, C. (2005, December 7). Data Warehousing Meeting – December 7, 2005. Retrieved November 16, 2008, from DAMA UK – Data Management Association:http://www.damauk.org/Building%20the%20adaptive%20data%20warehouse%20-%20Cliff%20Longman.pdf

A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.

The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:

1. Site – The location where the building sits
2. Structure – The foundation and frame
3. Skin – The outer shell of the building
4. Services – Water, electric, sewage, etc.
5. Space – The interior layout and configuration
6. Stuff – Lighting, colors, flooring, decor, etc.

The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.

While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:

• What is our design objective? (to improve understanding or to communicate the solution to others)
• Have we accomplished our objective yet? (i.e., have we done enough for now?)
• If so, what’s keeping us from getting started developing?
• If not, what is the smallest/simplest thing we can do to accomplish our objective?

Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.

Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier

One of the many things that agile leaders do is motivate teams. Dan Pink is the author of the recent best seller, Drive: The Surprising Truth About What Motivates Us. You may have seen Dan Pink’s “Motivation” presentation on Ted. This video went viral a couple of months ago. But, even more compelling is this artistic animation of Dan’s earlier talk. Give it a look, it’s well worth 10 minutes of your time.

——

It’s beginning to look like 2010 may be the year of agile business intelligence. I’ve been preaching about agile BI since I first started doing it in 2004. Although there are a few converts, the BI and data warehousing community hasn’t been fast to adopt. I’m suddenly hearing a lot more buzz in BI/DW circles about agile this year, including the agile theme of the upcoming TDWI World Conference in San Diego. I’m excited about this because I’ve personally experienced the joy of iteratively building BI systems in small increments with tons of collaboration with business experts and end users, and I want you to experience the same joy.

Since 2004 I’ve had one foot firmly planted in the agile software development community and the other in the data community. Agile seems to have “crossed the chasm” in the software community, and there is an interesting (and undesirable) tendency emerging that the BI/DW community can learn from. Some software organizations are falling into the trap of looking to agile as a prescriptive methodology that can be imposed on software engineering teams to make them faster and better. Guess what — it isn’t working!

It isn’t working for these organizations because they have fundamentally misunderstood agile. Agile is a style of working, not a methodology or process. Agile is founded on a shared set of core values including:

• Individuals and interactions
• Working software
• Customer collaboration
• Responding to change

(For more, see agilemanifesto.org.) Agile espouses a set of guiding principles including:

• The early and continuous delivery of value
• Welcoming changing requirements
• Frequent delivery of a working system
• Daily collaboration between developers and business experts
• Self-organizing teams of experts

among others. (To learn more, visit manifesto.org/principles.html).

Sure, there are methodologies such as Scrum, eXtreme Programming, and Lean that introduce practices that are consistent with these values and principles, but agile is inherently a way of “being” that involves the entire organization, not just development teams. The specific practices and processes are simply manifestations of agile behaviors. Agile’s core values and guiding principles give rise to good process, not the other way around. The great agile success stories (such as Google and Salesforce.com) reflect organizations that embrace this notion and have found agility to be very powerful. Organizations that adopt agile practices without embracing the values and principles often struggle.

As an agile consultant I get to work with many companies in various stages of agile transformation with varying corporate cultures and experiencing different challenges and successes. Through these experiences, success patterns and failure patterns emerge. I’ve begun referring to success patterns as being agile and failure patterns as doing agile. Doing agile refers to organizations that fail to move beyond the simpler trappings of agility: iterations, daily stand up meetings, and so on. Being agile refers to organizations with inherent values, behaviors, and guiding principles that exhibit the essence and spirit of agility: adaptive, evolutionary, value-driven, and quality-driven development. Organizations that are agile also do agile, but the inverse is not necessarily true. Many organizations are decidedly non-agile while still utilizing many agile practices.

Symptoms of Doing, Not Being

I was recently asked to help a struggling agile organization figure out why they weren’t seeing the benefits of their investment in agile adoption. Senior management in this company had expectations of improvements in quality, increased productivity, better customer satisfaction, and happier developers. Unfortunately, their progress was going backwards in every area. After some digging, I began to see some familiar anti-patterns emerging. These are the symptoms of an organization that is doing, but not being, agile.

Iron Triangle Planning: Agile BI projects deserve sufficient planning, but agile plans are projections, not promises. Experienced managers sometimes have difficulty breaking the habit of expecting a fixed price, fixed scope, and fixed schedule project plan. An agile plan reflects the teams’ best projections based only on the information that is currently available. As requirements change and uncertainty is uncovered, those projections are likely to become obsolete. Agile managers anticipate and adapt to this.

Management Styles Don’t Change: The best agile BI teams are self-organizing, self-managing, and self-responsible. This doesn’t mean that the role of management is subverted — instead, it changes. Managers are enablers, decision makers, and facilitators. They work with development teams to remove barriers and protect the team from unwanted outside pressures. Agile managers are effective at replacing command and control leadership styles with more collaborative ones.

Emphasis on Productivity: The promise of agile BI is the delivery of a high-value, high-quality working system, not increased productivity. Leaders who emphasize productivity are surprised when quality suffers and end users are unsatisfied with results. Yet, when developers are pressured to be more productive, they take shortcuts such as reduced testing and hurried workmanship. Conversely, when the emphasis is on high-value and high-quality BI features, user’s needs are often met after only 60 to 70 percent of the planned features are complete, effectively shortening the project cycle by shrinking the scope. Agile managers emphasize quality and value; they trust their teams to be productive.

Adapting to Change is Only Lip Service: Change is inevitable, especially in today’s climate. Unfortunately, embracing and adapting to change is not a normal part of human nature. We go to great lengths to control and limit changes. We establish change control boards and change management processes. Instead of controlling changes, these processes only made them more disruptive. Embracing change means seeking it out, inviting it in, and encouraging more of it to assure that we build the right solution for our customers. Agile managers are eager to add new requirements, eliminate unnecessary requirements, overhaul project plans, rearrange priorities, and even discard working BI features in order to respond to — and embrace — change.

Customer Collaboration is Short-Circuited: One of the four agile core values is customer collaboration. Unfortunately, effective customer collaboration is hard. Our customers are busy and hard to pin down. They sometimes tell us things that don’t make sense or won’t work. Customer collaboration cuts into development time, so we create roles such as product owner or business analyst to be the “voice of the customer.” It’s easier to do this, but it’s almost never as effective as real customer collaboration. BI customers have interesting stories to tell, and when developers get to hear these stories first-hand, they get a more holistic understanding of the BI system they are building – and they build it better. Agile leaders insist on frequent collaboration between BI development teams and end users, and they enable effective collaboration between these groups.

The Last Word

What is the lesson for us in the BI/DW community? Although agile BI developers are responsible for many agile engineering practices, the ultimate success of agility cuts across the entire organization. The management community, the customer community, the stakeholder community, and the development community are equally responsible for adopting agile attitudes and behaviors. Only then will an organization be agile rather than simply do agile.

I’ve been practicing and consulting on Agile BI since 2004 and have worked with teams that are “dislocated”. Since Agile BI values individuals and interactions as a more effective way to build the right thing, colocation is favored. However, outsourcing and geographic distribution are unavoidable realities in today’s business. Take a look at Scott Ambler’s recent blog about Agile and Geographic Distribution in which he discusses 3 key challenges caused by dislocation:

1. Communication – Just being in two separate buildings reduces the preference for face-to-face communication.
2. Temporal – The farther apart the time zones, the greater the impact on synchronous collaboration.
3. Cultural – Different work ethics, values, etc. affect the effectiveness of team interactions.

So, outsourced BI in India for example, is about an 11 hour difference from North American timezones, communication is asynchronous, and there are distinct cultural differences. All risk factors to true agility.

But, all is not lost – The best example of a high-functioning but dislocated agile team I’ve worked with was a situation that had experienced/talented people in both the US and India. The company brought 3 of the senior Indian developers to the US for 2 months to work face-to-face with the US team. Two BIG things occured during this 2 months:

First, the project envisioning, chartering, speculating and planning was a shared face-to-face process that got everyone galvanized around a shared vision and set of expectations (including senior management). Plus the first three project sprints were face-to-face, so the team established shared working agreements.

Second, The US members hosted the Indian members, including showing them the local attractions; taking them to shows; and generally hanging out with them (this was done voluntarily by the US members). The result was an invaluable building of friendships and trust between both groups.

After the Indian contingent returned home, the team figured out ways to divide the work in each sprint to minimize the dependencies between teams; and maximize the separation of concerns so that each team could work relatively autonomously. Every sprint both contingents integrated their work into one system and partnered in the feature showcase for the user community.

If you discovered a critical defect in your production BI system, how long would it take to revert to a previous version while you resolve the problem?

Like other mission critical systems, it should take no more than a few days to rebuild your BI system from scratch – including reconfiguring new servers and reloading data. The actual redeployment of your warehouse implementation (database schemata, ETL scripts, BI applications, etc.) should range from minutes to hours, not days or weeks.

Rapid deployment is not only essential due to the mission critical nature of today’s BI systems; it is also a critical aspect of agility. Remember that our highest priority is to satisfy the user community through early and continuous delivery of BI features. This means releasing new BI features into production every iteration or every few iterations. Doing so requires a highly optimized deployment process.

Optimizing data warehouse deployment time requires a combination of several good engineering and IT practices. Central to this goal is proper code management and version control, concepts with which many seasoned BI professionals remain unfamiliar. Unfortunately, it is all too easy to establish bad habits and practices in code management if we aren’t careful. Fortunately, our colleagues in the software development community have been managing code for decades, and have paved the way with effective tools and techniques that we can utilize.

Traditional BI development methods push system and acceptance testing to the end of the project cycle. This backend testing is typically manually intensive, possibly supplemented by the use of semi-automated tools. We need an entirely different testing discipline for Agile BI development.

First and foremost testing is integrated into the development process. Each development iteration must include plans for QA activities. One of the great things about this is that bugs don’t get a chance to accumulate over the project lifecycle. Quality feedback is immediate and frequent, and bugs are handled as they arise. QA specialists become integral members of the development team rather than gatekeepers at the end of the development cycle. Developers become integral to the QA process and learn sound testing practices as an extension of their technical skills. When I first introduce this notion to new agile teams I often get pushback from developers who say things like, “I don’t have time to test.” Or, “Testing is not my job.” I generally quell the urge to say something like, “If building a high quality BI system is not your job, then what exactly is your job?” Once developers establish the rhythm of making testing an integral part of development they usually love it. I’ve had a number of BI developers wonder why they didn’t learn to integrate testing and development long ago.

Essential to integrated testing is test automation. Manual testing is just not practical in a highly iterative and adaptive development environment. There are two key problems with manual testing. First, it takes too damn long and is an inhibitor to the delivery of frequent working software. Teams that rely on manual testing ultimately end up deferring testing until dedicated testing periods, which allows bugs to accumulate. Second, it is not sufficiently repeatable for regression testing. While we seek to embrace and adapt to change, we must always be confident that features that were “Done, done!” in a previous iteration retain their high quality in light of the changing system around them. Test automation requires some initial effort and ongoing diligence, but once technical teams get the hang of it, they can’t live without it.

Teams that do not practice integrated, automated testing are not really agile. It just isn’t feasible to create production quality, working features for user acceptance every 1-3 weeks without integrated and automated testing.

And by the way, quality assurance must be more comprehensive than system level testing. I’m always surprised at BI teams who treat final system testing as the only testing that is required in BI development. Agile BI developers test every unit of code, every integration point, every data structure, every user feature, and ultimately the entire working system not matter how embryonic. Unit testing involves testing the lowest level components that make up the BI system such as SQL scripts, ETL modules, stored procedures, etc. Integration testing involves testing all of the data transition points and wherever commercial tools are receiving or returning data. As data is pumped from source systems into staging databases; or from staging into multidimensional databases or OLAP engines, each data structure along the dataflow path must be tested to ensure that data integrity is preserved or enhanced. Simple mistakes like copying a VARCHAR(50) value into a VARCHAR(30) field in the staging database can wreak havoc on data integrity. Finally, each newly developed feature must be tested for acceptance and accuracy. Does it do what the user wants, needs, and expects; and does it do it correctly? While this is the ultimate acid test, we need confidence that our system is behaving well throughout the process flow.

The benefits of integrated, automated testing are even further boosting by using test-driven BI development. TDD is as much an implementation practice as it is a testing practice. In this approach test cases are written first, and then the code (or script, or configuration, etc.) is written to pass those test cases. When the system passes all of the test cases, and the BI practitioners can’t think of any new test cases, then the implementation work is “done”. That is it works as the developers think it should, and is of production quality. It is now ready for user acceptance to consider it “done, done”. While test-driven development may not be as mandatory as test automation and test integration, this is a technical practice that yields tremendous benefits since testing and development are inextricably linked. The test suite grows alongside the system, and since testing is automated the suite can be rerun frequently to maintain a high level of confidence in BI product quality.

TDD is applied most prominently at the lowest component level, or unit level, which ensures that high quality exists in the building blocks that make up the system. Story test driven development will help ensure that the user acceptance criteria are clearly defined for each user story before development begins.

Integrated automated testing in database development presents a unique set of challenges. Current automated testing tools designed for software development are not easily adaptable to database development, and large data volumes can make automated testing a daunting task. Data warehouse architectures further complicate these challenges since they involve multiple databases (staging, presentation, and sometimes even pre-staging); special code for data extraction, transformation, and loading (ETL); data cleansing code; and reporting engines and applications.

They insisted that this was the necessary precursor to building BI applications for end users. I asked them what specific business problems the data warehouse was required to solve. They speculated on a lot of possible ideas, but admitted that they had no business requirements. They explained that their project was an IT initiative, and their first job was to consolidate the data. After that, they planned to begin building BI applications against the warehouse. This was how this group had always worked. When I asked how often their customer/user community was completely delighted with the resulting BI applications, the team chuckled and reluctantly admitted that users had never been “delighted”.

This story is too often the way data warehousing projects go. They fail to focus on the early delivery of business value and lose end user trust and acceptance. We “data geeks” take a data-centric approach to building a data warehouse, and we convince ourselves that we have to solve lots of thorny data issues before we can build the applications for users. It’s all about the data! Many data experts make the wrong assumption that if they get the data right, they can meet all possible future user requirements. Oddly enough, when you talk to the business users, it’s not about the data. The data is just a means to the end goal of handling business problems and supporting business decisions. Users have stories to tell, and they are usually NOT about the data. The data merely plays a supporting role.

Agile data warehousing is a feature-driven or story-driven approach. We are eager to produce user features that enable our customers to do their jobs better or more efficiently. The data just supports this goal. And it just so happens that the best way to deliver that data is via a data warehouse.

Story-driven development is a very gratifying way to work once you get the hang of it. Agile BI developers know how to write good user stories for BI systems; how to make the stories small enough and simple enough to complete them in short iterations; how to prioritize stories; and estimate effort. By the way, Mike Cohn has written extensively about “user stories” among other agile topics. You should read Mike’s book for a deeper dive into this topic. My goal is to show how to do story-driven data warehouse development, and to help you wrestle with the common question, “How can we build anything meaningful in only two weeks?”