Design excellence is critical to the success of Agile Analytics. The right design choices will help minimize technical debt, facilitate adapting to changes, improve quality, and provide the agile team with a cohesive technical framework. The wrong design choices can lead to overbuilt systems and high technical debt, severely hindering the teams’ ability to be agile. This applies to the design of data models, system architectures, ETL code, BI applications, and other components of the data warehouse and business intelligence solution. Agile DW/BI presents a difficult paradox – the ability to quickly respond to change and frequently deliver new features requires excellent data models and system design; yet excellent design takes time to develop. How do we deliver business value early and frequently without doing a lot of the design up-front?

A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.

The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:

1. Site – The location where the building sits
2. Structure – The foundation and frame
3. Skin – The outer shell of the building
4. Services – Water, electric, sewage, etc.
5. Space – The interior layout and configuration
6. Stuff – Lighting, colors, flooring, decor, etc.

The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.

While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:

• What is our design objective? (to improve understanding or to communicate the solution to others)
• Have we accomplished our objective yet? (i.e., have we done enough for now?)
• If so, what’s keeping us from getting started developing?
• If not, what is the smallest/simplest thing we can do to accomplish our objective?

Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.

Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier