The following article was originally published by TDWI in the May 15, 2012 issue of BI This Week.Startups know that what customers say they want isn’t necessarily what they want. The same principle is critical to the success of your BI project.
As a proponent of agile data warehousing and business intelligence, I am constantly looking for new techniques for delivering value to customers faster, adapting to their feedback, and evolving toward the right business solutions (regardless of initial requirements). I recently read the new book, Lean Startup by Eric Ries (Ries, 2011) and it has rocked my world. In the short time since this book hit the shelves in September, 2011, it has exploded in popularity. Be forewarned, this book is about entrepreneurship and high-tech startups. It isn’t about data warehousing, BI, or analytics … or is it?
The following post was originally published by the Cutter Consortium as an e-mail advisor for the Data Insight and Social BI practice in December, 2011. Hence, the reference to the holiday season. That publication can be found here.
In honor of the holiday season, this month’s topic is focused on an all-too-familiar problem: the common hangover. No, I’m not speaking of an “adult beverage” hangover, although if I find a cure for that I’ll be sure to share it with you. The type of hangover I’m speaking of is the routine failure of your agile delivery team to complete the user stories it commits to during sprint planning.
I’m surprised at what a common occurrence this is for many agile analytics teams. Here are a few example scenarios:
- The team completes development but testing hangs over into the next sprint.
- The team completes the data integration elements of a story, but the demonstrable corresponding BI features hang over to the next sprint.
- The story is too big to be completed in a single sprint, so the team just allows it to hang over into the next sprint.
There are other examples, but you get the idea. A hangover is not the same as a BI feature that is technically done, but the customer review calls for further modification or refinement (it isn’t “Done! done!”). A hangover is simply a failure to have the completed story ready for customer review at the end of the sprint. Continue reading “Curing the Common Hangover” »
Agile adoption for data warehouse and BI is on the rise. Agile Analytics can shorten development cycle time, improve quality, and help ensure that you build the right BI solutions for business decision makers. However, conventional IT organizational structures, policies, processes, and procedures are sometimes inconsistent with the tenets of agility. Values like customer collaboration, face-to-face interaction, and continuous delivery of value are often impeded by IT organizational protocols. Here are the top 10 points of friction that I frequently see in companies that I assist in Agile Analytics adoption. While many of these also impact software teams, DW/BI departments are often more directly impacted by them. Continue reading “Organizational Friction Impedes Agility” »
TDWI’s BI This Week newsletter ran this short article on June 22, 2011, and I’ve included it below. It provides a glimpse at the important technical practices that enable you to safely evolve your data models in ways that improve the design quality rather than degrading it. My book Agile Analytics has an entire chapter devoted to evolving excellent design, but this article provides an overview. I’d love your feedback on this piece.
The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.
What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.
Design excellence is critical to the success of Agile Analytics. The right design choices will help minimize technical debt, facilitate adapting to changes, improve quality, and provide the agile team with a cohesive technical framework. The wrong design choices can lead to overbuilt systems and high technical debt, severely hindering the teams’ ability to be agile. This applies to the design of data models, system architectures, ETL code, BI applications, and other components of the data warehouse and business intelligence solution. Agile DW/BI presents a difficult paradox – the ability to quickly respond to change and frequently deliver new features requires excellent data models and system design; yet excellent design takes time to develop. How do we deliver business value early and frequently without doing a lot of the design up-front?
A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.
The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:
- Site – The location where the building sits
- Structure – The foundation and frame
- Skin – The outer shell of the building
- Services – Water, electric, sewage, etc.
- Space – The interior layout and configuration
- Stuff – Lighting, colors, flooring, decor, etc.
The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.
While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:
- What is our design objective? (to improve understanding or to communicate the solution to others)
- Have we accomplished our objective yet? (i.e., have we done enough for now?)
- If so, what’s keeping us from getting started developing?
- If not, what is the smallest/simplest thing we can do to accomplish our objective?
Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.
Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier
Agile leadership is sometimes portrayed as, “Buy pizza and get out of the way.” Or, “Empowering teams to be more self-organizing and self-managing.” Neither of these descriptions is particularly accurate. Agile leaders have a uniquely challenging role. In fact, in many ways command and control, “my way or the highway” leadership is much simpler. The manager gives direction and the minions follow it. However, effective agile leadership results in much higher performance, greater innovation, and exciting new ideas.
One of the many things that agile leaders do is motivate teams. Dan Pink is the author of the recent best seller, Drive: The Surprising Truth About What Motivates Us. You may have seen Dan Pink’s “Motivation” presentation on Ted. This video went viral a couple of months ago. But, even more compelling is this artistic animation of Dan’s earlier talk. Give it a look, it’s well worth 10 minutes of your time.
In June, 2010 The Data Warehousing Institute (TDWI) published my article entitled “Being” Agile vs. “Doing” Agile as part of their “This Week in BI” series. Although I wrote the article for the Agile BI community, the concepts apply to any agile development organization. Please give it a read and let me know what you think.
(The following is a repost of my response to a question about outsourced BI and geographically distributed teams on LinkedIn’s TDWI BI and DW Discussion Forum. Here is the entire thread.)
I’ve been practicing and consulting on Agile BI since 2004 and have worked with teams that are “dislocated”. Since Agile BI values individuals and interactions as a more effective way to build the right thing, colocation is favored. However, outsourcing and geographic distribution are unavoidable realities in today’s business. Take a look at Scott Ambler’s recent blog about Agile and Geographic Distribution in which he discusses 3 key challenges caused by dislocation:
1. Communication – Just being in two separate buildings reduces the preference for face-to-face communication.
2. Temporal – The farther apart the time zones, the greater the impact on synchronous collaboration.
3. Cultural – Different work ethics, values, etc. affect the effectiveness of team interactions.