Contrary to popular opinion, the best business intelligence systems are not driven by the data, or the operational source systems. I recently had a conversation with a group of data warehouse developers who were completely baffled by the notion of building a BI application without first extracting all of the source system data into a single, normalized data model.

They insisted that this was the necessary precursor to building BI applications for end users. I asked them what specific business problems the data warehouse was required to solve. They speculated on a lot of possible ideas, but admitted that they had no business requirements. They explained that their project was an IT initiative, and their first job was to consolidate the data. After that, they planned to begin building BI applications against the warehouse. This was how this group had always worked. When I asked how often their customer/user community was completely delighted with the resulting BI applications, the team chuckled and reluctantly admitted that users had never been “delighted”.

This story is too often the way data warehousing projects go. They fail to focus on the early delivery of business value and lose end user trust and acceptance. We “data geeks” take a data-centric approach to building a data warehouse, and we convince ourselves that we have to solve lots of thorny data issues before we can build the applications for users. It’s all about the data! Many data experts make the wrong assumption that if they get the data right, they can meet all possible future user requirements. Oddly enough, when you talk to the business users, it’s not about the data. The data is just a means to the end goal of handling business problems and supporting business decisions. Users have stories to tell, and they are usually NOT about the data. The data merely plays a supporting role.

Agile data warehousing is a feature-driven or story-driven approach. We are eager to produce user features that enable our customers to do their jobs better or more efficiently. The data just supports this goal. And it just so happens that the best way to deliver that data is via a data warehouse.

Story-driven development is a very gratifying way to work once you get the hang of it. Agile BI developers know how to write good user stories for BI systems; how to make the stories small enough and simple enough to complete them in short iterations; how to prioritize stories; and estimate effort. By the way, Mike Cohn has written extensively about “user stories” among other agile topics. You should read Mike’s book for a deeper dive into this topic. My goal is to show how to do story-driven data warehouse development, and to help you wrestle with the common question, “How can we build anything meaningful in only two weeks?”

6 Responses to “Feature Driven Data Warehousing”