I first met my good friend Jim Highsmith in 2003. Jim is a well-respected author, speaker, consultant, and a founding member of the Agile Alliance. In 2000, the Alliance collectively outlined a set of core values, principles, and practices for developing software that better meets users’ needs and expectations: The Agile Manifesto.
Because my roots are in software engineering, I was aware of the Agile Alliance during the early years. Since I had spent the previous five years leading business intelligence practices for other consultancies, my attention was largely focused on advancements in data warehousing and BI rather than in software engineering. Then I met Jim. I began to see the possible connection between Agile principles and data warehousing practices.
Jim and I, along with another colleague meet weekly to share coffee, experiences, and ideas. When we first started these weekly meetings, Jim was working on Agile Project Management, his latest book. He would talk about his APM principles as they unfolded and I would lament about a data warehouse project I was working on that was understaffed, over-scoped, and user requirements were a moving target. My development team was working long, hard hours and the users never seemed quite satisfied enough.
My “Aha! Experience” occurred during one of our weekly coffee meetings when it dawned on me that all of this Agile stuff that Jim and others have been talking and writing about have a direct application to data warehousing and business intelligence in general. Thus began my current journey, which I am currently referring to as Agile Business Intelligence for lack of a more creative title. I have envisioned a set of values, principles, and practices for infusing agility in all BI projects ranging from canned reporting, to data visualization, to data mining and quantitative analytics; and, of course, data warehousing. I am confident that these principles apply across the BI spectrum. I have used the Agile BI principles in practice, and I am very enthusiastic about the results.
Using Agile BI practices, I was able to lead three inexperienced (in data warehousing) developers to complete a data warehouse in 2.5 months. Our team didn’t pull a single all-nighter in the process! Although our resulting data warehouse will continue to mature over time, the complete end-to-end architecture is established and multiple OLAP reports are available for use. My friend Luke Hohmann describes Agile products in their early stages as being like a baby, they are complete but immature. This is a very effective way of thinking about Agile BI. Our goal is to get a working system in the hands of users as early as possible, so that we can start getting their feedback and mature the system.
In the spirit of full-disclosure, and with due respect to the Agile pioneers, I’m not certain there are any unique notions in my development of ADW. I have unabashedly “borrowed” ideas from the works of Scott Ambler, Kent Beck, Alistair Cockburn, Martin Fowler, Jim Highsmith, Luke Hohmann, Ken Schwaber, and others. My contribution is in adapting the ideas of these thought leaders to industry standard data warehousing best practices. Seasoned with my own knowledge and experience, and much coaching from Jim, my aim is to establish a practical, applicable, and more successful data warehouse development alternative.