Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing
- by Ken W. Collier
Buy It!
- Safari Books Online
- Amazon
- Barnes & Noble
Archive for the ‘Agile Business Intelligence’ Category
Agile adoption for data warehouse and BI is on the rise. Agile Analytics can shorten development cycle time, improve quality, and help ensure that you build the right BI solutions for business decision makers. However, conventional IT organizational structures, policies, processes, and procedures are sometimes inconsistent with the tenets of agility. Values like customer collaboration, face-to-face interaction, and continuous delivery of value are often impeded by IT organizational protocols. Here are the top 10 points of friction that I frequently see in companies that I assist in Agile Analytics adoption. While many of these also impact software teams, DW/BI departments are often more directly impacted by them. Continue reading “Organizational Friction Impedes Agility” »
TDWI’s BI This Week newsletter ran this short article on June 22, 2011, and I’ve included it below. It provides a glimpse at the important technical practices that enable you to safely evolve your data models in ways that improve the design quality rather than degrading it. My book Agile Analytics has an entire chapter devoted to evolving excellent design, but this article provides an overview. I’d love your feedback on this piece.
The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.
What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.
Continue reading “Agile Data Modeling: Evolving Toward Excellence” »
Design excellence is critical to the success of Agile Analytics. The right design choices will help minimize technical debt, facilitate adapting to changes, improve quality, and provide the agile team with a cohesive technical framework. The wrong design choices can lead to overbuilt systems and high technical debt, severely hindering the teams’ ability to be agile. This applies to the design of data models, system architectures, ETL code, BI applications, and other components of the data warehouse and business intelligence solution. Agile DW/BI presents a difficult paradox – the ability to quickly respond to change and frequently deliver new features requires excellent data models and system design; yet excellent design takes time to develop. How do we deliver business value early and frequently without doing a lot of the design up-front?
A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.
The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:
- Site – The location where the building sits
- Structure – The foundation and frame
- Skin – The outer shell of the building
- Services – Water, electric, sewage, etc.
- Space – The interior layout and configuration
- Stuff – Lighting, colors, flooring, decor, etc.
The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.
While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:
- What is our design objective? (to improve understanding or to communicate the solution to others)
- Have we accomplished our objective yet? (i.e., have we done enough for now?)
- If so, what’s keeping us from getting started developing?
- If not, what is the smallest/simplest thing we can do to accomplish our objective?
Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.
Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier
In June, 2010 The Data Warehousing Institute (TDWI) published my article entitled “Being” Agile vs. “Doing” Agile as part of their “This Week in BI” series. Although I wrote the article for the Agile BI community, the concepts apply to any agile development organization. Please give it a read and let me know what you think.
(The following is a repost of my response to a question about outsourced BI and geographically distributed teams on LinkedIn’s TDWI BI and DW Discussion Forum. Here is the entire thread.)
I’ve been practicing and consulting on Agile BI since 2004 and have worked with teams that are “dislocated”. Since Agile BI values individuals and interactions as a more effective way to build the right thing, colocation is favored. However, outsourcing and geographic distribution are unavoidable realities in today’s business. Take a look at Scott Ambler’s recent blog about Agile and Geographic Distribution in which he discusses 3 key challenges caused by dislocation:
1. Communication – Just being in two separate buildings reduces the preference for face-to-face communication.
2. Temporal – The farther apart the time zones, the greater the impact on synchronous collaboration.
3. Cultural – Different work ethics, values, etc. affect the effectiveness of team interactions.
If your data warehouse server(s) burned up in a fire, how long would it take to redeploy the system into production?
If you discovered a critical defect in your production BI system, how long would it take to revert to a previous version while you resolve the problem?
Like other mission critical systems, it should take no more than a few days to rebuild your BI system from scratch – including reconfiguring new servers and reloading data. The actual redeployment of your warehouse implementation (database schemata, ETL scripts, BI applications, etc.) should range from minutes to hours, not days or weeks. Continue reading “BI System Version Control and Code Management” »
Remember that our goal in agile BI development is the frequent release of production quality working software for user feedback and acceptance. At the end of each iteration or sprint, and each release our working product is expected to be of shippable quality, even in its most embryonic stages. This objective requires an entirely different approach to our quality assurance methods. Foremost it means integrating QA efforts right into our iterations.
Traditional BI development methods push system and acceptance testing to the end of the project cycle. This backend testing is typically manually intensive, possibly supplemented by the use of semi-automated tools. We need an entirely different testing discipline for Agile BI development. Continue reading “Automated Data Warehouse Testing” »
Contrary to popular opinion, the best business intelligence systems are not driven by the data, or the operational source systems. I recently had a conversation with a group of data warehouse developers who were completely baffled by the notion of building a BI application without first extracting all of the source system data into a single, normalized data model.
I occasionally get asked to describe agile project failures and struggles. I haven’t formally studied root causes of failure but have worked with enough struggling agile teams to gain a qualitative sense of these causes. Agile struggles are commonly caused by non-agile behaviors masked behind agile trappings and terminology. Failure to collaborate is a common problem. People tend to revert to the asynchronous communication (e-mail and written documents); and “throw-it-over-the-wall” habits with which they’ve grown familiar. Continue reading “On Community, Customers, and Collaboration” »
So here is a summarization of the key characteristics of Agile BI. This is simply a high level glimpse at the key project traits that are the mark of agility; not an exhaustive list of practices. Moreover, Agile BI is a development style not a prescriptive methodology that tells you precisely what you must do and how you must do it. The dynamics of each project within each organization require practices that can be tailored appropriately to the environment. Remember, the primary objective is a high quality, high value, working BI system. These characteristics simply serve that goal:
I fear that, “agile as the latest magic bullet” has crossed the chasm, but that “agile as a different way of behaving” has not."
- Ken Collier
