“Current data warehouse projects simply take too long to produce value, says CEO Michael Whitehead. All too often projects are built on the assumption that they won’t need to change. Instead, Whitehead advocates delivering value quickly, thus winning the time and budget to continue to improve design.

In this interview, Whitehead shares his thoughts on current data warehouse design, agile development, and the best route to revamping a data warehouse. WhereScape, a data warehousing company, offers WhereScape 3D, a data warehouse planning tool, and WhereScape RED, an integrated development environment (IDE) for building, deploying, managing, and renovating data warehouses and data marts.”

The carpenter is a skilled craftsman who is adept at reading architectural plans and building what is prescribed. A good carpenter has a well-developed set of skills, high-quality tools, and the experience to build high-quality structures that will last.

The cabinet maker’s work is designed to be seen and must be visually appealing. The joints must appear seamless, and the finish flawless. A good cabinet maker works with the customer to design a functionally effective configuration and select styles, color, and appearance. Unlike the carpenter’s coarse tools, the cabinet maker’s tools are precise and delicate.

The furniture maker’s work must serve a specific purpose, but its actual design and appearance can vary widely. A chair might have arms or not, have a high or low back, be symmetrical or asymmetrical. An innovative furniture maker’s vision is not tightly bound by the appearance of the chair, only by its functionality. The artistic furniture maker is not directed by the customer, but instead measures success by how many customers buy his work.

All three are skilled craftspersons who possess the right skills, tools, and experience. Data warehouse, BI, and analytics practitioners have historically been most like the carpenter, building what the architects prescribe. This talk examines a blend of Lean Startup, Kanban, and agile techniques that offer the opportunity to work more like cabinet makers and furniture makers, infusing creativity, vision, and innovation into our results–-and measuring how well our customers like what we build.

A quick glance at Google Trends data confirms that Big Data–at least as a marketing phrase–is at its peak. Over the past year the phrase has skyrocketed in popularity, most likely thanks to searchers both curious about what Big Data means, and how they should approach data management when information has more volume, variety and velocity than ever before.

But analyst firm Gartner is quick to condition users that with great technology investment, comes great responsibility. Gartner found that almost three-quarters of Business Intelligence (BI) projects fail, and that only 30 percent of projects will have successfully aligned business objectives and measures by the end of next year. So how can organizations work to ensure their BI investment is as successful as possible?

I recently had the opportunity to chat with Loren Bast, Director of Business Intelligence at online humor network Cheezburger. Cheezburger invested in a new BI tool because it was unable to keep up with the amount of data it was producing across its popular humor websites such as I Can Has Cheezburger and FAIL Blog.

I asked Bast how he would have managed the project differently if he could start again. He said in an ideal world, he would have focused more on the following three things:

#1 Even with Big Data, the Small Stuff Matters — When Cheezburger introduces new data sources to its BI tools today, it ensures that the data is accurate and valid by comparing against legacy systems and closely looking at each feed before reporting is automated. But at first, this wasn’t always the case.

“We would spend a ton of time analyzing one month or three months’ worth of data, only to realize the data we were looking at wasn’t exactly what we were looking for,” says Bast.

From the beginning, organizations should invest in both people and technology to ensure that the data being analyzed is, in fact, accurate and as intended.

#2 Act as Internal Consultants When Preparing Reports — KPI-bloat was a real issue for the Cheezburger team when they started being inundated with report requests. But Bast and his department found value in not just selecting the best performance indicators, but working directly with product and engineering teams to ensure the reporting will lead to positive actions for the business.

“Now, we always ask if something valuable to the business is going to come out of the report,” says Bast. “If we can’t agree on a clear answer to that question, we just won’t build the report.”

#3 Be a Leader, Increase Buy-In — Finally, the last point Bast shared to me was a bit unexpected–when when he introduced it less than a year ago. The BI department at Cheezburger sends out daily reports on its data, but Bast had the idea to add a section at the top that focused more on the possible issues behind the data, and opened the floor for commentary from other stakeholders.

For an online business where things change daily, the new “Insanely Insightful Commentary” section is now one of the more important catalysts for action in the company today.  “Opening up the conversation gets the most productive chatter going,” says Bast. But this makes sense–BI is at it best when it provides action, not just analysis. Bast simply opened the door.

For Cheezburger, improving on these three areas has allowed the organization to become what Bast calls a “service group” of the Cheezburger business–helping not only analyze its data, but direct the organization to invest more time and resources into what’s working the most.

Michael Koploy is an analyst and managing editor for Software Advice, a company that reports on news and trends within enterprise software. You can read more about the Cheezburger interview here: Big BI Lessons from Cheezburger’s Notorious B.I.T. For more information, he can be reached at michael@softwareadvice.com.

As a proponent of agile data warehousing and business intelligence, I am constantly looking for new techniques for delivering value to customers faster, adapting to their feedback, and evolving toward the right business solutions (regardless of initial requirements). I recently read the new book, Lean Startup by Eric Ries (Ries, 2011) and it has rocked my world. In the short time since this book hit the shelves in September, 2011, it has exploded in popularity. Be forewarned, this book is about entrepreneurship and high-tech startups. It isn’t about data warehousing, BI, or analytics … or is it?

Most startup companies fail. Startups typically rely on a good plan, a solid strategy, and thorough market research. The problem is that startups operate with a high degree of uncertainty. They don’t yet know who their customers are or what their product should be. They can’t predict the future. “Planning and forecasting are only accurate when based on a long, stable operating history and a relatively static environment. Startups have neither,” Ries points out.

Corporate DW/BI/analytics initiatives have much of this same uncertainty. Our customers are highly diverse, from across the enterprise, acting in varying roles, and having rapidly changing goals. For many reasons they can’t accurately tell us what “products” they want, and they don’t fully understand what our BI systems can do. Moreover, our BI strategies and “market research” are based on yesterday’s business needs, which are in a constant state of change.

My clients routinely tell me stories about building dozens of BI reports based on customer spreadsheets or requests, only to discover that most of these are rarely or never used. Like startups, enterprise BI initiatives don’t have a long, stable operating history or a static environment.

Ries points out that “The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to pivot or persevere.” Lean Startup techniques follow a Build-Measure-Learn feedback cycle. This cycle begins with an idea or hypothesis immediately followed by building a minimal viable product (MVP). Customer response to this MVP is carefully measured and the resulting data provides the basis for learning and adjustment. The goal is to move through this cycle as fast as possible, and as many times as necessary to converge on the product that customers want.

This cycle is aimed at quickly building something, getting it in the hands of customers, and measuring their behaviors. “Instead of making complex plans that are based on a lot of assumptions, you can make constant adjustments with a steering wheel called the Build-Measure-Learn feedback loop.,” Ries explains. In this cycle, we start with assumptions, build the smallest/simplest product, test our assumptions, and decide whether to pivot or persevere.

Two critical elements of this cycle are the MVP and the validated learning that is based on scientific testing of customer acceptance. The MVP is the very smallest, fastest thing you can introduce to your customers to gauge their response. For a business intelligence “product,” this might be a disposable prototype report or dashboard mockup populated with snapshot data. For analytics, it might be a mockup of a scoring algorithm based on a rudimentary predictive model. It is the simplest version of what we think customers want, so that we can find out if our assumptions are correct.

It is essential to quickly get the MVP in front of your customers and scientifically measure their behaviors. Is there a growth in the number of business users who demand access to the new “product”? Is there a deepening in the usage of the new “product” by existing users? Designing the right metrics will provide you with the evidence you need to either persevere (continuing to invest in the maturity of the “product”) or pivot in favor of a different hypothesis. Designing the right metrics for validated learning is difficult but essential. Ries’ book provides a great set of guidelines for developing these measures that we can apply in our DW/BI/analytics environments.

Giving the Customer What They Want

Sometimes we accidentally build something that nobody wants, in which case it doesn’t matter if we do it on time and on budget. The goal of a startup is to figure out the right thing to build as quickly as possible. Agile BI development helps accomplish this by delivering a few working BI features every few weeks. Agile BI calls for a prioritized backlog of user stories, which are the smallest aspects of business value that we can get our customers to articulate. Agile BI calls for building production-quality features during every iteration and deploying them into production as often as it makes business sense to do so.

This is highly effective way of evolving toward the right solution, assuming you can get your customers to accurately articulate and prioritize their stories. Lean Startup techniques make no such assumptions. The question in Lean Startup is: How fast can we decide to pivot, and how many times must we pivot, so we can ensure that we are building what customers want?

Once we have correctly discovered what customers want, then we can use agile BI techniques to build, refine, and mature the solution. Ries describes this approach as “…killing things that don’t make sense fast and doubling down on the ones that do.” This theme makes as much sense for BI directors as for startup entrepreneurs.

Final Words

We must learn what customers really want, not what they say they want or what we think theyshould want. We must discover whether we are on a path that will lead to growing a sustainable data warehousing, business intelligence, or analytics program.

Data warehouse and BI program leaders are entrepreneurs within the enterprise. It is the job of these entrepreneurs to quickly determine which efforts are value-creating and which are wasteful. By fostering Lean Startup thinking within your BI department, you can effectively establish a pattern of learning and adapting, which is the essential measure of progress for startups. It’s time to start thinking of your DW/BI/Analytics initiative as a startup. Will it be a successful one or not?

References:

Eric Ries, The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. New York: Random House, 2011.

In honor of the holiday season, this month’s topic is focused on an all-too-familiar problem: the common hangover. No, I’m not speaking of an “adult beverage” hangover, although if I find a cure for that I’ll be sure to share it with you. The type of hangover I’m speaking of is the routine failure of your agile delivery team to complete the user stories it commits to during sprint planning.

I’m surprised at what a common occurrence this is for many agile analytics teams. Here are a few example scenarios:

1. The team completes development but testing hangs over into the next sprint.
2. The team completes the data integration elements of a story, but the demonstrable corresponding BI features hang over to the next sprint.
3. The story is too big to be completed in a single sprint, so the team just allows it to hang over into the next sprint.

There are other examples, but you get the idea. A hangover is not the same as a BI feature that is technically done, but the customer review calls for further modification or refinement (it isn’t “Done! done!”). A hangover is simply a failure to have the completed story ready for customer review at the end of the sprint.

Back in 1975, software engineer Fred Brooks observed that, “A project gets late one day at a time.”¹Similarly, an agile project gets late one iteration at a time, when teams fail to honor their sprint commitments. During project chartering/planning the entire project community speculates on an anticipated project outcome. While this outcome is subject to the routine fluxuations of requirements uncertainty, technical uncertainty, and risk, it is hindered by fluxuations caused by the failure of the team to meet shared expectations.

Agile development works in large part because we are continuously realigning the expectations of management stakeholders, customers, and delivery team members. These three constituencies collaborate during sprint planning to ensure that everyone has a shared understanding of the expected sprint outcome. They also collaborate at the end of every sprint to validate that the results are valuable and acceptable. Many times they collaborate during midsprint to provide guidance and clarity.

Delivery teams must accept that customers and product owners know what is important and can focus the team on the next most important stories. Customers and product owners must accept that the sprint outcome is constrained by the team’s limited capacity. However, when delivery teams fail to honor their end of the bargain or when the customer team puts unreasonable pressure on the delivery team, the trust relationship between customer community and delivery team erodes. The customer community and product owner can no longer rely on the team to meet their mutual expectations, and the process loses predictability and reliability.

The root of the problem is what Jim Highsmith refers to as “wish-based planning” instead of “capacity-based planning.”² The team succumbs to wishful thinking, overly optimistic estimating, and the pressure to do more than is realistic. Moreover, they fail to base their plans on previously demonstrated actual capacity. The problem snowballs when the team does this routinely sprint after sprint after sprint. Anticipated project outcomes are no longer valid, but they cannot be replaced with any degree of certainty.

So, now to the cure for the common hangover:

Ingredients Include

• Two parts acknowledgement
• One part self-discipline
• Two parts effective user story decomposition
• Two parts shared responsibility
• One part team honor
• Two parts celebration

Recipe

1. The customer team (including product owner) and the delivery team collaboratively acknowledge the symptoms of hangover. The delivery team acknowledges the undesirable consequences of wish-based planning and routine hangover. The customer team acknowledges any undue pressure placed on the team to plan beyond its capacity. Both teams recommit to the principle of continuous delivery of business value at a sustainable pace, and within capacity.
2. The delivery team explicitly recognizes that with the freedom of self-management comes the responsibility of self-discipline. Agile team self-discipline calls for moderation and self-control. The delivery team recommits to more thoughtful estimating and more realistic sprint planning. This involves pushing back on the pressure to overcommit, while still focusing on the delivery of high-valued, production quality working BI features.
3. Both the delivery team and the customer team seek to improve the art of user story decomposition. The customer team learns to see value in small, simple, immature, early working BI features rather than always expecting fully embellished and mature BI reports. The delivery team uses story-driven development to slice the architecture as thinly as possible, doing the minimal amount of data modeling, database development, ETL development, BI development, and so on, needed to complete the story. The delivery team does no more than is necessary for each story, and learns that each incremental change in the implementation is not rework, but instead is an opportunity to evolve a high quality DW/BI system.
4. The customer team takes responsibility for helping the delivery team plan within its capacity, and the delivery team takes responsibility for making realistic sprint commitments and keeping those commitments. Both teams take responsibility for the mutual goal of establishing a steady and effective cadence of successful sprints.
5. Despite Steps 1 through 4, the delivery team may occasionally still overcommit during sprint planning. When this happens, the delivery team must make every effort to honor those commitments, even if it means working at an unsustainable pace for a short period of time. Afterwards the team must seek to rebalance its planning for the next sprint to achieve a more realistic commitment.
6. Finally, every successful sprint should end in a celebration by both the customer team and the delivery team. When both teams get a taste for celebrating success rather than bemoaning failure, the agile project takes on a positive and energetic tone that carries forward into successive iterations. This practice builds a powerful camaraderie between customers and developers and a shared sense of purpose. But don’t celebrate too hard; that can cause hangovers of a different sort.

Finally, I wish each of you a happy New Year and continued growth and improvement in your agile analytics efforts. As agile luminary Mike Cohn says, “Agile is not something you become. It is something you become more of.” Best wishes on curing your project hangovers in the process of becoming more agile. I’ll see you in 2012.

Note

¹Brooks, F.J. Mythical Man Month: Essays on Software Engineering. Addison-Wesley, 1975.

²Highsmith, Jim. “A Recipe for Success, Part 3.” Cutter Agile Product & Project Management Advisor, 8 March 2007.

1. Conventional Leadership – Agile expert Jim Highsmith says, “Conventional leaders expect projects to be on-track early and off-track at the end. Agile leaders expect projects to off-track early and on-track at the end.” Agile leaders measure, monitor, and manage in ways that are very different from conventional leaders. Successful Agile adoption involves leadership training as well as technical and project management training. Significant friction occurs when organizational leaders maintain conventional mindsets, habits, and behaviors, while expecting project teams to “go agile”.
2. Failure to Limit Work in Progress – Friction occurs when IT leaders try to execute too many simultaneous projects. People are saddled with multiple assignments and their time is split between too many activities. This causes significant “context switching” resulting in diminished effectiveness of individuals and teams. The perception that parallel projects will achieve goals faster is false. Cycle times are shorter and results are better when teams focus on one project at a time. Agile leaders prioritize program portfolios and limit the number of parallel projects based on organizational capacity.
3. Shuffling of Team Members – When IT leaders treat project staffing as a matter of shifting resources between projects based on skillset demands, every project suffers. People are much more than resources with specialized skills. When people are dedicated to a project team for the duration, they naturally develop a sense of ownership of the project. When people are constantly shifted in and out of an Agile project, it is impossible for the team to establish a steady cadence and predictable velocity that can be used for effective iteration planning. Agile Analytics projects should be staffed with a dedicated, cross-functional, self-organizing team.
4. Functional “Resource Pools” – Friction occurs when developers, architects, modelers, testers, project managers, and others are organized into functional silos or “resource pools”. People are pulled from their functional pool as-needed and returned to the pool when their skill is no longer needed. This organizational structure further contributes to the problems previously discussed. While skills specialization is important, Agile projects benefit from cross-functional teams with a lots of collaboration and interaction. Specialists assigned to dedicated, cross-functional teams naturally broaden their skillsets and become more general-purpose. This serves to boost the performance of the Agile team. Agile leaders seek to deconstruct functional silos and encourage team cross-functionality.
5. Off-shoring – Agile Analytics favors team colocation, but can still be highly effective in distributed teams under the right conditions. Inhibitors to distributed agility include: large time zone differences, language barriers, and cultural mismatches. Off-shoring in places at opposite ends of the globe introduces all of these characteristics. When considering the benefits of off-shoring, be sure to also consider the cost of rework and the difficulty in team collaboration. Since geographical distribution unavoidable in most companies, Agile IT leaders seek to maximize team interaction by investing in the appropriate technologies and techniques to boost “virtual colocation” wherever possible.
6. IT Policies and Restrictions – Friction occurs when DW/BI developers do not have unfettered control over their development environments, such as when developers must submit data model change requests rather than making such changes directly in the development sandbox. Such restrictions create undue friction for Agile teams that must deliver business value frequently. Agile leaders seek to strike the best balance between important governance policies where it matters, and the freedom from such policies where it does not.
7. Shared Services – Many IT organizations have shared services groups to handle activities like deployment, administration, maintenance, etc. These groups are most familiar with conventional phased-sequential projects rather than iterative-incremental ones. Unlike conventional methods in which development is delayed until later in the project, Agile projects call for an immediate start of development. Therefore, support services are needed earlier and in a timely fashion. To prevent the mismatch between shared services’ conventional policies and the iterative needs of Agile teams, Agile IT leaders invest in appropriate Agile training for these ancillary support groups.
8. Agile is a Project Management Approach – Friction occurs when IT leaders send project managers to Agile training but fail to invest in similar training for delivery teams and management. Suddenly project managers must be the Agile trainers as well as the “enforcers of agility”. Delivery team members lack the context and fundamentals necessary to understand agility, and often resent the disruption that this new change causes. Managers retain conventional habits, which are often at odds with Agile tenets.
9. Agile is a Technical Method – Friction occurs when IT leaders perceive Agile Analytics as a technical methodology. Some IT leaders believe that if the delivery team receives Agile training, then they will be more productive and able to do more with less. Every project involves management stakeholders, business users, and delivery team members. It is nearly impossible to align the expectations of these constituencies when Agile adoption is relegated to the delivery team only. Everyone must have a shared understanding of Agile Analytics.
10. Minimal Customer Collaboration – Friction occurs when actual end users of the DW/BI system are not actively involved throughout the Agile Analytics project. Many DW/BI programs rely too heavily on business analysts to be the “voice of the customer”. This is caused by concerns about intruding on busy end users, and users unwilling to commit to frequent review and feedback on new BI features. If a project is worth doing, it is worth doing right – which requires the active participation of actual business users.

How does your IT organization stack up with respect to these points of friction? As an IT leader can you influence chances that will help reduce or eliminate these friction points? If you could only eliminate one of these friction points, which one would produce the greatest benefits to your Agile DW/BI teams? How soon can you get started?

The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.

What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.

Why is agile data modeling a good idea? To paraphrase Ron Jeffries, one of Extreme Programming’s founders, the best way to implement a DW/BI system is to implement less of it. The best way to have fewer defects in your DW/BI system is to have a smaller/simpler one. The problem with comprehensive up-front modeling is that you must design for all contingent requirements, both known and speculated. This inevitably results in an overdesigned model that costs more to implement, is costlier to maintain, is more likely to contain defects, and is more difficult to understand.

Agile data models are as simple as possible while being sufficiently detailed, accurate, and consistent. They also fulfill a well-understood purpose and provide positive value (i.e., their benefit outweighs the cost of keeping them updated).

How to Safely Evolve Data Models

These concerns about evolutionary modeling resulting unnecessary rework, unintended side effects, and design degradation are legitimate. Additionally, the prospect of making a data model change to a high-volume data warehouse in production can be scary. Agile data modeling calls for a new set of practices that enable the safe evolution of models, even those in production. I’ll summarize those practices here. Consider this list a brief introduction; each deserves a deeper study to gain proficiency.

Data Model Patterns: Data models evolve toward excellence when we take advantage of tried and proven designs. Design patterns enable us to benefit from mature solutions that have previously been developed. Effective application of patterns relies on familiarity and awareness of patterns catalogs, and the ability to use them appropriately and sparingly.

Michael Blaha’s Patterns of Data Modeling is the most recent catalog of patterns, but David Hay first started cataloging data model patterns in 1996 with his Data Model Patterns: Conventions of Thought, and followed in 2006 with Data Model Patterns: A Metadata Map. An extension of data modeling patterns is the adaptive data model (ADM), a generalized data model designed to accommodate multiple domains. The ADM has been successfully used in data warehouse design and I write about it in detail in the Cutter executive report, The Message Driven Warehouse.

Technical Debt Management: Data models evolve toward excellence when changes are easy to make due to low technical debt. Technical debt is common in data warehousing. It is the entropy that occurs in any system over time due to development shortcuts, suboptimal design choices, maintenance activities, and so on. Like financial debt, a little technical debt is acceptable as long as we monitor it and pay it back quickly. When technical debt accrues unabated, the cost of change becomes unacceptably high. Agile data modelers continuously identify, prioritize, and monitor technical debt in the data model, seeking to eliminate it to assist with fast response to new requirements.

Database Test Automation: Data models evolve toward excellence when we have continuous confidence that our ideas are working. Agile BI practitioners work in short iterations delivering business value every few weeks. We need confirmation that what we build in later iterations doesn’t break what we built in early ones. The only practical way to accomplish this is with an automated test suite. Automated database tests validate data structures, data content and quality, schema constraints and integrity, data derivations, and so on. We can run automated tests quickly and simply at any time to confirm that everything still works. Tests are added as data model changes are made, so the test suite grows alongside the model. I devote an entire chapter to this topic in my book, Agile Analytics.

Database Refactoring: Data models evolve toward excellence when we can safely make changes to the design, even if it is in production. Database refactoring is a technical discipline that enables the safe evolution of data models without breaking previously working features and components. In their book Refactoring Databases, Scott Ambler and Pramod Sadalage explain database refactoring as “… a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.” Refactoring combines automated regression testing with a change transition period to ensure that changes haven’t broken anything. The change transition period is a window of time during which the model revision lives alongside the former version to ensure that nothing breaks. Refactoring combined with test automation are the central disciplines for effectively evolving data models.

Recognizing Data Model Smells: Data models evolve toward excellence when we recognize where they need improvement. Experienced data modelers develop a nose for elegant designs … and stinky ones. Discovering smells in the model is an essential precursor to improving it. Smells may include smart keys, multipurpose columns and tables, data redundancy, very large tables, and so on. By learning to pay attention to these smells, you can focus your attention on possible problem areas and consider them as candidates for technical debt management.

Change Deployment: Data models evolve toward excellence when we can quickly deploy changes at any time without fear. Agile BI practitioners develop data model changes using techniques that safeguard production deployment. All data model changes are scripted and those scripts are kept under version control. Database schemas are versioned and scripts are developed to roll forward to the next version, and roll back to the previous version in case things don’t work as expected. Data migration scripts ensure no loss or corruption of production data. Everything is automated and tested carefully in preproduction. Automated deployment, automated testing, and database refactoring disciplines support frequent, fast and fearless deployments.

Take Small Steps: Data models evolve toward excellence as a series of small, easily understood changes. It’s easier to undo a sequence of little changes than one big, complicated one. A side benefit of agile BI is that short iterations force us to plan in small steps. Agile data modelers quickly learn to change only what is needed to support the BI features currently in development.

The Final Word

All of these techniques combined form a strong safety net for evolutionary data modeling. Moreover, these techniques aren’t exclusive to agile BI, they should be considered as modern data warehousing practices that should be among the skills of every data modeler and data warehouse practitioner — agile or otherwise.

Recommended Reading

Ambler, S. W. (2002). Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process. New York: John Wiley & Sons, Inc.

Ambler, S. W., & Sadalage, P. J. (2006). Refactoring Databases: Evolutionary Database Design. Boston: Addison Wesley.

Blaha, M. (2010). Patterns of Data Modeling. Boca Raton: CRC Press.

Collier, K. (2011). Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing. Boston: Addison-Wesley Professional.

Collier, K., & O’Leary, D. (2009). The Message Driven Warehouse. Cambridge: Cutter Consortium, Inc.

Hay, D. C. (1996). Data Model Patterns: Conventions of Thought. New York: Dorset House Publishing.

Hay, D. C. (2006). Data Model Patterns: A Metadata Map. San Francisco: Morgan Kaufman.

Longman, C. (2005, December 7). Data Warehousing Meeting – December 7, 2005. Retrieved November 16, 2008, from DAMA UK – Data Management Association:http://www.damauk.org/Building%20the%20adaptive%20data%20warehouse%20-%20Cliff%20Longman.pdf

A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.

The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:

1. Site – The location where the building sits
2. Structure – The foundation and frame
3. Skin – The outer shell of the building
4. Services – Water, electric, sewage, etc.
5. Space – The interior layout and configuration
6. Stuff – Lighting, colors, flooring, decor, etc.

The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.

While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:

• What is our design objective? (to improve understanding or to communicate the solution to others)
• Have we accomplished our objective yet? (i.e., have we done enough for now?)
• If so, what’s keeping us from getting started developing?
• If not, what is the smallest/simplest thing we can do to accomplish our objective?

Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.

Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier

One of the many things that agile leaders do is motivate teams. Dan Pink is the author of the recent best seller, Drive: The Surprising Truth About What Motivates Us. You may have seen Dan Pink’s “Motivation” presentation on Ted. This video went viral a couple of months ago. But, even more compelling is this artistic animation of Dan’s earlier talk. Give it a look, it’s well worth 10 minutes of your time.

It’s beginning to look like 2010 may be the year of agile business intelligence. I’ve been preaching about agile BI since I first started doing it in 2004. Although there are a few converts, the BI and data warehousing community hasn’t been fast to adopt. I’m suddenly hearing a lot more buzz in BI/DW circles about agile this year, including the agile theme of the upcoming TDWI World Conference in San Diego. I’m excited about this because I’ve personally experienced the joy of iteratively building BI systems in small increments with tons of collaboration with business experts and end users, and I want you to experience the same joy.

Since 2004 I’ve had one foot firmly planted in the agile software development community and the other in the data community. Agile seems to have “crossed the chasm” in the software community, and there is an interesting (and undesirable) tendency emerging that the BI/DW community can learn from. Some software organizations are falling into the trap of looking to agile as a prescriptive methodology that can be imposed on software engineering teams to make them faster and better. Guess what — it isn’t working!

It isn’t working for these organizations because they have fundamentally misunderstood agile. Agile is a style of working, not a methodology or process. Agile is founded on a shared set of core values including:

• Individuals and interactions
• Working software
• Customer collaboration
• Responding to change

(For more, see agilemanifesto.org.) Agile espouses a set of guiding principles including:

• The early and continuous delivery of value
• Welcoming changing requirements
• Frequent delivery of a working system
• Daily collaboration between developers and business experts
• Self-organizing teams of experts

among others. (To learn more, visit manifesto.org/principles.html).

Sure, there are methodologies such as Scrum, eXtreme Programming, and Lean that introduce practices that are consistent with these values and principles, but agile is inherently a way of “being” that involves the entire organization, not just development teams. The specific practices and processes are simply manifestations of agile behaviors. Agile’s core values and guiding principles give rise to good process, not the other way around. The great agile success stories (such as Google and Salesforce.com) reflect organizations that embrace this notion and have found agility to be very powerful. Organizations that adopt agile practices without embracing the values and principles often struggle.

As an agile consultant I get to work with many companies in various stages of agile transformation with varying corporate cultures and experiencing different challenges and successes. Through these experiences, success patterns and failure patterns emerge. I’ve begun referring to success patterns as being agile and failure patterns as doing agile. Doing agile refers to organizations that fail to move beyond the simpler trappings of agility: iterations, daily stand up meetings, and so on. Being agile refers to organizations with inherent values, behaviors, and guiding principles that exhibit the essence and spirit of agility: adaptive, evolutionary, value-driven, and quality-driven development. Organizations that are agile also do agile, but the inverse is not necessarily true. Many organizations are decidedly non-agile while still utilizing many agile practices.

Symptoms of Doing, Not Being

I was recently asked to help a struggling agile organization figure out why they weren’t seeing the benefits of their investment in agile adoption. Senior management in this company had expectations of improvements in quality, increased productivity, better customer satisfaction, and happier developers. Unfortunately, their progress was going backwards in every area. After some digging, I began to see some familiar anti-patterns emerging. These are the symptoms of an organization that is doing, but not being, agile.

Iron Triangle Planning: Agile BI projects deserve sufficient planning, but agile plans are projections, not promises. Experienced managers sometimes have difficulty breaking the habit of expecting a fixed price, fixed scope, and fixed schedule project plan. An agile plan reflects the teams’ best projections based only on the information that is currently available. As requirements change and uncertainty is uncovered, those projections are likely to become obsolete. Agile managers anticipate and adapt to this.

Management Styles Don’t Change: The best agile BI teams are self-organizing, self-managing, and self-responsible. This doesn’t mean that the role of management is subverted — instead, it changes. Managers are enablers, decision makers, and facilitators. They work with development teams to remove barriers and protect the team from unwanted outside pressures. Agile managers are effective at replacing command and control leadership styles with more collaborative ones.

Emphasis on Productivity: The promise of agile BI is the delivery of a high-value, high-quality working system, not increased productivity. Leaders who emphasize productivity are surprised when quality suffers and end users are unsatisfied with results. Yet, when developers are pressured to be more productive, they take shortcuts such as reduced testing and hurried workmanship. Conversely, when the emphasis is on high-value and high-quality BI features, user’s needs are often met after only 60 to 70 percent of the planned features are complete, effectively shortening the project cycle by shrinking the scope. Agile managers emphasize quality and value; they trust their teams to be productive.

Adapting to Change is Only Lip Service: Change is inevitable, especially in today’s climate. Unfortunately, embracing and adapting to change is not a normal part of human nature. We go to great lengths to control and limit changes. We establish change control boards and change management processes. Instead of controlling changes, these processes only made them more disruptive. Embracing change means seeking it out, inviting it in, and encouraging more of it to assure that we build the right solution for our customers. Agile managers are eager to add new requirements, eliminate unnecessary requirements, overhaul project plans, rearrange priorities, and even discard working BI features in order to respond to — and embrace — change.

Customer Collaboration is Short-Circuited: One of the four agile core values is customer collaboration. Unfortunately, effective customer collaboration is hard. Our customers are busy and hard to pin down. They sometimes tell us things that don’t make sense or won’t work. Customer collaboration cuts into development time, so we create roles such as product owner or business analyst to be the “voice of the customer.” It’s easier to do this, but it’s almost never as effective as real customer collaboration. BI customers have interesting stories to tell, and when developers get to hear these stories first-hand, they get a more holistic understanding of the BI system they are building – and they build it better. Agile leaders insist on frequent collaboration between BI development teams and end users, and they enable effective collaboration between these groups.

The Last Word

What is the lesson for us in the BI/DW community? Although agile BI developers are responsible for many agile engineering practices, the ultimate success of agility cuts across the entire organization. The management community, the customer community, the stakeholder community, and the development community are equally responsible for adopting agile attitudes and behaviors. Only then will an organization be agile rather than simply do agile.