Author Archive

Check out this TDWI BI This Week interview of Michael Whitehead by Linda Briggs. Michael is a friend of mine and the very astute CEO of Wherescape, a company that produces tools that enable agile data warehousing. Michael’s comments reflect much of what I mean when I talk about “being agile” rather than “doing agile”. Here is the opening teaser…

“Current data warehouse projects simply take too long to produce value, says CEO Michael Whitehead. All too often projects are built on the assumption that they won’t need to change. Instead, Whitehead advocates delivering value quickly, thus winning the time and budget to continue to improve design.

In this interview, Whitehead shares his thoughts on current data warehouse design, agile development, and the best route to revamping a data warehouse. WhereScape, a data warehousing company, offers WhereScape 3D, a data warehouse planning tool, and WhereScape RED, an integrated development environment (IDE) for building, deploying, managing, and renovating data warehouses and data marts.”

In September, 2012 I had the honor of presenting the keynote address at The Data Warehousing Institute (TDWI) World Conference in Boston. Here is the abstract and video recording of that presentation. I hope you enjoy it.

The carpenter is a skilled craftsman who is adept at reading architectural plans and building what is prescribed. A good carpenter has a well-developed set of skills, high-quality tools, and the experience to build high-quality structures that will last.

The cabinet maker’s work is designed to be seen and must be visually appealing. The joints must appear seamless, and the finish flawless. A good cabinet maker works with the customer to design a functionally effective configuration and select styles, color, and appearance. Unlike the carpenter’s coarse tools, the cabinet maker’s tools are precise and delicate.

The furniture maker’s work must serve a specific purpose, but its actual design and appearance can vary widely. A chair might have arms or not, have a high or low back, be symmetrical or asymmetrical. An innovative furniture maker’s vision is not tightly bound by the appearance of the chair, only by its functionality. The artistic furniture maker is not directed by the customer, but instead measures success by how many customers buy his work.

All three are skilled craftspersons who possess the right skills, tools, and experience. Data warehouse, BI, and analytics practitioners have historically been most like the carpenter, building what the architects prescribe. This talk examines a blend of Lean Startup, Kanban, and agile techniques that offer the opportunity to work more like cabinet makers and furniture makers, infusing creativity, vision, and innovation into our results–-and measuring how well our customers like what we build.

The following is a guest post written by Michael Koploy, an analyst for Software Advice. This post is a summary of a more in-depth interview that Michael conducted with Loren Bast, Director of Business Intelligence at Cheezburger.com. I always love hearing what’s working for BI directors, and I’m never surprised when the things that work have the essence of agility. The Cheezburger story has these characteristics. Please let me know how you like this guest post. If it’s sufficiently popular, I will consider more of these in the future. Cheers, Ken

A quick glance at Google Trends data confirms that Big Data–at least as a marketing phrase–is at its peak. Over the past year the phrase has skyrocketed in popularity, most likely thanks to searchers both curious about what Big Data means, and how they should approach data management when information has more volume, variety and velocity than ever before.

But analyst firm Gartner is quick to condition users that with great technology investment, comes great responsibility. Gartner found that almost three-quarters of Business Intelligence (BI) projects fail, and that only 30 percent of projects will have successfully aligned business objectives and measures by the end of next year. So how can organizations work to ensure their BI investment is as successful as possible?

I recently had the opportunity to chat with Loren Bast, Director of Business Intelligence at online humor network Cheezburger. Cheezburger invested in a new BI tool because it was unable to keep up with the amount of data it was producing across its popular humor websites such as I Can Has Cheezburger and FAIL Blog.

Continue reading “The BI McTriple: 3 Business Intelligence Lessons from Cheezburger” »

The following article was originally published by TDWI in the May 15, 2012 issue of BI This Week.

As a proponent of agile data warehousing and business intelligence, I am constantly looking for new techniques for delivering value to customers faster, adapting to their feedback, and evolving toward the right business solutions (regardless of initial requirements). I recently read the new book, Lean Startup by Eric Ries (Ries, 2011) and it has rocked my world. In the short time since this book hit the shelves in September, 2011, it has exploded in popularity. Be forewarned, this book is about entrepreneurship and high-tech startups. It isn’t about data warehousing, BI, or analytics … or is it?

Continue reading “Treating Your BI Initiative Like an Entrepreneurial Startup” »

The following post was originally published by the Cutter Consortium as an e-mail advisor for the Data Insight and Social BI practice  in December, 2011. Hence, the reference to the holiday season. That publication can be found here.

In honor of the holiday season, this month’s topic is focused on an all-too-familiar problem: the common hangover. No, I’m not speaking of an “adult beverage” hangover, although if I find a cure for that I’ll be sure to share it with you. The type of hangover I’m speaking of is the routine failure of your agile delivery team to complete the user stories it commits to during sprint planning.

I’m surprised at what a common occurrence this is for many agile analytics teams. Here are a few example scenarios:

1. The team completes development but testing hangs over into the next sprint.
2. The team completes the data integration elements of a story, but the demonstrable corresponding BI features hang over to the next sprint.
3. The story is too big to be completed in a single sprint, so the team just allows it to hang over into the next sprint.

There are other examples, but you get the idea. A hangover is not the same as a BI feature that is technically done, but the customer review calls for further modification or refinement (it isn’t “Done! done!”). A hangover is simply a failure to have the completed story ready for customer review at the end of the sprint. Continue reading “Curing the Common Hangover” »

Agile adoption for data warehouse and BI is on the rise. Agile Analytics can shorten development cycle time, improve quality, and help ensure that you build the right BI solutions for business decision makers. However, conventional IT organizational structures, policies, processes, and procedures are sometimes inconsistent with the tenets of agility. Values like customer collaboration, face-to-face interaction, and continuous delivery of value are often impeded by IT organizational protocols. Here are the top 10 points of friction that I frequently see in companies that I assist in Agile Analytics adoption. While many of these also impact software teams, DW/BI departments are often more directly impacted by them. Continue reading “Organizational Friction Impedes Agility” »

TDWI’s BI This Week newsletter ran this short article on June 22, 2011, and I’ve included it below. It provides a glimpse at the important technical practices that enable you to safely evolve your data models in ways that improve the design quality rather than degrading it. My book Agile Analytics has an entire chapter devoted to evolving excellent design, but this article provides an overview. I’d love your feedback on this piece.

The number one resistance I encounter helping organizations adopt Agile BI is from technical architects and data modelers who say something like, “Agile makes sense as long as the data models are developed and settled.” Modelers often incorrectly believe that data models must be designed, developed, and locked down before building applications that use the data. The idea of evolving a data model incrementally can strike fear in the hearts of modelers and architects. Sometimes it is a fear of rework; data modelers prefer the goal of getting it right once. Other times it is the fear of unintended side effects or the risk of creating a spaghetti mess.

What’s agile data modeling? Agile modeling calls for a minimally sufficient design up front to establish a reference model that guides the delivery team’s incremental development activities. Aspects of the logical and physical models are completed just in time to support the BI features under development. Agile modelers avoid detailing aspects of the model that aren’t immediately needed. Combined with good data modeling discipline, this style produces the right data model for its intended purpose. The model evolves to support future requirements as those become reality. Scott Ambler’s book Agile Modeling (see recommended reading at end of article) covers agile modeling principles and practices in-depth.

Continue reading “Agile Data Modeling: Evolving Toward Excellence” »

Design excellence is critical to the success of Agile Analytics. The right design choices will help minimize technical debt, facilitate adapting to changes, improve quality, and provide the agile team with a cohesive technical framework. The wrong design choices can lead to overbuilt systems and high technical debt, severely hindering the teams’ ability to be agile. This applies to the design of data models, system architectures, ETL code, BI applications, and other components of the data warehouse and business intelligence solution. Agile DW/BI presents a difficult paradox – the ability to quickly respond to change and frequently deliver new features requires excellent data models and system design; yet excellent design takes time to develop. How do we deliver business value early and frequently without doing a lot of the design up-front?

A common misconception among agile critics is that agile development involves zero design up front, and therefore has a high risk of resulting in a poorly designed product. Conversely, agilists dislike the big design up front (BDUF) nature of plan-driven development, preferring instead to begin building something sooner to which customers can evaluate and react. Uncertainty early in a project makes BDUF too costly and risky. However, experienced agile developers also know that “No design upfront” leads to poor quality and high technical debt. What is needed is sufficient design up front design (SDUF) – enough to galvanize developers around a shared understanding of problem domain, architecture, user experience, and data. Agile development doesn’t require a whole new set of modeling techniques. What is required is a new way of applying good modeling methods in an incremental, iterative, and evolutionary manner. Establishing a minimally sufficient conceptual model up-front, and then incrementally evolving the physical model as the system is built helps limit technical debt, and increase design quality. Good evolutionary design requires team discipline, design expertise, and technical excellence. In other words Agile DW/BI is not a magic alternative to proper training, techniques, and experience.

The goal of agile design and modeling is to strike the right balance between too little and too much. Our objective is to model just enough up front to ensure that all developers have a shared understanding of the solutions approach, and can commence with building the working components in a common and cohesive way. We can take a lesson from Stewart Brand’s observations in How Buildings Learn: What Happens After They’re Built (Brand 1995). Brand identifies six layers that exist in any building:

1. Site – The location where the building sits
2. Structure – The foundation and frame
3. Skin – The outer shell of the building
4. Services – Water, electric, sewage, etc.
5. Space – The interior layout and configuration
6. Stuff – Lighting, colors, flooring, decor, etc.

The order of this list of layers is important. Each successive layer is increasingly easier, and less costly to change than the one before it, with site being the hardest to change and stuff being easiest. Like buildings, systems have these layers as well. The underlying hardware and technology infrastructure is much like the site; the systems architecture is the structure; and so on up to the look and feel of BI applications, which is the stuff.

While it is not impossible the change a DW/BI system’s infrastructure or systems architecture after it’s built, it is difficult and costly. Therefore, it is important to get these layers right as early as possible. Note that getting it right is not the same as getting it finished. In other words, we need to design these layers to a sufficient level of detail to convince ourselves that our design choices are viable, sustainable, robust, scalable, flexible, etc. We do not need a complete and comprehensive detailed design before we can start building the warehouse. During the early stages of design on a new project, before development has started I like to continuously ask the following questions:

• What is our design objective? (to improve understanding or to communicate the solution to others)
• Have we accomplished our objective yet? (i.e., have we done enough for now?)
• If so, what’s keeping us from getting started developing?
• If not, what is the smallest/simplest thing we can do to accomplish our objective?

Continuously asking this sequence of questions will help the Agile DW/BI team avoid the temptation to spend too much time doing up-front design while helping ensure that they don’t get started developing without the important prerequisite design decisions.

Excerpted from forthcoming book – Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier

Agile leadership is sometimes portrayed as, “Buy pizza and get out of the way.” Or, “Empowering teams to be more self-organizing and self-managing.” Neither of these descriptions is particularly accurate. Agile leaders have a uniquely challenging role. In fact, in many ways command and control, “my way or the highway” leadership is much simpler. The manager gives direction and the minions follow it. However, effective agile leadership results in much higher performance, greater innovation, and exciting new ideas.

One of the many things that agile leaders do is motivate teams. Dan Pink is the author of the recent best seller, Drive: The Surprising Truth About What Motivates Us. You may have seen Dan Pink’s “Motivation” presentation on Ted. This video went viral a couple of months ago. But, even more compelling is this artistic animation of Dan’s earlier talk. Give it a look, it’s well worth 10 minutes of your time.

In June, 2010 The Data Warehousing Institute (TDWI) published my article entitled “Being” Agile vs. “Doing” Agile as part of their “This Week in BI” series. Although I wrote the article for the Agile BI community, the concepts apply to any agile development organization. Please give it a read and let me know what you think.

Continue reading ““Being” Agile vs. “Doing” Agile” »