by Ken Collier
This article is a reprint from ThoughtWorks’ Big Data Analytics Insights.
Agile Analytics is a blend of sophisticated analytics techniques, agile delivery methods and lean learning principles, but doing it well involves a bit more than that. I recently gave a talk in Singapore where I outlined the essential elements of effective agile analytics. I summarize these seven dimensions here, which accompanies this slide deck from my original talk.
Big Data. While I’ve grown cautious of this buzzword, it is important to acknowledge that today’s data is more diverse and messy than ever before. Agile analytics involves the use of both SQL (relational) and NoSQL technologies to create a polyglot persistence infrastructure for data management. The ability to align your data storage technology to the nature of your data is key to flexibility and speed.
Data Science. Creating true value from data requires going well beyond conventional BI reporting, which is important but rearward facing. Competitive disruption comes from predictive and prescriptive analytics (Data Science) and visual storytelling.
Agile Discovery. Agile analytics starts with a high value business goal and then chunks up the data science into tiny incremental goals that can be presented to stakeholders every few days. Each tiny analytical discovery can be reviewed and used by business decision makers and is a step toward achieving the bigger original goal.
Lean Learning. Each tiny business action or decision based on analysis is rigorously measured so that we can learn whether the analytical model has the intended consequence. This build-measure-learn cycle is key to ensuring that we are converging on the right outcome even if it diverges from the original goal.
Impact. Agile analytics is continuously focused on actionable, insightful, and disruptive knowledge discovery. It is very easy to find yourself doing interesting data analysis that is not necessarily impactful analysis. Delivering results every few days enables us to continuously evaluate the usefulness and impactfulness of the results.
Solutions Thinking. The best analytics are not simple answers to singular questions. Rather they are the complex heart of powerful solutions to challenging business problems. Examples like using machine learning on traffic flow data in real-time to reduce gridlock and traffic jams or using dynamic, real-time bioinformatics sensors embedded in a sports bra to detect breast cancer in women are at the pinnacle of such solutions thinking.
Ethics. High ethical standards are needed to endure the inevitable scrutiny that analytics undergo. Standards include ensuring personal privacy and enabling individuals to control their own data. They also include radical transparency by the organization conducting the analysis. Democratic access to available data is key as is the use of open data standards among other considerations.
Consider your analytics programs and processes against these disciplines. How well are you creating frequent and continuous value from the data today? How excited are your business stakeholders about the disruptive things they can do because of analytics? How well are your analytical models and discoveries core to innovative solutions to complex business problems?
What can you do to get better?
by Ken Collier and Lynn Winterboer
This article was first published in TDWI BI This Week on July 9, 2013
When quality and testing is moved up front on a project, everyone enjoys a higher quality project. Here’s how your team can be happier with an agile approach to testing, even before implementing key automation tools.
Rather than explain the well-understood theory and benefits of agile, let’s examine how agile testing works in the real world and see how agile business and delivery teams reap the benefits of the methodology and have more fun in the process.
If you’ve been following me, then you probably know that I consider Jim Highsmith my mentor. Jim is one of the 17 signers of the Agile Manifesto, a prolific author and blogger, and a truly original thinker. I first met Jim during my last (and horrifying) waterfall project. Jim gently encouraged me to think about how agile might apply to data warehousing. He also taught me a lot about freelance consulting and book writing.
Many of you know who Jim Highsmith is. But for those of you who don’t, Jim has more than 30 years of experience in IT and is the recipient of the 2005 Stevens Award for outstanding contributions to systems development. Jim is also the author of Adaptive Software Development (2000 Jolt Award winner), Agile Project Management: Creating Innovative Products, Agile Software Development Ecosystems and the just-released Adaptive Leadership: Accelerating Enterprise Agility.
In Adaptive Leadership, Highsmith has compiled, updated, and extended his best writings about agile and lean methods for a management audience. Key themes include:
- Drawing on what’s been learned in application development, this guide shows how to use adaptive leadership techniques to transform the way you deliver complete solutions, whatever form they take.
- How enterprise agility can enable the ambitious organizational missions that matter most.
- How leaders can deliver a continuous stream of value.
- How to think disruptively about opportunities and how to respond quickly by creating more adaptive, innovative organizations.
- Integrating economics, products, and social responsibility.
- Understanding the financial implications of technical debt.
“We are at a tipping point. Technology—cloud, big data, mobility, social media—tops CEOs’ list of concerns per a recent IBM study. Companies that emerge successfully from this havoc will need to build agility into the very fabric of their organizations and develop the technological savvy to enable that agility. Adaptive Leadership provides a framework for such a transformation,” Highsmith said.
Read more in a recent Q&A with Jim on the ThoughtWorks Insights page where he talks about the evolution of agile and the role of senior management in an enterprise-wide Agile transformation.
Check out ThoughtWorks’ resources on Agile Project Management.
By Lynn Winterboer and Ken Collier (first published in TDWI BI This Week on July 30, 2013)
Agile leaders focus more on the continuous delivery of high-priority value to business stakeholders and less on making sure each worker is always busy. We explain how your BI program will benefit by limiting developer multi-tasking.
Ben and Susan are each a DW/BI director at their respective companies; each leads a department of roughly the same size. Both have the staffing necessary to organize three dedicated, cross-functional teams. Ben’s department is organized into dedicated teams while Susan has functional skills “pools” from which project teams are staffed. Both directors are planning for the coming year and have a similar set of 12 projects to schedule.
Through collaboration with business stakeholders, product owners, project managers, and the program management office (PMO), Ben has a prioritized backlog of funded project requests to schedule into the calendar. With help from technical leaders, he has completed some high-level project sizing and assigned durations of two, three, or four months per project. Projects longer than four months are divided into multiple phases, each designed to fit into one of these smaller time periods. This ensures that long-running projects do not block other projects by dominating the department’s capacity.
Ben assigns a dedicated team to each of the first three projects in the backlog. He then estimates when each team might be free to start its next project, continuing in this manner until he has a reasonable program plan for the coming year:
Startups know that what customers say they want isn’t necessarily what they want. The same principle is critical to the success of your BI project.
As a proponent of agile data warehousing and business intelligence, I am constantly looking for new techniques for delivering value to customers faster, adapting to their feedback, and evolving toward the right business solutions (regardless of initial requirements). The recent book, Lean Startup by Eric Ries (Ries, 2011) and it has rocked my world. In the short time since this book hit the shelves in September, 2011, it has exploded in popularity. Be forewarned, this book is about entrepreneurship and high-tech startups. It isn’t about data warehousing, BI, or analytics … or is it?
It’s been said that agile BI is simple but not easy. That is, the concept of working in short iterations, delivering your BI system in small increments, and evolving the solution based on feedback is easy enough for most people to understand. However, as with many things, the devil is in the details.
At its core, agile BI requires a fundamental shift in attitude, mindset, and culture—motivated by the core values and guiding principles outlined in the Agile Manifesto. This mindset shift is supported by a set of concrete practices and disciplines tailored to make up the agile BI methodology that best fits your organization. In my work with dozens of BI teams attempting to “go agile,” I’ve seen lots of mistakes. This Ten Mistakes to Avoid outlines the most common and recurring errors to help you avoid repeating the mistakes of others.
- Believing agile is just a project management thing.
- Thinking agile is just a developer thing.
- Confusing agile with Scrum.
- Failing to cultivate agile mindsets and attitudes.
- Using wish-based planning.
- Over-relying on specialized roles.
- Limiting customer collaboration.
- Not embracing adaptive scoping.
- Measuring the wrong things.
- Cherry picking agile practices.
It has been far too long since my last post, and frankly I’ve been up to my ears in alligators in my new role as director of Agile Analytics at ThoughtWorks and haven’t had enough time to write. So, if you’re still out there following this blog, I appreciate your patience. I have a lot of topics in the queue to write about and I’ll be making that a higher priority in the coming months.
Meanwhile, I’d like to tell you about a wonderful experience I had in June in Melbourne, Victoria, Australia. Over the months prior to my trip I had virtually met Em Campbell-Pretty (@prettyagile) on the Agile Data Warehousing LinkedIn forum. Em leads the data warehousing group at Telstra, one of Australia’s largest telecommunications and media companies. I can’t quite figure out whether to refer to Em as the DW director or manager. You see she is the epitome of a servant leader, and titles don’t seem to matter much to her. I know this because I got to meet Em in person when she invited me to visit Telstra’s data warehousing group when I was in Melbourne. I got to see what they were doing and participate in a discussion about what is working well and where challenges lie. The following week Em and I were both speaking at the Agile Australia 2013 conference where we became even better friends.
I was very impressed by what I saw and learned while I was there. This is probably the most well-scaled Agile Data Warehousing group I’ve ever observed, and they are still getting better. Em’s group is composed of 6 DBT (design, build, test) teams working across approximately 24 separate projects as well as an offshore product support & maintenance team – all working on a shared code base. That’s a lot of moving parts that need to be in sync. They are using aspects of Scrum, XP, Kanban, and Lean development, and they have been very successful at adapting these methods to data warehousing and business intelligence and it is working very well.
One of the most notable techniques Em’s group has adopted is the Scaled Agile Framework (SAFe) developed by Dean Leffingwell and described in Dean’s book, Scaling Software Agility. Now Dean is not a data warehousing guy but, like many great agile techniques, his SAFe adapts very nicely to the complexities of data warehousing programs that consist of multiple teams sharing the same code-base. I urge you to read Em’s recent blog post, A Perspective on the Scaled Agile Framework, which summarizes her teams’ journey toward the effective scaling of agile data warehousing and BI. It’s working extremely well at Telstra, and it might just work for you.
And while you’re on Em’s blog site, read her other posts about her teams’ agile DW journey. It’s a good story. I look forward to my next trip to Melbourne.
Check out this TDWI BI This Week interview of Michael Whitehead by Linda Briggs. Michael is a friend of mine and the very astute CEO of Wherescape, a company that produces tools that enable agile data warehousing. Michael’s comments reflect much of what I mean when I talk about “being agile” rather than “doing agile”. Here is the opening teaser…
“Current data warehouse projects simply take too long to produce value, says CEO Michael Whitehead. All too often projects are built on the assumption that they won’t need to change. Instead, Whitehead advocates delivering value quickly, thus winning the time and budget to continue to improve design.
In this interview, Whitehead shares his thoughts on current data warehouse design, agile development, and the best route to revamping a data warehouse. WhereScape, a data warehousing company, offers WhereScape 3D, a data warehouse planning tool, and WhereScape RED, an integrated development environment (IDE) for building, deploying, managing, and renovating data warehouses and data marts.”
In September, 2012 I had the honor of presenting the keynote address at The Data Warehousing Institute (TDWI) World Conference in Boston. Here is the abstract and video recording of that presentation. I hope you enjoy it.
The carpenter is a skilled craftsman who is adept at reading architectural plans and building what is prescribed. A good carpenter has a well-developed set of skills, high-quality tools, and the experience to build high-quality structures that will last.
The cabinet maker’s work is designed to be seen and must be visually appealing. The joints must appear seamless, and the finish flawless. A good cabinet maker works with the customer to design a functionally effective configuration and select styles, color, and appearance. Unlike the carpenter’s coarse tools, the cabinet maker’s tools are precise and delicate.
The furniture maker’s work must serve a specific purpose, but its actual design and appearance can vary widely. A chair might have arms or not, have a high or low back, be symmetrical or asymmetrical. An innovative furniture maker’s vision is not tightly bound by the appearance of the chair, only by its functionality. The artistic furniture maker is not directed by the customer, but instead measures success by how many customers buy his work.
All three are skilled craftspersons who possess the right skills, tools, and experience. Data warehouse, BI, and analytics practitioners have historically been most like the carpenter, building what the architects prescribe. This talk examines a blend of Lean Startup, Kanban, and agile techniques that offer the opportunity to work more like cabinet makers and furniture makers, infusing creativity, vision, and innovation into our results–-and measuring how well our customers like what we build.
The following is a guest post written by Michael Koploy, an analyst for Software Advice. This post is a summary of a more in-depth interview that Michael conducted with Loren Bast, Director of Business Intelligence at Cheezburger.com. I always love hearing what’s working for BI directors, and I’m never surprised when the things that work have the essence of agility. The Cheezburger story has these characteristics. Please let me know how you like this guest post. If it’s sufficiently popular, I will consider more of these in the future. Cheers, Ken
A quick glance at Google Trends data confirms that Big Data–at least as a marketing phrase–is at its peak. Over the past year the phrase has skyrocketed in popularity, most likely thanks to searchers both curious about what Big Data means, and how they should approach data management when information has more volume, variety and velocity than ever before.
But analyst firm Gartner is quick to condition users that with great technology investment, comes great responsibility. Gartner found that almost three-quarters of Business Intelligence (BI) projects fail, and that only 30 percent of projects will have successfully aligned business objectives and measures by the end of next year. So how can organizations work to ensure their BI investment is as successful as possible?
I recently had the opportunity to chat with Loren Bast, Director of Business Intelligence at online humor network Cheezburger. Cheezburger invested in a new BI tool because it was unable to keep up with the amount of data it was producing across its popular humor websites such as I Can Has Cheezburger and FAIL Blog.