A new (data) world order

23 billion dollars of shadow IT

When you think of shadow IT, what comes to mind? Your neighbor’s son running your company’s website? The flash drive that you used to get your latest sales figures? An online app that you paid for with your credit card because the one sanctioned by your company doesn’t work most of the time? Or a Dropbox account that your department uses to exchange customer information?

In layman’s terms, shadow IT is the result of businesspeople solving their problems outside of the IT organization. It’s the business adopting tools to get stuff done and it all sounds very pragmatic and fairly insignificant.

But what if I told you that there is a major, 23 billion dollar industry that is completely based on the notion that everything that comes from IT is completely wrong? It is an industry that relies on a network of smart MBAs with flash drives and Dropbox accounts full of Excel and CSV files. It depends on desktop tools paid by personal credit cards and boutique consulting services paid from departmental budgets.

And yet it is through these shadowy attributes that we accomplish anything with business intelligence tools today.

How did we get here?

A decade ago, if a CMO wanted to learn more about their customer’s journey, they needed to submit a request to the IT team and wait for a flip-flop-wearing data warehouse specialist to extract meaning and insights. That meaning, however, was based on IT’s definition of the customer — and no one else’s. And given that their definition had to be the same for the whole company, no one was ever happy with the results.

So instead of relying on IT, the CMO decided that it made much more sense to work with their own definition of the customer — and therefore, with their own BI tool. And thus the modern Business Intelligence industry was born.

What is the downside?

The value in taking BI outside of IT lies in the flexibility. But for the company as a whole, that flexibility proves to be an inhibitor. The widespread usage of desktop visualization tools results in hundreds or even thousands of versions of the truth — everyone defines data in different ways, and discussions end up having different meanings, resulting in faulty decision making.

As of today, IT has no control over business intelligence. Governance is non-existent, as is trust in data and its resulting insights. As with traditional shadow IT, shadow BI is fragmented and siloed; each department has its own tools, and those tools don’t agree or work well together.

The ticking bomb

Data is business critical, now more than ever. Companies like Netflix, StitchFix, and John Deere show that data analytics are a means to competitive advantage, and every company has increasing amounts of data. The difference is in how they manage it.

With the privacy regulations that have taken hold over the last decade, data management is no longer up to interpretation. One wrong move (like storing sensitive data in a spreadsheet) could cause monumental chaos for your company. The need for governance will only grow, and the consequences of mismanagement will only worsen with time. In the same vein, the external threat landscape is evolving in nature as well, and individual criminals can adapt more quickly than the average small business or enterprise.

Urgency needs to run high — private data sets moonlighting as business intelligence are a ticking time bomb. It’s only a matter of time before something goes wrong: a phishing attack on an unsuspecting employee could result in a fine worth 4% of your annual global revenue.

The data value chain will realign

As I wrote in a previous post, Snowflake’s record-breaking IPO has signaled a shift in the data industry ecosystem: the data value chain is evolving and this realignment will prompt the transformation of business intelligence as we know it. The new data value chain must balance the need for governance and flexibility.

Snowflake’s success may serve as a catalyst for this change — when companies have flexible, modern, and standardized places to store, manage and govern data, the need for the shadow BI will decrease.

The consumption layer

The availability of trusted enterprise cloud data doesn’t solve the problem of data consumption: not enough people use data to make decisions. While the last generation of BI tools focused on the avoidance of any type of governance, the next generation of BI tools will have to focus on making governed cloud data and insights 100% pervasive and accessible. The next generation of BI will be a consumption layer for insights that are embedded into every decision. Strategic or tactical. Global or local. Headquarters or field. Every single decision will be based on data.

The pendulum is swinging back

Early business intelligence efforts were run entirely by IT, while today’s BI is a free-for-all; we swung the pendulum from one side to the other, and neither works. The answer is somewhere in the middle. This is why the future of analytics must allow everyone to leverage data in the cloud to build their own personalized dashboards, create insight applications, and most importantly, to share and collaborate.

The future of data depends on collaboration between IT and business teams. IT will focus on governance, consistent business rules, data asset development, accelerated response and real time data, and businesses will have the freedom to innovate with low-code/no-code analytics tools. In this new world, everyone wins.

Cover image by Tom Barrett on Unsplash.