PODCAST
Filtering the BI Fire Hose

How can we make the most of Web data services for business intelligence (BI)? As enterprises seek to gain better insights into their markets, processes, and business development opportunities, they face a daunting challenge -- how to identify, gather, cleanse, and manage all of the relevant data and content being generated across the Web.

In Part 1 of our series we discussed how external data has grown in both volume and importance across internal Internet, social networks, portals, and applications in recent years. As the recession forces the need to identify and evaluate new revenue sources, businesses need to capture such web data services for their BI to work better and fuller.

Enterprises need to know what's going on and what's being said about their markets across those markets. They need to share those Web data service inferences quickly and easily across their internal users. The more relevant and useful content that enters into BI tools, the more powerful the BI outcomes -- especially as we look outside the enterprise for fast shifting trends and business opportunities.

In this podcast, Part 2 of the series with Kapow Technologies, we identify how BI and Web data services come together and explore such additional subjects as text analytics and cloud computing.

Listen to the podcast (40:32 minutes).

So, how to get started and how to affordably bring web data services to BI and business consumers as intelligence and insights? Here to help us explain the benefits of web data services and BI, is Jim Kobielus, senior analyst at Forrester Research. We're also joined by Stefan Andreasen, cofounder and chief technology officer at Kapow Technologies.

Jim, let's start with you. Let's take a look at what's going on in the wider BI field. Is it true that the more content you bring into BI the better, or are there trade-offs, and how do we manage those trade-offs?

Jim Kobielus: It's true that the more relevant content you bring into your analytic environment the better, in terms of having a single view or access in a unified fashion to all the information that might be relevant to any possible decision you might make within any business area. But clearly, there are lots of caveats, "gotchas," and trade-offs there.

One of these is that it becomes very expensive to discover, to capture, and to do all the relevant transformation, cleansing, storage, and delivery of all of that content. Obviously, from the point of view of laying in bandwidth, buying servers and implementing storage, it becomes very expensive, especially as you bring more unstructured information from your content management system (CMS) or various applications from desktops and from social networks.

So, the more information of various sorts that you bring into your BI or analytic environment, it becomes more expensive from a dollars-and-cents standpoint. It also becomes a real burden from the point of view of the end user, a consumer of this information. They are swamped. There's all manner of information.

If you don't implement your BI environment, your advanced analytic environment, or applications in a way that helps them to be more productive, they're just going to be swamped. They're not going to know what to do with it -- what's relevant or not relevant, what's the master reference, what's the golden record versus what's just pure noise?

So, there is that whole cost on productivity, if you don't bring together all these disparate sources in a unified way, and then package them up and deliver them in a way that feeds directly into decision processes throughout your organization, whether HR, finance or the like.

Dana Gardner: So, as we look outside the organization to gain insights into what market challenges organizations face and how they need to shift and track customer preferences, we need to be mindful that the fire hose can't just be turned on. We need to bring in some tools and technologies to help us get the right information and put it in a format that's consumable.

Kobielus: Yes, filter the fire hose. Filtering the fire hose is where this topic of Web data services for BI comes in. Web data services describes that end-to-end analytic information pipe-lining process. It's really a fire hose that you filter at various points, so that the end users turn on their tap and they're not blown away by a massive stream. Rather, it's a stream of liquid intelligence that is palatable and consumable.

Gardner: Stefan, from your perspective in working with customers, how wide and deep do they want to go when they look to web data services? What are we actually talking about in terms of the type of content?

Stefan Andreasen: Referring back to your original question, where you talk about whether we need more content, and whether that improves the analysis and results that analysts are getting, it's all about, as Jim also mentioned, the relevance and timeliness of the data.

There is a fire hose of data out there, but some of that data is flowing easily, but some of it might only be dripping and some might be inaccessible at all. Maybe I should explain the concept.

Think about it this way: The relevant data for your BI applications is located in various places. One is in your internal business applications. Another is your Software-as-a-Service (SaaS) business application, like Salesforce, etc. Others are at your business partners, your retailers, or your suppliers. Another one is at government. The last one is on the World Wide Web in those tens of millions of applications and data sources. There is very often some relevant information there.

Today, all of this data that I just described is more or less accessible in a Web browser. Web data services allow you to access all these data sources, using the interface that the Web browser is already using. It delivers that result in a real-time, relative and relevant way into SQL databases, directly into BI tools, or to even service enabled and encapsulated data. It delivers the benefits that IT can now better serve the analysts need for new data, which is almost always the case.

BI projects happen in two ways. One is that you make a completely new BI. You get a completely new BI system, and then make brand-new reports, and new data sources. That's the typical BI project.

What's even more important is that incremental daily improvement of existing reports. Analysts sit there, they find some new data source, they have their report, and they say, "It would be really good, if I could add this column of data to my report, maybe replace this data, or if I could get this amount of data in real-time rather than just once a week." So it's those kinds of improvements that Web data services also really can help with.