Business Intelligence and the Semantic Web

Analytics strategist Seth Grimes was in town last week speaking in Covent Garden on the subject of Web 3.0. For those still catching up with what this means then the evolution of the web is generally thought of as;

Web 1.0		Retronym which refers to the web as largely a publishing paradigm
Web 2.0		The interactive web characterised by the rise of social networking
Web 3.0		The semantic web. Functionally rich and understandable to machines as well as people
Die Hard 4.0		Fourth instalment of the Bruce Willis franchise released in the US as Live Free or Die Hard

Whilst web 3.0 is some way off there are some early glimpses into the world of possibilities with search engine innovations from Google and Wolfram Alpha. Fellow BI blogger, Peter Thomas who also writes about this subject in his blog Literary Calculus uses the example http://www.google.co.uk/search?&q=age+of+the+pope.

What interests me about the semantic web though, has much less to do with what might be described as contextually aware searching and more to do with the impact on Business Intelligence.

BI, since the 1970's, has been almost entirely focused on numbers. This is understandable given that these tend to be highly structured, organised and available in databases. Arguably though, this wasn't the original vision. In his 1958 (yes, 1958) article "A Business Intelligence System", IBM visionary, Hans Peter Luhn describes the objective as "to supply suitable information to support specific activities carried out by individuals, groups, departments, divisions.." and what Luhn goes on to describe is statistical analysis of text and documents as information sources.

Seth Grimes, in his Intelligent Enterprises article BI at 50 Turns Back to the Future makes the point that BI has more latterly revisited this original vision on the analysis of text and document sources which, after all, reflects the vast bulk of corporate data.

Early innovators, including my own company Artesian, are already making progress in the field of analysing documents for the purpose of business intelligence. This new breed of semantically aware business intelligence technology can "supply suitable information" to "support activity" by answering questions like;

• 'which of my competitors are growing and which are declining?'
  
• 'are my customers launching initiatives that could be supported by our products or services?'
  
• 'do market behaviours indicate a declining need for our products?'
  
• 'how did customers respond to our competitors when they changed their business in a way that we are also considering?'
  

This isn't to suggest that semantically aware BI should function like search engines. Indeed, I would strongly argue that it should not. Search engines deliver a single set of answers to a single ad-hoc question. Business processes are much more frequent, diverse, repeatable and involve wider audiences. Further, businesses require scalability and high degrees of automation. Here, the business questions need to be regularly monitored, visualised, distributed, shared and collaborated around. Interestingly this has more in common with today's best practice BI systems. So there it is. The shape of semantically aware BI is emerging through the fog of future developments and it is unsurprisingly an evolution of what is best about business intelligence as we know it today with some breakthrough thinking that will unlock meaning from the colossal volume of corporate and online documents.

BI Project Managers and Eyebrows

Like eyebrows, you don't really notice project managers when they are there but if you are rash enough to let them go you will end up looking startled and stupid.

I point this out because over a period of more than 10 years I have had the opportunity to observe many, many BI projects and one of the most surprising patterns is the scaling back of project management largely because the project is going well!

The openly declared reason is usually cost or some other misdirection but it is invariably preceded with pointed questions about what value the project manager has been adding to a project that is going so well. Perversely, the better the project is doing, the higher the risk that there will be murmurings about things like the overhead of project reporting and that project management activity will ultimately be reduced or even removed altogether. It has become as common and predictable as it is deeply and logically flawed.

Perhaps this is one of the phenomena that explains why the trend for project failure is not getting any better. According to the latest Standish Group report which is covered by Peter Taylor, author of 'The Lazy Project Manager', in his blog 'Are your Project Managers working too hard to be successful?' instances of challenged (late, over budget or reduced deliverable) projects continues to rise.

As BI practitioners we often value technical skills, competency in the reporting tool and the deep musing of the data architect and yet have a blind spot when it comes to project management. This may be partly because early BI projects were often departmental in scale. It may also be because many of today's BI Competency Centres originated as 'skunk works' initiatives and see project management as all methodology and meetings but we ignore it at our peril.

It is true that project management can be at its most obviously valuable when priorities need resetting, additional resources have to be secured or controlled management escalation is called for. However, we shouldn't assume that if a Project Manager is not doing these things that they are not doing anything.

Planned projects with predictable timescales along with accurate project reporting are rewarded with confidence from our business sponsors. A considered set of risks based on real-life experience of BI projects will mitigate against them becoming time sucking issues and properly managed issues will prevent them becoming show-stoppers.

A good Project Manager may make it look easy but don't take the lack of fire fighting and crisis meetings as an indication that nothing is being done. Look deeper for the benefits of order over chaos or be prepared to invest in an eyebrow pencil for a look that is decidedly a poor second best.

Small Children, Energy and Efficient Data Warehousing

Last week, I referred to Peter Thomas, and his article Using multiple BI tools in a BI implementation – Part II, In the article, Peter points out that the way to drive consistency across dimensions and measures is to define as much logic as possible in the data warehouse.

I was musing over this again this weekend (I hear the cries of what an interesting life you have) whilst out for lunch with friends and their small children (ages 9 and 7) On the short walk to the restaurant I was amused by how different their approach at getting from A to B was to the 'grown ups'. We were focused on getting to the destination in a relatively direct and efficient way. However, the children ran to and fro, stopped, doubled-back, looped a few circles and even randomly waved their arms in the air. They generally spent as much time in a state of motion as possible. Clearly the criteria of small children, when en-route to a destination, is to use the maximum amount of energy possible!

This, if you will go with me on this, is rather like trying to implement data warehouse consistency in the BI tools rather than further downstream in the data warehouse. You do eventually get to the destination, but will probably be exhausted, out of breath and hungry. This is fine if you are two children out for dinner with some stuffy grown-ups but not an efficient use of the somewhat limited time of a BI practitioner.

A typical BI architecture comprises tiers that include;

• Source Systems
  
• ETL
  
• Data (Data Warehouse, Data Marts, OLAP Cubes)
  
• BI Metadata
  
• BI Application (Reports, Scorecards, Dashboards)
  

A properly architected data warehouse (more on this in later blogs) should have been built against an enterprise schema and is therefore *the* consistent representation of business information. Common definitions of customers, departments, profit and products live here. If there is one good reason for this (although there are many) then it is simply that there these can all be defined once in the data warehouse but would have to be defined many, many times in what can often be hundreds of reports that comprise a BI solution.

One of the reasons that we fail to do this is that inconsistency is often made visible for the first time by the BI tools. At this point the project momentum is around building metadata models and reports. Inconsistencies are fixed where the resources are focused ... in reports. Add to this that revisiting the design may need involvement from the ETL developer, the DW designer and the business analyst and, if there is a lack of clarity, the business users. It is no surprise that the report author is inclined to fix it where they stand. After all, the tools make the fix simple and it is only when the report author has built the same calculation for the tenth time that they become suspicious about starting it in the first place. And of course, the initial build of the BI application is only the beginning. Many more reports will be built over the life of the BI application.

So work hard to establish the correct definitions during the design of the data warehouse and it will reap productivity gains. Leave it to the BI application only if you have the carefree attitude, the free-time and the energy levels of an eight year old.

BI Tools: Just because you can ...

Peter Thomas, in his article Using multiple BI tools in a BI implementation – Part II, points out that the way to drive consistency across dimensions and measures is to define as much logic as possible in the data warehouse.

There was a challenge to this approach because it misses the opportunity to exploit some of the features of BI tools. This challenge, I think, invites the maxim that just because you can ... doesn't mean you should!

BI Software vendors frequently build functionality into their tools that may not always be appropriate to exploit regardless of what the marketing message implores us to do. As an example, many reporting tools are able to integrate multiple data sources in metadata but is this really the best place to integrate?

Recently, I was engaged by a public sector organisation because their BI vendor had implemented this very feature. They outsourced their Data Warehouse and were sensitive to the costs associated with Data Warehouse and ETL changes so saw this as an opportunity to implement a 'virtual' warehouse and take greater control of their reporting solution whilst reducing data warehouse maintenance costs. Sensibly, before they embarked on a significant project to adapt this approach they sought external guidance on a best practice approach. Unsurprisingly, the conclusion was that it remains best practice to transform data from operational systems into a persistent store (data warehouse) rather than a transient (virtual) equivalent because;

• Application Performance. Queries from the reporting tool will have an unpredictable and likely negative performance on operational systems.
  
• Transformation Efficiency. Transformation in a typical DW architecture is typically designed to run once, usually overnight in a controlled and predictable environment rather than many times as each report is required or run.
  
• Maintainability. Transformation is better done as a series of small, manageable and auditable steps rather than a single complex process.
  
• Best Practice. In our experience, persistent, multi-tier data architecture is the most commonly adopted approach by successful BI installations.
  

Over the last ten years, BI tools have grown up into rich, enterprise technologies. However, some boast capabilities far beyond what we, as BI practitioners, should consider using in a properly architected solution.

Six Ways to Justify your BI Implementation

The R in ROI is Hard to Find

Most organisations seem to have the I part of this calculation well covered. In fact the investment is so well understood that it can typically be analysed by project and ongoing costs and by staffing, hardware, software, consultancy assistance and training. Indeed, we live in a climate where knowing these costs is vital.

Calculating the R is somewhat more problematic though. This is partly because the return is often a mix of the tangible and intangible but also because there are often multiple ways to justify a BI implementation. The first step in putting together a rock solid business case that will secure a green light is to consider which approach to use. What follows are some of the justification approaches that I have seen used successfully.

The Productivity Justification

One of the least unambiguous methods of calculating a return is productivity. If a monthly management report pack takes two heads (or FTE's) the whole month to prepare then there is an obvious and ongoing cost saving to be made by replacing this effort with an automated BI solution. However, even this can be a complex justification. A typical finance department deals with such a variety of tasks and activities that identifying the effort in this way is not always straightforward. Then there is all the hidden effort as countless cost centre managers produce their own subtle variations in Excel. Not only is this undermining the integrity of business information it is hiding the cost involved in sharing it. I have lost count of the number of business review meetings where managers spend more time debating the accuracy of their own spreadsheet than assessing what should be the single business view and making decisions. Generally speaking though, the cost of business productivity improvements can be assessed to a sufficient level of accuracy that it makes a credible business case.

However, whilst there are invariably productivity benefits to all Business Intelligence project, they are rarely the primary ones.

The 'Go to Jail' Justification

Another justification is that the information is mandatory and defies an ROI calculation because without the information the business cannot operate. Statutory obligations under acts like Sarbanes Oxley include titles that deal with enhanced disclosure and corporate fraud accountability that could even result in criminal penalties including prison. There is nothing like the threat of a fall all the way from the boardroom to the prison yard to focus the mind on reporting transparency. There are often many other benefits realised as a result of meeting these obligations but the production of legally required reports and the continued liberty of senior executives is often justification enough.

The Flying Blind Justification

This typically comes from someone from the senior management team. Multiple, poorly integrated ERP's and outdated information systems comprising a thousand manually manipulated spreadsheets can create the feeling of flying a very expensive aircraft without any instrumentation. I recently worked with a customer who had a completely new management team with ambitious plans to revitalise the business. The ultimate goal was to double the share price and to do this they had a solid understanding of where they needed to be. They had worked with a consultancy partner to determine where cost savings could be made and which revenue streams to accelerate or choke depending on their profitability profile. As a team they had a tightly defined set of key performance indicators and targets but could not get the information from existing systems which were operationally strong but KPI weak. The project was entirely justified on the strategic needs of the management team but also delivered consequential benefits to senior managers and middle managers too.

The Information as a Product or Service Justification

Using management information to add value to customer service offerings is an increasingly common project rationale. Information, typically available in a variety of operational systems has value to the customer once it has been integrated, cleaned, sanitised, summarised and made available as an extranet solution. An environmental organisation we are currently working with provides information back to their customers on the amount of waste that is being recycled on their behalf allowing them to demonstrate that they are meeting government and local authority targets and their duty of care. The data is a subset of information already being extracted into the corporate data warehouse and aside from winning new business is also becoming a mandatory requirement for some local authorities thereby freezing out some of the less information capable competitors.

The Big Problem Justification

The richest rewards are typically the least tangible though and are associated with having access to timely and accurate insight.

The return is a product of better decisions made as a result of having this insight so it is difficult to generalise and problematic when it comes to articulating it in numbers. Highly visible business issues can help here. A few years I partnered with an on-line retailer who had customer returns that exceeded £10m without having a handle on exactly why. In a wafer-thin margin business, a small percentage reduction in this number was a significant enough prize to justify a project to help them understand which product categories were being returned and for what reasons.

Large, visible, business issues that require information to understand and fix have been the mother of some great innovations in the field of business intelligence.

The BYO (Build Your Own) Justification

BI Implementations don't always have a big problem to hang their hats on or a new senior management team hungry for better, faster information. Indeed most do not. For these, the justification has to come from a considered review of the types of decisions that are being supported and the financial benefits of making better ones. Dorothy Miller, in her article Measuring the Return on Investment for Business Intelligence, describes the challenges of justifying BI more clearly than I ever could and recommends a Benefits Audit approach to assessing the benefits and therefore the value.

As a minimum, there should always be a section in the requirements documentation which articulates the requirements as business questions. These might include what is the order backlog?, what is the trend in sales win/lose?, how many deals are being discounted more than the average of 15% and what is the billable utilisation of each consulting team compared to the company average?
Again, the impact of accurate information to improve the business based on answering these questions is highly subjective but represents a good starting point for articulating the benefits and potential returns.

Business Intelligence ... no glaze please

When asked "what is BI ?" it is so easy to illicit a look so glazed that it could compete with a Krispy Kreme donut.

The problem is we tend to explain BI in terms of the technology. Don't misunderstand me, this is perfectly understandable. After all, it was the convergence of a number of technologies over the last 20 years that made so much of what Business Intelligence is today possible. Also, the technology does some amazing things. It allows users to navigate their information with a slice, dice, drag or drop and then visualise it in any number of charts, maps or diagrams.

However, before I launch into an enthusiastic guided (or misguided) tour of OLAP, data warehousing and metadata with a detailed explanation of the difference between scorecards and dashboards I usually remind myself that it is possible to get through the BI elevator pitch after you have pressed the button and before the doors have closed.

At it's simplest, Business Intelligence is business information for the purpose of decision making. That's it.

But let me just explain a little more about master data management ...