Death of the BI Generalist

I have fond memories of the 90s. I spent a fair amount of it being a 'Cognos Consultant' travelling the length and breadth of the UK being an expert in all things Cognos. There was a great sense of independence and autonomy. It was possible to pitch up at a new Client with 2 CDs (One for PowerPlay, one for Impromptu) and install the software on a handful of PCs on Monday morning and by Friday I would have interviewed for requirements, built a metadata model, cubes and reports and would be huddled around a desktop with the users putting the final touches to what I dared to call a BI application.

It was the day of the renaissance BI Consultant, the Polymath. We all felt like experts from the time of we opened the jewel case to the point the user signed off their brand new Business Intelligence application.

As much as I enjoyed this, I don't yearn after these as simpler times. The solutions we built were great and delivered real and incremental value. They didn't, however, have the breadth, depth and coverage as the best implementations today. These are deployed over the web to hundreds, sometimes thousands of users covering multiple business domains. Modern performance management solutions have bought together Sales, Marketing, Operations, Finance and the Senior Management Team together in ways that were less achievable perhaps even impossible ten years ago.

There is a cost though. The range of skills required to deploy a modern BI (or Business Analytics) Application is broad. Take the individual that installs the software. What used to require a working knowledge of Windows and an installation manual now also needs an understanding of web servers, operating systems, virtualisation, application servers and security all in multiple flavours. It may also require systems management conversations to configure for fail-over and load-balancing in a way that wasn't required before because it wasn't possible. And whilst the internet means we are all effectively sharing a single network it stands to reason that it is important to understand all the different ways of communicating and processing on this single, global network.

So far, we have only unwrapped the cellophane but things get really interesting when we get our hands on data. The possibilities for delivering value from the vast array of corporate data assets as improved dramatically but this means that they need to be integrated too. This required an in-depth understanding of relational and dimensional modelling techniques, databases sql and at least one ETL tool. Whilst there are some tools that simplify this process with tools and (usually) OLAP cubes, they are typically departmental in scale. Modern solutions may also required a mix of on-premises and cloud data and a mix of  structured, and unstructured which increases the richness of the solution but also the number of things a developer needs to keep in their head.

So where does this leave us? If you have a team of homogeneous 'BI Developers' then their skills may be too general. I am working with one client in an advisory capacity at the moment and they have one consultant that has implemented everything. He has been very successful too. However, their business, though, is of a size where he can wrap his arms around the requirement, the infrastructure and the data. He also has exceptional aptitude and is an outstanding consultant throwing herculean effort at their projects. For the rest of us mortals we need to divvy up the responsibilities.

Many of my clients tend to do this across technical/systems, data and application. It's not the only way but works really well. The IS team pick up the install and maintenance of the application server environment, a data team create a single reporting database/data warehouse/marts and a BI team maintain metadata and the reporting application in all it's variants of reports, analysis, dashboards, scorecards etc.

There are interesting implications for the modern, complex out-sourcing and off-shoring organisation. Another of my clients based in the City out-source their databases including data warehouse development. Because there is a cost each time they want to add a new data item for the purpose of BI they tended to try to resolve new requirements in metadata or the reporting application. Inevitably the quality of their solution deteriorated over time because they were not always extending or fixing in the right place. Eventually, the sticking plasters gave out and they were forced to back to first principles which had implications of cost and re-work.

The more common issue is one of a shared design. Changes to data impact ETL, metadata, reports and the application. A new requirement might need to be changed in one or all of these places. Whilst usually a number of small and straightforward changes, they do need to applied consistently. To co-ordinate activity when the solution is expanded or enhanced there needs to be a common data model. In our experience this needs to be two models. One, a logical (and relational) model that represents the business data in a perfect and integrated world. Secondly a dimensional model that represents the data as the business see it in their reporting application through reports and metadata. These are surprisingly rare in our experience and usually not because they take a long time to draw (they don't) Our suspicion is that what makes these time consuming is the need for consensus. However, if there is no consensus then there is a risk that the solution is already flawed.

So, a BI/Analytics solution cannot be built single-handedly. It requires a range of skills and it's difficult to be an expert in all of them. The generalist then is fading away and being replaced by a team and a shared design who can deploy solutions with greater reach and richness than when we could ever have believed a few short years ago.

What 127 Hours Tells us About Social Networks

In an interview this week for Mark Kermode's Film Review show, Danny Boyle made it clear that he used the success of Slumdog to make a film that might otherwise not have been made. To leverage the success of 2010 most acclaimed film is an indication that 127 hours is more than this years 'would you?' movie.

For those few that have not yet heard about it, it is the true story of Aron Ralston who gets trapped under a boulder whilst canyoneering alone in Utah. The desperate measure that he takes to free himself is well documented so it is not giving anything away to say that he was trapped by his arm, he has a multi-tool (a really cheap one) and a little under 127 hours to debate if he should ... or should not.

Before, I go on you might be wondering what's the connection between BI and Social Networks let alone the connection between Danny Boyle's latest movie and Social Networks. Those that follow my posts and tweets will know that Social platforms interest me because I think they are changing the way we share and use information in business and will profoundly change the Business Analytics space over the coming years. A social platform has already made it into IBM Cognos 10 because these guys, again, are ahead of the game. Many don't see it yet because the original use of social platforms have trivialised their significance but it's there nonetheless.

Back to the connection. Aron Ralston, played by James Franco, is an all-American hero. He's young, fit, strong, intrepid and independent. He is good at what he does, he has spent a lot of time in his chosen wilderness and is able to navigate it with speed and ease. In fact, at one point in his story, he briefly but convincingly takes the role of park guide. The hopelessness of his literal and figurative fall takes a long time to sink in for our hero. Indeed, when it does dawn on him that he could have shared his hiking plan with his friends or family it wouldn't be exaggerating to call it an epiphany. It's clearly a powerful realisation for Aron that he's not a hero, he's an arse.

There is a moment in the movie where Aron says 'thank you'. It's a strange moment. I don't want to give away why it is strange but if you see the movie, you will know why. For me, it was significant because he knew that if he made it home alive (which was still, by no means certain) then he would be changed forever. He would live the rest of his life in the knowledge that however strong, smart and experienced he was that those tiny connections we all make each day matter. Sometimes in small ways because it's just about about sharing. Sometimes in significant and surprising ways.

For me, I am continually and pleasantly surprised by what I learn on the subjects of analytics, organisational leadership, productivity, start-ups and social media in my twitter stream. It's full of links to content that cover important ideas from solid thinkers. Admittedly none of them are life-saving but, at a stretch, a rare few might be described as life-changing. Each of them make a tiny but positive difference and sometimes someone in my network helps me (or me them) in a game changing way.