jump to navigation

Unstructured Information in BI – Only for Spooks? What about Business Analysts? July 30, 2007

Posted by Cyril Brookes in General, Tacit (soft) information for BI, Unstructured Information.

There’s a big marketing and consultant push on UIMA, unstructured information management architecture. But I think it is largely missing the point for the real world corporate BI people, i.e. those not spooks or librarians. The critical concept, ignored by many, is that unstructured information is of two kinds, explicit and tacit. Even Wikipedia gets it wrong, ignoring the latter.

I believe BI gains most from tacit intelligence, but that’s not where the product marketing thrust lies. The importance of tacit unstructured information in BI is summed up well by Timo Elliott in a cartoon and by James Taylor’s recent blog post.

The heavy hitters in BI software, as evidenced by the recent takeover activity, e.g. Business Objects and Inxight for one, are pressing home apparent advantage to be gained by corporations with analysis of masses of emails, news, documents, etc. See, for example, the description of UIMA.

Well and good. But you can’t make a silk purse out of a sow’s ear, as Jonathan Swift said. And you can’t create relevant action oriented information for executives out of data that has no embedded useful information in it. The ocean of documents, with some exceptions, is a BI desert for most companies. But I mix my metaphors.

Basically, I believe that a corporation’s vast compendium of historical documents has little BI relevance. It may be useful to track or assess a person’s background, or to isolate the cause of a problem. But history rarely contains the up-to-date information that’s relevant to managing a business, assessing current performance and finding problems. The real lies in exploiting the tacit stuff.

I’ve quoted Henry Minzberg often before, but it bears repeating, as the message hasn’t yet been fully understood in the mainstream of BI: “The strategic database of an organization is in the minds of its managers, not in the databases of its computers”. This is as true today as it was in 1974. Today, one can add: “Or in the morass of historical documents and emails”.

Of course recent emails and documents often contain important information that can, and should, be part of a BI context; but usually only as the seed for a collaborative knowledge building process. This is the nub of the issue; it’s hard to identify, collate, disseminate and collaborate on tacit unstructured information. Perhaps this is why most authors steer clear of the issue. But we need to address it if we are to be effective. More on this in my next post.

I wrote last year detailing some of my research in the 90s on the subject of “hard” and “soft” information, how valuable it is in many BI contexts particularly CRM, but also how difficult it is to exploit. In this context hard information refers to the structured, numeric, formatted, BI reports. Soft information is the unformatted, unarticulated, information in managers’ and professionals’ minds.

An interesting article by Rick Taylor deals with unstructured information is relevant here. It says, in part:

The key to defining knowledge management is to make sure you are separating “explicit” knowledge from “tacit” knowledge. Explicit knowledge is anything easy to quantify, write down, document or explain. Tacit knowledge is everything else. The knowledge based on ones experiences, and often times, at a subconscious level. It is information that you don’t necessarily know you know until you are reminded of it. If you were asked to write down everything you know, could you do it?

The explicit and tacit labels were used first in this context, I believe, by Nonaka and Takeuchi in The Knowledge-Creating Company.

The BI key questions that arise from this discussion are, I believe:

  1. What are the most useful sources of unstructured information in our business? Explicit or Tacit?
  2. If Explicit, how do we best marshal the information and report it?
  3. If Tacit, ditto?
  4. Is the information we get from our unstructured sources complete, and ready for promulgation, or do we need to amplify or build on it before it’s useful?

I believe that the above analysis outlines the problem of utilizing tacit unstructured information reasonably well. I’ll offer my answers to these issues in the next post.



Implementing Decision Automation BI Projects July 13, 2007

Posted by Cyril Brookes in Decision Automation, General, Issues in building BI reporting systems.
1 comment so far

The feedback on my three earlier posts on specification guidelines for automated decision capability in BI systems has been both positive and heartening. My objective has been to show how these BI projects for operational business processes may be built relatively simply, and to generate enthusiasm for this among the legions of business analysts. You can indeed try this at home!

This post summarizes the major issues that received favorable comment and then deals briefly with profitability, feasibility and implementation techniques for these systems. It concludes the series of (now) four posts on decision automation for BI that commenced here.

I haven’t attempted to place this subject in its context, or to cite various examples of success or otherwise. Thomas Davenport, Jeanne Harris, James Taylor and Neil Raden have done this comprehensively in their recent articles and books.

You may recall, Dear Reader, the underlying principles for this methodology are:

Selecting the right targets is critical to success:

Doing the right thing is much better than doing things right (thanks Peter Drucker!). My prescription is to avoid trying to pick winners from the various business processes that may be candidates for BI automated decisions. Rather we should look to a different set of candidates as the start point.

Identify the controllable variables – the levers that are adjustable in the business process that can shift performance towards improvement

These are easy to pick. There are relatively few options available in most businesses, variables like changing prices, adjusting credit ratings, buying or selling stock or materials, approving or rejecting a policy proposal, etc. A more complete discussion is in my second post on this subject.

Only consider automated decision making BI systems where controllable variables exist

This is a no-brainer, I guess. It’s only possible to automate when automation is possible. If we can’t control the process when there’s a problem, because nothing is available to be done (e.g. we can’t raise prices if all customers are on fixed contracts), then don’t let’s start automating.

Segment the design processes into logical sub-projects so the project doesn’t run away uncontrolled

I suggest in the third post that the Herbert Simon style decision process elements are an effective segmentation. This allows focus on (say) finding problems and then on deriving the relationship between adjusting a control variable and the resultant outcomes.

Enough of a recap: here are some basic suggestions for project management.

Implementation of a decision automation project is always tricky. In most cases it is not possible to “parallel run” the new with the old, since only one option can be tried each time in real life, and comparison is not possible longitudinally.

I suggest that an iterative implementation is therefore appropriate. It should incorporate feasibility and profitability analyzes as well.

Referring to the more detailed methodology in the third post:

  • Build the status assessment and problem finding sections first, and leave the action part to the management.
  • Then design the diagnosis and alternative selection modules and instruct a human manager what to do (always leaving the override option, of course). This is simple as long as there is only one controllable variable available for the business process and only one KPI metric, or a set of related KPIs and metrics, that are out of specification, hence signifying the problem. If there are more than one of these, then it can (almost certainly will) become complex. Certainly it’s achievable, but there’s a good deal of modeling and use case analysis required that is beyond the scope of a blog post.
  • Finally, link the alternative action chosen to the automatic adjustment of the control variable(s) and you’re home free.

I hope, Dear Reader, you’ve been infected in some small way with the enthusiasm I have for automated decisions in BI applications. In many ways they are the most satisfying aspects of Business Analyst work, since you get to design the system, and then get to see it perform. Working on high level strategic projects is often more intellectually challenging, but you rarely get to have full closure, it’s the executive users who have that pleasure, often long after you’ve left the scene.