jump to navigation

Implications for BI from Gartner’s Five Discontinuities for IT November 4, 2007

Posted by Cyril Brookes in BI metadata documentation, Issues in building BI reporting systems, Tacit (soft) information for BI, Taxonomies, Tags, Corporate Vocabularies.
8 comments

Those following the technical press will know of the five discontinuities to orderly IT business evolution as expounded by Gartner in Orlando in early October. I have no intention of adding my thoughts to the copious analyses and assessments of these as regards IT generally. However, it may be of interest to you, Dear Reader, to embark with me in exploring the Business Intelligence impact of these Famous Five.

The press release and examples of the commentary are here and here, but to make it easier to judge your interest level, the five movers and shakers envisaged by the gurus are: Web 2.0, Open Source Software, increasing consumerization, Global Computing and SaaS.

All of these factors have been considered in earlier posts for this blog, but in this one I will give a short overview of my opinions on the BI implications, and point to former, often more detailed, discussion.

The two immediately obvious observations I draw are:

SOA has finally been tipped out of the IT analyst headlines – and about time too. Thanks be!

The inter-dependence of the factors, and their cumulative impact, are at last being recognized.

I’ll comment on the Five in order of my perceived significance to the BI community.

Web 2.0 and Enterprise 2.0

Most of us already know that social networking has huge potential to reshape the BI environment. I’m not sure of the precise tools that will become dominant; I don’t believe they are in complete form as yet. However the key resource is, in my view, TACIT information. And the key process is SHARING. Sharing information, up till now almost solely done effectively in face-to-face or video/phone meetings, is the only way that knowledge is created.

Therefore, in my view, Enterprise 2.0 will become the new all purpose marketing term to replace Knowledge Management. This much is unavoidable, the vendors will see to that.

The importance of My Space, Facebook, Wikis, and their ilk is that they are rapidly reshaping the culture of communication. I have outlined my views on the cultural roadblocks to effective information sharing. The new generation of professionals, each one with his/her own personal net space, will presumably more readily accept collaboration imperatives from the executive suite. More readily, that is, than the current set of professionals who are decidedly reluctant to do so. I must admit that I do not think the culture will shift that much, and we still will have this problem in a decade.

It is in the interests of young people to share details of their interests, activities, etc. This is part of the current social norm. However, when corporate life intrudes older survival instincts may well prevail such as: the perceived power of knowledge (say, customer preferences, possible new clients, etc.); the desire to remain part of a team (not being the whistle blower); and the danger of being the messenger of bad tidings who is shot.

Up till now, it appears to me that the personal related Web 2.0 applications have little relevance in the corporate context, other than this culture shift opportunity. However, the evolution of Wikis is harder to assess.

Wikis are collaboration tools. They can be used as knowledge creation devices, provided that the classification, categorization and alerting mechanisms are effective. In corporate BI, as opposed to the social context, speed of response is critical. Knowledge must be created rapidly, and this implies almost instant initiation of collaboration processes that enable this creation.

Information that is not categorized cannot be searched or personalized for alerting.

I have often argued that a universally agreed taxonomy is important if people with information are to find people who can build on it, and those who need to use the knowledge created by collaboration. There is much talk of folksonomies, where classification is determined by the individuals. They are useful on the Internet, but I maintain that they are unsuitable for tacit information management. Newer categorization and retrieval software may well alter my opinion, but as yet, I believe that the taxonomy of approved, standard, preferred terms is an essential building block if wikis and similar knowledge stores are to be part of evolving BI applications based on Web and Enterprise 2.0 technology.

Open Source and SaaS

These are the tools of the Rogue BI professional I have blogged about earlier. For corporate BI these emerging movements, they’re not really technologies, have major implications. They’re not really discontinuities either, since they been evolving since timesharing terminals were invented some 40 years ago. And think spreadsheets, the most common Rogue tool at present.

Drawing from that earlier post:

I define rogue BI as the user self-creation of unofficial, unapproved corporate performance reporting, competitive analysis and customer behavior assessment.

To mix my metaphors: Blessed are the BI rogues, for they shall know what they want much sooner than IT will get around to filling their request.

The pressure for instant gratification in BI is intense, perhaps more so than the teenager’s need for the new Playstation game. Or maybe it’s just the grown-up version of the same.

Rogue BI is a management nightmare if left unaddressed by IT and senior executives. However, if managed properly, it will be a salvation, since much of the demand for new BI services will be redirected from conventional, and probably over-stretched, IT capability.

The solution for the over-stressed IT executive lies with effective metadata control and management. Provided the data is under control, the rogue can do little harm to the corporate entity. Of course, the rogue’s BI reports may be rubbish, but that ought only cause localized difficulty; as long as the data used is correct, of course. But then, any reporting from crook data is going to be rubbish.

The Rogue BI movement, driven by Open Source and SaaS, will probably have most impact on the vendor community. Heavy hitting BI software will have to become far more user developed application friendly. It will be a difficult shift for the marketers, as it threatens product differentiation. But it must happen. Spreadsheets are already there.

Consumerization

Dreadful word, it is, but it’s the one Gartner use in the press release, so who am I…. It relates to the common, now anyway, problem of system developers generally that uses are so used to high class presentation from their internet experiences that they demand the same quality user interfaces and capabilities from their in-house corporate BI systems.

If the Open Source and SaaS vendors (is an Open Source supplier also a vendor?) capitalize on this user demand then it will be another factor in promoting Rogue BI. And with good justification too.

Global Computing

I’m not terribly sure what the gurus mean by this. Perhaps they wanted an odd number of discontinuities? The most common example given in the commentaries is “Google Apps”.

I assume that the main implication to be drawn for BI proponents is that the pervasive and high response nature of Google Earth etc. is giving us new opportunities for BI projects, outside the normal scope of BI reporting applications.

I guess that says it all.

What else is there in the BI discontinuity department?

Well, the drive to automated systems doesn’t rate a Gartner mention, but undoubtedly the new rule management systems are discontinuities in the BI environment as I explored in some detail a few weeks back. Automation in IT generally is taking a back seat, but it’s full steam ahead in the BI space, see, for example, the work of James Taylor. There is a consequential imperative for IT and BI analysts. Revisit your current set of BI applications to assess where the automated decision making capabilities can be used profitably.

The rise of capability with unstructured information systems covered in my last posts (here and here) are also important. Web 2.0 is part of the story, but there’s much more technology at work in the move to enhance BI applications with both explicit and tacit unstructured information.

Next post, I’ll consider the BI related management implications of these trends, since Gartner’s symposium presentations have also generated considerable comment on how enterprises ought respond to the discontinuities.

Rogue BI will be like the Poor, Always with Us. Metadata to the Rescue. April 1, 2007

Posted by Cyril Brookes in BI metadata documentation, General, Metadata management.
1 comment so far

Two recent articles on the place of the maverick in conventional IT, by Joe McKendrick and Ramon Padilla stimulated me to consider the role of the Rogue in BI systems creation, modification and use. It’s more benign in BI than in IT mainstream, that’s for sure.

To mix my metaphors: Blessed are the BI Rogues, for they shall know what they want much sooner than IT will get around to filling their request.

I define rogue BI as the user self-creation of unofficial, unapproved corporate performance reporting, competitive analysis and customer behavior assessment.

The pressure for instant gratification in BI is intense, perhaps more so than the teenager’s need for the new Playstation game. Or maybe it’s just the grown-up version of the same.

The data is there, the report building is now easier, or the same, as building a spreadsheet, and no one is directly affected by poor execution, except that the Rogue gets rubbish instead of the goodies. Of course, the business meeting where the rubbish is put forward will be interesting; much like used to happen in the old old Visicalc days when ingénue spreadsheet creators battled it out with garbage versus rubbish.

It’s the “no one knows” that’s the issue for IT administration. If no one knows that a report exists, no account will be taken of it when the data cubes are redesigned, deleted, re-dimensioned, or whatever.

Reuse is also impossible, obviously, but then again it’s likely undesirable except for the brilliantly capable Rogue. SOA followers don’t want to go here!

Some form of monitoring and control is required of the BI environment, but the jury will be out for some time yet. In the meantime, it is important for us to realize Rogues are part of the shift to the BI brown-field.

In January I blogged that BI issues in 2007 would be focused on metadata because of the brown-field effect. In the brown-field new BI applications would be re-workings and extensions, not new systems for executives without current BI, the green-field. BI Rogues proliferate in the brown-field.

This leads us irrevocably to the metadata management issue. As I see it, it is imperative that IT administration control the Rogues through regular, maybe daily, snapshots of the BI metadata, not just the SQL but also the (using Microsoft examples) the Analysis Services, Integration and Reporting Services content. Lineage that links report content to cube content and maybe data sources is also useful.

Dear Reader, if you’re a regular, you will know that I’ve been working on a metadata documentation and snapshot tool – BI Documenter, which does most or all of what is needed in the BI metadata area, if you’re a Microsoft user. You may care to check the website www.bidocumenter.com.

Metadata snapshots tell the story, the Rogues leave an auditable trail, and all modifications are likewise exposed. Assuming they don’t do something completely stupid, the Rogues can be allowed to create their “thousand flowers” to bloom.

If you cannot, or don’t want to, monitor BI metadata snapshots then I believe your only option is to use governance to manage the inevitable BI Rogues. Good luck, they’re smart!

Next time I’ll look at a code of good practice for BI Rogues.

2007, the Year Metadata will become King of the BI Brown-field January 3, 2007

Posted by Cyril Brookes in BI metadata documentation, General.
2 comments

New Year brings both reflection and forward thinking impulses.  On the former theme it may be interesting, Dear Reader, to consider how the fundamentals of BI design, implementation and operation have shifted these past 12 months; and what is in store for the next 12.  

Of course, center stage in 2006 was the SOA hype and its implications for BI analysts. 

You may know from my last post that I believe there are only two aspects of the SOA hype that are material to the BI designer:

  • Standardised web interfaces between sub-systems, and

  • A well designed, implemented and maintained data warehouse.

Re-usability isn’t going anywhere much. 

Much more interesting, judging by the pleasing reaction to my blog entry of September 28 is the sea-change wrought by the shift in emphasis from green-field to brown-field BI implementations.  This shift is inescapable as BI implementations mature.  The impact on the BI analyst, consulting and marketing scene will be huge.   

Remember the shock to IT professionals when accountants discovered Visicalc?  Anarchy set in, with multiple representations of transactions data, each with its own set of errors and bias.  Substantial, but comic, arguments raged over whose data was the right basis for analysis. 

We’re in the same situation today with data warehouse content.   The ease with which cubes are created, modified, deleted has the potential for much confusion and angst.  Reporting from cubes is also approaching anarchy in many businesses; and the common lack of understanding about the ETL mechanisms will lead to the aforementioned substantial, but comic, situations being repeated. 

Metadata documentation and repository monitoring is the new audit and information integrity imperative.  Data lineage documentation is the new key analysis tool.  We need to know where it came from, what it has been doing, and who has been looking at it.   

You may care to check out the documentation tool I’ve been developing to meet this challenge:   BI Documenter. 

Happy New Year, it is going to be interesting!

Soft (Tacit) Information Metadata for CRM and Competitive Intelligence BI October 18, 2006

Posted by Cyril Brookes in BI metadata documentation, General, Tacit (soft) information for BI, Taxonomies, Tags, Corporate Vocabularies.
5 comments

We’ve explored why CRM and Competitive Intelligence isn’t shared in my previous blog post of October 9. A major contribution factor is that few people understand that soft information has metadata, just like the hard stuff. If you come here often you’ll know I’m a fan of BI oriented metadata repositories (to the foolish extent of creating a new one for BI analysts!).

Business analysts need a metadata repository for the soft stuff too, or at least they need to understand what they’re working with. Without this understanding you can’t distinguish between, and treat appropriately, one rumor on customer disaffection, a second opinion on the impact of new patent, or a third comment on what’s wrong with the new marketing campaign.

As with most abortive knowledge collaboration initiatives, enterprises usually do not address the underlying cultural factors inhibiting sharing. They simply rely on altruistic motivation: “It’s good for the company, so the teams will embrace it”.

Well, of course they won’t, or not often. “Altruism is a disease”, someone perspicacious once said (actually it was my PA when I was a Professor). Whatever the truth of this, good faith certainly is not a big motivator in a context where the staff members face potential downside when they open their mouths and blow the whistle on their boss, product, peers or whomever.

Let’s examine what soft (tacit) information actually is, and from whence it is usually derived. The metadata if you will. We can then move on to synthesize and prescribe design principles that encourage sharing knowledge, or at least ameliorate the cultural issues.

There are three categories of soft information in my experience. These form a basis of soft information metadata categorization:

Independent Items;

Comment, or Qualification, Items; and

Reference Items

Independent soft information exists in a “stand-alone”, hopefully self-explanatory, format and context; much like hard data objects such as customer names, ID codes, Product SKUs, Employee address, etc.

  • Examples include: rumors, ideas, suggestions, proposals, news, web site content, knowledge created during meetings, results of informal “corridor” meetings, etc.
  • These are often the most valuable BI items in a CRM or CI context, as they can portend a paradigm shift – say the disaffection of a major customer, or the rumored breakthrough features of a new competitive product.
  • They often are discovered accidentally, and originate from unlikely sources. Therefore, they’re difficult to collect, assess and disseminate.
  • Validation of their accuracy is always an issue, often relying on source identification to establish reliability.
  • Without a formal effort to collect this valuable resource, the business is relying on serendipity. You may as well buy a ticket in the lottery.

Comment soft information items qualify other items of soft information or known facts.

  • Examples include: assessments of a change in ordering pattern of a major customer, reasons why a market share has increased sharply, opinions on the validity of rumors about competitor marketing plans, evaluation of a forecast sales plan, possible solutions to a plant production problem, possible implications of an event – say the effect on sales and customer loyalty due to a fire in a production facility, etc.
  • Clearly these items have no value without the context of the item they qualify.
  • Knowledge is created as qualification assessments are made, and this know-how needs to be made explicit and managed so that the added value of the soft information is always retrievable whenever the main item is accessed (a non-trivial data management issue!). They are most valuable when they come from a known, designated, subject expert.
  • Inadequate mobilization of this resource is inexcusable. The business knows not what the business knows – you’re condemned to meet and improperly deal with the same problems, over and over.

Reference soft information describes likely sources, and the quality of those sources, of other intelligence, hard and soft. These items are only as good as the reliability of the quality assessments of the sources. Essentially they embody the second part of Samuel Johnson’s famous adage:

“Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information on it.”

Or more colloquially: “It’s who you know, not what you know, that counts.”

  • Examples include: a list of URLs of “good” blogs on a topic, ranked lists of investment advisors, suggestions for better than average travel guides, etc.
  • In the CRM and CI context, examples include lists of people with inside knowledge of customers and competitors, industry analysts, former sales representatives, relevant web sites, and useful publications on corporations, etc.
  • This inventory of sources, and the quality and reliability of same, is clearly a vital information resource. But how many enterprises devote even minimal effort to it? Most rely on the goodwill and efforts of individuals, some of whom are inhibited by the cultural factors of my October 9 blog post.
  • Reference items are particularly valuable for Drilldown in problem solving situations – see my blog post of August 27. They tell us, for example, who may know how serious the problem is and what happened last time it occurred.

Hopefully, I’ve convinced you that it’s worth doing something substantial to facilitate capture and dissemination of this forward looking business intelligence.

We have an understanding of the issues, both cultural and definitional. All we have to do is create requirements specification principles that embrace the positives and either ameliorate or circumnavigate the problems!

What does seem certain is that we can’t wait for the all sharing, all collaborating, Web 2 generation to come to our aid; they’re too busy in MySpace downloading and assessing music and videos and exposing their foibles to the world. The problem is now.

Thanks for sticking with me this far, Dear Reader. Enough of analysis: synthesis and prescription is where it’s at. We’ll have a go at synthesis next time.

Green-field to Brown-field BI Environments – metadata analysis holds the key September 28, 2006

Posted by Cyril Brookes in BI metadata documentation, General, Metadata management.
3 comments

Our BI systems are getting on in years.  What was an appealing virgin territory to the BI enthusiast has become a difficult pre-teen.  Maybe bringing up BI systems is like raising children – every stage is the worst.   Client executives are getting more aware of what’s possible as well and, of course, the data is still messy.

But the biggest challenge could well be the need to renovate existing systems, rather than have the luxury of starting afresh.  Welcome to the brown-field.

I outlined the metadata complexity issue for BI analysts in my September 21 post.  Earlier, in my June 30 post, I described the stages and tasks involved in a bottom-up approach to BI system design. I opined that the bottom-up, or rapid development, methods will remain dominant in the BI systems space for the foreseeable future.  Now I am attempting to bring these two issues together.

To recap, with some adjustments to fit my current convictions and practice, the bottom-up approach to BI system development requires answers to the following queries:

1.  Where are we now?  E.g.

  • What is the current, synchronized, state of the data warehouse and BI tables: the SQL database documentation, ETL (covering all the space from source transaction system to data warehouse), cubes and dimensions? 
  • What reports and query specifications are out there, what do they contain?
  • What are the outstanding BI specification requests and user complaints/suggestions?

2.  What is stopping us perform better in our BI reporting? E.g.

  • Bad source data content?
  • Inadequate ETL packages corrupting data warehouse content?
  • Under-utilized data?
  • Unused, wasted, data collection?
  • Unavailable data to satisfy specified requirements?

3.  What can we do to perform better?  E.g.

  • Repair the data sources
  • Find and fix the ETL errors and omissions, starting with the most dysfunctional
  • Check with the client executive base to see if the under utilized data should be reported more or better
  • Review the available data with client executives to improve satisfaction
  • Design new data warehouse content and associated reporting where possible and desirable (Of course, in this business everything is desirable, only a few things are essential!)

4.  Prioritize the work schedule and get going – probably in a series of iterations with highest/fastest ROI first.

I know this is obvious, motherhood and flag, stuff to the experienced BI analyst.  However, it is often not understood – I believe – that we are completely dependent on the stage 1 being done right.  We must have comprehensive, accurate, up-to-date, access to the synchronized metadata.  Further, we need to be able to explore this metadata, its relationships and sensitivities. 

Iterative BI design is impossible if we cannot document the metadata adequately.  Green-field sites are all migrating to the evolutionary mode as they age, and metadata is king of the brown-field BI environment.  Once we have this documentation and analysis capability the rest of the project is fairly straightforward.

Documenting the relatively narrow BI software context isn’t enough; we must be able to reach out to the source data locations wherever possible.

Plus, passive documentation itself is also short of the full whack.  It’s like being in the supermarket without the meal plan and recipes to tell us what we need to buy. 

I reckon we need to have a “what-if” capability to allow us to explore the metadata world as documented, checking on data existence, link existence and integrity; and reviewing the Drill-down and Drill-through capabilities. 

Of course, this capability is also very useful in the virgin, green-field site as well, but we usually have more flexibility to maneuver then. 

There are many software documentation tools on the market, and I leave you, or others, to evaluate them for this purpose.  However, it is my view that a more fully featured tool is required than those currently available.

To provide an evaluation yardstick, and to act as a design guide for my specific software design purposes, I’ve put together my checklist for a metadata repository tool that serves BI analysts in both their support and design roles.  It also avoids the problems, e.g. the dog’s breakfast syndrome, raised in my last post and has the required “what-if” exploring capability.

My repository checklist is based on the Microsoft products, but others would be similar:

  1. Take regular snapshots of the relational database contents relevant to BI – Tables, Columns, etc.
  2. Also, snapshot the Integration Services packages: Data sources, Data Flows, Data destinations, Tasks
  3. Ditto the Analysis Services tables: Cubes, Dimensions, Report Specifications
  4. Ditto the Report Services content; Datasources, Datasets, parameters, drill-through reports
  5. Facilitate HTML and diagrammatic documentation for each of these snapshots as needed, with extended diagramming capability to capture design ideas and extensions.
  6. Provide a server application based analysis tool that:
  • Tracks data lineage from SSRS, through SSAS and SSIS to the relational, legacy or other data source, and displays the transformations that occur on the way.

  • Provides cross component impact analysis capabilities, i.e. what is affected if I change the definition of this field – in the database, in the cube, in the report?

It’s taking a while to build this, but even the current status should be useful to the designer.  If you’re interested you can monitor the progress and try it out yourself at www.bidocumenter.com.  There is a new release due next week.

The BI Metadata Enigma – where did these data come from; and where have they been? September 21, 2006

Posted by Cyril Brookes in BI metadata documentation, General, Metadata management.
1 comment so far

These are the kinds of questions we ask our children – where were you, what happened last night?  And to the putative, hopeful, BI support analyst and developer, the DW content, the cubes, the dimensions; they are our children.  Strangely, the answers are just as hard to get on metadata as with the kids, or at least the right up-to-date answer.   

The issue is going to get much worse.  Why?  Because we are rapidly moving from BI “Greenfield” design situations to “Brownfield” support and re-design environments.  We’re stuck with  the treadmill of history as well as client executives who want better BI. 

There are two different sets of metadata related problems here:

  • Support of operational BI systems, and
  • Design of new or improved BI 

But, they both have the same solution. 

Everything is related to the BI metadata, sql database documentation and associated BI development system tables, cubes, etc.  This is a problem to us all because: 

  • Metadata is not in one place – it’s a real dog’s breakfast; even when all the BI tools come from the same vendor
  • It is dynamic, so we need to be able to monitor the sql documentation to see what the DW architects have done since last time we looked
  • Comprehensive commercial repositories, at least those available today, are cumbersome and not kept current
  • Synchronization of the metadata component documentation is super-important to the analyst – without it your project must surely fail.
  • BI design is not concerned with much of the metadata available for a database, the focus is on the business related items and not the process type – schedules, update counts, etc.  Often, the unrelated stuff dilutes that which is important.
  • Metadata standards are too complex, and not a useful categorization tool for BI analysts at present
  • Essentially, the issues for BI analysts are:  What is in the current DW, the cubes, KPIs, tables, columns, dimensions, reports, etc.; and where did they come from.  Most database and BI software analysis tools are inadequate for this purpose. 

Support analysts find it difficult to track down the sources of bad data.  Clients complain about inaccurate reports, but why is it so?  We need, indeed must have, better BI related metadata repositories.  We don’t want all the crap, pardon my expletive, just the business related stuff, so we can find out …..Where it came from and where has it been – to make such lousy, and inscrutable, content. 

That’s probably enough stating the issue for support analysts. 

Next post I’ll detail how these same issues impact the BI developer, especially the person trying to improve on an existing BI system.

I’ll then outline my proposed solution, a specialist BI metadata repository tool.

BI for CRM – Much hyped as easy, but hard to build effectively September 10, 2006

Posted by Cyril Brookes in BI Requirements Definition, BI metadata documentation, General, Tacit (soft) information for BI.
add a comment

We all know that every enterprise needs automated CRM, and BI is a big part of that.  All the IT gurus on or off payola, marketing white papers, and newsletters tell us so.  As the saying goes: “It must be true, it’s in all the papers”.

Most of this advice avalanche accepts as axiomatic:

  1. Successful CRM, like Minerva, springs full-panoplied from the brow of Jove , or

  2. Implement any CRM software package and, bingo, we have instant lift-off, etc.

But, of course, CRM success isn’t automatic, there’s a lot of hard work.  As I have posted before, my mantra in these situations is drawn from the late Peter Drucker:

Efficiency is doing things right.

Effectiveness is doing the right things

This is most important when considering how to design or, more likely in today’s enterprise, make more effective, the CRM environment.   There are very many options, and they can often be implemented quickly, so they are efficient – but they don’t do the job as it needs to be done.  Effective CRM is complex, elusive and requires careful thought, not rushed procurement.

This post is rather lengthy, but I strongly believe that there is too much written on the net about this topic that is a series of common knowledge checklists, without the accompanying detail that suggests, or occasionally prescribes, how the essential components can/should/must be implemented.

Of course, I’m banging on again about the need to define the requirements before you build BI solutions.  Here, however, it is especially true, for defining CRM is essentially about defining its BI component. Why?  Let’s consider what is involved.

CRM‘s context and solutions involve much more than BI reporting.  We all know, truly this time, that for good CRM we potentially need some or all:

  • Data Warehouse content detailing some or all transactions and static profiles with:
  • Customer “touches” including SFA data; the statistics plus the sales representative contact reports
  • Sales transactions and associated accounts receivable
  • Customer intelligence;  the statistics, order patterns, etc. plus what the customers themselves and the industry is saying about our customers
  • Service calls and results; not just numbers, but also the comments from service personnel on issues as they arise
  • Customer complaints and resolutions; not just the numbers, but also the content and especially the trends
  • Call-center and telemarketing activity and intelligence; what the operators are learning from contacts, and the relevant statistics
  • Marketing promotion performance; how our marketing and promotional efforts are paying off,
  • Competitor intelligence; what competitors are doing and saying and what the industry is saying about them

Note that these data, in full scope, are drawn from both hard and soft (tacit) sources. Regular readers will know my conviction, and emphasis, that tacit information is a vital part of an effective BI environment, and CRM is no exception – in fact it is a prime example of a field where collection and dissemination of tacit information plus collaboration on its significance, Knowledge Management if you will, is absolutely key.

 

I’ll deal with all the above categories of data collection in more detail in later posts.

 

It is obvious that only the largest corporation can afford to collect all the above material in its finest detail.  So the “some or all” filter is critical.  Even with all the data, the “can’t see the wood for the trees” syndrome often leads to failure.

 

CRM for each enterprise, large or small, needs access to some or all of these databases, BUT it is the executive interface to them for reporting, inquiry, alerting and analysis that determines success or otherwise – effectiveness or not.  Make no mistake, the executive interface determines what must be collected and stored in the data warehouse.

 

THEREFORE, the BI specification for on-demand reports, pre-formatted inquiry responses, alerting messages, collaboration between sales, marketing, service and product professionals, decision support environment, etc. must drive the specification for the data warehouse content and the CRM application software.

 

BI requirements definition is not everything in CRM planning, but when it comes to ensuring success, it is truly the only thing that’s important.  All the rest is relatively straightforward; just connect the dots and you’re done, courtesy of the plethora of software that’s just waiting to be turned on.

 

So how do we go about defining the CRM BI requirements?  We have the two basic options I’ve discussed in earlier posts, but I repeat some of it here – see the June 30 and some earlier contributions.

 

Top-down, or “waterfall” is the logical process for the green-field site with time and resource for planning activity.  Bottom-up is for everyone else.

 

Both approaches have the same objectives.  They require that we interview executives and carry out research into data availability to establish:

  •  What information needs to be reported routinely, or available to be reported on-demand?

  • How can potential problems or opportunities be detected and executives alerted to them?

  • What reporting and models are required for Drilldown – especially diagnosis analysis and decision support (see post of Aug 27)?

  • What data items are required in the data warehouse cubes to satisfy the above reporting needs?

  • What degree of detail, i.e. the data dimensions, is required for the above?

  • What additional dimensions are required for Drilldown?

The comprehensive nature of top-down analysis should produce the best result, but it will take time and may unacceptably delay that Holy Grail – the quick ROI.  Bottom-up will almost always give a sub-optimal result, but quicker.  I’ll compare them briefly after presenting the general principles.

 

The differences are in the effort involved and the starting point. CRM BI reporting specifications include the following:

 

  • Pre-formatted reporting for routine and on-demand inquires: This is intended to give the client executives comfort that they are fully informed about the customer relationship status. 

  • Pre-specified alerting: This is intended keep executives up-to-date with any unusual existing or forecast situations that could imply problems. 

Prespecified CRM BI therefore involves some or all of:

Summarization of KPIs and other metrics dealing with customer relations

Comparitive analysis of KPIs and metrics with CRM pre-specified benchmarks

Forecasts of key variables

Trend analysis of relevant KPIs and metrics

Alerting to unusual situations

Determination of the data items and dimensions required for the above

 

In all cases, it is vital to ensure that the tacit information – collected from sales people from calls and industry gossip, meeting minutes, rumors, opinions, etc. is assembled, assessed, communicated and escalated in a structured context.  It is the early warning system for the enterprise, and especially important when the topics are customer related.

 

  • Drilldown reporting is the other CRM BI specification component:  modeling and forecasting desirably (IMHO) follows the principles discussed in my August 27 post.  It involves some or all of:

Scenario building to predict potential problem situations

Assessment of the inquiry, statistical analysis and data mining capabilities required for resolving situations in those scenarios

Assessment of the additional data and data dimensions required by these capabilities

Assessment of the additional access to tacit information and collaboration capability; particularly the identification of subject experts who can comment on emerging problem situations.

 

I return now to the methodologies involved. Top-down methods for requirements definition involve significant interaction with executives, to understand the business processes and how the executives use information to manage them adequately.   The data resource and dimension design for the data warehouse cubes follow naturally from the information requirements as they are determined.  Of course, the overall design is iterative, as some of the required data and its dimensions will not be readily available at the outset.

 

For those interested in the detail of this approach, the processes and documentation I use in CRM and other BI projects is described at www.bipathfinder.com.

 

Bottom-up methods focus on what is available in the data warehouse cubes for reporting and the existing reports.  Although the process is abbreviated, compared with top-down, it follows the same general steps, and has the same objective – that is to prepare a requirements definition to guide the CRM environment development. 

Basically, Bottom-up methods involve answering the questions:

  • Where are we with CRM reporting now?

  • What data is available, in what detail?

  • Where does it come from, how good is it; who uses it?

  • What better use can be made of the existing resources?

  • How should we structure the project to obtain maximum ROI in the shortest time?

The first step in Bottom-up approaches to CRM BI must be documentation of the existing data warehouse meta data, including the links to source data resources – such as legacy transactions processing systems and their databases.

 

 

There are several products that do this.  However, recognizing this imperative, I have helped specify, and now use, a new product – it works at this time only for Microsoft environments – called BI Documenter to document the SQL meta data and the links to cubes, reports and sources.  If you want more detail it is given at www.bidocumenter.com.  There is a free version that documents SQL database content, and an Enterprise version that includes all the BI metadata content for Analysis Services and Integration Services.

I’ll amplify these concepts, especially the data requirements and pre-specificied reporting content in a later post.

Keeping score – What’s on all those dashboards? July 17, 2006

Posted by Cyril Brookes in BI metadata documentation, General, Metadata management.
add a comment

Who thinks they know how fast we’re going? As the plurality of available reporting proceeds apace for most businesses, we are losing track of what we have, who has it, and what we could/should be doing with it all. It’s likely to be the spreadsheet stakes of the 1970s all over again. Remember? You turn up at a sales meeting and six different people have six different numbers all created on Visicalc on their personal Apple IIs.

Effective BI with adequate controls. We need this, but it will be elusive.

My preferred direction for improving BI is to encourage more creativity in report design, with emphasis on closer matching of information presentation and the executive’s cognitive decision processes. Regular readers will know that I have a hobby horse about inadequate requirements definition for BI, and its consequences.

But, I’m turning my attention, as an interim gambit, towards BI documentation. A form of modified bottom-up development methodology seems to be the most viable approach, given the existing situation in most IT departments and the exigencies of corporate life.

This requires accurate, clear, comprehensive and up-to-date documentation of the BI context. What is in the cubes, what KPIs are active, what measures? And who is using them. Which KPIs aren’t being used, why? If you like, what is in peoples dashboards – read 1970s Visicalc sheets. I’ve been working on a new project, BI Documenter for this, you may care to check it out at www.bidocumenter.com.

With this first step we can analyze the present, and offer executives a better way of looking at it.. I’ll discuss how we can do this in a later post.

BIg questions; Where did all that bad data come from? How do we fix it? April 21, 2006

Posted by Cyril Brookes in BI metadata documentation, General, Metadata management.
add a comment

We all have bad data, and there’s no such thing as good bad data, but we don’t have to fix it all.  Bad data is like curved roads, its not in the nature of the human race to create perfect data and more than it is to build straight roads.   But not every road needs to be straight, some never get travelled.

Where should we start fixing bad data?  Are we wasting our time on frivolous pursuits correcting data we don’t need?  Are we ignoring the data we should be getting right? 

If you’ve dipped into my earlier posts, you’ll know I’ve a fixation on requirements definition, or rather the lack of same, when starting out building BI systems.  Surely, though, knowing what we want to report, and the manner in which we want to report it, is an essential prerequisite for any data cleansing operation. 

It’s pretty clear now that the IT corporate standard approach to BI systems development is driven by weight of marketing effort by key suppliers – playing the “Keep up with the CIO Jones” syndrome to absolute perfection.  If you don’t have DW, ETL, Scorecarding, Dashboards, blah you’re not in the game.  Data accuracy isn’t an issue at this purchasing point.  Its like buying a several packets of blueberries at the supermarket, then finding out they’re all soft and fungicidal when you get home.  

The resultant evolving continuum of BI environment is something like: 

Buy data warehousing softwareBuy ETL software

Try and figure out what the hell to do with the above

Buy, or implement “included with server”, software to do BI reporting, data mining, data analysis.

Try and figure out what the hell to do with the above [especially since a lot of the blueberries are crook]

Analyze lots of existing reportsSpecify and then create a set of corporate standard KPIs by aggregating everything everybody gets now [regardless of data quality]

Realize how bad the data is.  Start a data integrity project.

Create the ETL and computation transforms, cubes and other magic things that should have integrity.

Build new BI systems that replicate the existing reports, with flashy presentation, but poor data.

Wonder why no one is excited after spending all that money – just like no one really wants a jazzed up version of Ben Hur – it was bad enough first time.

Or, more accurately, as Peter Drucker opined:  Efficiency is doing things right, effectiveness is doing the right things. 

What is wrong?  I reckon that the problem can be summed up by the age old IT adage:  We’re building solutions looking for problems.   

Information delivery is not matched to the business process. 

Worse, information presentation is not matched to executive mental processes 

Data collection, and integrity expenditure, is not directed at the real, important, useful information needs 

Much effort is wasted on cubes, ETL, etc. that are never to be needed.   

No effort is spent to identify new information metrics not yet collected, or not yet present in a report. 

Get the specifications right first, then find out what data we really need and how to present the information, then collect clean versions of the same.  Simple really.