Middleware and/or Data Integration

Many of my colleagues reached out to be recently asking about both these terms. Although they knew about these terms independently, but when used together to compare, they struggled.

So thought of pulling out some simple explanation to see if it helps:

Middleware; is software that allows applications to communicate with one another, or send messages back and forth, often in real time, to initiate an action of some kind. It is also sometimes called enterprise application integration (EAI).

Middleware is at use, for example, when a bank assesses a customer’s creditworthiness. One application considers the customer’s credit history, and if certain conditions are met, middleware allows the application to communicate with another application to continue the process, and so on.

Middleware, or EAI, is the “glue” that enables applications running on different platforms to communicate. In a classic middleware transaction, data itself is not transferred from one location to another, however. That’s where Data Integration — software that identifies, transforms and moves data from point A, like a transactional database, to point B, such as a data warehouse – comes into play.

But the two disciplines (middleware and data integration) have a number of commonalities, “The two worlds are definitely overlapping, and over time, we will distinguish between them less.”

For example, extract, transform and load (ETL) tools are traditionally used to integrate data in batch from one source to another. But ETL tools are not ideal for real-time movement of data. Middleware solutions such as Oracle BEA’s Weblogic and IBM’s WebSphereMQ, which facilitate real-time communication between applications, can be used to supplement this process.

Likewise, when data, not just messages, needs to be shared between applications that communicate via middleware, data integration tools can help.

Experts therefore encourage application development and data integration teams to initiate a conversation about the tools and software at use in both groups to avoid making costly, redundant investments in technology. And at a time when IT budgets are tight, making better use of the technology an organization already has is a must.

Data Governance

We talk about this term in all possible strategic discussions because of the importance of the topic. So for the benefit of masses, thought of pulling out the simplest explanation that I found during my study.

Data governance is actually the policy making and oversight from a business perspective of common enterprise master data or data in general, actually. So, what we’re seeing with data governance is it’s a business driven initiative where business people, ideally cross-functionally across organizations, are actually sitting down together through a common process, developing policies around the data, making decisions around the data, defining the data often for the first time in a consensus-driven way and actually sanctioning enterprise data and the use of that data. So, (it’s) very much a policy-making set of processes.

We distinguish that from data management which is the tactical execution of those policies. So, we’ve got data governance as the business-driven initiative or set of initiatives and then we’ve got data management as the IT initiative for actually managing the data and a day-to-day basis.

7 important considerations towards a MDM strategy

MDM has come a long way from being just discussed as an organisation challenge few years ago. Now this is being seen (or getting considered) as a separate solution stream and many organisation are making their move towards establishing the solution (and the strategy).

Based on my discussion with various senior executives and MDM solution evangelists, following are the 7 important building blocks (like 7 Habits, 7 wonders etc. ) towards an effective MDM strategy:

• Data Quality

A strong level of data quality must be achieved before attempting to initiate a MDM strategy. The success of any MDM system deployment will be determined by the strength of internal process to develop:

1. Common data definitions
2. Policies and business rules
3. Quality measurements
4. Adoption and adherence to set practices

 

• Owner

It is an absolute necessity to understand the needs of business in relation to master data, especially with regard to decide who need to be able to access and control the information. This will affect the amount of technology and internal processes required as part of the MDM strategy.

The owner ensures system processes and data quality routines are consistent with the need of the programme.

• Alignment to business vision

Gartner analyst John Radcliffe, author of a March 2009 research note on creating an MDM strategy and roadmap, thinks it should be fairly straightforward to make the case for MDM as an enabler of an organisation’s business vision.

“If the business vision is very much about being customer-centric, then it will be very difficult to deliver on that promise if there is no single view of the customer,” Radcliffe said. “Equally, if it is all about operational efficiency, then it will be difficult to do that without single views of the product, customer and supplier.”

By following that premise, he added, a programme manager should be able to create an MDM project plan that aligns directly with the organisation’s business strategy.

 

• Quick win

As the saying goes, for a smooth run, “the squeaky wheel really should get the grease”.  The MDM team identifies the business stakeholders who can best articulate how poor data quality is affecting their operations, and address their needs first.

MDM initiatives when started as a small pilot projects before a board approval has been seen getting a far receptive response, as they can now see and predict the anticipated benefits of the programme. It is important to use the quick win approach but without forgetting the bigger picture.

 

• Spot opportunity loss

Experts recommend demonstrating that business success is unachievable without MDM – for example, by showing that an organisation lacking an MDM strategy could fail to recognise who its best customers are and have trouble retaining them.

Other potential business problems include losing out on revenue opportunities and facing fines or worse as a result of not meeting regulatory compliance requirements. Linking those possible outcomes to a lack of investment in MDM “should gain the attention and sponsorship of executive-level management,” Radcliffe said.

 

• Vendor RFP

The market for MDM technologies is extremely confusing, with MDM vendors offering mixed messages and very different solutions to similar problems. If companies have a good idea of what they want to achieve upfront, they can narrow the field more quickly and avoid possible missteps when they evaluate and buy MDM tools.

 

Companies in their request for proposal are very specific about their needs. E.g. some would want a best-of-breed product with “the technical capability to support their current needs, the operational support to ensure a successful initial implementation and on-going continuous improvement. They also look for a vendor that had a stated vision for staying ahead of the curve on product development in the evolving MDM market.

 

• MDM Utopia

Decide what you would like to achieve with MDM in an ideal world before taking the first steps towards adopting it. One company had a utopian model for a real-time enterprise MDM integrated with a service-oriented architecture (SOA).

Before proposing the MDM project, they discussed this vision with key executives in the organisation to get a feel for how much compromise might be needed. Doing so “provided them with an understanding of the field they had to play in, how much selling was required and how far could they go with their initial MDM pitch,”.

Next, they conducted a gap analysis between the green-field version of the MDM vision and the company’s existing data quality capabilities in order to help it develop a viable MDM strategy “that was relevant to their business at that time.”

 

4 Bedrock requirements of BI and 11 Questions to ask a BI Solution !!

4 bedrock requirements of BI

1. Historical analysis and reporting.

Fundamentally, BI should give you the insight into both business performance and the drivers of that performance. An understanding of business influencers and results is the foundation for successful, proactive decision making. Technically, this capability requires the mapping and analysis of data over multiple years. This can also often mean the modelling and manipulation of hundreds of millions of database rows.

2. Forecasting and future projection.

While understanding historical data is a first step, it is also vital to project those findings into the future. For example, once you know how different types of sales deals have progressed in the past, you can examine current opportunities from that perspective and make future forecasts. The ability to forecast and align your business resources accordingly are key to success.

3. Ability to integrate information from multiple business functions.

Strategic insight often requires data from multiple systems. For example, operational results require a financial perspective to show the full picture. Sales management benefits from a comprehensive view of the demand funnel. Targeted, customized marketing efforts require analysis compiled from customer, marketing, and purchasing data. Your solution needs to be able to easily integrate information from multiple sources in order to get answers to broad business questions.

4. Easily explored reporting and analysis.

Decision makers need to understand overarching business views and trends. They also need to examine increasing levels of detail to understand what actions can be taken to achieve further success. It’s not enough to simply have a report; if that report is not explorable, it might raise critical issues but not satisfy the need to know more detail in order to make a decision. A full range of drill-down and drill-across capabilities make it possible for decision makers to fully understand an issue at hand and make critical decisions.

11 Key Question to ask of a BI Solution

1)      Can I get a comprehensive view of my business?

2)      Does it provide full features at an affordable price?

3)      Can I start seeing value within 90 days?

4)      Can I be assured that my data is secure and available?

5)      Can I proceed with limited IT resources?

6)      Does it avoid risky integrations?

7)      Can users easily create and explore their own dashboards and reports?

8)      Can the solution scale to a large, diverse user base?

9)      Can the solution keep up with my business as its needs change?

10)   Can the solution easily serve my entire ecosystem?

11)   Is the solution provider dedicated to my ongoing success in BI?

Top 2012 trends observed in BI

1. Big data gets even bigger
One of the highly spoken trends in BI today is growing data which is growing and further growing. What we expect to see is more organisations actually starting to use the data, rather than just collecting it. This will put pressure on various products to deliver valuable information that really works with big data in multiple ways; such an in-memory analytics, data discovery / science and general performance improvements.

2. Self-reliance is the new self-service
The idea of self-service BI where IT opens up a small menu of capabilities for employees is no more sufficient. Giving employees an environment where they can get the data they need to answer questions on their own will become the norm. The consumerisation of enterprise software is part of the story here, but the real driver is the increasing pace of business across industries. Business users are starting to believe and expect that they can modify and create reports as needed, when they can’t, their frustration with existing tools will lead to a change in their organisations.

3. The pace toward the “Consumerisation of Enterprise Software” accelerates
We all have already heard it: consumer software is faster, easier, and often more sophisticated than enterprise software. Why? Consumer software typically puts more thought into design. And software that’s well-designed with fewer features is more useful than poorly-designed software that is packed with options. People want their business software to work as easily and as smoothly as their personal software – to the point where they use personal software to accomplish business objectives. This trend is going to speed up and IT needs to be ready. Traditional enterprise software deployments beware.

4. Mobile adoption goes mainstream
Apple claims that 92% of the Fortune 500 will be testing or deploying iPads in the 2012 timeframe. Companies are moving from the experimentation stage with mobile into real, IT-supported deployments. And the tablet finally offers a form factor that makes sense for BI.

5. Companies get (a little more) comfortable with social
Alerting has been around forever, long enough to clog our inboxes with too many alert emails. Social platforms like Facebook, Twitter, Blogs, Salesforce’s Chatter offer the promise of disseminating information in a more consumable, useful way. We see social BI as a nascent trend in 2012. It will take several years before most organisations are willing to change their patterns of communication to support more grassroots, interconnected communication.

6. Companies explore the cloud
Lower TCO, easier setup—these factors will drive some companies to the cloud for business intelligence. In 2012 we see adopters primarily in small- and medium-sized businesses that don’t have a lot of IT resources. BI cloud offerings will also get more diverse and more mature.

7. Analytical talent will become a required part of many jobs…leading to talent shortages
The McKinsey Global Institute released a study in 2011 predicting that by 2018, the US would face a shortage of up to “1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.” In 2012, despite the soft job environment, we’ll start to see tightness in the market for analytical skills and demand for tools to help make up the difference.

8. IT and business users continue to dance around “alignment”
Companies with enlightened IT staffs and business leaders who recognise the business impact of IT are achieving spectacular results from BI projects. In these organisations, IT no longer tries to squash business-driven BI projects rather they are asking how they can support and improve them. These organisations will continue to see gains from their alignment efforts.
But in less enlightened environments, BI still represents a battleground for control. And, unfortunately, IT just isn’t in a position to win.

9. Interactive data visualisation becomes a requirement
The wide variety of data visualised on the web will make business users expect that capability inside their organisation as well. And with the trend of bigger data, interactive data visualisation will become a critical tool in sifting through all that information. Reports filled with endless rows and columns of numbers or static, boring charts that take hours to sift through and weeks to change won’t cut it anymore.

10. Hadoop gathers momentum – unstructured data isn’t going anywhere
Hadoop is the best way to deal with massive amounts of data especially if it’s unstructured. While still a nascent technology, vendors like Cloudera are pushing Hadoop forward. We don’t see it becoming mainstream in 2012, but we do see it getting much closer.

Key demand factor for a BI platform

Before selecting any BI platform for your business, its very important to understand the major demand side factors. These factors later become tools to measure the success of a BI implementation and a true measure of ROI.

The primary usage & need of a Business Implementation has seen a shift from measurement to analysis, forecasting and optimisation. There are majorly 3 factors that stand out clearly on the demand aspect from a good BI platform (Please note these are not the evaluation parameters for any BI Platform selection, but guidance of evaluation of post implementation success):

1. BI Consumerisation
2. Performance orientation & emerging data support
3. Unified decision platform

Let’s look in detail about these factors:

1. BI Consumerisation:

The BI tool should be business friendly, intuitive and mobile. Everyone wants a BI tool that is as simple, fast and intuitive as Google. There should be a minimum need of IT assistance in terms of operating them similar to facebook or google+. Ease of use has always been the top need of the business, but with the emerging technology, iphonesque, intuitiveness, graphical and fast are soon getting as primary business needs as well. Organisations are yet to embrace mobile BI completely, but this has already moved from “Wish to have” category to “Good to have” last year and very soon would move into the “Must to have” category. This represent the inflection point where the portability and presence location not only enhance the analytic experience, but the touchscreen and interactivity of the tablet makes mobile device suitable for BI analysis beyond simple BI report viewing.

 

Business user data mashup is a key capability that accelerates the analytic process and also extends the BI Platform usage to a broader set of users and use cases.

2. Performance orientation & Emerging data support:

Another important driver of BI growth is the capability that enables the analysis of large, volatile and diverse data sets.  As a step towards improving the performance of a BI solution, “in-memory” technology got introduced. Although “in-memory” in itself cannot drive a BI growth, but it has to be in combination with a consumer oriented approach. This combination expands BI to broad range of users.

Organisations are “Data rich but information Poor” today. Thus in addition to the performance, support of emerging and extreme volume of data has become vital. Along with the traditional structured source of data, there is a need to address emerging diverse data that resides outside the confines of organisation’s structured corporate data sources. i.e. unstructured, Hadoop, social network, web services and device data. This integration enables in truly sense a complete perspective of a subject (of analysis) combining cloud (the unlimited possibilities) with the traditional solution (the defined & designed objective).

With the ever increasing popularity, influence and reach of internet into people, analysis techniques around social filtering, social – network analysis, sentimental analysis and social media analytics are gaining momentum. These analysis are used to measure, analyse and interpret the results of interactions and associations among people, topics and ideas that occur on social-software applications used in workplace, in internally or externally facing communities, or on the social web. Content analytics enables users to combine and analyse structure and unstructured data that may or may not be available on social network sites.

3. Unified decision platform

Improved decision making is a key driver of BI purchases. Key capabilities that will evolve BI from and information delivery system to a decision making platform are:

• Closing the loop by enabling “Insight to Action” framework and completing (and automating at places) the information lifecycle (Data -> Information -> knowledge ->Action) by embedding with the relevant business processes. The primary objective of closing the loop is to optimise and affect the business process improving the business productivity. It’s important to make BI actionable at the point of decision with increasing industry specific analysis, predictive and prescriptive capabilities.
• Integrated and a unified platform capability is key in today’s world when every industry has multiple specialised sources specific for certain business operations. It is very important (infact mandatory) to have a common metadata area combining the planning, simulation, forecasting, prediction and prescription. The chosen platform should also combine the departmental BI solutions to an enterprise one. This convergence enables to incorporate predictive analytics and simulation, not only into planning and forecasting process, but also for predicting what target thresholds should be, for analysing and prescribing different course of action, and in identifying leading and weak indicators.
• Collaborative decision making capability enables decisions to be made at all levels of the organisation. This differentiates the highly performing companies from the poorly performing ones. A collaborative system with the integration of BI and social software capabilities can alert decision makers to events and changing patterns that indicate and early need to make a decision. It also enables bringing together the right people with right information, discuss issue, assess and capture assumptions, brainstorm and evaluate options to agree on the most appropriate course of action. This would convert the meeting rooms into resolution arena rather than issue identification centre.

Indicators, Metrics and Measures

We often hear the terms KPIs, metrics & measures used interchangeably and most of the times the semantic differences between the three are very small.

A business should have at the most 7-8 KPI’s that are aligned with the Business Strategy and targeted to the business main objectives. More KPI’s makes it hard to steer your business in the right direction. On the other hand you will have PI’s, that is just simple Performance Indicators. These you can have more of just to follow up on the business, but these are not the most important ones. The KPI’s should change overtime, in the business lifecycle, when the Strategy of the business is changed or the objectives need to be adjusted. PIs and KPIs are created from metrics. A KPI can be, a ‘complex’ aggregation of metrics.

In continuation to the term KPI that I have put in my earlier posting, let us try understanding the terms Metrics and Measurements here.

Keep in mind, mathematically; a metric is a ‘distance’ function. Meaning non-negative result, and comes from the mathematical system called a “Metric Space”. Thus a metric is a mathematical set of relevant, quantifiable, attributes (measures) taken over time. And a measure is a numerical value assigned to an attribute according to defined criteria quantified against a standard at a point in time.

For example, if we calculate stores sales for August 2010, the standard is the Pounds (GBP), and the “Total Sales for August 2010” is the measure. A single measure usually has little value without some context. How do we know if GBP 48,000 in sales for that month is good or bad?

Suppose we track monthly sales over a 13 month period between August 2009 and August 2010. Now, we have some context to see our August 2010 sales trending up or down?

A metric is the degree to which a particular subject possesses the quality that is being measured. It is based upon two or more measures. Total Monthly Sales for 08/2009-08/2010 is our metric.

In our example, Total Monthly Sales gives us more context and understanding of the relationship between measures. However, the fact that this stores sales are trending up may not give us the complete perspective of how this store is performing.

However, if we measure the stores sales against a baseline such as budgeted sales, then we get a truer indicator of performance. Total Monthly Sales to Budget for 08/2009-08/2010 is our “indicator”. If this is providing the business with actionable information that is/would be used to drive the business, then we would consider this to be a “Key Performance Indicator”.

So, the KPI should tell you if something is “Good” or “Bad” (or moving one way or the other).
And A KPI does not tell the “where” or the “cause”. It is the underlying ‘metrics’ making up the KPI that you need for that.

Key Performance Indicators

 Over the last few years, I have heard a lot of definitions and interpretation to one of the most used (or I would say Abused) mantra in BI: “KPI” or “Key Performance Indicators”.  Everybody uses this term to their own interpretation and advantage. Be it customer or consultant, it is very easy to play with the scope of a project using this term KPI.

Having been on the receiving end as a professional consultant for so many years, I finally thought of putting together a reasonable meaning of this terms “KPI”. As they help show the progress (or lack of it) toward realizing the firm’s objectives or strategic plans by monitoring activities which (if not properly performed) would likely cause severe losses or outright failure.

Some of the most common definitions that I find on the internet are:

1. A quantifiable measurement that can be used to track the progress in achieving important goals within a company. 
2. Key Performance Indicators are the goals or targets set by an entity in their strategic plan. Also known as KRAs (Key Result Areas).
3. Key Performance Indicators – Measurements that represent the status of an operational area and progress made to reach operational objectives.
4. A measure (quantitative or qualitative) that enables the overall delivery of a service to be assessed by both business and IT representatives.
5. A short list of important financial or operational metrics that provide a measurement for results.

If I could comment, all of the above definitions are correct but still not conclusive on what a KPI should be. I then went on reading more articles and found very interesting facts and indeed good explanation that I could chew upon and my brain cells actually started absorbing them.

A company must establish its strategic and operational goals and then choose the KPIs which best reflect those goals. For example, if a software company’s goal is to have the fastest growth in its industry, its main performance indicator may be the measure of revenue growth year-on-year.

Every business would have few (or many) KPIs, but Whatever KPIs are selected, they must reflect the organization’s business strategy and Goals. They must be the key to its success, and should be quantifiable (measurable). KPIs usually are long-term considerations, and the definition of what they are and how they are measured do not change often. However, the goals for a particular KPI may change as the organization’s goals change, or as it gets closer to achieving a goal.

 

Key Performance Indicators Reflect The Organizational Goals

As mentioned above the KPIs should (and MUST) reflect an organization’s business strategy and its short term and long term goals.

An organization that has as one of its goals “to be the most profitable company in our industry” will have Key Performance Indicators that measure profit and related fiscal measures. “Pre-tax Profit” and “Shareholder Equity” will be among them. However, “Percent of Profit Contributed to Community Causes” probably will not be one of its Key Performance Indicators. On the other hand, an educational institution is not concerned with making profit, so its Key Performance Indicators will be different like “Graduation Rate” and “Success in finding employment after graduation” and these accurately reflect the institution’s mission and goals.

The act of monitoring KPIs in real-time is known as business activity monitoring (BAM). KPIs are typically tied to an organization’s strategy using concepts or techniques such as the Balanced Scorecard.

KPIs should not be confused with a Critical Success Factor. For the example above, a critical success factor would be something that needs to be in place to achieve that objective; for example, an attractive new product.

A KPI can follow the SMART criteria. This means the measure has a Specific purpose for the business, it is Measurable to really get a value of the KPI, the defined norms have to be Achievable, the KPI has to be Relevant to measure (and thereby to manage) and it must be Time phased, which means the value or outcomes are shown for a predefined and relevant period.

Key Performance Indicators Must Be Quantifiable

If a KPI is going to be of any value, there must be a way to accurately define and measure it. “Generate More Repeat Customers” is useless as a KPI without some way to distinguish between new and repeat customers. “Be The Most Popular Company” won’t work as a KPI because there is no way to measure the company’s popularity or compare it to others.

It is also important to define the Key Performance Indicators and stay with the same definition from year to year. For a KPI of “Increase Sales”, you need to address considerations like whether to measure by units sold or by dollar value of sales. Will returns be deducted from sales in the month of the sale or the month of the return? Will sales be recorded for the KPI at list price or at the actual sales price?

Good Key Performance Indicators vs. Bad

Bad:

• Title of KPI: Increase Sales
• Defined: Change in Sales volume from month to month
• Measured: Total of Sales By Region for all region
• Target: Increase each month

What’s missing? Does this measure increases in sales volume by dollars or units? If by dollars, does it measure list price or sales price? Are returns considered and if so do the appear as an adjustment to the KPI for the month of the sale or are they counted in the month the return happens? How do we make sure each sales office’s volume numbers are counted in one region, i.e. that none are skipped or double counted? How much, by percentage or dollars or units, do we want to increase sales volumes each month?

Good:

• Title of KPI: Employee Turnover
• Defined: The total of the number of employees who resign for whatever reason, plus the number of employees terminated for performance reasons, and that total divided by the number of employees at the beginning of the year. Employees lost due to Reductions in Force (RIF) will not be included in this calculation.
• Measured: The HRIS contains records of each employee. The separation section lists reason and date of separation for each employee. Monthly, or when requested by the SVP, the HRIS group will query the database and provide Department Heads with Turnover Reports. HRIS will post graphs of each report on the Intranet.
• Target: Reduce Employee Turnover by 5% per year.

 

Now that we have a Business Strategy and appropriate KPIs, it’s now turn to understand what Metrics and Measures mean exactly.

Look at this page in next few days for more on Metrics and Measures……