CLARITY BI

Clarity Business Intelligence (BI) specialises in providing a centralised enterprise BI capability. An enterprise BI capability caters for both standard reporting and ad-hoc query/publishing that empowers the BI administrators and distributed analysts to work from the same data platform.

Business Intelligence

Business Intelligence is the process of incorporating a range of data management and information collection techniques using appropriate tools to standardise data from various departments into a more meaningful and understandable form that can be used across an organisation. In a nutshell a BI process transforms data into information used for making informed business decisions and creating better strategic objectives.

It is important to note the Business Intelligence discipline should focus on publishing an understandable data repository accompanied by effective training, support and communications strategies. These strategies are effective if they encourage and foster an ad-hoc reporting environment that empowers business users and analysts to extract their own views of information. Contrary to many thoughts it is not about producing reports! Reports can be used to prove the data, tools and technologies will meet the business need but they should not be the sole output of a BI project.

Why cater for ad-hoc queries?

An ad-hoc query is any request for information from a subject area that is currently reported from the data warehouse. If a data warehouse does not support direct access to the underlying data, business users will go direct to the source systems or they will create their own un-managed data repository.

The reality in most organisations is that there are many people who do a variety of ad-hoc queries using tools ranging in complexity from Excel or Word documents through to direct database access. A lot of this reporting is overly time consuming and re-creates unique business logic for each report which raises the risk of inconsistent reporting. Much of this repeated effort can be eliminated if time is put into creating a central reporting repository that can be leveraged by many analysts.

If a BI project is tasked with delivering only reports based on certain specifications the organisation will receive what we call "Just A Few Reports". This will be fine until a month or two down the track as employees understand the data better or the business or market changes and someone asks for "Just another report" or to "Just add another field". In a report focussed delivery this will likely require major rework to identify and acquire the additional data required for the change. If a centralised enterprise BI capability has been delivered that can cater for ad-hoc and structured reporting, most new requests can be fulfilled within minutes as the data is already prepared for reporting!

Successful Business Intelligence

A successful BI implementation can be achieved through a combination of understanding the business and their information requirements, creating an architectural design that complements the requirements, identifying the proper BI tools, creating a user-friendly interface and adopting a suitable project management (agile project management methodology suits better for a BI project) and implementation methodology.

A typical BI process flow

Once there is the initiative and commitment for a BI strategy and implementation, the first step is to assess the readiness of the organisation - are the business ready to make use of improved access to information?. Step two involves creating the IT architecture to facilitate data storage and extraction from the source data systems. The next step implies a data governance program to manage the access to data, verifying data accuracy and reliability, and use of data. Finally we can begin the BI implementation! A step-by-step process flow in a typical Business Intelligence implementation is listed below discussing briefly each individual process. The first step in a typical BI implementation is to identify key business areas and prioritising or grouping the areas together for which Business Intelligence is required. Some simple examples can be found by looking at the parts of the business that process the most 'things' or spend/generate the most money. After identifying the business areas, each of the individual processes can be mapped using a simple diagram or flow chart. This will form the basis to gathering and understanding the business requirements of the organisation. Talk to business users to understand how they do their job and what information they use or need. The information needs should be defined at an entity level at this stage of the process. Identify the organisations key performance indicators (KPI) as these will serve as a way to prioritise the information requirements. Research the master sources of the relevant entities in the organisation to understand the key attributes that are captured. Typically a business area will have been using a variety of reports to run their business over many years. Identifying a list of the top 10 or 20 most used reports for each business area will form the base for initial analysis. The list of top reports will help in identifying the data elements used to monitor each business process. This in turn will define the measures (facts) and categories (dimensions) used during the data-modeling phase. Once the measures and categories are identified, a BUS matrix should be created. In the BUS matrix common measures can be grouped into facts and categories from a single entity can be grouped into dimensions. Measures, which are numerical values, will be required when an analysis has to be performed. The measures can be derived by way of calculations or come from the source directly. Attributes, which are individual elements, have to be identified. These will be used to filter or group data during an analysis. Now is a good time to detail the Business Requirements. This is not done earlier in the process because you need to understand the information constraints (you can only report on information that is captured), and current state reporting as this will be most users frame of reference. There are 2 parts to the business requirements: generic information requirements and specific report requirements. Remember to validate requirements with other business users. The next step is creating a data model for the BI solution. This will typically translate the business requirements and processes into a logical data model based on established dimensional modeling principles. The data model will help in validating the design by key stakeholders or business users. A successful design should be open and scalable to meet future and evolving business requirements. At this stage it's important to include as many attributes as possible but be careful not to over-complicate the model with complex derivations that are not required yet as this will only lengthen your build and test phases. The data model should be converted into a physical data model by implementing the model into the database. These physical tables are the targets for any ETL scripts that will be created to load the data into the tables from different sources. Once the physical data layer is created it is now time to create Extract, Transform and Load (ETL) scripts. In this ETL process data will be extracted from various data sources into a staging area. Then the data will be cleansed and checked for data quality issues. The data will also be altered so that it can be fit into the tables that have been designed for reporting and querying. This is known as Transformation. The final step in an ETL is to load the transformed data onto the individual tables. Once the data is available in the warehouse, the next step is to create business views of the subject areas, author metadata and create various reports per the business requirements. These reports can be static in nature and can also be surfaced via interactive dashboards. The report development process must be highly iterative in nature as business users often change their original report ideas once they can see the data and reporting capability you are delivering. The final step in a BI implementation is very critical. This is where the end users of the BI solution are to be trained. A BI implementation can be termed as successful only when the intended audience finds it easy to use and continue to use the solution as part of their functional responsibilities. As part of the BI solution the end users will have to be give the facility to create ad hoc reports. This will eliminate the dependencies on the IT department. For this the end users must be trained on how to create ad hoc reports and they must understand how the business data is represented in the solution for them to be able to use the correct data to generate reports.

In addition to the above steps, adopting best practices will reduce the risks and enhance the opportunity to build and implement a successful BI solution. Also, A BI solution will be complete only if the solution goes beyond the basic usages and offers the organisational users the capability to perform complex statistical analysis like trend analysis, predictive analysis, forecasting and pattern detection.

Best practices for developing a BI solution

Following industry best practices will always help any development process to successfully implement a solution. Best practices in a Business Intelligence (BI) and Data Warehousing (DW) solution will greatly bring down the risk and raise the chances to successfully implement the solution by successfully meeting the BI/DW requirements. The below best practices, philosophies and principles are best suited for a development life cycle that follows Agile methodology. Remember, the success of any BI implementation is measured by the adoption of the tools and data by end users!! You are there to create a business difference, not to create a theoretically correct or puristic data store. Understand that requirements are evolving in nature and be ready to embrace the changes in requirement as the project progresses from stage to stage. Any good BI/DW implementation starts with a good architecture and data modeling. Imagining an initial architecture will provide a potential vision for the development team to work on. This can even be a simple sketch done on a white board! Follow the agile principle of creating a model design “Just In time” and not at the beginning of the project. Create a working model, that is end-to-end, using either Agile Unified process (AUP) or Rational Unified Process (RUP) that will support your envisioned architecture. Adopt a usage centered development approach aided by use case scenarios. Though it is desirable to have a single version of data, it may not be always possible due to the differing functional ways of storing data by the various departments in an organisation and business applications. Hence be open to think otherwise and prevent the development team from being hitting this roadblock. Keep the releases coming. Plan for short iterations and deliver a working application at regular intervals, ideally one or two weeks. Testing should be an integral part of the development cycle and this will prevent major blind spots in data quality management. It is very much advocated that a “test-first” methodology is employed during the development of the solution. Involve all key stakeholders of the project right from the start. This must include all the support and operations staff that will form the majority of the target audience for the developed solution. Once you do this, you can be sure that you gather all requirements of the project in a timely manner. Software tools are key in a BI/DW implementation. Shortlist and choosing proper ETL tools, reporting and dash boarding interfaces that fit the requirements of the organisation will lessen the development burden. These tools should also make the developers life less complex. Make the project light by having less documentation. By involving all stakeholders from the beginning, the need to create an intensive documentation for the reporting needs can be avoided. Adopt commonly used development standards and take an agile approach towards data governance. This will create a less complex environment for the development team and will create an environment where each developer will follow the standards willingly. A true and complete business intelligence implementation will have to extend its usage beyond basic reporting needs. It has to offer the organisation the ability to perform statistical operations that are complex by nature, like forecasting, trend and predictive analysis and pattern identification. These operations should help an organisation to optimize its resources and guide an organisation to make informed decisions. A useful add-on to the BI implementation would be a knowledge management portal that can identify strategies and share innovative ideas as best practices across the organisation.

Keys to a successful implementation

Despite the array of complex technical challenges that must be overcome to deliver an effective BI solution, the success of a BI initiative is reliant on people and personal interactions. It is reliant on skilled developers, good business relationships, committed business sponsors and enthusiastic end users. Of all these points probably the end user usage is the most important aspect, as it is they who must use the information to make better business decisions - which is the whole point of BI in the first place!