Ten Reasons Why Your Data Preparation is Failing
August 08, 2016 by Anne-Frances Hutchinson
Make no mistake about it: collecting, housing and prepping data on spreadsheets is still a thing, and it’s probably going to stay that way for a while. That’s just one of the findings of a meaty Data Warehousing Institute (TDWI) report, Improving Data Preparation for Business Analytics.
As they begin to yield to the demands of a connected and clever customer base, life insurers have become well aware of the bounty of legacy data in their organizations -- much of which is maddeningly out of relatively easy or budget-friendly reach. TDWI’s sponsored*, industry-agnostic survey report quickly shows that insurers are in fine company with organizations of all sizes with the same thorny data prep issues. According to the report, 86% of the 411 respondents are not completely satisfied with the quality of their data and 94% are unsatisfied by their organization’s process for handling duplicate data.
Lumbering extraction, transformation and loading (ETL) routines are shown to cause abundant misery for many, and the aforementioned use of spreadsheets to create “spreadmarts” is likely to be causing problems for companies with excruciatingly tight IT budgets for years to come.
From most important to least significant, here’s how respondents ranked these common barriers to improving data prep:
- Data is difficult to access across system or data silos
- Unsatisfactory inbound data quality
- Poor integration of data prep with BI/analytics tools
- Existing ETL and data integration lack speed and agility
- Modifying existing data warehouse/ETL processes is too difficult
- Managing the entire data preparation workflow is overwhelming
- The sheer volume of incoming data makes it impossible to prepare
- The effect on operational systems is a concern
- Concern that self-service access may break the consistency of organization’s metrics
- Cloud-based data is difficult to access and manage
“Organizations need data preparation processes that can help them record tribal wisdom about data assets, including data definitions, best practices about data usage, and the data’s applicability for certain metrics and algorithms,” report author David Stodder writes. “The value of data preparation processes can be measured by how well the resulting data meets users’ BI and analytics requirements for clean, relevant, and trustworthy data.”
Those requirements, in addition to accuracy, quality and validity, include the frequency of data refresh, accessibility, conformity across data formats and data sets, completeness and depth, consistency across data sets, flexibility, and duplication levels.
Of all the benefits of improving data preparation, TWDI found that shortening the time between data preparation and deriving business insights is the most important benefit that companies want to derive from improving their data prep practices. Another key benefit respondents perceive is a reduction in the time between preparing the data and delivering business-ready data.
Captricity is the non-disruptive solution to getting the data you need, when you need it, while extending the investments of your existing business process solutions. Watch our latest video on how New York Life leverages Captricity and see what our industry leading technologies can bring to your organization.
(*A list of the report’s sponsors can be found here. Captricity is not one of them.)