BI Professionals Spend 50-90% of Their Time ‘Cleaning’ Raw Data for Analytics

Sharing this…

Last year, the NYT shined a light on big data’s “janitor” problem – that data scientists and business intelligence pros spend too much time cleaning, not evaluating data. But how big of an issue is it, really?

Xplenty just wrapped a commissioned study of +200 BI pros and found that a third spend 50-90% of their time just cleaning raw data. This is one of the first reports to tie an actual # to the ETL process.


From my days at Wells Fargo being an analyst I know how hard it was to maximize your analysis and communication time and minimize time spent finding and cleaning data. This was especially true for me as I was using more unstructured data to do things like competitive intelligence then structured data.

I see it being even more of a challenge now because the % of unstructured data in any business has exploded the past few years. Being able to mine valuable insights from unstructured data is a time consumer, at least until you get a process in place to extract and refresh the data using some kind of technology.

In addition, businesses continue to find new data points to bring into their data warehouses, dramatically increasing the amount of structured data.

What this means is a lot of analysts are spending a lot more time looking through mountains of data to figure out exactly which data to use. Its not going to get easier.

Good data gathering methodologies and nimble BI tools can help cut down on some of the workload, but in the end we just keep making data faster then we have the ability to truly process it.

There is just no replacing the human factor of someone knowledgeable about the business who can interpret the data and decide what data to use and what not to use.

Which makes life even more challenging, because once we determine what data we want to use, we still often have to take the raw data and clean it up so it is valid and so it will fit nicely into our BI tools.


If you have having trouble figuring out what data to use in your business and if you find yourself spending far too much time cleaning the data, perhaps DMAI can help. We have a Data Science team ready to assist your organization with just these types of challenges.

Thank Your Mentors!

“Although I consider you more friend then mentor, you are one of the key people in my professional life. Your help, guidance and inspiration, especially during my first year here in the Philippines helped set the stage for the new level of success I have been blessed with this year. For that I owe you my humble thanks for being a mentor to me as well as a trusted friend.”

This is a message I shared with some of my mentors here in Manila.

LinkedIn recently sent members an email suggesting they send thanks to their mentors. Pretty cool idea.

Mentor is defined as someone who teaches or gives help and advice to a less experienced and often younger person. I love being a mentor and have many less experienced and younger people I have taught/helped over the years, but even mentors need mentors.

Which makes me truly blessed as I have a long list of people helping guide me.