Finding the Right Data

“Data! Data! Data! I can’t make bricks without clay!”

-Sir Arthur Conan Doyle

Sir Conan Doyle’s famous fictional detective, Sherlock Holmes, couldn’t form any theories or draw any conclusions until he had sufficient data. Data is the basic building block of everything we do in analytics: the reports we build, the analysis we perform, the decisions we influence, and the optimizations we derive.

Finding the Right Data at the Right Time

Back at Wells Fargo, the single greatest attribute that made me successful was my ability to size up how long it would take to deliver something. Knowing what data I would need, where I would find it and how long it would take to analyze it to come up with something useful made me somewhat of a wizard in the minds of the team.

Finding the right data at the right time requires one to first know ends and outs of their data. You have to know how the data is captured, where it is stored and how it makes its way to you. Knowing the data architecture in your business is the key.

So you have to get to know the people who know where your data comes from and how it gets there. Learn from them. Partner with them. Buy them doughnuts.

A few years ago I came across an analogy being used to describe data in a business. That of a data lake. A data lake is the living, breathing, evolving pool of all the data in a business. If you have a good data architecture, and you can navigate it fairly easily, then you have a data lake. Ideally, your business has data structured in such a way you can live off it. Data to a business is like water to living things… it sustains life

So once you have the lake mapped out, then you have to learn how to fish it. Knowing where the fish are biting is another key. Once you know what data you need, you have to know how to get to it quickly.

I can’t stress this enough. No matter how good you are at analysis, or what tool you are using to do the analysis, if you don’t have an understanding of what happens to the data before it gets to you then you are probably not drinking from a clean lake.

Business Intelligence tools help us here. As does coding languages to extract data from a database. These are your fishing tools. You have to practice using them to be good at getting the right data at the right time.

Another way to optimize your data search is to save your work. Of as I call it leave yourself breadcrumbs. Save the query. Cut and paste the code into a document and save it. Write down the steps. Whatever you need to do to replicate what you just did so you can do it again in the future without starting over from scratch.

So to recap, if you know data structure, you understand how data is stored and you leave yourself clues to do things faster next time.

Now the other part of the equation is knowing if the data you are using is the right data. Finding data quickly doesn’t do you any good if you bring back the wrong data.

So, how do you know if the data you are using is the right data to be using?

I can’t count the number of times I asked myself that question. In general, just about every new analysis or project or research or whatever it is you are using data for, you have to ask that question at some point.

Even data you have used a hundred times and comes from a highly trusted source needs to be scrutinized.

Now if you work with data every day in a familiar format, from the same source and with no changes to the data gathering and storage process you don’t have to spend much time validating it. Usually you will see problems when something just doesn’t look right when you are doing the analysis.

On the other hand, things get a whole lot trickier when you are using data from a source you don’t use often, or something has changed in the way the data is populated or if it’s the first time you are using the data.

When this happens, I have a few suggestions on how to validate the data.

· First off, pull the data, do your analysis and draw some conclusions. If it passed the eye test and it feels ok to you, then your job is just to validate it.

· One simple way to do this is pull the data again the exact same way to make sure you get the exact same data. Or change one parameter like the dates used in the query. See if that significantly alters the way the data looks and feels.

· Another option is to have someone else do the same thing independently. See if they get the same results you do. You can also find someone who knows the data to look over your work to see if it makes sense to them.

In the end, whatever you do, make sure you have the right data.

I will cover all these concepts in more in upcoming my training classes. For a list of training events, please visit

I’ll be conducting the following business analytics trainings over the next few months:

· June 5 in Ortigas (Metro Manila, Philippines)

· July 17, in Pleasant Hlll, CA (San Francisco Bay Area, US)

· August 22, in Bonifacio Global City (Metro Manila, Philippines)

Dan Meyer heads Sonic Analytics, an analytics training, consulting and outsourcing company with offices in Manila and the San Francisco Bay Area. With over 20 years in Big Data, Dan is one of the most sought after public speakers in Asia and has recently begun offering public training seminars in the United States.

We need to look at the data (analytics), plan a course of action (strategy) and share our data-driven viewpoints (presentation). So he has started an internship program under Sonic Analytics to empower the youth the use Analytic, plan Strategy and Present their views… ASP!

Sonic Analytics( brings big data analytics solutions like business intelligence, business dashboards and data storytelling to small and medium sized business looking to enhance their data-driven decision-making capabilities.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s