Finding the Right Data at the Right Time

Sir Conan Doyle’s famous fictional detective, Sherlock Holmes, couldn’t form any theories or draw any conclusions until he had sufficient data. Data is the basic building block of everything we do in analytics: the reports we build, the analysis we perform, the decisions we influence, and the optimizations we derive.

Basil Rathbone as Sherlock Holmes

Several years ago I came across a book called the Accidental Analyst (* The book opens with the questions, “Are you drowning in a sea of data? Would you like to take control of your data and analysis to quickly answer your business questions and make critical decisions? Do you want to confidentially present results and solutions to your managers, colleagues, clients and the public?”

Written by two Stanford professors, the book explores how and why people become good analysts and goes into detail about how to approach analytics successfully. After reading the book I was inspired to come up with a way to teach analytics to college students and fresh graduates.

The core of both the book and my program hinges on the ability of an analyst to find the right data at the right time. The authors suggested that identifying your data is where it all starts. Identifying exactly what you need to address whatever it is that you need to report.

Back at Wells Fargo, the single greatest attribute that I had that made me successful was my ability to size up how long it would take to deliver something. Knowing what data I would need, where I would find it and how long it would take to analyze it to come up with something useful made me somewhat of a wizard in the minds of the team.

Finding the right data at the right time requires one to first know ends and outs of their data. You have to know how the data is captured, where it is stored and how it makes its way to you. Knowing the data architecture in your business is the key.

So you have to get to know the people who know where your data comes from and how it gets there. Learn from them. Partner with them. Buy them doughnuts.

A couple of years ago I came across an analogy being used to describe data in a business. That of a data lake. A data lake is the living, breathing, evolving pool of all the data in a business. If you have a good data architecture, and you can navigate it fairly easily, then you have a data lake. Ideally, your business has data structured in such a way you can live off it. Data to a business is like water to living things… it sustains life

So once you have the lake mapped out, then you have to learn how to fish it. Knowing where the fish are biting is another key. Once you know what data you need, you have to know how to get to it quickly.

Business Intelligence tools help us here. As does coding languages to extract data from a database. These are your fishing tools. You have to practice using them to be good at getting the right data at the right time.

Another way to optimize your data search is to save your work. Of as I call it leave yourself breadcrumbs. Save the query. Cut and paste the code into a document and save it. Write down the steps. Whatever you need to do to replicate what you just did so you can do it again in the future without starting over from scratch.

So to recap, if you know data structure, you understand how data is stored and you leave yourself clues to do things faster next time.
Now the other part of the equation is knowing if the data you are using is the right data. Finding data quickly doesn’t do you any good if you bring back the wrong data.

So, how do you know if the data you are using is the right data to be using?
I can’t count the number of times I asked myself that question. In general, just about every new analysis or project or research or whatever it is you are using data for, you have to ask that question at some point.

Even data you have used a hundred times and comes from a highly trusted source needs to be scrutinized.

Now if you work with data every day in a familiar format, from the same source and with no changes to the data gathering and storage process you don’t have to spend much time validating it. Usually you will see problems when something just doesn’t look right when you are doing the analysis.

On the other hand, things get a whole lot trickier when you are using data from a source you don’t use often, or something has changed in the way the data is populated or if it’s the first time you are using the data.

When this happens, I have a few suggestions on how to validate the data.

  • First off, pull the data, do your analysis and draw some conclusions. If it passed the eye test and it feels ok to you, then your job is just to validate it.
  • One simple way to do this is pull the data again the exact same way to make sure you get the exact same data. Or change one parameter like the dates used in the query. See if that significantly alters the way the data looks and feels.
  • Another option is to have someone else do the same thing independently. See if they get the same results you do. You can also find someone who knows the data to look over your work to see if it makes sense to them.
  • Whatever you do, the best way to prevent publishing or using bad data is to involve someone else. Not always possible, I know, but it’s the best way to go.

Another suggestion is to (1) get the data, (2) do some analysis, and then (3) step away for a while. Come back to it with fresh eyes. Don’t let our minds play tricks on us by making us see what we want to see and not what is really there.

I have seen several articles showing research that most time doing data analysis is actually spent cleaning data. In a lot of businesses, the data lake has become a data swamp, clogged with bad or unusable data. As the % of unstructured data increases daily, it’s easy to see how data swamps have become the norm. Even the most robust data collection and mining can run afoul if the data is not trustworthy.

I can’t stress this enough. No matter how good you are at analysis, or what tool you are using to do the analysis, if you don’t have an understanding of what happens to the data before it gets to you then you are probably not drinking from a clean lake.


DMAIPH – Decision-making, Analytics & Intelligence Philppines

Over the past few years businesses in the Philippines have invested heavily in big data, analytics and data science, but still have not achieved the expected outcomes of data-driven companies.

Based on our learnings from the 100’s of Filipino businesses and 10,000s of Filipinos who have taken part in DMAIPH Analytics trainings all across the country, we have crafted a proven,  non-technical approach to upskilling your team in analytics.

In 2019, we will be launching two new training programs: (1) Our DMAIPH Applied Analytics Master Class series for executives, leaders and decision-makers and our (2) DMAIPH Applied Analytics Boot Camp series for practicing analysts.

We will feature case studies of real Filipino run business, exercises based on actual analytics challenges being solved by Filipino analysts, and provide you with a copy of my book, Putting Your Data to Work, an analytics guidebook for the Filipino professional

Connect with us via our marketing partner, to learn about upcoming analytics trainings and events. 



Data Science and Big Data Analytics For Finance

Was recently asked to do a one hour talk on data science and big data analytics for Finance… so I created A Step By Step Process To Get More Value Out Of Your Finance and Accounting Data.

To get started, we will discuss at a high level what is analytics, big data and data science and how it can be used in Finance and Accounting to get more value out of all the numbers you have in your business.

Per Deloitte, “In today’s highly competitive business environment, companies need more from Finance than accurate financial statements and reports. They need forward-looking, predictive insights that can help shape tomorrow’s business strategy and improve day-to-day decision-making in real time. “

New IT applications and infrastructure such as big data technologies, predictive analytics, as well as modern mathematical methods are opening up new possibilities for gathering and processing large amounts of data and opportunities for generating value.

They keys to a sound data science and analytics approach to Finance include the following:

  • A Process for Using Big Data to Answer Business Questions
  • A Well Mapped Data Lake of all the Data Finance Needs
  • The Right Mix of Analytics Talent, Technique and Technology
  • A Top Down Embrace of an Analytics Centric Culture

By translating data into insights around financial statements and operations, the finance team can unlock and create new value. Being able to identify unrealized and often unexpected potential as well as quickly and decisively mitigating risk, data science and analytics can take your team to a new level of insight and performance.

This in turn supports the finance function to make better decisions by being able to understand what has happened and why, and then predict what may happen next. The end result is a strategy built on data and one with a much higher rate of success then ones based on intuition or gut feel.


The Fundamental of Business Analytics – Business Analytics is the application of talent, technology and technique on business data for the purpose of extracting insights and discovering opportunities. DMAIPH specializes in empowering organizations, schools, and businesses with a mastery of the fundamentals of business analytics. Contact DMAIPH now at or connect with me directly to find out how you can strengthen your business analytics fundamentals.

APEC Data Science & Analytics Key Competency #1: Operational Analytics

According to the APEC (Asia Pacific Economic Cooperation)  Advisory Group, Operational Analytics is one of the key competencies of a Data Science & Analytics professional working in the region.

By definition, a DSA professional uses data analytics and specialized business analytics (i.e. business intelligence) techniques for the investigation of all relevant data to derive insight in support of decision-making.

Operational analytics is made up of all the analytics processes within an organization that take data and transforms the data into actionable intelligence. In short, this is management reporting.

Without a doubt the most widely used form of analytics, management reporting is deeply ingrained into the culture of data-driven organizations.

I often liken management reporting to a pyramid. The bottom of the pyramid is the data or the base of decision-making in an organization.

The middle of the pyramid is the processes of operational analytics. Where the data is transformed.

The top of the pyramid is the decision-making. Managers need intelligence that comes in the form of insights. Great analysts deliver these insights in reports, dashboards and visualizations.

DMAIPH offers a wide range of analytics centric training solutions for professionals and students via public, in-house, on-site, and academic settings. We tailor each training event to meet the unique needs of the audience. If you need empowerment and skills enhancement to optimize the use of analytics in your organization, we are here to help.

Contact DMAIPH now at or connect with me directly to set up a free consultation to learn which of our DMAIPH analytics training solutions is best for you.

Actionable Management Reporting

Except from my upcoming book on analytics for the small business owner…

One question I get asked a lot is what should someone do when they know the data they are reporting and/or using in their analysis is not the best data available?

  • Typically, Excel and PowerPoint are the primary tools used to provide management reporting to a company’s leadership. In the past few years there have been major technology innovations in business intelligence applications and data visualization software that have taken management reporting to a whole new level.
  • Recruiting has seen a huge increase in number and types of reporting tools available to deliver very fast and very detailed recruitment analytics.
  • This leads up to the concept of a business dashboard… which we will get to later.

No matter what part of the business you work in, the first thing to do is to define the current Key Performance Indicators (KPIs) being used in decision-making.  Often right off the bat, some of the KPIs being reported aren’t even being used.

You can do a simple survey, asking end users to rank in order of importance the KPIs they get. Also ask if the ones at the bottom are even useful or should they be eliminated if no one is using them.

At the same time you should be working on understanding what computations go into each KPI. Often we just do simple counts, total and averages that mask more important data. On the flip side, we tend to over complicate things with extravagant weighing and scoring. Either way, we need to make sure we know exactly what is being reported and how does the final data point come to its end state.

The next step is to look at the data architecture to make sure there is nothing happening upstream that might impact the data we are using in the KPIs. Before making changes to the KPIs we want to have the full view of what happens before the data gets to the end user.

Now we are at the point where we can start experimenting. What happens when we swap out data points? Or if we change a variable in a calculation? Or we pull the data from a different source? The questions are endless. Pick a few, make some changes in a test environment and start sharing the updated KPI data. See if it has more value with the end users.

Again, this shouldn’t be hard. But of course in many organizations a lot of consequences can result from a simple change to just one KPI. Spreadsheets may have to be reformatted, review processes may have to be updated, and dashboards may have to be redesigned. But in the end, what is more important? Making decisions with crappy data or setting a standard to let the reporting process evolve as the business evolves?

This come back to my point earlier, changing KPIs is as much sales as it is analysis… that you have to be ready to share a story, back it up with data, and really influence the minds of senior management that updating the KPIs makes good business sense.

If you are at a point where you are trying to figure out what KPIs aren’t working anymore or you need help in building a business case to change some KPIs, let me know. I’m here to help.


Business Strategy with Analytics – Aligning a business strategy to drive an organization forward requires a robust analytics solution. Businesses who have good analytics tend to be much more profitable and efficient then ones that do not. DMAIPH has helped dozens of companies in both the U.S. and the Philippines with adding more data analysis in their business strategy. Contact DMAIPH now at or connect with me directly to find out what we can do to help you align your business strategy with analytics.



















Most Analysts Are Spending Only 20% of Their Time on Reporting

In many cases reporting is either something that is set and stone and just needs to be filled or there is a time crunch forcing quick and dirty reporting.

Little time is devoted to using the data for storytelling, maximizing data visualization and really providing the audience exactly what its needs.

% Finding Analyzing Reporting
10 12% 6% 33%
20 14% 10% 39%
30 20% 31% 24%
40 6% 14% 2%
50 31% 16% 2%
60 14% 18% 0
70 0% 0% 0
80 0% 2% 0
90 0% 0 0
100 0% 0 0

Ideally, at least a third of the time should be spent post data gathering and analysis to really give the end user of the data the things they need for intelligent decision-making.

A full one-third only spend 10% of their time on reporting, which to me means that there is a lot of the waste in their analytics process.

If you take a full 40 hour week to complete a high priority, high value report but only have Friday afternoon to boil down your finding into a report, it is highly unlikely that your report will fully capture the fruits of your labor.

However, if the time frame is even shorter… you have to do all this in one day, you are just getting to the reporting phase at around 3:30pm.

You have less than an hour and a half to summarize you methods and boil your findings into a few points.

Making sure you craft a compelling story to really influence decision-making based on intelligent data analysis is likely impossible.

Data is based on a survey I sent to 3,000 of my LinkedIn connections who are either analysts or work closely with data and analysis.

Analytics Survey – DMAIPH conducts quarterly analytics surveys to collect data on current trends in analytics. We specialize in surveys that assess analytics culture and measuring how aligned an organization is to using data and analytics  in its decision-making. Contact DMAIPH now at or connect with me directly to find out more about how DMAIPH can conduct surveys to help you assess the analytics culture in your business.



Data Analytics Training on Feb 21

Did you know that most successful businesses have solid data analytics in place across their entire organization?

Organizations that invest in data analytics generally make much better business decisions then one’s that don’t.

In fact, IBM found a few years back that companies who use data analytics are up to 10x more efficient and 33% more profitable the ones who don’t.

By bringing data together data from across the business, companies can get real-time insights into finance, sales, marketing, product development and much more.

Data analytics enables each team within the business to collaborate, achieve better results and outsell the competition.

Join us on February 21, 2017 in Ortigas, and learn how to turn your business data into insightful and actionable analysis.


Analytics Training – DMAIPH offers a wide range of analytics centric training solutions for professionals and students via public, in-house, on-site, and academic settings. We tailor each training event to meet the unique needs of the audience. If you need empowerment and skills enhancement to optimize the use of analytics in your organization, we are here to help. Contact DMAIPH now at or connect with me directly to set up a free consultation on which of our DMAIPH analytics training solutions is best for you.