Tag: Big Data

BI Professionals Spend 50-90% of Their Time ‘Cleaning’ Raw Data for Analytics

Sharing this…

BY ON AUGUST 26, 2015 • ( LEAVE A COMMENT )
Last year, the NYT shined a light on big data’s “janitor” problem – that data scientists and business intelligence pros spend too much time cleaning, not evaluating data. But how big of an issue is it, really?

Xplenty just wrapped a commissioned study of +200 BI pros and found that a third spend 50-90% of their time just cleaning raw data. This is one of the first reports to tie an actual # to the ETL process.

Source: bigdataanalyticsnews.com

From my days at Wells Fargo being an analyst I know how hard it was to maximize your analysis and communication time and minimize time spent finding and cleaning data. This was especially true for me as I was using more unstructured data to do things like competitive intelligence then structured data.

I see it being even more of a challenge now because the % of unstructured data in any business has exploded the past few years. Being able to mine valuable insights from unstructured data is a time consumer, at least until you get a process in place to extract and refresh the data using some kind of technology.

In addition, businesses continue to find new data points to bring into their data warehouses, dramatically increasing the amount of structured data.

What this means is a lot of analysts are spending a lot more time looking through mountains of data to figure out exactly which data to use. Its not going to get easier.

Good data gathering methodologies and nimble BI tools can help cut down on some of the workload, but in the end we just keep making data faster then we have the ability to truly process it.

There is just no replacing the human factor of someone knowledgeable about the business who can interpret the data and decide what data to use and what not to use.

Which makes life even more challenging, because once we determine what data we want to use, we still often have to take the raw data and clean it up so it is valid and so it will fit nicely into our BI tools.

3.8.2

If you have having trouble figuring out what data to use in your business and if you find yourself spending far too much time cleaning the data, perhaps DMAI can help. We have a Data Science team ready to assist your organization with just these types of challenges.

Analytics 3.0, Big Data Equals Big Insights: Learning to Use Big Data to Build a Smarter Global Workforce

That was the title of my quick introduction to Big Data for HR to a crowd of about 1,000 HR professionals yesterday. My agenda was to talk about:

  • Using Big Data in HR to empower more Data-Driven Decision-Making
  • Extracting Key Business Insights using Big Data
  • A Big Data Analytics centered approach to building A Smarter Global Workforce

I must say it was pretty awesome as the topic generated a lot of discussion about the biggest challenges facing HR professionals when it comes to Big Data.

11960185_923787737670982_7880910083394419466_n

Not surprisingly, one of the top challenges the attendees are facing back at work is getting their managers to support Big Data initiatives. There is such a great need for awareness of what Big Data is all about and how analytics is used to extract the right data to give decision-makers the ability to make smarter decision.

To start off, I suggestesd they think of HR Analytics like a Pyramid

Start with the base and gather all the HR Big Data

Build to the middle of using HR specific Analytics

Strategic Focus comes out ofthe top and you get Actionable Insights

If you can show that a Big Data approach adds value, optimizes processes and provide a strong return on investment. Basically you need to use data to support the use of more data.

Identifying data sources and analytical resources can provide guidance in understanding your organization’s needs and capability to adopt a talent-centric data-driven approach.

Having a data-centric culture is the first step in optimizing the Big Data in your business.

And my final word of advice was that you have to be the one to champion Big Data. You can’t wait for someone else to. As a leader in HR, you need to be the one pushing the issue of how to use Big Data to to the forefront.

The Number Of Solutions Is Just Not Enough

I just came across a couple of companies like DMAI that provide Philippines based analytics outsourcing to overseas clients. I guess that makes about a half dozen companies that I know of that are seriously trying to take advantage of the huge opportunities out there to push the Philippines to the forefront of global analytics solutions.

However, its just not enough. I see more and more Filipinos everyday employed in analytics for a wide range of companies. The number of analysts out there has mushroomed from a few thousand to tens of thousands in just a few years. Yet, a large percentage of these analysts need a lot of help to optimize the analytics in their businesses.

The efforts of big industry, working with the government and higher education to include analytics training within college curriculums is really picking up steam with dozens of schools in the early implementation stages of preparing tomorrows analytics talent.  Yet, the projections are so staggering that even if every schools filled every planned class to the max we will stall have a huge talent shortage.

1508982_603376689807778_4892012849582809050_n

I am equally excited to dive deeper into the midst of this opportunity as I am sometimes a little overwhelmed with where to focus most of my energy.

Writing books, teaching courses, training, public speaking, setting up data science teams, taking on more outsourcing clients, the list just keeps getting bigger.

The number of solutions is just not enough.. .talk about being at the right place at the right time. No wonder I am having the time of my life.

DMAI Data Science > Where Dreams and Demand Meet

Building a data science team tasked with helping other organizations build data science teams is equal parts dream and demand.

There is a quickly growing need for data science capabilities in the Philippines, but there are few ways for Filipinos to learn how to be data scientists. Almost over night it seems that people are posting job requirements for high powered analytics talent with very little idea of what data science is all about.

Business analytics is just now taking root in academia and being offered as a series of elective classes. Big data is just one class. Predictive and prescriptive analytics are also just one 3-5 month class. Its just not enough.

The big companies who are committed to building their own team are scrambling to find talent in the already hyper competitive BPO industry.

That’s the demand.

DMAI_GrowMoreDMAI_shemac081815 copy

Data science as a discipline is still quite new. In the U.S. and India you are starting to see a significant number of degree programs in analytics and data science. I learned a lot about data science before it even had a name. Analytics is deeply rooted at Wells Fargo and I benefited from being in the right place at the right time to get exposed to some pretty awesome analytics efforts.

This experience unlocked an opportunity to become one of top analytic minds in my adopted home, the Philippines. The opportunity of a life time really. Now I am at a point in the evolution of my business, DMAI, where I need to find 3 people like me to join me in my quest. My quest to help organizations in the Philippines set up data science teams.

I need a dream team. Like the Eath’s Mightiest Heroes the Avengers or the NBA Champion Golden State Warriors. the DMAI Data Science Team needs the best of the best who excel in complimenting each other.

We need a big data analyst strong man in the paint, we need a visionary data modeling expert who can create great data models and pass them off to the shooter of the team, the business analyst.

That’s the dream!

It’s time to join the right and be at the forefront of spreading data science across this great island nation so full of potential.

If you feel the call that I feel and are interested then connect with me on LinkedIn and/or send me you resume at danmeyer@dmaiph.com ,

Big Data Analyst > The Guy Making Sure We Have The Data We Need

If you don’t know where that information is coming from and whether you can trust it, then it’s useless.

Imagine your data as water.

The same idea applies to big data analytics. If you don’t know where the data is coming from, your data lake will quickly start to resemble a swamp instead of what it should resemble: a reservoir, something that guarantees access, quality, and provenance.

DMAI_DataGovernance

The role of the DMAI big data analyst is at the guy managing the dam at the mouth of a big river. Data analysts constitute the foundation of a data science project and they are trusted with the responsibility of capturing, storing and processing the relevant data. Data Collection, Data Warehousing, Data Transformation and Data Analysis – these are typical tasks of a data analyst.

They are the professionals who play with the tools and frameworks, like Hadoop or HBase, in a distributed environment to ensure that all the raw data points are captured and processed correctly. The processed data is then handed over to the next group of people, the machine learning experts, for taking it further.

In order to call your data a true “reservoir” or “lake,” you big data analyst needs to be able to provide the business-level guarantees that one comes to expect from a data warehouse.

If you are able to create this type of environment the you should have no problem using data analytics in your business, then you are the ideal Big Data Analyst candidate. You are a pro with apps Hadoop, MapReduce or HBase and have the analytical skills required to become a successful data analyst.

A data analyst should be flexible to learn new tools according to the changing business needs and always be willing to upgrade to specialized techniques related to data analysis. Just like the guy controlling the flow of water from a lake to the community that lives off it.

Once we have the guy who makes sure we have the data we need, when we need it, then the DMAI Data Science Team will be complete.

Calling All Analysts! It’s Time To Step Up And Do More With Your Skills. Join The DMAI Data Science Team.

The DMAI Data Science Team

The DMAI Data Science Team is being assembled to offer companies and schools with the training and consulting they need to implement analytics strategies in their organizations.

Headed by analytics guru Daniel Meyer, this team of analytics professionals with diversified skill-sets will guide organizations as they build analytics teams, design analytics programs and empower the use of analytics to drive more data-driven decisions.

For your data science project to be on the right track, you need to ensure that the team has skilled professionals capable of playing three essential roles – Big Data Analyst, Data Modeling Analyst and a seasoned Business Analyst. The presence of these three types of analytics professionals, working together for a common goal, will result in proper analysis of relevant information for predicting the behavior of consumers, in line with the business objective.

522

With this end goal in mind, we are looking for three super analysts to join our team and fill each of the components. Here are the roles:

Big Data Analyst:

The role of big data analyst is at the base of the pyramid. Data analysts constitute the foundation of a data science project and they are trusted with the responsibility of capturing, storing and processing the relevant data. Data Collection, Data Warehousing, Data Transformation and Data Analysis – these are typical tasks of a data analyst.

They are the professionals who play with the tools and frameworks, like Hadoop or HBase, in a distributed environment to ensure that all the raw data points are captured and processed correctly. The processed data is then handed over to the next group of people, the machine learning experts, for taking it further.

Ideal Candidate for the Big Data Analyst role: A Big Data Analyst is predominantly a technical role. The ideal candidate does not need to be very academic but must possess technical competency on the back-end frameworks and tools used for capturing the data points. If you are pro with Hadoop, MapReduce or HBase, then the role of a data analyst would perfectly match your profile. Besides technical acumen, analytical skills are also required to become a successful data analyst. A data analyst should be flexible to learn new tools according to the changing business needs and always be willing to upgrade to specialized techniques related to data analysis.

Component 2 – Data Modeling Analyst

Analytics modeling experts play the role of a link between the data analyst and the business analysts. They are primarily responsible for building data models and developing algorithms to draw conclusive information. Their job is to ensure that the derived information is well researched, accurate, easy to understand and unbiased.

Ideal Candidate for the Data Modeling Analyst role: Candidates with statistical background, having a deep interest in quantitative topics, and are usually preferred for the role of machine learning experts. The ideal professional must have a solid understanding of data algorithms and data structures in specific, and software engineering concepts in general. Knowledge and experience with both predictive and prescriptive analytics is a plus. Capability of handling computational complexity can be considered as an added bonus.

Component 3 – Business Analyst:

Data exploration and data visualization are the two most important responsibilities associated with the role of a business analyst. Business analysts work with front-end tools related to the core business and interact with the higher management of an organization. They further analyze business-level data provided by the data modeling analyst to find out insights related to the organization’s core business interests.

Another important responsibility of a business analyst is to coordinate with the big data analyst and the data modeling analyst to make them understand the business objectives and identify possible focus areas. The ultimate responsibility of a business analyst is to produce actionable insights based on the processed data and help the company leadership in their decision making process.

Ideal Candidate for the Business Analyst role: Business analysts should have expert level knowledge on the underlying business data and source systems. The ideal candidate should have an eye for details and must possess exceptional analytical skills. Moreover, solid understanding of the organization’s business model and the ability to think out of the box are two important qualities that all business analysts should definitely have. It is also important to have sufficient technical skills to come up with precise dashboards for representing business data in a structured manner. Experience with Tableau a plus.

If you are interested in any of these roles with DMAI, please email me directly @ danmeyer@dmaiph.com

Compensation packages will be negotiated based on experience and availability. A part-time arrangement is possible for a pre-defined time period as we build out the capabilities in the team. Potential ownership in a spin-off of DMAI is also a possible form of compensation.

The primary job functions of the team will be related to consulting and training organizations on areas of expertise as well as working together on analytics projects for clients.

Our end goal is to come into an organization and empower those in the organization to address needs in their analytics usage and to grow more competent analytics teams. We will do this for both companies using analytics and schools teaching people to be analysts.

It All Starts With A Business Question…

A Business Question is any type of question that is asked to help understand the business better.

Business Questions are generally asked by executives and senior leaders to help them make more educated decisions.

Often middle management and professional staff are required to provide the answer to the question.

There is no bad question.

However, there are many bad ways to try to answer the question.

Good analysts are able to take the question and find data to answer the question, analyze the results and report their findings.

This is true with just about any kind of analysis work.

Learning how to best tackle business questions is what DMAI specializes in.

We have several upcoming events where I will be showing people how to answer business questions using the data they have in and around there business.

  • August 4 – Recruitment Analytics @ the DMAI  office in Ortigas
  • August 6 – Fundamentals of Business Analytics with Inspire @ the Richmond Hotel in Ortigas
  • August 11 – Data Analytics  @ the DMAI  office in Ortigas
  • August 13 – Big Data and Social Media Analytics @ PNP Headquarters in QC
  • August 27 – Big Data and HR Analytics @ SMX with Ariva  
  • September 1 – Recruitment Analytics @ the DMAI  office in Ortigas
  • September 15 – Data Analytics  @ the DMAI  office in Ortigas

Lots of opportunities to hear more about analytics in the coming weeks!

10592010_10152674958362425_1982237172_n

In Demand > DMAI Data Analytics Trainings

Next week I will be conducting a training on data analytics.

Data analytics is generally considered to be the examining raw data with the purpose of drawing conclusions about that information. Data analytics is used in many industries to allow companies and organization to make better business decisions.

Topics covered in the training include:

  • What is Data Analytics?
  • The current state of Data Analytics in the Philippines
  • Self-Assessment of our own Data Analytics
  • Finding the Right Data at the Right Time
  • Big Data and Data Warehousing
  • Descriptive, Predictive, and Prescriptive Analytics
  • Business Intelligence and Data Visualization
  • Using Data Analytics to Drive Decisions

What’s extra cool about this training is that is already full. We are going to add a second batch in August.

The need for having good data analytics in a business continues to grow at a pace much faster than the supply of talent can keep up with.

9.0.2a

Besides the topics above, participants will also learn:

  1. How to do public data mining
  2. How to provide data for business intelligence
  3. How to build better reports in Excel
  4. How to manage data in business dashboard

So if you are interested in getting a better handle on the data in your business and how to build a good data analytics solution, DMAI can help.

If Only They Would Look At The Data

Humans by nature are not very data-driven.

Often we act before we think.

In general we go with the gut over the information we are presented with.

It’s just who we are as human beings.

And you see it everywhere.

We come across an inefficiency, a broken process or a half-baked idea and ask ourselves who thought this up?

Why didn’t they look at the data before they made a decision that impacted so many people?

So just for a minute I want to daydream a little.

If only X people would look at and listen too Y data before making a decision, then the world would be a better place.

If only politicians would look at data before spending so much money on that big project.

If only people would really listen to the stories about how dangerous it is to text while driving.

If ever we actually stopped and took the time to calculate the high cost of living facing our children.

Try it yourself. If everyone did a little more of this, think of how awesome it would be.

We have enough data to help us decide just about anything.

In fact we now have whats called Big Data… an almost infinite amount of information to guide us.

We just need to get more people to understand how.

The then humans would really be able to solve the problems we face.

It is a simple hope magnified by 7 billion people and the data they create every day.

20140322-093659.jpg

PATIENCE AND FAITH ARE OUR BEST FRIENDS

I wish i could remember where i found this, but it definitely speaks to me so I wanted to share it again.

“No one can determine the final destination of our life’s journey. Therefore, the next best thing to do is to keep our cool and have faith in ourselves. Whenever I feel I am detouring from my destiny, I try to remind myself that my journey is my path. It’s a journey full of peaks and valleys, sunshine, and rain. This faith allows me to explore unchartered territory with confidence. I am at ease to fail forward. Because even when we fail, we do not lose it all—we can learn valuable lessons, and build the foundation for our next chapter.”

Keeping a can-do attitude helps us to never give up on ourselves. It allows us to recharge, reinvest, and reinvent ourselves by melting down our fear.

And that’s how I will start 2017!

jobspicture2Analytics Leadership – DMAIPH specializes in arming the Data-Driven Leader with the tools and techniques they need to build and empower an analytics centric organization. Analytics leadership requires a mastery of not just analytics skill, but also of nurturing an analytics culture. We have guided thousands of Filipino professionals to become better analytics leaders. Contact DMAIPH now at analytics@dmaiph.com or connect with me directly to discuss a uniquely tailored strategy to ensure you are the top of your game when it comes to Analytics Leadership.