Q9: Can you please describe the concepts of storing data in a data ware house?

Twenty years ago data was mostly stored in databases. These databases housed all the data a business would need to do analytics. Transaction data, sales data, customer data, demographic data was all neatly collected, stored and analyzed in databases.

A surprising number of companies still store most of their data in databases. It works well for business that just need to look at historical data to conduct basic descriptive analytics.

About ten years ago the amount of data captured in a business and the growing diversity in date sources and data storage brought about the mainstream use of data warehouses in the business world.

Data warehouse are often a collection of databases interconnected so that data can be brought together into one place for reporting and analysis.

Whether you are working with a data base or a data warehouse, you should have a basic understanding of how data is stored. It should be in table format, with header columns and data rows.

A good way to quickly assess the analytics culture of a business is to look at how data is shared among management. Does it look table like? Or is it obvious that most of the time spent by the author was put into decorating? If you can’t easy sort something, then you are not dealing with a good data culture.

The best way to have a good data culture is to have well documented data structures. Any dB admin worth a grain of salt has the data hierarchy mapped out and has a knowledge base to help users know what data is in each field.

Like with finding data, being good at storing data starts with knowing the environment. Any good analyst should have a basic understanding of how to use SQL to pull a query for a data table. Even if you cant do hard core coding, know how data is generally stored in a structure is key.

1075177_10151826941667425_1417094118_n

Another important concept about data warehouses if you have to know how to join or blend data from different sources. When you have multiple data tables in a warehouse you often need to join the data on a common field. Data blending goes on step further as you are often trying to take data that doesn’t have a natural point on common that is easy to join on. Advanced data warehouses and data management tools can blend things easily, but its still important to understand the core concepts of how to join and blend data.

As I mentioned in earlier posts, there is now a new concept taking root that one up data warehouses. Data lakes are being used to address the fact that we have more unstructured data then we have structured data. Data bases and data warehouses were designed only to handle structured data the easily fits into a data able.

Now we have to collect data from images, videos, blogs, comments and other places that are not easily converted to a value. Data blending across both traditional structured data warehouses and new types of data is not easily done in most data warehouses so tools are being developed to bridge this gap.

The lake is no longer a place just to fish, but also to do all the other things a lake can be used for.

So, when it comes to understanding data warehouses, learn who built and/or maintains it and buy them a cup of coffee. Get your hands on the data dictionary, knowledge base, FAQ, metadata.. whatever you can to map out the data environment. If you do that then you can find use the big data stored in a data warehouse to find the right data at the right time.

Q7: What exactly is data science and why the rapid rise of data scientists?

A year ago I might have found it challenging to really answer this question. The first time I had heard of the term data science and a data scientist wasn’t that long ago. And I have been doing some pretty advanced analytics for close to 20 years now. I know the term has been around in academic and research circles awhile longer, but 2014 is the first time I ever saw a job posting for data scientist in big business.

So what is data science? Besides simply being the study of data, it generally refers to using complex models, machine learning, predictive and prescriptive analytics and powerful technology to analyze business data in much greater volume, velocity and variety then possible a few years ago.

And of course the ones charged with doing the data science are data scientists. They understand math, statistics, and theories that can be applied to business data using new technologies and methodologies.

The biggest challenge to being a true data scientist is that you have to be adapt at both technology and working with people. Being a business data expert, knowing how to code and doing higher math are only half the job. You have to also share your data, communicate it in ways that drive action, share and engage with non-data centric people. It’s hard to find people who are good at both.

Image from Forbes Magazine.

In addition, whole some data scientists are educated to be data scientists, very, very few actually have any kind of degree in data science. That kind of degree really didn’t exist until very recently. Instead most data scientists have advanced degrees is related subjects and have migrated into the business world do to market demand.

That demand has been growing at a staggering rate the past few years as every day we generate more and more data across the planet. President Obama first employed a data scientist for his campaign in 2012. The White House now has a chief data scientist position.

If you were to compare results from job board searches form 2012, you’d see maybe 100 data scientist job postings. Now its easily in the 1000’s. So that’s why the job market for data scientist is one of the hottest around. Lack of training programs, having both tech and people skills, and the booming demand due to unending new data to being analyzed.

Some people ask me if I’m a data scientist I am careful with my answer. True data science is not something I am academically prepared for nor I have never published anything in a scholarly journal. But my real world experience working with data has made me an expert on many aspects of data science.

I guess I feel more like an analyst, but a freakin awesome analyst who can do a lot of things using data that are super important to a business.

Analytics Education – Facilitating a mastery of the fundamentals of analytics is what DMAIPH does best. As a key parnter of the Data Science Philippines Meetup Group, DMAIPH champions the use of using data. All across the world, companies are scrambling to hire analytics talent to optimize the big data they have in their businesses.

We can empower students and their instructors with the knowledge they need to prepare for careers in analytics. Contact DMAIPH now at analytics@dmaiph.com or connect with me directly so we can set a guest lecturer date, On-the-Job Training experience or other analytics education solution specifically tailored to your needs.

Q3: What are some of the current trends in analytics?

Every few months I devote a day to discover what are the current trends in analytics. I do this both to refresh the slides in my presentation and to refresh my mind to see what I may have missed.

The amount of literature out there on analytics continues to blossom at an amazing rate, making it a true challenge to stay well versed on what’s hot and what’s not. I read a new analytics themed book about once a month and I have well over 200 blogs, web sites and social media groups cataloged. So I like to think I’m pretty well versed on what is current.

Every time I go to list the top 5 analytics trends, I find that some things change and some stay the same. Ever since I have been doing this, data visualization is near the top. Business dashboards continue to be a big need. Business intelligence tools evolve and new ones’ pop up, but Tableau continues to be a market leader. 90% of us still use Excel for 90% of our analytics work.

Still a lot has changed. When I started this just 5 years ago no one was really talking about Big Data or Data Science. People just stared discussing using predictive analytics and now its all about prescriptive, even though most of us are still just doing descriptive analytics. For the newbie, descriptive = historical, predictive = forecast models, and prescriptive = really complicated models with a lot of variables to not just predict the future but to show a lot of alternatives as well.

Now if you talk to experts they make think nothing I have mentioned so far is new. But to the novice analyst or to the manager who really doesn’t care what’s it called, she just want’s results… its all new to them.

So I try each time to really find something really new not just to me but truly new to analytics. Six months ago that was the idea of using a data lake instead of a data warehouse. For those still unsure what a data warehouse is, it’s a collection of databases stored and/or connected centrally. Data lakes are used to describe the reality that more and more data is now unstructured data.

The discussion on what is unstructured data and how best to mine it and integrate it with structured data has really been at the forefront for a while now. Going from 80% structured to 90% unstructured in in just a few short years as mankind generates unprecedented amounts of data not easily captured in a database every day.

As of today, if I had to pick 5 topics to talk about it would be (1) Hiring Data Science and Analytics Talent, (2) Big Data Analytics, (3) Data Warehousing and Data Lakes, (4) Data Blending and (5) Mining Public Unstructured Data

Check back with me in a few weeks and this list will change.

The Fundamental of Business Analytics – Business Analytics is the application of talent, technology and technique on business data for the purpose of extracting insights and discovering opportunities. DMAIPH specializes in empowering organizations, schools, and businesses with a mastery of the fundamentals of business analytics. Contact DMAIPH now at analytics@dmaiph.com or connect with me directly to find out how you can strengthen your business analytics fundamentals.

The Number Of Solutions Is Just Not Enough

I just came across a couple of companies like DMAI that provide Philippines based analytics outsourcing to overseas clients. I guess that makes about a half dozen companies that I know of that are seriously trying to take advantage of the huge opportunities out there to push the Philippines to the forefront of global analytics solutions.

However, its just not enough. I see more and more Filipinos everyday employed in analytics for a wide range of companies. The number of analysts out there has mushroomed from a few thousand to tens of thousands in just a few years. Yet, a large percentage of these analysts need a lot of help to optimize the analytics in their businesses.

The efforts of big industry, working with the government and higher education to include analytics training within college curriculums is really picking up steam with dozens of schools in the early implementation stages of preparing tomorrows analytics talent. Yet, the projections are so staggering that even if every schools filled every planned class to the max we will stall have a huge talent shortage.

I am equally excited to dive deeper into the midst of this opportunity as I am sometimes a little overwhelmed with where to focus most of my energy.

Writing books, teaching courses, training, public speaking, setting up data science teams, taking on more outsourcing clients, the list just keeps getting bigger.

The number of solutions is just not enough.. .talk about being at the right place at the right time. No wonder I am having the time of my life.

DMAI Data Science > Where Dreams and Demand Meet

Building a data science team tasked with helping other organizations build data science teams is equal parts dream and demand.

There is a quickly growing need for data science capabilities in the Philippines, but there are few ways for Filipinos to learn how to be data scientists. Almost over night it seems that people are posting job requirements for high powered analytics talent with very little idea of what data science is all about.

Business analytics is just now taking root in academia and being offered as a series of elective classes. Big data is just one class. Predictive and prescriptive analytics are also just one 3-5 month class. Its just not enough.

The big companies who are committed to building their own team are scrambling to find talent in the already hyper competitive BPO industry.

That’s the demand.

Data science as a discipline is still quite new. In the U.S. and India you are starting to see a significant number of degree programs in analytics and data science. I learned a lot about data science before it even had a name. Analytics is deeply rooted at Wells Fargo and I benefited from being in the right place at the right time to get exposed to some pretty awesome analytics efforts.

This experience unlocked an opportunity to become one of top analytic minds in my adopted home, the Philippines. The opportunity of a life time really. Now I am at a point in the evolution of my business, DMAI, where I need to find 3 people like me to join me in my quest. My quest to help organizations in the Philippines set up data science teams.

I need a dream team. Like the Eath’s Mightiest Heroes the Avengers or the NBA Champion Golden State Warriors. the DMAI Data Science Team needs the best of the best who excel in complimenting each other.

We need a big data analyst strong man in the paint, we need a visionary data modeling expert who can create great data models and pass them off to the shooter of the team, the business analyst.

That’s the dream!

It’s time to join the right and be at the forefront of spreading data science across this great island nation so full of potential.

If you feel the call that I feel and are interested then connect with me on LinkedIn and/or send me you resume at danmeyer@dmaiph.com ,

Big Data Analyst > The Guy Making Sure We Have The Data We Need

If you don’t know where that information is coming from and whether you can trust it, then it’s useless.

Imagine your data as water.

The same idea applies to big data analytics. If you don’t know where the data is coming from, your data lake will quickly start to resemble a swamp instead of what it should resemble: a reservoir, something that guarantees access, quality, and provenance.

The role of the DMAI big data analyst is at the guy managing the dam at the mouth of a big river. Data analysts constitute the foundation of a data science project and they are trusted with the responsibility of capturing, storing and processing the relevant data. Data Collection, Data Warehousing, Data Transformation and Data Analysis – these are typical tasks of a data analyst.

They are the professionals who play with the tools and frameworks, like Hadoop or HBase, in a distributed environment to ensure that all the raw data points are captured and processed correctly. The processed data is then handed over to the next group of people, the machine learning experts, for taking it further.

In order to call your data a true “reservoir” or “lake,” you big data analyst needs to be able to provide the business-level guarantees that one comes to expect from a data warehouse.

If you are able to create this type of environment the you should have no problem using data analytics in your business, then you are the ideal Big Data Analyst candidate. You are a pro with apps Hadoop, MapReduce or HBase and have the analytical skills required to become a successful data analyst.

A data analyst should be flexible to learn new tools according to the changing business needs and always be willing to upgrade to specialized techniques related to data analysis. Just like the guy controlling the flow of water from a lake to the community that lives off it.

Once we have the guy who makes sure we have the data we need, when we need it, then the DMAI Data Science Team will be complete.

Data Modeling Analyst > The DMAI Data Science Team Middle Man

The person is the middle is often the most important one. When it comes to data science, the person who takes the data provided by the big data analyst and then gives the output of refined data to the business analyst is often the data science team MVP.

As modeling experts play the role of a link between the data analyst and the business analysts.They have to know both the business and the data and then also know which type of analytics to apply.

Modeling experts are primarily responsible for building data models and developing algorithms to draw conclusive information. Their job is to ensure that the derived information is well researched, accurate, easy to understand and unbiased.

Ideal Candidates with statistical background, having a deep interest in quantitative topics, and are usually preferred for the role of machine learning experts. The ideal professional must have a solid understanding of data algorithms and data structures in specific, and software engineering concepts in general.

Knowledge and experience with not only descriptive analytics, but also both predictive and prescriptive analytics is a plus.

Descriptive Analytics looks at the past to explain the present.
Predictive Analytics uses past data to model potential futures.
Prescriptive Analytics use past data to direct variable present and future options.

If you know someone looking to join the DMAI Data Science team to help businesses and schools around the Philippines set-up and/or build out data science capabilities then please tell them about this post.

What Is Data Science and Who are Data Scientists?

Per Wikipedia, Data Science is the extraction of knowledge from large volumes of data that are structured or unstructured, which is a continuation of the field data mining and predictive analytics, also known as knowledge discovery and data mining (KDD).

Does anyone know a “data scientist”? Data scientists work with large data sets, analysis models, and technological solutions to help businesses drive more data-driven decisions. This is known as data science. Data scientists should have these six skill sets:

Tech Skills

Programmer
Statistician
Domain SME

People Skills

Artist
Client Facing
Communicator

As you can imagine, it is very difficult to find people who have expertise in all 6 skills sets.

The unique blend of skills required for a role on a data science team is being debated and almost everyone around the globe who is associated with Big Data, Analytics and Visualization has opinion on this topic.

DMAI has determined that the best lineup for our clients in the Philippines is a veteran business analyast, a big data analyst and a data modeling expert.

Ask me how you can get a data science team set up in your business.

Calling All Analysts! It’s Time To Step Up And Do More With Your Skills. Join The DMAI Data Science Team.

The DMAI Data Science Team

The DMAI Data Science Team is being assembled to offer companies and schools with the training and consulting they need to implement analytics strategies in their organizations.

Headed by analytics guru Daniel Meyer, this team of analytics professionals with diversified skill-sets will guide organizations as they build analytics teams, design analytics programs and empower the use of analytics to drive more data-driven decisions.

For your data science project to be on the right track, you need to ensure that the team has skilled professionals capable of playing three essential roles – Big Data Analyst, Data Modeling Analyst and a seasoned Business Analyst. The presence of these three types of analytics professionals, working together for a common goal, will result in proper analysis of relevant information for predicting the behavior of consumers, in line with the business objective.

With this end goal in mind, we are looking for three super analysts to join our team and fill each of the components. Here are the roles:

Big Data Analyst:

The role of big data analyst is at the base of the pyramid. Data analysts constitute the foundation of a data science project and they are trusted with the responsibility of capturing, storing and processing the relevant data. Data Collection, Data Warehousing, Data Transformation and Data Analysis – these are typical tasks of a data analyst.

Ideal Candidate for the Big Data Analyst role: A Big Data Analyst is predominantly a technical role. The ideal candidate does not need to be very academic but must possess technical competency on the back-end frameworks and tools used for capturing the data points. If you are pro with Hadoop, MapReduce or HBase, then the role of a data analyst would perfectly match your profile. Besides technical acumen, analytical skills are also required to become a successful data analyst. A data analyst should be flexible to learn new tools according to the changing business needs and always be willing to upgrade to specialized techniques related to data analysis.

Component 2 – Data Modeling Analyst

Analytics modeling experts play the role of a link between the data analyst and the business analysts. They are primarily responsible for building data models and developing algorithms to draw conclusive information. Their job is to ensure that the derived information is well researched, accurate, easy to understand and unbiased.

Ideal Candidate for the Data Modeling Analyst role: Candidates with statistical background, having a deep interest in quantitative topics, and are usually preferred for the role of machine learning experts. The ideal professional must have a solid understanding of data algorithms and data structures in specific, and software engineering concepts in general. Knowledge and experience with both predictive and prescriptive analytics is a plus. Capability of handling computational complexity can be considered as an added bonus.

Component 3 – Business Analyst:

Data exploration and data visualization are the two most important responsibilities associated with the role of a business analyst. Business analysts work with front-end tools related to the core business and interact with the higher management of an organization. They further analyze business-level data provided by the data modeling analyst to find out insights related to the organization’s core business interests.

Another important responsibility of a business analyst is to coordinate with the big data analyst and the data modeling analyst to make them understand the business objectives and identify possible focus areas. The ultimate responsibility of a business analyst is to produce actionable insights based on the processed data and help the company leadership in their decision making process.

Ideal Candidate for the Business Analyst role: Business analysts should have expert level knowledge on the underlying business data and source systems. The ideal candidate should have an eye for details and must possess exceptional analytical skills. Moreover, solid understanding of the organization’s business model and the ability to think out of the box are two important qualities that all business analysts should definitely have. It is also important to have sufficient technical skills to come up with precise dashboards for representing business data in a structured manner. Experience with Tableau a plus.

If you are interested in any of these roles with DMAI, please email me directly @ danmeyer@dmaiph.com

Compensation packages will be negotiated based on experience and availability. A part-time arrangement is possible for a pre-defined time period as we build out the capabilities in the team. Potential ownership in a spin-off of DMAI is also a possible form of compensation.

The primary job functions of the team will be related to consulting and training organizations on areas of expertise as well as working together on analytics projects for clients.

Our end goal is to come into an organization and empower those in the organization to address needs in their analytics usage and to grow more competent analytics teams. We will do this for both companies using analytics and schools teaching people to be analysts.

The Current Analytics Talent Landscape in the Philippines – Updated

Updated on 10/26/16

Something I was working on for a potential client, that I thought interesting enough to share.

Here are some of the common characteristics of the three types of analytics talent you will find in the Philippines. Keep in mind that analytics is still a fairly very new concept in the Philippines, but I am convinced its primed for continued growth.

From my experience the analytics talent in the Philippines can be broken into three groups.

Fresh Grads
Entry Level Analysts
Experienced Analysts

For the sake of comparison, I will speak mainly of analytics generalist positions like business analysts and operations analysts. More niche analytics jobs like financial analysts or quality analysts often come from different backgrounds then the bulk of the analytics talent I have worked with.

The first group of analysts are fresh grads. By and large they come from programs like IT, ComSci, Marketing, Business and other related courses. The ones with technical degrees all have some coding skills, know a few programming languages, are very comfortable with Excel and Access and have a general theoretical knowledge of databases, data warehouses and how big data is collected, stored and managed. Generally the business and marketing grads, have less technical skill, but are better prepared for the communication and data sharing side of analytics.

There are several thousand of these graduates entering the workforce every year, but a small % of them are really prepared for analyst careers. Several schools have launched Business Analytics elective tracts, but they are just getting started.

Most fresh grads with analytics talent find jobs in retail or in call centers, many as tech support or IT staff. The ones that do end up with the word analyst in the title are actually more like data encoders or just run reports. They do very little actual analysis.

From this pool, the English and communications requirements of many analyst jobs screen out 75% of the applicants. Fresh grads who start in analyst roles make salaries of between 10-20,000 PHP a month.

The second group of analysts comes the way most people have become analysts… accidentally. They have very similar educational experiences to the fresh grad batch, but started as a CSR or IT or Tech Support and rose above their peers to take on more complex duties.

These analysts find ways to add value to their reports, or have a propensity to solve hard problems or have a tech skill that really stands out. They are promoted to entry level analyst jobs where they are generally used with business lines to do metrics and KPI reporting, assist with process improvement initiatives.

They start to become very skilled, but generally in only 1-2 applications. Their talent is very concentrated and they are not generally pushed to do more. Whether it be working with certain BI or analytics tools, CRMs or other applications, they become very proficient end users but rarely learn the concepts that allow them to move easily between companies. They generally make salaries of about 20-30,000 PHP a month.

However, that doesn’t stop them from hopping around quite a bit. The severe talent shortage for analysts in Metro Manila see a lot of analysts with 1-2 years under their belt get pirated and over a 5-6 year period you see they working for 3-4 companies, each time chasing more money. And rarely does this make them a better analysts as they have a lot of knowledge about a few things, but have not really mastered a competency in anything.

The only training they receive is in-house training on new systems that is generally rolled out from abroad/above. This is one of the biggest difference between analysts in the US and India versus the Philippines. There is little investment in the analyst to grow. There is no encouragement to innovate they get bored pretty quickly which is why they are so susceptible to jump for a little more money.

Based on my observations, maybe 20% of this pool of thousands stick with the same company and rise up to be an experienced analyst in their original company. And you find the ones who stick are generally employed by US, Indian and European BPOs. They make about 25,000 and up PHP a month.

The final group, the more experience analysts are a rare breed. They have the skill similar to what you would expect from an Indian or US analysts with 5 years of experience. They have mastered a couple of disciplines (apps, systems, dbases, etc) and have carved out a good niche. They get paid at least 30-40,000 PHP a month and are firmly established with their employers.

They don’t hop for more money and they tend to be pretty loyal. The best way to pry them away is to offer them something new and different to play with. When you appeal to their curiosity, then they will consider hoping for more money. This is the play the HP, IBM, Google, Citibank, and others who have set up analytics teams are doing. They are trying to entice top talent with both money and new opportunity.

There is also a small, but growing number people in the Philippines who are at the level I was when I left Wells Fargo. Analytics Experts who can offer you a wide range of analytics solutions, understand how complex analytics works and are truly on the cutting edge. A lot of these analysts are now being classified as data scientists. The salaries for these positions can be 50,000 PHP or more a month.

Traditionally data scientist have advanced degrees in statistics, math or some other heavily technical field of study. They generally focus on building models and mining big data using advanced software. They have mastered several coding languages and use predictive and prescriptive modeling techniques. If I had to put a guess on this, there might be a couple thousand true data scientists in the Philippines right now. Hardly enough to go around.

In reality, many of the job postings across the Philippines for data scientists are actually looking for something different. The term is the current hot buzz word and many traditional analyst jobs are being mislabeled as data science jobs. It is very important when hiring someone who has data science in their background to make sure they really have the level of expertise you need.

Bottom line though, if you are looking for someone who is curious, adept at technology, loves solving problems and is data hungry, you can find them in the Philippines.

These thoughts are solely based on my observations and research; I would love to hear others either validate or counter any/all of my conclusions.

If you would like to know about the current state of analytics in the Philippines, please check out my new book, Putting Your Data to Work. The book serves as a guidebook for Filipino professionals to better understand how to get more data in their business. Connect with me and I’ll let you know how to get a copy.

Analytics in the Philippines – The Philippines is at the center of the action when it comes to solutions to the global need for analytics. Blessed with a solid foundation of young, educated and English speaking workforce, companies around the world are look for Filipino analytics talent to fill analytics positions. DMAIPH was set up to facilitate these solutions and bring the talent and the business together. Contact DMAIPH now at analytics@dmaiph.com or connect with me directly so we can help you take advantage of this unique global opportunity.

	DMAIPH on You Don’t Build A Busine…
	Ioliveiraserpa on You Don’t Build A Busine…
	DMAIPH on The Current Analytics Talent L…
	John on The Current Analytics Talent L…
	Dominic on Outsourcing Tip > Size the…