The Good, the Bad, the Technical: Considerations for Big Data and Data Science Integration for Small and Medium Enterprises

Big data- it’s a buzz word that is bandied around. In the article “What is Big Data”?, we laid out what the term “Big Data” means in plain language. Now, let’s get a little more technical and talk about how small and medium enterprises (SME’s) can leverage Big Data and data science tools to help them remain competitive. In the world of data analytics, Big Data refers to a combination of structured, semi-structured, and unstructured data. Data points can be mined for use in machine learning projects, predictive modeling, and other advanced analysis applications.

A report by the National Institute of Standards and Technology defines big data as “extensive datasets—primarily in the characteristics of volume, velocity, and/or variability—that require a scalable architecture for efficient storage, manipulation, and analysis.” Other definitions of Big Data assert that it is data exceeding one petabyte, that means one million gigabytes!

The “Four Vs” of Big Data

Now that we have defined Big Data (once again), let's talk about its characteristics. Big Data is characterized by 4’vs, namely, volume, velocity, variety, and veracity. Let’s start by unpacking the first V, Volume.

• Volume
  An unprecedented data explosion in recent years means that by 2025, the digital universe will reach 180 zettabytes (180 1021 ZB). For the last decade, the biggest challenge facing most companies were costs associated with storage. This issue has been addressed with technological advances that will be discussed in this article. That said, SMEs should give thought to a methodology to identify, prioritize, systematize, and analyze relevant data in huge data sets.
• Velocity
  Data is generated at an ever-faster rate. For example, Google receives 3.8 million search queries every minute. Email users send 156 million messages per minute. Facebook users upload 243,000 photos per minute. The challenge for SMEs is that they must find ways to collect, process, and utilize the large amounts of data available for analysis.
• Variety
  Big Data comes in different forms. Structured data is data that can be neatly organized. These types of data are relatively easy to enter, store, query, and analyze. Unstructured data is more difficult to classify and extract value from. Examples of unstructured data include emails, social media posts, word processing documents, audio, video, and photo files.
• Veracity
  This refers to the quality of the data collected. If the source data is incorrect, the analysis will be worthless. As the world moves toward automated decision-making (computers making choices instead of humans), it becomes crucial that SMEs be able to access quality data.

Beyond the Big Four Vs

Recently, Big Data practitioners and thought leaders have proposed additional Vs. They are variability and visualization. Now let’s take a closer look at each.

• Variability
  The meaning of data is constantly changing. For example, since words often have multiple meanings, it is very difficult for computers to perform language processing. Data scientists must resolve this issue of variability in such instances by creating complex programs that understand context and meaning.
• Visualization
  Non-technical stakeholders and decision-makers must understand the data. Visualization is the process of creating complex charts that tell a story, turning data into information, turning information into insights, turning insights into knowledge, and turning knowledge into advantages.

Where does Big Data come from?

Big Data comes from multiple sources. The following are the three primary sources of Big Data.

• Social Data:
  Social data come from likes, tweets (and retweets), comments, video uploads, and general media that are uploaded and shared via popular social media platforms. These kinds of data provide invaluable insights into consumer behavior and sentiment. They can provide enormous insights for SME marketing analytics. The public web is another good source of social data. Tools like Google Trends can be used to increase the volume of quality Big Data an SME can access.
• Machine Data:
  Machine data is defined as information which is generated by industrial equipment, sensors that are installed in machinery, and even web logs which track user behavior. This type of data is expected to grow exponentially as the Internet of Things (IoT) grows ever more pervasive and expands around the world. Sensors such as medical devices, smart meters, road cameras, satellites, games, and the rapidly growing IoT will deliver high velocity, value, volume, and variety of data in the very near future.
• Transactional Data:
  Transactional data are generated from all the daily business transactions that take place both online and in brick and mortar locations. Invoices, payment orders, storage records, delivery receipts – all are characterized as transactional data. These data alone are almost meaningless if SMEs don’t leverage them correctly. Most SMEs struggle to make sense of the data that they are generating. Moreover, they do not have the tools to put these data to good use. With Big Data analytics techniques, SMEs can draw meaning from these datasets using machine learning techniques derived from the field of data science.

What is Data Science?

Since 2010, harnessing the power of Big Data to improve business processes and intelligence has been the focus of multinational companies. During this decade-long quest, the main focus of these businesses- think Google, Amazon, Facebook, even Intuit and Target - has been to build a solution for storing data. Recently, frameworks such as Hadoop have successfully solved the storage problem, so the focus has shifted to processing these data. This phenomenon has paved the way for the field of data science to blossom. Artificial intelligence aka machine learning has taken the massive amounts of data generated by commerce, both electronic and through in-person transactions, and helped both corporations and SMEs sharpen their competitive edge.

Note: Frameworks are the open-source software for storing data and running applications on clusters of commodity hardware. These software’s provide massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks.

Data Science is a blend of various tools, algorithms, and machine learning principles focused on discovering hidden patterns from the raw data. How is this different from what statisticians have been doing for years, you might ask? The answer lies in the difference between explaining and predicting. The image above illustrates how data analysts use the historic record captured in a dataset to explain a pattern or phenomenon – they explain. Conversely, data scientists not only perform exploratory analysis to discover insights from datasets, but also use various advanced machine learning algorithms to identify the occurrence of specific events in the future. In other words, data science is mainly used to make decisions and predictions through predictive causal analysis, descriptive analysis (predictive and decision science), and machine learning.

Big Data vs Data Science

Let’s wrap things up to reveal the real difference between these two terms.

Data science is an evolutionary extension of statistics. Data science uses computer science technology to process large data sets while statistical methodologies are limited to relatively smaller datasets. On the other hand, Big Data deals with a large number of heterogeneous data collections from different sources and is not available in the standard database format as we know it. This means that data will not be made into tables or graphs.

Big Data classifies data into unstructured, semi-structured, and structured data.

• Unstructured data – social networks, emails, blogs, digital images, and contents
• Semi-structured data – XML files, text files, etc.
• Structured data – RDBMS, OLTP, and other structured formats.

Although structured data is easy to understand, unstructured data requires customized modeling techniques to extract information from the data with the help of computer tools, statistics, and other data science methods.

Some Key Takeaways – Big Data vs. Data Science

There are some major differences which we should talk about when comparing Big Data to Data Science.

• Big Data can be used by SMEs to improve efficiency, understand untapped markets, and enhance competitiveness while data science concentrates on providing modeling techniques and methods to evaluate the potential of big data in a precise way.
• The amounts of data that can be collected by firms are huge. Value can only be extracted from Big Data through data science techniques.
• Data Science uses theoretical as well as practical approaches to extract information from Big Data which plays an important role in harnessing its potential. That said, Big Data can be looked at as a pool of data which has no credibility unless analyzed with deductive and inductive reasoning.
• Big Data analysis focuses on large datasets (also known as data mining). Data science makes use of machine learning algorithms to design and develop models to generate knowledge from these large datasets.
• Data Science focuses more on business decisions whereas Big Data relates more with technology, computer tools, and software.

To learn more about how to deploy data science and Big Data to optimize your business, find useful resources in the cloud computing section of our blog. 

‍