li {margin: 1em 0}

When To Use Big Data

With all the current excitement around big data, many are eager to get started with a big data project. But not all projects are big data projects. Remember, just having large amounts of data does not necessarily require big data technologies. Thus, project managers must be capable of advising management not only about how, but also when, the business can and should take advantage of big data technologies.

Do You Really Need a Big Data Solution?

Initial Considerations

Before beginning a big data project, companies must ask the following basic questions:

  1. Why do this at all?

    Tamara Dull of the best practices team at SAS advises:

    “Keep in mind that you don’t always need big data; you just need the right data (see Building Off Best Practices).

    “The first, most obvious question is “Why do this at all?” There should be a compelling use case, a competitive driver, cost driver, or some other issue that has been identified where the application of big data technologies is in the critical path to solving the problem. Typical drivers include the information type (for example, under-utilized structured information sources), or the volume of information (retention of IP logs), but in any case, you need to identify exactly why you are pursuing this path.”

    “One of the most important things you should look for is a compelling ROI (return on investment). That is to say, find something for which you can put a value on the cost of the problem before you plan a solution (see Selecting Your First Big Data Project).

  2. Are you using the basic data you already have in a way that engages consumers?

    “The biggest reason that investments in big data fail to pay off, though, is that most companies don’t do a good job with the information they already have. They don’t know how to manage it, analyze it in ways that enhance their understanding, and then make changes in response to new insights. Companies don’t magically develop those competencies just because they’ve invested in high-end analytics tools. They first need to learn how to use the data already embedded in their core operating systems, much the way people must master arithmetic before they tackle algebra. Until a company learns how to use data and analysis to support its operating decisions, it will not be in a position to benefit from big data (see You May Not Need Big Data After All).

Additional Considerations to Help Determine if a Big Data Solution Is Required

Once you have answered the above questions, there are additional considerations. Remember, just because your data is too big for Excel does not necessarily mean your data is “big data”.

A Test to Determine If You Need a Big Data Solution

The following offers a 4-point test based on volume, velocity, variety, and variability of data to help determine if you really need a big data solution:

4 Point Test

Points to Consider When Determining If You Need a Big Data Solution

The following excerpt from SyonCloud provides several points for consideration to help determine if a Big Data solution is required.

More General Tips to Consider Before Starting a Big Data Project

And here are a few more general tips to consider before undertaking a big data project:

Limitations of Hadoop

Since much of big data is "unstructured", it must be formatted to make it suitable for data mining and analysis. Hadoop solves this problem, and is the core platform for structuring big data.

To help you determine if a big data solution would be beneficial, you should also consider Hadoop’s limitations.

About Patti Gilchrist

Patti Gilchrist

Patti Gilchrist is a Sr. Technical Manager with 25 years experience implementing strategic enterprise initiatives.

Patti has a reputation for effectively translating business problems into innovative solutions and creating strategic roadmaps to achieve business goals.

Visit Patti Gilchrist's Other Sites

Free PM Study

The Entrepreneurial PM

See Also

You May Not Need Big Data After All

8 Proven Steps to Starting a Big Data Analytics Project

4 Point Test to Determine If You Need a Big Data Solution

Selecting Your First Big Data Project

Building Off Best Practices

How To Know if a Big Data Solution is Right For Your Organization