How to evaluate big data products?

In the current world of advanced communications, there is a significant rise in the amount of data that is generated daily. Data can come from multiple sources, such as social media, sensory devices such as thermostats, and from personal devices such as mobile phones.

These data that are generated can actually be put into use if there are effective tools in mining them. However, how do we know what data can be mined and what useful information can be gathered from them?



Big enterprises, such as banks and major chain stores, will often hire data analysts to improve their business operations or to predict a market trend. They may spend hundreds of thousands dollars to collect data, and yet another hundreds of thousands to purchase analytics software or hire data analysts. In smaller companies, this may seem like a luxury and at times not feasible to them due to costs concerns.

There are two questions one should ask before getting any analytics software or hiring any data analysts for their company:

1. What do I aim in getting from analyzing the data? 
2. What are the most cost effective methods to analyze data in the long run? 

Data if untapped is wasted, but tapping on the data should not be a costly investment. Big enterprises often need to invest in a high upfront cost for collecting data, as well as getting the right people and software to analyze the data. On the other hand, smaller companies have trouble justifying to themselves how useful will analyzing data be, and how much money they should spend.

These two market sectors can actually be serviced by having a single product if it meets the following criteria:

• reduces upfront setup costs, 
• eases data collection processes, 
• extensible analytics that can be added across time 
• specific customized applications that can easily be created, possibly by customers themselves or 3rd party vendors, in future 

When selecting such a product, look for one that enables low upfront costs or one that has a monthly payment offering. This will often allow for a faster budget approval from finance department, as well as give you a chance to try out the product before committing any further investments.

The product should include sensory devices such as cameras, or an automated way of data collection. As patterns in data are not easily representative if the sample space is small, consistent data collection over a period of time is absolutely needed for accurate data analysis. Employing staff to do manual data collection or ad-hoc sampling is a very inefficient and time consuming method which is strongly not recommended.

As technology improves over time very quickly, the product will also need to be future proof by having extensible components without the need of direct supplier involvement. It will be best if the product allows 3rd party vendors or developers within a company's IT team to extend and create what is needed down the road.

 It may be hard to find a product that fulfills every single criteria, but rather than committing to a costly white elephant solution, effort should really be spent searching for such a good product.

Happy finding and do remember all data collected is useful. It only depends on how we actually use it.