Adam Gartenberg's Blog

Business Analytics and Optimization, IBM and Social Marketing

A deeper look at Big Data

My fellow bloggers and I sat down with a number of the people in IBM responsible for our Big Data strategy, as well as Joe Cunningham from Visa.

Some highlights from the discussion:

  • Jeff Jonas:  Big data is like a new physics.  False positives, false negatives both collapse with big data.    E.g., last few puzzle pieces when do a puzzle at home are easier than first few pieces.
  • Michael Curry:  There are three legs to the big data story:  unstructured/semi-structured spread across multiple systems (Hadoop good for), streams of data from sensors, etc., analytics used to measure business.  All are growing.  
  • Anant Jhingran:  The biggest of big data is only 1/3 of use case.... Need to factor in size, analytic flexibility, schema flexibility.  Big Data doesn't need to be macho on all three.
  • Anant Jhingran:  It's important to factor in th ecollaboration aspect to intelligence - like what did in Cognos 10. What we found in Many Eyes was that it's not just powerful for visualization, but for collaboration.  Someone says "look what I found," and someone else says "look what happens when add this."  Breakthroughs happen when machines and people come together.
  • Tom Deutsch: Knowing why financial meltdown happened is useful, but knowing how to take action at an individual level is even better.  Used to have to shrink or homogenize to point where lose that.
  • David Barnes: Big Insights, Big Sheets is new class of applications on top of Hadoop.  Don't hear others talking about it as application foundation, but it is.  (See below for a video demo from David)
  • Joe Cunningham (Visa) - Half of his job is looking at Hadoop and if there is business applicability for Hadoop.  The research is now done - very positive results, have identified use cases internally.  Are now looking at how to take out of research and put into Visa's production facilities (where the bar is very high for reliability and security).  Required looking at which parts of the Hadoop stack are mature and which aren't, and building a rock-solid use case with business folks.

All in all, a very interesting discussion, and I'll leave you with David's video on how he used Big Sheets to gauge sentiment for smart phones, culled from 1.4M twitter comments from a 36 hour period.