Information has always been central to the functioning of an enterprise. Today, with the fast pace of business, access to the right information at the right time is critical. Enterprises need information to track the status of the organization; to answer questions; to alert it to changes, emergencies, trends, opportunities or risks; to predict, model and forecast their business.
To this, we must add one more information goal, one that is so valuable but so elusive that it has been little more than a dream: to find the unexpected: the unknown threat, the unknown opportunity. These so-called black swans lurk on the edge of our understanding, obscured by the over-abundance and scattered nature of information in the organization today.
Big data tools and technologies have been developed to help manage, access, analyze and use vast quantities of information. Big data is often defined by the three V’s: Volume (amount of data); Velocity (the speed at which it arrives); and Variety ( the number of data types or formats). But the value in big data is not really rooted in its abundance, but rather in how it is used. Big data tools enable us to understand trends and answer questions with a degree of certainty that was not possible before—because we did not have enough data to support our findings. Big data approaches to healthcare are starting to enable treatments that take into consideration the particular characteristics of a patient– their age, history, or genetic makeup. We use these characteristics as a filter or lens on the medical research literature, focusing what we know within the context of that patient. Given enough data, we can also find unexpected patterns. For instance, one project uncovered previously unknown markers for predicting hospital readmissions for congestive heart failure, saving a health organization millions of dollars. Big data techniques have helped predict the next holiday retail season, uncovered patterns of insurance fraud, and emerging trends in the stock market. We use these tools to find out if customers are satisfied with our products, and if not, why not. Political campaigns use them and so do managers of baseball teams.
Briefly, then, big data gives us plenty of data to analyze overall trends and demands, but it also helps us understand individuals within the context of a solid set of information about people who are like them. Instead of aiming at a mythical “average”, it lets us treat customers, patients and voters as individuals.
With new technologies like big data, we are at the beginning of a very complex new relationship between man and machine. Machines can find patterns and make recommendations; but people need to test these patterns for reality, and they also need to be able to hypothesize and test results. Used wisely, big data could improve customer service, healthcare, or government by allowing us to dig more deeply. Used wisely, these tools will also help us to make our organizations more flexible and adaptable in a fast-changing world.
Big Data and Cognitive Computing: The Next Industrial Revolution? updates the trends we covered in The Answer Machine, published by Morgan & Claypool last year. This webcast on Jan. 30, 2014 was given to the Cornell Entrepreneur Network, but was open to all. You can listen to the recording at https://cornell.webex.com/cornell/lsr.php?RCID=616468230cc9b30a45ddd07d778325e2.
In updating the book, we found that the nascent trends we discussed in 2012 have quickly exploded. Applications that aggregate information and integrate technologies are becoming common. Task-centered design is almost a requirement. The market, driven by the buzz around big data, and bombarded by information has started to demand what vendors foresaw: there’s immense value in putting together the pieces from disparate sources, and we need help in doing this. IBM’s Watson may have been the first to define cognitive computing, but we see others positioning themselves in this marketplace as the interest grows. We’ll be covering some of these new companies in the months ahead.
During the past year, as we work with vendors and technology buyers, we have found that one of the most difficult concepts to get across is probabilistic computing. Where does it fit in the current IT landscape? Does it replace traditional BI? We expect to explore this topic also in the coming months. Please contact me directly if you’d like to discuss it in depth, or to schedule a briefing for your company. I can be reached at email@example.com.