Smart and Dumb Data
It is a tragedy when smart people create nothing more than smart data.
Years ago I worked for Honeywell making sensors used in steel production. Our premium systems were the market leader in high speed, precise measurements. My role at the time was manager of the “cost effective” product line. One day, while visiting a customer using our equipment, I made a discovery: Our worst sensor was already five times faster than they needed! Their control systems connected to the sensors were so slow that any increase in data acquisition was just a waste. Yes, at the same time, our ‘most educated’ customers were pushing us to build faster sensors, our competitors were trying to build sensors as fast as ours, and we were proudly announcing our latest speed triumphs each year. How could this be?
It was true. My discovery was that slower and more stable was the better product for the customer. This insight earned me a promotion to Director of Engineering charged with the task of completely redesigning our product line, reeducating our customer-facing staff, and re-educating our customers. The insight was in asking a customer how (in detail) they acted upon the data rather than asking what data they wanted.
In the physical world we make sensors that collect data. More and faster sensors create more data that can be manipulated. In the IT world, connectors, aggregation and filters create data. Just like physical sensors, more aggregation means more data that can be manipulated using new high tech algorithms. Most people would describe this processed data as information: I think it is, at best, just “Smart Data”. From my perspective, data is information only if it provides the ability to take effective action based on its content.
In recent months, I found myself dealing with two companies with just such a problem. One made sensors for sporting equipment; the other seeks to find defects and fraud in financial companies. In both cases, they had proved that their Smart Data was valid. These two companies reported that lots of people were excited by their technology, but they were both having problems converting it into a revenue stream. As you might guess, their problems had similar roots– how could their customers take actions based on the new data they provided?
The financial IT product could identify anomalies and create suspicious cases for investigation, but the investigation teams and associated support processes could not cost effectively process the additional cases. In order to sell more anomaly detection, the product maker needed to switch its focus from anomaly identification to optimization of the case-clearing process: the investigators, lawyers, and support center people had to first become more efficient before the additional data could become useful. Just like my industrial sensors, their first version of financial software was already too competent for the rest of the system to effectively use.
The team making sensors for sport activities initially received great interest from Olympic and professional teams literally from around the world. Actual demonstrations on their equipment generated even more excitement when they saw data related to speed and flow extracted from their sports equipment. Unfortunately, most lost interest and expressed vague comments of “no budget” after the first enthusiasm ran out.
Sportsmen are trained to use their sports equipment based on a tradition of what is fast and what is slow. This is based on what they see and feel. Combined, engineers would describe this as a “system model”. People who design sports equipment think within the sportsman’s system models and are usually ex-sportsman themselves. Since these systems did not include the data provided by the new electronic sensors, there simply was no known action to take based on the newly available speed and flow data. More and better sensors could never change this fact. As a result, the sports sensor company must now invest in creating new sports equipment models by either calibrating the new sensor data into the legacy system models or by creating new systems models based on other scientific principles that can use the new sensor data directly to calibrate the results. Only when they can create a new paradigm to replace the legacy sports traditions will the market for their speed and flow sensors break open.
Look at the recent advances in swimming equipment — it is a radical, science based, change from the previous tradition based of 50 years of swimming equipment. The developers of the “shark skin” suit became rich because they were able to transition smart data derived from shark motion efficiency to swimming– the dolphin kick combined with the new suits proved substantially better. Now everyone is copying in an effort to catch up.
All three of the companies (Honeywell, sports sensors, and financial anomaly identification) discussed were busy working on more advanced algorithms to process even more data. However, by looking at how their customers used this data, they discovered that it was the customer-side bottlenecks that were keeping their innovations from being valuable.
The final insight is that their own customers were leading them the wrong direction. When asked if improved data would be of interest, all potential customers emphatically responded with a yes. All three companies had to probe deep into their customers’ processes to discover that the customers could not meaningfully use the additional data that they were creating.
When customers can use your smart data to take valuable actions, then you are providing information that has real value in the marketplace.. Smart data that is not yet actionable information is called a science experiment — technical people regularly overestimate a customer’s appetite for science experiments. Instead, real information is far easier to value and by extension, becomes much easier to sell.
What data does your systems produce? Do you know how your customers actually use this data or do you rely on what could be done with the data? Your product or service might just already be too good for your customer.