In my last post Dew Computing and Smart Machine, I explained what is “Dew Computing” and how peer-to-peer communication will bring Cloud computing to Fog Computing and finally to Dew Computing.
Dew Computing will be network intense, where millions of IoT ( Internet of Things) devices will talk to each other and share data. In this blog, I will discuss about data compression which will ease network and improve data quality for IoT devices.
Today, we have multitudes of compression algorithms available which do good job as saving space on disk and in-memory calculation – and that precisely their objective is. but we have limited compression algorithms available to do compression “over the network” that will be crucial for peer-to-peer or Dew Computing.
Nature of Data : IoT devices will be emitting data at regular time interval, where barring some measure value, all other dimensional data ( like location, make, network info etc) will be same. Unfortunately, static data will also be in the same emitted dataframe, which will choke network with duplicate values. Also the granularity of time intervals will be in milli-seconds, seconds or minute so measure values are also expected not be very abrupt. An out of range value will trigger data quality concerns.
Logic for Compression: If the static data is separated from dynamic data and send once, it will reduce network load tremendously. The compression algorithm will received snapshot (boiler-plate) of data-frames at regular intervals and only changed values will be conveyed over the network, continuously. The algorithm will know how to build full data frame ( if needed) from incremental values. An example below
Time Series Compression
Data Quality Trigger : The algorithm can be configured to check if dynamic values are range bound and within accepted norm of variables from previous value. Compression Algorithm anyway checks the time frame stream of values for creating incremental data and has access to historical value. A data quality check here will save a round trip to server to validate the data – saves both latency and load
DQCompression
Conclusion : An understanding and consortium of IoT device maker will define the protocols of time series compression algorithms which will hugely benefit all involved parties.
Vivek Singh is chief contributor to Open Source Data Quality and Data Preparation tool.
viveksingh36 