Fujitsu Enlists Smart Gateways to Cut Big Data Bloat

By Pedro Hernandez

Big Data is about to get squeezed by a distributed processing technology that Fujitsu says can slash network traffic by up to 99 percent. The company hopes to commercialize the technology in its 2013 fiscal year, or by March 2014. (Fujitsu's fiscal year runs from April through March.)

Storing and processing Big Data can present challenges in and of themselves, but transmitting it can really clog networks. An ever growing, data-hungry swarm of sensors, mobile devices and social networks have made traditional networking architectures ill-equipped to handle Big Data, said Fujitsu.

Fujitsu joins an industry-wide efforts to optimizing networks that handle Big Data with projects like OpenStack, a software defined network protocol. For its part, Fujitsu claims to have solved Big Data's appetite for networking resources with a new algorithm that allows gateways to parcel out data that would otherwise end up getting processed by the cloud.

Gateway to Leaner Big Data Processing

The goal is to bring a portion of Big Data's processing workload closer to the data's point of origin rather than taxing a network's bandwidth and resources by transferring that data wholesale. Fujitsu's technology enables gateways to perform a measure of pre-processing, like filtering or statistical processing, and streamline the data flows that wind up being transferred to -- and processed by -- cloud infrastructures.

It offers an example of how it would work in an electric utility where energy consumption data is collected from power distribution boards or power supply taps.

Instead of transferring that data as-is to headquarters, smart gateways at each branch office performs some pre-processing and only the pertinent data required for billing and grid maintenance is sent to the cloud. If necessary, said Fujitsu, raw data can be compressed after the fact and transmitted to the cloud.

For this to work, Fujitsu Laboratories developed a network auto-discovery framework that aids administrators in dividing up the work performed by the gateways. Once a network's topography and data flows are determined, gateway configurations are generated for each unit's assigned workload. And in order to manage data grouping by processing type (like averaging or tallying), the company developed an aggregation key that groups data according to their business function, as well as a way to trace sensors and machines to optimize data flows to the gateways.

Ultimately, the Fujitsu envisions cloud services that can handle, "huge volumes of real-world data can operate while holding down communications costs."

Pedro Hernandez is a contributor to the IT Business Edge Network, the network for technology professionals. Previously, he served as a managing editor for the Internet.com network of IT-related websites and as the Green IT curator for GigaOM Pro. Follow him on Twitter @ecoINSITE.

This article was originally published on March 16, 2012