iWay Unites On-Demand Integration, Big Data for Real-Time Analytics 
iWay Software, a unit of Information Builders, is bringing together the worlds of on-demand integration with Big Data. IDN explores how iWay has optimized its long-standing any-to-any integration framework and connector architecture to make it easier for IT to design and deploy solutions based on MapReduce, Hadoop and other new data and analytics technologies.
iWay Software, a unit of Information Builders, is bringing together the worlds of on-demand integration with Big Data.
IDN speaks with iWay’s marketing director Vincent Lam to learn how iWay has optimized its long-standing any-to-any integration framework and connector architecture to make it easier for IT to design and deploy solutions based on MapReduce, Hadoop and other new data and analytics technologies.
“We’re definitely seeing among our customers a trend where unstructured data with technologies such as Hadoop and MongoDB are becoming a bigger component of what companies are using, and these customers need those to work together with their RDMBS and other resources,” Lam told IDN. “Supporting these solutions within iWay was important because to us, the whole crux of integration is to get all the things a customer needs to work together to work together.”
“Our support for MapReduce allows iWay to federate the query [across distributed data sources] very quickly.”
Vincent Lam
Marketing Director
iWay Software
Supporting Hadoop, MongoDB and other new unstructured data sources was not a major problem for iWay’s core architecture, he added.
“iWay has always used an agnostic approach to integration, where endpoints were always separated at the functional layer. So, integrating with these new types of data only meant that we needed to make sure we had new adapters that would talk to these sources,” Lam said.
That said, iWay also wanted to go one step further and make it easier for various IT professionals (from both the data and the integration side) to design and deploy these types of solutions.
“Because MapReduce and other Hadoop and ‘Big Data’ tools are so new, IT often needs to learn specialized skills. Even beyond training. Sometimes it’s also not clear how IT should address a project or even who should take responsibility,” Lam added. “That’s why we added MapReduce-caliber capabilities to its iWay Parallel Service Manager.”
The iWay Parallel Service Manager 6.1 adds MapReduce-style functionality to provide simplified tooling and out-of-the-box integration support to power Big Data solutions, such as “federated search” across unstructured data stores, as well as many popular ones now used by enterprises including Oracle, MUMPS, and HL7.
Lam added, “For some customers, the ability to search across a large pool of physically distributed data sources and obtain near-real time results can be a monumental issue,” adding MapReduce allows iWay to federate the query very quickly, supported by agents in the iWay Parallel Service Manager.
The latest Parallel Service Manager also extends ETL with “parallel” extracts, transforms, and loads, as well to batch-oriented managed file transfer (MFT) where the processing and distribution of files takes place in parallel
Under the covers, the iWay Parallel Service Manager will accelerate the processing of complex queries and/or computations by breaking them up and distributing them across multiple technology assets.
It will perform these computing functions in parallel (rather than sequentially) because inside the iWay Parallel Service Manager, “map” and “reduce” functions are done in a process flow, which is executed in parallel for each item in the list. So, when a user makes a query that needs to access multiple data stores simultaneously, iWay launches a “parallel control agent,” which will simultaneously execute process flows configured with information for what connections are required based on information in the list. The outcome of each query can then be amassed into a final results document.
This approach, Lam said, will deliver key MapReduce-optimized benefits, without the need for IT complexity. Among the benefits are:
- Reduce complexity of setting up and executing MapReduce-type federated queries
- Seamless integration with the Hadoop Distributed File System
- No need to write custom Java code to divide large queries across multiple systems (and then collect and aggregate the results)
- A drag-and-drop interface to set up and execute parallel processing jobs
- Pre-packaged integration components providing native access to 300+ backend source / target systems
This iWay approach to bringing together Big Data and integration comes as Gartner predicts that real-time analytics will explode for structured and unstructured data across distributed systems.
“Analytics has become a major driving application for data warehousing, with the use of MapReduce outside and inside the DBMS, and the use of self-service data marts, according to Gartner’s just-rleeased report on the Top 10 Technologies for 2012. “One major implication of big data is that in the future users will not be able to put all useful information into a single data warehouse. Logical data warehouses bringing together information from multiple sources as needed will replace the single data warehouse model.“
