Flashback Friday: Cloud and Technical Data Integration Challenges Waning

Back in 2008, when cloud was just a localized phenomenon, it was questionable as to whether or not the (then) nascent model would be able to support the data management (and integration) required for enterprises to really take advantage of its capacity and cost models.

Back then I noted: “The availability of and ease of access to data stored in the public cloud for integration, data mining, business intelligence, and reporting - all common enterprise application use of data - will certainly affect adoption of cloud computing in general.”

It’s been almost ten years now (which is like 100 in technology years) and many of the technical challenges that existed then are now waning. The proof of that can be seen in the significant percentage of enterprise organizations that are (or plan to) take advantage of cloud for its business intelligence (BI) efforts. According to BARC research released early in 2017, “46% of organizations prefer public cloud platforms for cloud BI, analytics and data management deployments.” A non-trivial 30% employ a multi-cloud approach, while 24% prefer private cloud.

Regardless, cloud is definitely a favorite for “big data” initiatives. Given the bigness of big data – particularly those warehouses built up on which BI and its increasingly real-time dashboards are based – that’s not nearly as surprising as the results indicating the most prevalent use case for cross-cloud data integration today is that of applications. The BARC research showed that “Data integration between on-premises and cloud applications dominates use cases across all company sizes, with 48% of enterprises leading in adoption.”  

In 2008 the technical limitations on both sides of the equation – on-premises and public cloud – were a deterrent to this type of integration. Today, thanks to (always) evolving application architectures and moves toward API-based integration (instead of service buses and integration hubs) combined with dramatic increases in network speeds and feeds, a variety of use cases for data integration that were technically infeasible just a decade ago are now within easy reach of most organizations. 

Challenges remain. Of those cited in the BARC study, security is number one (45%) followed by legal issues (37%) and performance (29%). That’s understandable, as myriad regulations and laws regarding privacy and data protection have arisen to muddy the cloud waters, making it risky (if not impossible) for global implementation of data integration and management in public cloud environments.

So we’ve still got some issues to deal with when it comes to data integration and management in public cloud environments, but the biggest technical ones seem to have been addressed by the steady onward march of technology.