Big Data Research Agenda and Trends are Bolder in 2015

Big data has become a big deal as the technology industry has invested tens of billions of dollars to create the next generation of databases and data processing. After the accompanying flood of new categories and marketing terminology from vendors, most in the IT community are now beginning to understand the potential of big data. Ventana Research thoroughly covered the evolving state of the big data and information optimization sector in 2014 and will continue this research in 2015 and beyond. As it progresses the importance of making big data systems interoperate with existing enterprise and information architecture along with digital transformation strategiesVentanaResearchLogo300pxbecomes critical. Done properly companies can take advantage of big data innovations to optimize their established business processes and execute new business strategies. But just deploying big data and applying analytics to understand it is just the beginning. Innovative organizations must go beyond the usual exploratory and root-cause analyses through applied analytic discovery and other techniques. This of course requires them to develop competencies in information management for big data.

Among big data technologies, the open source Hadoop has been commercialized by now established providers including Cloudera, Hortonworks and MapR and made available in the cloud through platforms such as Qubole, which received a Ventana Research Technology Innovation Award in 2014. Other big data technologies are growing as well; for example, use of in-memory and vr_BDI_03_plans_for_big_data_technologyspecialized databases also is growing like Hadoop in more than 40 percent of organizations, according to our big data integration benchmark research. These technologies have been integrated into databases or what I call hybrid big data appliances like those from IBM, Oracle, SAP and Teradata that bring the power of Hadoop to the RDBMS and exploit in-memory processing to perform ever faster computing. When placed into hosted and cloud environments these appliances can virtualize big data processing. Another new provider, Splice Machine, brings the power of SQL processing in a scalable approach that uses Hadoop in a cloud-based approach; it received a Ventana Research Technology Leadership Award last year. Likewise advances in NoSQL approaches help organizations process and utilize semistructured information along with other information and blend them with analytics as Datawatch does. These examples show that disruptive technologies still have the potential to revolutionize our approaches to managing information.

Our firm also explores what we call information optimization,Ventana_Research_2014_Tech_Innovation_Award_Main which assesses techniques for gaining full value from business information. Big data is one of these when used effectively in an enterprise information architecture. In this context the  “data lake” analogy is not helpful in representing the full scope of big data, suggesting simply a container like a data marts or data warehouse. With big data, taking an architectural approach is critical. This viewpoint is evident in our 2014 Ventana Research Technology Innovation Award in Information Management to Teradata for its Unified Data Architecture. Another award winner, Software AG, blends big data and information optimization using its real-time and in-memory processing technologies.

Businesses need to process data in rapid cycles, many in real time and what we call operational intelligence, which utilizes events and streams and provides the ability to sense and respond immediately to issues and opportunities in organizations that adapt to a data-driven culture.vr_oi_how_operational_intellegence_is_used Our operational intelligence research finds that monitoring, alerting and notification are the top use cases for deployment, in more than half of organizations. Also machine data can help businesses optimize not just IT processes but business processes that help govern and control the security of data in the enterprise. This imperative is evident in the dramatic growth of suppliers such as Splunk, Sumo Logic and Savi Technology, all of which won Ventana Research Technology Innovation awards for how they process machine and business data in large volumes at rapid velocity.

Another increasing trend in big data is presenting it in ways that ordinary users can understand quickly. Discovery and advanced visualization is not enough for business users who are not trained to interpret these presentations. Some vendors can present locationvr_Big_Data_Analytics_08_top_capabilities_of_big_data_analyticsand geospatial data on maps that are easier to understand. At the other end of the user spectrum data scientists and analysts need more robust analytic and discovery tools, including predictive analytics, which is a priority for many organizations, according toour big data analytics research. In 2015 we will examine the next generation of predictive analytics in new benchmark research. But there is more work to do to present insights from information that are easy to understand. Some analytics vendors are telling stories by linking pages of content, but these narratives don’t as yet help individuals assess and act. Most analytics tools can’t match the simple functionality of Microsoft PowerPoint, placing descriptive titles, bullets and recommendations on a page with a graphic that represents something important to these business professional who reads it. Deeper insights may come from advances in machine learning and cognitive computing that have arrived on the market and bring more science to analytics.

So we strong potential for the outputs of big data, but they don’t arrive just by loading data into these new computing environments. Pragmatic and experienced professionals realize that information management processes do not disappear. A key one in this area is data preparation, which helps  ready vr_BDI_12_managing_big_data_integrationdata sets for processing into big data environments. Preparing data is the second-most important task for 46 percent of organizations in our big data integration research. A second is data integration, which some new tools can automate. This can enable lines of business and IT to work together on big data integration, as 41 percent of organizations in our research are planning to do. To address this need a new generation of technologies came into their own in 2014 including those that received Ventana Research Technology Innovation Awards like Paxata and Tamr but also Trifacta.

Yet another area to watch is the convergence of big data and cloud computing. The proliferation of data sources in the cloud forces organizations to managed and integrate data from a variety of cloud and Internet sources, hence the rise of information as a service for business needs. Ventana Research Technology Innovation Award winner DataSift provides information as a service to blend social media data with other big data and analytics. Such techniques require more flexible environments for integration that can operate anywhere at any time. Dell Boomi, MuleSoft, SnapLogic and others now challenge established data integration providers such as Informatica and others including IBM, Oracle and SAP. Advances in master data management, data governance, data quality and integration backbones, and Informatica and Information Builders help provide better consistency of any type of big data for any business purpose. In addition our research finds that data security is critical for big data in 61 percent of organizations; only 14 percent said that is very adequate in their organization.

There is no doubt that big data is now widespread; vr_Info_Optimization_12_big_data_is_widely_usedalmost 80 percent of organizations in our information optimization research, for example, will be using it some form by the end of 2015. This is partly due to increased use across the lines of business; our research on next-generation customer analytics in 2014 shows that it is important to improving understanding customers in 60 percent of organizations, is being used in one-fifth of organizations and will be in 46 percent by the end of this year. Similarly our next-generation finance analytics research in 2014 finds big data important to 37 percent of organizations, with 13 percent using it today and 42 percent planning to by the end of 2015. And we have already measured how it will impact human capital management and HR and where organizations are leveraging it in this area of importance.

I invite you to download and peruse our big data agenda for 2015. We will examine how organizations can vr_BDI_08_benefits_of_big_data_integrationinstrument information optimization processes that use big data and pass this guidance along. We will explore big data’s role in sales and product areas and produce new research on data and analytics in the cloud. Our research will uncover best practices that innovative organizations use not only to prepare and integrate big data but also more tightly unify it with analytics and operations across enterprise and cloud computing environments. For many organizations taking on this challenge and seeking its benefits will require new information platforms and methods to access and provide information as part of their big data deployments. (Getting consistent information across the enterprise is the top benefit of big data integration according to 39 percent of organizations.) We expect 2015 to be a big year for big data and information optimization. I look forward to providing more insights and information about big data and helping everyone get the most from their time and investments in it.

Regards,

Mark Smith

CEO and Chief Research Officer

Datawatch Enables a New Generation of Information Optimization

When organizations need to optimize their business processes and improve operations and decisions, the often speak of having the right information at the right time, but don’t always make that a priority. This information optimization is often thought to be expensive and time-consuming, especially with advent of big data and disparate data VR_2012_TechAward_Winner_Logosources across cloud and on-premises environments, as I have articulated. Datawatch can help business get to information of any variety or volume at any time through its access and integration tools. When I published my last analysis of Datawatch, it had made significant advancements in its platform, with enterprise-class reliability and support for business analytics through its data discovery and virtualization processes. Over the last year Datawatch continued to grow its business worldwide, and through investments into its marketing, sales and product efforts is finding more potential from existing and new customers. The company’s energized product efforts earned it our 2012 Technology Innovation Award for Information Applications for its Information Optimization Suite.

Datawatch has simplified its product portfolio over the last year, focusing on how organizations transform, distribute and optimize information. Its Monarch Professional, Data Pump and Enterprise Server products respectively support these common functions. It has expanded its support for big data to ensure that no matter where information exists, it can be optimized for use across business and IT. In its 11.6 release Datawatch added support for Hadoop and Hive through its Data Pump product. It also works with commercialized Hadoop providers such as MapR, which provides enterprise-scale deployments. Datawatch supports other types of vr_bigdata_big_data_technologies_plannedbig data technologies, including RDBMS, appliances and systems, which a third of organizations in our big data research are planning to adopt.

Datawatch brings information into business processes through support for a range of environments. For instance, in cloud computing, it partners with Amazon Web Services, a rapidly growing Infrastructure as a Service (IaaS) provider, for a range of applications and tools. Our research finds that business has led the way to cloud computing. Datawatch can operate safely and securely across these environments with little impact to IT.

Datawatch continues to advance in many small but critical capabilities, such as document approval and state management. It optimizes information processing through prefetching data needed by the operating environment. It has added visual presentation methods to its product, including traffic lights and thermometer gauges, and lets users drill down to any level of detail. It now provides Section 508 compliance for supporting the disabled, for which it has created a template that can be adapted to an organization’s specific needs. Datawatch products are used for a wide range of governance and compliance needs.  Our research finds that the cost of compliance is rising faster in the last three years, according to 53 percent of organizations in heavily regulated industries.

Datawatch now provides more power to analysts and individuals whovr_infoappbench_current_technology_shortcomings need to facilitate information optimization through the Monarch Power Client, a visual environment that was part of the Monarch Professional 11.5 release. This product helps address assembling information into a view, a process that almost half (47%) of organizations in our information applications research found challenging. Monarch Context for Excel helps address the issues in using personal productivity tools with support for secured embedding of information inside spreadsheets and for data lineage. Data is always more valuable when it is a click away, rather than accessible only upon request from a separate analyst.

To support specific needs of IT, Datawatch supports machine data that is generated by applications and systems, which, if shaped in the right format, can help optimize not only IT systems and resources but also business processes. Datawatch can take data in log files and database and combine it with information in reports, documents and HTML pages. Datawatch recently announced it can utilize machine data from Microsoft Windows.

Datawatch has grown through partnerships with software providers such as Qlikview, helping them get access to semistructured data, vr_bti_br_whats_driving_change_to_technology_selectionand solution providers such as Asta Systems. Resellers use Datawatch as a new business enabler to empower the optimized use of information across an enterprise. For global deployments, Datawatch supports languages like Japanese and Chinese and unique character sets. Support for and focus on partners is a critical investment for Datawatch as it seeks to grow globally. I would like to see Datawatch provide a version of its product for free trial on its website, operating either in the cloud or on the desktop.

Datawatch makes information optimization more readily available at an affordable price. Its software’s ability to access content and semistructured information and blend it with structured data is what organizations require to optimize business processes and make more informed actions and decisions. Our research into business technology innovation finds the needs to improve and drive better quality in processes are important to more than half of organizations. We are busy researching information optimization to see how the best practices and efforts of organizations are changing how technologies are used for business.

Datawatch finds itself at the intersection of information needs for anVR_2012_LeadershipAward_Winner_Logo enterprise. I would like to see more support from the company for mobile technology, and simpler methods to flip through information assets and even collaborate on them, but with its current focus on its foundation and enterprise-class requirements, those features represent potential for providing more value by harvesting its investments in big data and cloud computing. Organizations should examine Datawatch to see how it can help them leverage investments to access and integrate information and meet business needs while meeting IT requirements for security and policy compliance. Its progressive software earned Datawatch our 2012 Ventana Research Leadership Award for Information Applications for its deployment at Piedmont Henry Hospital.  If you are looking to get information from any source to any form for any business need, see how Datawatch meets the requirements of the next generation of information optimization.

Regards,

Mark Smith

CEO & Chief Research Officer