Datawatch Enables Business Analytics through Data Discovery and Virtualization

Thanks to Datawatch, businesses may find themselves buying and deploying fewer new business analytics and business intelligence tools. As I noted in my analysis last year, the company’s new management has begun to transition it to a more appealing market position in which it helps enterprises use analytics to improve operational efficiency and decision-making. The Datawatch products focus on data virtualization – that is, accessing data from multiple sources not usually seen as accessible for analysis, such as reports and documents in Adobe Acrobat format, production systems, streams coming from cloud computing or print output, or even legacy systems. The value of this lies in reducing the time it takes to access such data and integrate it with other data to create enriched information and analytics.

Datawatch is a public company, and its recent second-quarter results show continued growth on both quarterly and year-to-year bases. The company has more than 40,000 customers in 100 countries, and its technology is used by Fortune 100 to midsize and small organizations. At the core of its growing enterprise business is Monarch Enterprise Server version 11, which was made generally available at the end of 2011. With this version the server uses Microsoft .NET and can operate in 64-bit computing environments, which significantly enhances its computing power. It offers XML export capabilities to help programmers process data further into other data streams or use it within other products; XML is the second-most important technology standard after SQL, according to our research. Version 11 also improves the presentation of information, supporting a larger display of information and multiple charts in a grid. It adds a drag-and-drop approach to help simplify the creation and distribution of information and analytics. Our business analytics benchmark research shows that analysts spend upward of two-thirds of analytic process time on data-related activities, which Datawatch can help them decrease.

Monarch Enterprise Server is part of Datawatch’s Monarch Report Analytics product family, which also includes Monarch Data Pump, Monarch Report Mining Server and Monarch Report Manager On Demand. As business analytics software, it falls in the category of what we call information applications, enabling the access, assembly and publishing of information – areas that are critical for most organizations to improve. Our information applications benchmark research found that for more than half of organizations, existing approaches are too slow and not adaptable or flexible.

Datawatch has jumped on the growing trend in enterprise software of offering its software, notably Monarch Professional, for trial to enable more individuals to see the power of the software. Report Manager On Demand lets organizations maintain an easily accessible repository of reports and documents. Well-known for its Data Pump software, the company has a large base of existing customers that may be enticed to upgrade to the enterprise versions of the Monarch technology.

I expect Datawatch to make its technology further available via cloud computing and software as a service (SaaS), a means of access increasingly required by organizations. I would like to see it further its accessibility through search and flexibility by providing access from mobile technologies such as smartphones and tablets to extend its simplicity to a larger number of business users.

Organizations that are looking to leverage investments in aging applications or recent documents should look at Datawatch’s software. Its simple approach is aimed at businesses that do not want to buy a new system to perform analytics when they know that the data they need is already in existing systems; these businesses want to implement a continuous flow of information to support their analytics needs instead of copying and pasting data into spreadsheets for one-off analysis. In our benchmark research, 42 percent of organizations said they can significantly improve their use of analytics, and only 23 percent were satisfied with their current analytics efforts.

Datawatch is aligned with the current trend in the technology industry of finding methods to extend the value and use of existing systems, and one of these ways is business analytics. Its methods to virtualize data via access to semistructured information and to simplify discovery of insights from analytics are what many businesses are looking for to get more value from existing data and investments.


Mark Smith – CEO & Chief Research Officer

Splunk: Big Data Machine for Operational Intelligence

Splunk recently entered the financial markets as a publicly traded company (NASDAQ: SPLK) and also entered a new phase in its corporate growth. Splunk combines the power of search and discovery with analytics on data generated by IT systems, that they call machine data, and provide insight for a new generation of operational intelligence that helps everyone in IT including the CIO determine the efficiency of its systems that support business. The company has built a platform that can index data on a large scale (“big data”) for rapid analysis and search. They also through its analytics provide the ability to perform visual and data discovery which is critical to reduce the time to determining unknown issues in existing IT systems. This helps IT staff ascertain not just the performance but the efficiency of systems that operate on a 24-by-7 basis. Splunk’s software operates in real time, surpassing the traditional methods of applying business intelligence against a data warehouse – a practice that’s ineffective for use in IT, where time is not the CIO’s friend when it comes to understanding issues or opportunities for improvement. Splunk has grown rapidly, partly because it’s simple to download and try, and then to license for use in production. It has more than 3,300 licensed customers in 75 countries. The management team is led by CEO Godfrey Sullivan, who has experience and a track record at companies such as Hyperion.

Where is Splunk finding its opportunity? IT organizations have been like the cobbler’s children who have no shoes. While they deliver applications and technology to business, IT departments seldom have time to analyze and optimize their own processes, systems, applications and projects. While this might sound counterintuitive, it is true. Almost seven years ago, I started investigating whether performance management being applied elsewhere in a business was occurring in IT. I discovered that CIOs lack critical tools to define goals and track the performance of their people and machines. Our IT Performance Management benchmark research identified lack of resources, budget and awareness as major contributing factors to this situation. It also showed that organizations that deploying appropriate tools were increasing IT effectiveness and supporting the needs of business more efficiently. Machine data or files were the top data source within 43 percent of IT organizations, and balancing loads between networks and servers, doing root cause analysis and doing impact analysis were key tasks in at least 52 percent of them. Splunk, which addresses this niche, has had significant growth over the last three years even though economic conditions have been shaky.

Last year we came out with new benchmark research on IT Analytics that investigated analytics related to IT organizations managing systems that support business. This research found that only 15 percent of IT organizations are at the highest level of maturity in IT analytics in the key categories of people, process, information and technology; the bulk of organizations are still struggling to function effectively. Interestingly, 91 percent of IT organizations want their analytics to be simpler – understandably so, as 79 percent are using spreadsheets universally or regularly and more than half are dissatisfied with their existing technology. The top issues with current efforts are that information is not readily actionable (for 46%) and not adaptable or flexible to change (44%); many organizations have reports or files but can’t assess and act on the data. Splunk also addresses that need of more than half (52%) that are not able to easily get the data they need for IT analytics. Considering that IT spends more than two-thirds of its time in the analytics process in data-related tasks as opposed to actual analysis, Splunk’s ability to get to machine data and index it for search and analysis is a welcome sign. Usability was identified as the top-ranked evaluation criteria, followed by the need to search and navigate machine data, which is exactly what Splunk offers.

Our analysis of Splunk’s latest release of softwareversion 4.3, found a series of advancements, including supporting access from tablets such as Apple’s iPad, support for data in cloud-based applications and the ability to operate in VMware environments, making it easier to deploy Splunk in the cloud. This version expands the enterprise scale of the platform, supporting more concurrent users and faster search. Users can preview machine data then start indexing against it for further discovery and analytics. These are all critical tasks as Splunk becomes a platform for harvesting machine data for search and discovery across the IT organization. Splunk must continue to expand its footprint with more collaborative capabilities to help IT understand issues and opportunities where analytics on the machine data are the starting point for dialogue and action. Introducing more collaborative capabilities and the ability to correlate data toward actions and notifications will help Splunk further differentiate its technology.

Since our analysis, Splunk has continued to support new machine data environments, including Hadoop. It can run MapReduce queries against data in Hadoop and then bring the data into Splunk for analysis. Our benchmark research into Hadoop and Information Management found that users of Hadoop are most often focused on integrating machine data from application and Web logs with the events and network monitoring data that Splunk is designed to address. Hadoop users need to analyze data much faster than others, with 77 percent of organizations indicating they need analysis within a day and 25 percent within an hour. One challenge for organizations using Hadoop is staffing and training; Splunk addresses those obstacles with a tool that is easier to use than creating custom programs, and it provides real-time and integration features, each sought by two-thirds of organizations. The growth of Hadoop comes from the need to address big data needs but also the simple access to download the software and then use it which is also what Splunk does with its download to use its software.

Splunk has a promising opportunity in the market, considering the size of IT spending on technology every year and the legacy of applications and systems that must be optimized for business use. Also, as IT organizations continue to advance their IT security and compliance and increase their use of cloud computing, the need not just to monitor but to analyze and act on issues and opportunities grows larger. Our last benchmark research on Operational Intelligence in IT found that IT has prioritized to manage IT performance through metrics in 71 percent of organizations, and more than half (58%) plan to improve their IT organization’s responsiveness. The research found more than half of organizations have at least 11 information sources to routinely interact with, which is part of why Splunk fits these needs on machine data. Splunk has little competition, with IT management software vendors like Computer Associates, Hewlett-Packard, IBM and Oracle providing little innovation or capabilities to address operational intelligence needs through discovery and analysis of machine data in IT. Eventually they or others will realize their shortcomings, but at that point I would expect to see Splunk be significantly larger with an expanded portfolio of capabilities for meeting a CIO’s need to truly become an information officer of his own IT organization.


Mark Smith – CEO & Chief Research Officer