Increasing volumes from the Internet of Things, 5G (the fifth generation of mobile networks), edge computing and cloud data, coupled with legacy relational databases, unstructured data lakes and event repositories are increasing the need for augmented preparation and analytical systems. Automated data mining and insight capabilities are being embedded in enterprise applications and data pipelines. Data is becoming more enriched and, from a data consumers perspective, less like a collection of databases and more like a federation of data streams. Confluent (creators of Apache Kafka), for example, provides an enterprise event streaming platform that ‘enables completely new ways of solving problems at scale’.

The Bad Old Days

Historically, we became accustomed to working with business data in tabular, structured forms, often generated and presented in operational spreadsheets and relational databases. The accumulation of extra rows, columns and new tables of historical information and data about new business processes have built-up, over time, and become increasingly voluminous and unwieldy. Corporate data warehouses and operational data stores became ‘operationalised’ and constrained so that ad-hoc analytical queries were forbidden, “until the books closed for the quarter”. IT departments became petrified that someone might run “the query from hell” that “joined everything to everything” and did an “order by”, thus bring down their entire estate!

A New Paradigm

The new paradigm is all about self-service data-wrangling and analytics, with data generated as continuous streams of events occurring in parallel. While each event may be small, they accumulate rapidly as both tabular (structured) and unstructured datasets. Machine learning is being applied to spot and highlight patterns, trends and conjunctions across these streams, to focus analysts’ attention and enable business users to take some of the burdens from data scientists (who are in short supply). Preparing, cleansing and the creation of relational, pan-silo views is essential for supporting the deep learning and artificial intelligence solutions of the future, so these enabling technologies are desirable to the investment community. Companies such as Dremio provide a Data-as-a-Service platform enabling data engineers to be more productive and data consumers more self-sufficient.

Technological advances such as global high-speed mobile internet access, increased adoption of ‘big data’ analytics and the move to cloud technology are all driving growth. Elastifile are building enterprise-grade file storage for the cloud, helping to secure increasing volumes of file and data object assets that underpin new capabilities. Statistical inference from vast datasets often referred to generically as “machine learning” is advancing and enabling artificial intelligence capabilities to come to the fore. Robotics and automation underpinned by appropriate data are also attracting significant interest, investment and adoption. Examples include Tray.io, providing a “central nervous system” for the automated enterprise and DEMISTO developing security orchestration, automation and response (SOAR).

Disrupting and Transforming the Division of Labour

These new, improving and evolving technologies will inevitably disrupt and transform current ways of working. Good examples of companies building capability in this space include FortressIQ, who are using data to drive digital transformation and build next-generation quantified workforce. Changes in the division of activity between people and machines require a shift in the skills needed to perform new job roles and the investment community are looking out for things that cannot be automated, as these will become extremely valuable. Human factors, creativity and imagination, and the start-up companies using data to understand interactions between humans and elements of technical systems in innovative ways are all in the investors sweet-spot this year.

Data as an enabling technology

Business competition is stronger than ever, and we all need to do more than just embrace these new trends in technology. Our goal should be to instill transformation in everything we do, especially when it comes to maximising the value and insights we get from data. Our team and delegates have now returned from the tour, and we look forward to reconciling these views and ideas with what they heard while they were away. One thing is sure, we’ve created an awful lot of data in the last few years, and there is a massive amount of untapped value to be extracted from it. However, the insight and value obtained from data is not the end of the story, unless there is action. At Ntegra, therefore, we view data as an enabler, not just an asset.

More insights and details about our findings during this year’s research tour will be published here. Meanwhile for help and support or to find out how Ntegra can help you with your digital transformation journey, please contact us.