Organisations and individuals are on a data journey.

This data journey represents the beginning of digital transformation for most of what we do in society and business: recreation, commerce, government and enterprise etcetera.

These different areas are adopting tech at an amazing rate, and is the culmination of 50 years of Moore’s Law — hardware and software are now cheap enough to be used on a wide-scale.

However, in both a business and consumer sense, we are not yet fully taking advantage of this. We are in fact, just getting started.

This is the most exciting time to be in technology, much like mechanisation during the Industrial Revolution; but happening much faster and across all aspects of society.

See also: Business metamorphosis: digital transformation of the enterprise – Implementing a digital transformation strategy, with the right technology and people in place, can transform a business from a caterpillar to a butterfly.

Data software

Data software is at the heart of this technological revolution. But, it has been changing recently — just ten years ago the dominant style of software, the relational database, was king. Today, it is a very different matter.

As we have greater quantities of data from different sources the relational database is now limited — a style limited to siloed systems that are difficult to connect.

It was “ripe for disruption, and the demand is there,” said Doug Cutting, Chief Architect at Cloudera, during his keynote at Big Data LDN.

“But, it is hard to design a solution and get it right the first time. An iterative approach to data software is better.”

Enter… Open source

Open source represents a better way to develop software for a more general purpose via collaboration.

It creates a huge advantage and was described as “steroids for software adoption,” by Cutting. Importantly, open source can be used by enterprise, and they are starting to realise this.

“It is a different environment from what it was ten years ago,” said Cutting.

See also: The value of open source software – Open source software is being integrated into an increasing number of organisations, most recently, the Pentagon.

“On top of Hadoop, there are now lots of tools that compliment and replace it — Apache Spark, for example. And, there are alternative storage engines.”

An ecosystem is developing that has a common thread of supporting better software development . These new tools are appropriate and needed for the kinds of data we now have — IoT, sensor data, network data, for example. These are not transactional data sources, so a relational database might not be the best tool.

“Progress is now in a much more productive style,” continued Cutting.

The rise of the software development house

Every industry is driven by software and every business needs to become a software development house, of sorts. This ability, or transformation, will provide unique value to businesses and their customers, in the fight against disruption. More often than not, this concerns data software.

Amazon and Tesla (and other disrupters), for example, built their companies around data systems. Data will be at the heart of business going forward.

Getting the most out of your data

Attaching AI methods to a data system might work, but there a lot of simpler things that can be done more rapidly: analytics (“basically counting”), data warehouses at huge scale, data science (“fancy counting, clever maths”), machine learning (“fancy counting with feedback and iteration — fairly evolved use case”).

“You need to have good data and understand that data for any of the above to work and provide real insights — what it’s telling you and not telling you. Most people are well advised to proceed slowly and not jump in the deep end,” said Cutting.

The fundamental challenge

Many organisations and indeed, governments, view the growing skills gap as the great mountain to climb. However, “people can learn fairly rapidly,” according to Cutting. “The skills gap is not a deep problem.”

Industry knowledge, where you know how improve your business, takes years to grow and develop: you can adopt technology in a matter of weeks, but it takes time to find out how this tech can align to business goals and solve regular challenges.

The fundamental challenge, however, surrounds mindset — changing business culture to take advantage of new technologies.

People underestimate how structured organisations are: how IT is managed, who owns what etcetera. To take full advantage means rethinking the organisation, its practice and philosophy.

Crucially, businesses should empower people throughout an organisation to use technology to make decisions. This idea of self-service systems goes against decades of IT culture.

The question is: Can enterprises adapt to take advantage?

New companies grow so quickly because they start from the outset with this required mindset — it this the fundamental challenge for more established, traditional industries.

A related challenge is to adapt the organisation in an ethical manner; maintaining the trust of users and not betraying it.

Only the beginning

Businesses and governments are only just beginning to figure out how to incorporate these technologies effectively. Importantly, there needs to be more regulation and for tech companies to push for it.

“If we are to become the digitally enabled society we want, we need to observe people’s rights,” said Cutting.

“The cloud is a good way to move away from restrictions and cultural problems that any classic IT organisation will have — it can make a lot of these things simpler and accelerate transformation.”

Related: Royal Mail’s data journey – Royal Mail’s Data Group has revolutionised the operational efficiency of the UK’s largest letter and parcel delivery company – and other businesses should take note.

The data journey

As organisation’s embark on their data journey, it is important to not aim to high from the offset. Business leaders need to envision a future with these technologies, but not over night. It is a slow revolution, a revolution over time.

“Let iterative success grow and the spread,” said Cutting. It’s about small wins, initially.

It’s necessary to have buy in from the top, and that they permit projects to happen — these transformation projects need to be carried out in an open manner.

It’s ultimately a process, but perfection is unachievable. It will always be a continuing journey.

“Don’t embrace any tech too tightly; always think how can we replace it and move forward. Change, after all, is the new normal,” concluded Cutting.

This is the beginning of the data journey and digital transformation efforts, which will empower organisations to improve; by getting data, understanding it and using it to optimise processes.