Reflections on tech, trust, and data adoption

Last December when I was writing my data engineering predictions, I knew 2021 would be a big year – and not just because we were in the throes of a global pandemic. 

At the time, Snowflake was going public, Google had just acquired Looker, and companies were investing in their data organizations more than ever before. It was exciting! 

Simultaneously, this demand for data and analytics translated into technical bottlenecks, process gaps, and cultural shifts reflective of an industry in flux. 

We knew we were on the cusp of something big, we just didn’t realize how big.

As in turns out, 2021 saw even more game-changing product launches, acquisitions and IPOs, in addition to kickstarting one of the most important conversations in our industry: now that we have all this data, how do we operationalize it – at scale?

In his 2021 Machine Learning, AI, and Data Landscape report, Matt Turck suggests that “every company is becoming not just a software company, but also a data company.”  

Echoing this, 451 Research predicts that the total data market is projected to reach $146B by the end of next year, compared to $89.4 just four years ago. Turck’s statement isn’t just clickbait; it’s the truth.

Compared to ten or even five years ago, data is more than just a siloed entity that informs the occasional planning dashboard – it’s a living, evolving thing with the power to transform how we do business. Executives want metrics in real time, and product teams rely on data to build new features and power digital services.

With great power comes great responsibility. To operationalize data and realize the full potential of its impact, data engineering leaders must invest in the right technologies to make data accessible, secure, and reliable.

In my opinion, though, technology isn’t the barrier to data adoption. It’s culture.

In 2022, I predict that the largest hurdle data engineers will face boils down to whether or not we can make data part of the company culture instead of an afterthought.

To do this, teams need to invest in education, expectation-setting, and processes that scale. We need to empower stakeholders by making data easier to work with, insights easier to share, and data products more democratized.

And it all starts with trust.

In our 2022 Data Engineering Predictions Guide, we touch on these topics and share what’s on our radar for next year across technology, process, and culture, and how some of the best data teams are leveraging them to drive impact at scale. Check it out.

Here’s wishing you a fabulous New Year – and no data downtime!