5 Data Predictions for 2024

Introduction

As we step into 2024, it is time to look back to a transformative year and look forward to an exciting year ahead. We’ve seen a big shift in the economy with many companies like Google and Amazon laying off thousands of employees. AI has emerged, beginning to replace certain roles and shifting the way people work.

There’s been a few trends in data that have emerged over the last year, even more so in the last few months, that will particularly shape 2024. Many of these trends are in reaction to the current economic climate, forever changing how teams function and focus their work.

Let’s dive into these trends and how we predict they will shape the data industry as a whole throughout 2024.

1. Embracing a transition from traditional 9-5 work to a consulting model for data engineers and analysts

With the economic downturn, many data practitioners were laid off from their corporate 9-5 jobs. While some have continued onto new roles, we see many who took this as an opportunity to take on consulting activities.

We even saw popular data folks like Benjamin Rogojan (a.k.a "Seattle Data Guy") and Zach Wilson leave their comfortable, 6-figure salaries at FANG companies to start their businesses. Benjamin Rogojan frequently talks about how he has made more money consulting than he did working at Meta.

On top of that, he teaches others to do just what he did and leaves the comforts of 9-5 to begin consulting. The number of people watching his live streams and liking his posts on data consulting shows a shift in the interest of data engineers and data analysts. Why work on someone else’s schedule when you can pick your clients and have the freedom to choose when you work?

Not to mention, many companies are hiring consultants for their data needs rather than hiring an in-house data team. With the economy being shaky, companies see consultants as less financial risk. They don’t need to worry about paying for health insurance or a salary and can use them as needed.

This trend of data engineers and data analysts shifting to consulting will only continue to grow as the job market becomes more competitive and 9-5 jobs become less stable. Most people and companies want to feel safe and secure and consulting may help them feel this way.

Chad Sanderson has been talking about data contracts for years, but in 2023 this topic became even more popular. It’s become so popular that he partnered with another data influencer, Mark Freeman, to start a company focused entirely on this problem - connecting data producers and consumers, ensuring data expectations are communicated and solving the problem so many data practitioners experience with enforcing data contracts.

However, it’s a difficult problem that can be attempted to be solved in many different ways. In 2024, companies will compete to be the first to solve it correctly, something that has yet to be done. We’ve seen this with many different data products, each solving the same problem in its unique way.

What’s to stop companies like dbt, Airbyte, and Y42 from building data contract solutions into their products? We’ve already seen dbt implement a lightweight version of data contracts in 2023. While there is much room for improvement, we can guarantee that this will be a large focus of theirs in the new year.

Y42 already has the strongest implementation of data contracts through its Turnkey Data Orchestrator with built-in Observability, never letting bad data enter production. Their Virtual Data Builds and embedded testing enforce data contract rules in a unique way.

In 2024, we’ll see more companies trying to catch up to the powerful data contract solutions already built out by these companies and new companies like Gable emerging solely to solve the data contracts problem.

3. Proliferation of AI-enabled features and advanced tooling in data management

It’s no secret that AI has been popping up everywhere in the last few months. Everyone is talking about how to use AI-powered tools like ChatGPT to get ahead in their work, and data experts are no exception.

Companies have been training their employees on how to use AI to write better code, build more secure products, and streamline projects like a data warehouse setup. If you aren’t learning how to use AI to improve the good work you are already doing, you are falling behind.

Databricks was one of the early data companies to pioneer AI features and even made a $1 billion+ acquisition to expand its capabilities. Modern data tools like Hex are launching AI-powered features that help business users write analytics queries and get their questions answered faster.

We will only continue to see more companies jump on the AI trend, baking in features to help users take better advantage of their products. We predict an uptick in AI features that help optimize queries, automate testing, streamline documentation, and build data models baked into every layer of the modern data stack.

4. Surge in the demand for analytics engineers to drive data-driven insights

With companies cutting back on the size of their data teams, the demand for more well-rounded data professionals increases. Luckily, the role of the analytics engineer has become popular in the last few years, addressing this emerging need.

Analytics engineers sit in between data engineers and data analysts, sharing the skills of both roles. They can build data models using SQL and dbt, build DAG-based orchestrations, and increase data quality through testing. This combined with modern data stack tooling allows teams to set up data warehouses and data pipelines without a specialized data engineer, or get actionable insights from data models without a specialized data analyst.

At Y42, we are observing a movement called “Shift Left” where we are seeing the need to shift away from highly specialized roles. Instead, better tooling will enable roles like the analytics engineer to shine. Instead of teams needing highly specialized individuals, they will need a few analytics engineers who know how to work with more user-friendly data platforms.

5. Witnessing a trend of comprehensive data platforms and vendor consolidation for streamlined operations

Similar to the reasoning for an increase in analytics engineers and data consultants, companies are going to want to use tools that help them succeed in analytics without all of the heavy lifting required from multiple tools. Using one tool or platform that covers a lot of ground is a lot easier to integrate than five different tools, each with its own inner workings.

Horizontal data platforms allow companies to easily scale their data while unlocking instant benefits. They require fewer people to set up, as most are easily managed by an analytics engineer or small data team. Not to mention that businesses don’t need to worry about hiring data practitioners with very specific skills and experience.

They also don’t need to talk to multiple vendors, keeping track of all the different contracts, payments, and terms. Data platforms like Y42 help to take a lot of the burden off the data team, allowing them to make an impact faster compared to the time it would take to set up a modern data stack consisting of 3-5 different vendors.

Y42 allows you to eliminate most 3rd party adapters and get started with just a data warehouse. Talk about hitting the ground running!

Outlook

We recommend getting ahead of the curve by looking into these different data trends sooner rather than later. Maybe it’s time you finally learn the ChatGPT prompts to write better SQL queries, or you learn how to use the new data platform that you’ve been seeing all over LinkedIn.

These trends are here to stay.

To keep up with the changing times, here are a few things to consider:

How can I use AI to improve my work?
Do I want to consider moving to data consulting or hiring a data consultant for my team?
How can I gain some of the key skills of analytics engineers?
Do I experience a problem with data contracts? How do I want to solve this?
Would an all-in-one data platform make sense for my team and our work?

Shifts will happen in data in 2024, so consider where you want to go. Whether there is one trend that interests you or it's making you think about your career in a different light, it’s never too late to start making a change to prepare you for the year ahead.