Why are data teams under pressure, and how do they prove their value?

Transcript

Hello everyone and welcome to the next episode of my video blog "Data Pipeline" where I smoke pipes and talk about data and leadership.

In today's episode we want to talk about the value of the data team. Times are tough for the tech industry nowadays: you have layoffs, you have budget cuts everywhere. So in many companies data teams are being questioned. Sometimes it's the number of people and sometimes even the existence of your team.

Let's take a few steps back and try to understand: how did we get there?

Obviously, 5 or 10 years ago there was a hype around data science. Every company I know was hiring data scientists like crazy. And even the term "data science" was not very well defined. So mostly machine learning people or people on the intersection between machine learning and data engineering got jobs easily.

Unfortunately, as with every hype, many companies were hiring data scientists for the sake of hiring data scientists. Nobody was looking into the business value they produce. Rarely there was a business justification for those roles.

I feel now we are doing the next iteration of this approach with AI. I am not questioning the value of AI-powered tools. I still feel many times it's more about having AI in your pitch deck and not so much about the content of the job. Hiring without a purpose always leads to waste.

So why do you actually need data to run your business?

Let's use this metaphor. When you ride a bike or a car or maybe even a plane, if you're a pilot, you need some data to make decisions, right? You need a dashboard. In a plane, it's obvious. For a bike, it's not that necessary. You can look around and make judgments based on what you see. Nevertheless, having the data helps a lot.

You can drive your car without looking at mirrors or on the dashboard for a while. You can even close your eyes for a while. But probably, in the long run, it won't be very successful. Without data you are flying blind. And in a big complex system like a modern company, it's challenging.

I'm not saying you always need data. You can make a lot of business decisions based on your gut, and sometimes those are the best decisions. But having the data and still going with your gut is a different thing than just having no data and flying blind.

I believe the value for a business is: you can save money by avoiding wrong decisions, and you can run safe experiments. Before you run experiments with your real users or your products, you can actually judge based on the data: historical data you have.

In some companies, data is part of their product or their offering, and here it's much easier to justify why you need a data team.

My own experience: I worked for a company which had B2C clients and basically sold data about used cars. Because when people make a decision if they want to buy a used car, they pretty much pay for background checks and for vehicle history reports, if you can provide those.

In another company, a great company I worked for, we had a lot of B2B clients. It was in e-health, and we had a lot of data about symptoms different people have around the world. Of course, you cannot sell the data of your patients or your users. But you can generate insights out of them, aggregated, anonymized data, which is pretty much valuable for many business partners. In that case, those were life science companies, big pharma, or health insurers all around the world.

In this kind of setup, it's easy to justify the data team, because basically the data team is producing your product: the "data insights".

What if you are in a typical product company, and data is just used for internal business decisions? In this kind of data business, data teams are first-class citizens, and you typically have an easy time justifying their existence.

Next point I would like to raise is invisible work and advocacy. In every data organization you have infrastructure roles: engineers who provide pipelines. Business often sees only analysts or scientists, because they are just facing the end clients. That's why everyone must advocate for their impact.

Here, my message to engineers: don't build tech for the tech's sake. Solve business problems. It's part of your job to show your value and your importance to the business people. And of course, it's a big part of the data leader's job to assure that the whole team gets visibility.

Now some final thoughts. You do need a data team nowadays, even though size and tooling can vary. AI won't replace all roles anytime soon. Metadata, documentation, governance still matter.

At this point, I would like to encourage every job seeker out there: you bring value, guys. You are important. Companies do need you. Keep trying, keep applying, keep pushing. Smart companies do realize the value of the data and the value of their data teams.

Thanks a lot for listening. See you in the next episode. Follow, like, share, let's connect, let's keep us visible!