How an Enterprise Intelligence Platform Delivered Self-Service Data Trust with Data Observability 

When it comes to the field of enterprise go-to-market intelligence, moving fast is critical for success.

We recently spoke with one platform’s data team who shared how they’re facilitating speed-to-insights for their data consumers.

After a period of rapid growth combined with several successful acquisitions, this particular platform—which delivers real-time data and insights to more than 35,000 companies worldwide—became one the first tech IPOs of the Covid-19 era. 

But, hyper-scale and pipeline reliability rarely go hand-in-hand—and that upward trajectory led to upward complexity on the data side as well.  

As the team and stakeholders grew, data quality ownership was dispersed across multiple domains. Lack of clarity around ownership rules combined with reactive data quality processes left data teams spending more time responding to downstream complaints than proactively maintaining the health of their pipelines.

Senior Manager of Data Governance Anne Fajkus knew it was time for a change. And, at the end of 2022, Anne set out to build a new data quality process that the entire organization could rally around. 

What Anne and her team needed was a solution that could facilitate prompt incident detection and resolution, while also acting as a single source of truth to help disparate teams understand the health of their data.

Let’s find out how she did it. 

The Challenges:

Inconsistent data quality processes

Organic growth combined with multiple acquisitions meant there was a lot of new data to manage. 

“When you go through acquisitions, you also acquire all that enterprise backend data from companies that didn’t do everything the same way you do it,” says Anne. “It can be a fun problem to solve.” 

But while navigating different systems can be a fun new challenge, it also comes with its fair share of data quality issues. 

“After investigating what was going on in the enterprise side of things, I realized we had a lot of disparate data quality processes,” said Anne. “We had people running Salesforce reports looking for data quality issues, we had people running Tableau reports looking for data quality issues. And the biggest pain point was that people weren’t doing these things every day.” 

These various data quality processes coupled with multiple data silos meant there were blindspots in the organization’s data quality coverage, and data health was impossible to measure and qualify over time. 

Ad-hoc incident management

Before Monte Carlo, data engineers were often spending time managing last minute, ad-hoc incident alerts on high-priority data products in the eleventh hour. 

“Our data quality management processes would occur on an ad-hoc basis,” said Anne. “It was inconsistent and ineffective.”

Inconsistent data quality monitoring also meant the team was only ever reacting to data quality issues instead of proactively improving those pipelines over time. 

Lack of proactive monitoring or stakeholder buy-in

“During the end of the month, [data engineers] used to be working 18 hour days trying to clean up data, because that’s the only time they were finding the issues – when they were looking,” said Anne.

To add fuel to the fire, both data consumers, like the FP&A team, and data producers, like Salesforce users, weren’t making things any easier. “They didn’t have to account for data quality issues, so there was some resistance,” said Anne. “People weren’t sure if they really needed to fix things.”

Faced with the prospect of enabling reliable pipelines across domains at scale, the organization turned to Monte Carlo’s data observability platform for help. 

Solution: Data observability with Monte Carlo

Immediate time-to-value with coverage out-of-the-box

The team needed an out-of-the-box solution to detect and resolve issues faster that could deliver immediate time-to-value and unify their disparate processes without wasting costly development time.

Prioritizing speed isn’t just about resolving data incidents faster – it’s also about onboarding faster as well. The sooner a team can be up and monitoring with data observability, the sooner that team will realize its value. A tool that’s difficult to deploy or onboard is unlikely to justify its cost in the near term. 

After evaluating various data observability solutions on the market, the team turned to category creator and leader Monte Carlo for help.

“It was really important to us that we had a tool that analysts could configure, set up, and get up and running. I didn’t want to have to do a lot of development work to get started,” says Anne. “That was really what set Monte Carlo apart. My team could do this themselves, and we could get pretty quick results out of it – and we did, right off the bat.” 

With Monte Carlo’s out-of-the-box monitors for key data quality pillars—like freshness, volume, and schema—the team was able to get up-and-running with data observability on day one – and then rapidly scale that coverage across all its domains.

Easy-to-use and optimized for self-service

For Anne, enabling a culture of self-service across the data organization was essential to evangelizing the importance of data quality. “I wanted to know how we could set this up so we could monitor things holistically, rather than people ‘doing data quality’ in one-off ways here and there,” said Anne.

“We wanted to get everyone on the same page [about the importance of data quality] and get some good metrics.” Metrics they’d be able to use as data quality improvement proof points to downstream stakeholders and executives.

Anne realized they needed to automate the incident detection and resolution process from end-to-end to showcase the importance of a data quality tool – and Monte Carlo’s automated monitoring was at the helm. 

First, a transition to SAP automated a significant amount of their data processes. From there, Anne worked with her team to build a monolithic spreadsheet listing of all the team’s pain points – and then used Monte Carlo to build monitors for each of them.

In order to present the impact of those monitors in a tangible way to skeptical data producers and consumers, Anne took their data quality initiative a step further. 

“We created a Tableau dashboard showing all the monitors we created, the resolver groups (the audiences), and what the incident and resolution rate was. We took that to our FP&A team to show them exactly what was happening. From then on, they could see the problems that needed to be solved, and the discussion around data quality changed. They got it.”

Results: A stronger culture of data accountability, more data trust, and less time wasted

Behavior change is often one of the biggest challenges for a data organization. Getting started with a tool doesn’t automatically mean that every user is ready and willing to dive in headfirst. It takes change management and effective strategy to market the impact that spurs action. 

For this organization, that behavior change started when the data team aggregated data quality incidents and the impact of their resolution. “[Data producers and consumers realized] they wouldn’t have to clean up the data every month because Monte Carlo had already prevented the issue. It saves them time too.”

And, it saved Anne’s team time. “We cut our FP&A monthly close rate in half. They went from over 10 days to 3 days, which has given their team the opportunity to add more checks where they want them and do more analysis. It’s better for the business.”

So, how does the organization scale this data quality strategy? Because their data team is decentralized, they also organized a committee to meet a couple times a month. “We identify a few monitors to see how we can prevent issues earlier,” said Annie. “We see the amount of failures going down each month.”

While Anne’s team controls the monitors that are created, they delegate resolution across the organization. “We use the Jira integration to create more than just messages,” she said. “Now, we create tickets.”

Moving forward: Scaling observability

So, what’s next for this go-to-market intelligence platform and their data observability journey?

“Now, we have other domains, like the marketing team, asking to learn more about what we’re doing with Monte Carlo and how it can help their work,” said Anne. “We’re also bringing our product data into our CRM, and we want to make sure that’s working, so we’re building some monitors to take care of that.”

Since getting started with Monte Carlo’s data observability solution, the team has been able to proactively manage data quality across their distributed data team. Now, accountability is clear and incident management is quicker than ever – bolstering trust in the data that underpins company operations. Data observability has given their team the pipeline visibility they needed to maintain data reliability across every domain. 

“If you get a couple of successes, then the [Monte Carlo] use cases only grow.”

And we can’t wait to see what’s next. 

Our promise: we will show you the product.