Governing Databricks and Democratizing Data Access with Atlan
The Active Metadata Pioneers series features Atlan customers who have recently completed a thorough evaluation of the Active Metadata Management market. Paying forward what you’ve learned to the next data leader is the true spirit of the Atlan community! So they’re here to share their hard-earned perspective on an evolving market, what makes up their modern data stack, innovative use cases for metadata, and more.
In this installment of the series, we meet Jorge Plasencia, Data Catalog & Data Observability Platform Lead at Yape, a fast-growing payment app from Financial Services holding company Credicorp, offering a P2P digital wallet to more than 13 million users across Peru. Jorge shares how Yape conducted a rigorous evaluation of modern data catalogs, and the capabilities and experiences that were critical for Yape to achieve its data governance goals.
This interview has been edited for brevity and clarity.
Could you tell us a bit about yourself, your background, and what drew you to Data & Analytics?
I’m an Industrial Engineer, and I started working in the BI world for Mondelez, a CPG company. Then, I learned low-code/no-code tools like Alteryx. Finally, four years ago, I had the opportunity to learn more about Data Governance and this incredible framework of improving the productivity of team members, guiding the work they do using policies, guidelines, and standards about data management.
I learned that people from all over need to be involved in that process. Not only IT needs context about data, understanding the meaning of a field or how data is flowing from one system to another, but also business users and teams like Marketing and HR. And if you can build a data culture in your company, the adoption of these users can increase exponentially.
Now, I finally have the opportunity to implement a data catalog, myself.
Would you mind describing Yape?
We are the largest digital wallet here in Peru. We offer an application that you can install on your mobile phone. Our core business is a P2P digital wallet where you can make a transaction using a QR code or just using your phone number, but we’re transforming right now and moving beyond just P2P wallets.
We want to be a digital ecosystem here in Peru. For example, we have a marketplace embedded in our app where you can purchase tech and household products from well-known sellers, and we are enabling other features such as gaming and ticketing, as well. Right now, we have more than 13 million users.
Could you describe your data team?
We have 4 specializations, Data Engineering, Data Science, Machine Learning Engineering, and Analytics Translators.
Data Engineers develop data pipelines and automate ETL workflows and maintain our data platform. Data Scientists are centered in modeling. ML Engineers are in charge of creating, deploying, and maintaining models and experiments in our MLOps platform. Translators help connect business users with analytical solutions, and identify and measure the impact generated.
The Data Governance team is embedded in Data Engineering. We’ve been in the market for six years. We’re a young company, and we’re just starting to increase our data literacy, and improve our data processes and maturity level. So we’re part of Data Engineering because both teams work closely together, and their leader knows a lot about data governance and how to drive value from it.
Could you describe your data stack?
We’re Microsoft Azure based, with Azure Event Hub, and Confluent Kafka to move streaming data into Databricks. For visualization, we’re implementing Power BI.
How did your search for an Active Metadata Management platform start? What was important to you?
With my data catalog experience, I started as an expert on validation of other tools like Alation, Collibra, and Informatica, and when I had the opportunity to join Yape this year, I was leading the evaluation and acquisition process of our new tool. So I started asking what tools we had, what tools we were evaluating, and if what we had was correct or if we had to change the scope a little bit.
At that time, we were evaluating Atlan, Ataccama, and Collibra, based on initial market research. Collibra is one of the catalogs with more years in-market, but I saw that it didn’t meet our expectations because by early 2023, their integration with Databricks Unity Catalog wasn’t the best. We needed a tool that had a great integration with Databricks. It’s our lakehouse, and is our main source.
But more than Databricks, we needed a platform for innovation to stay ahead of our competitors. We might know what we need right now, but if the market is moving in a new direction, with AI and Chat GPT, for example, we need to have an answer for that, and the opportunity to try these tools in our data catalog. That’s what I really liked about Atlan. You’re constantly innovating with the latest trends, you have Atlan AI, you support Data Mesh natively and enhance it with your new product, Atlan Mesh.
So I had to choose a new list of three tools to be part of our evaluation, and we moved on with Atlan in the first position, then Alation and Secoda.
We had a preliminary assessment with 20+ tools, with some important criteria that led us to those three choices. First was ease-of-use, because we need to drive adoption with our end users, and if they don’t use the tool confidently, this wouldn’t work. Second was we needed a tool that moves with us as a Startup. We have an agile mindset, and we move really fast to try new tools and integrate them into our data ecosystem. This was another point where the data culture of Atlan fit really well with us.
How did you structure your evaluation, and what were the results?
So we started a Proof of Concept with Atlan, and we really liked how you conducted it. We had the help of Ravi, who knows a lot about data, and helped me with technical items like integrations and bulk uploading metadata from Excel files. We also had the help of Jill, and as a Spanish-speaking company, I really liked that she introduced a member of your team who speaks Spanish that helped us with all the workshops during the proof of concept.
We implemented Atlan over a three-week phase with our own data by running 5 use cases with 21 activities in total, which drove a lot of value for us. We invited business users who use a lot of SQL queries and different data tools, and asked them to complete a survey, and they rated Atlan highly.
During that proof of concept, we scored Atlan against an evaluation matrix of different components, and the final score of Atlan was 4.8/5. We already knew that Atlan was a really good solution for us, and at that moment, we had to make a decision to do the same proof of concept with your competitors, Alation and Secoda, or to make a decision to stop the evaluation process and start the purchasing process. So we made the decision to move on with Atlan.
Atlan just excels in the things that were important to us. It was easy to use, your connectors with Databricks and our data ecosystem worked really well, and there was Atlan University, which I used as part of the evaluation and looked great for helping with data literacy.
We also talked with other Atlan customers, who spoke really well of you, and told us that your support team was great.
And that was it. With the three parts of our proof of concept, the evaluation with our power users, and the customer reference, we knew Atlan would be great. We think Atlan has a lot of potential, and we want to build something of a community of Atlan users here, and to help other customers choose the right tool for their business.
What stood out to you about Atlan, in particular?
First, it was Prukalpa’s direction. I’ve followed her for three years now, and I like the vision of her, Varun, and the Atlan team. I know that it’s a new company, but you’re growing exponentially, and I really like your data culture.
Also, any time I searched for documentation or information over the web, I saw something Atlan created. You have a clear explanation of what Data Mesh and Data Contracts are. You explain emerging technologies well. I really liked that, because yes, I have an Active Metadata Management tool, but I also want to integrate new tools and concepts in the market like Data Contracts, and you can help me with how to do that.
I also did some market research. I looked at Crunchbase, where I saw your funding and investors, and I looked at the Forrester Wave where you’re on top. I also looked at Gartner Peer Insights where you’re really well-rated, and the same goes for G2.
So there was the vision from your co-founders, all the research, all the resources, and then some of your customers like Nasdaq and Plaid. I knew we made the right decision, because it was important to us that Atlan worked with customers that had similar needs to us, and it gave us a lot of confidence in the tool we chose.
But to be honest, it’s that you have the best UI in the market right now. For me, the most important thing is that we chose a tool that’s not only for tech people, but for everybody so we can democratize access to data.
Photo by Jonas Leupe on Unsplash