From chaos to control: Building ML platform | Volvo Cars

Hello, everyone. Thanks for coming. My name is George. I'm engineering manager for the ML operations team at Volvo Cars. And this is Steve. Steve is a senior engineer in the team. We're not going to talk about cars today. We're going to talk about Abakus, which is the platform we've built internally. So less than two minutes is what it takes today for any engineer or data scientist to go from the idea to validation. When we started building Abakus, we started with a goal in mind. We wanted to reduce friction and improve efficiency and, as a byproduct, also accelerate some innovation. So we've eliminated long lead times, formal approvals, and a fragmented ecosystem in the company, among some other organizational and technical challenges. Today we're going to share with you our journey. We're going to show you the platform and go through the major design decisions that we took along the way. I'll hand it over to Steve to go through the tech stack.

Yeah, thanks, George. So we chose a modern cloud-native stack specifically because it provided the scalability, reliability, and cost efficiency that was required for us at an enterprise scale. The platform itself here is Abakus. It's the top layer here, and these are the components that we're going to talk about further today. Before we start, we'd like to acknowledge the other components that we build on. In particular, we use the Enterprise Container Platform, which is a common code base used to create many Kubernetes clusters onto which Volvo Cars deploys a myriad of container-based workflows. This platform of platforms approach allows us to focus on our primary concern, which is the ML platform. As you can see, we're fond of cloud-native and open-source technologies. Abakus is built around the Kubeflow ecosystem of software, and we've added several other products into this to complement and integrate it into the enterprise environment.

So let's start by looking at this platform and setting the context by seeing some numbers. This gives a hint of the type of scale at which we're working at in our company. The highlights are that we have around 200 monthly active users. We've been running production workloads here for around three years, including some time at the very start on a development cluster, which was not ideal. However, the figures that we are most proud of are the ones around the community. We use Slack for both announcements and support. We really believe that this transparency allows us to solve support issues out in the open together with the users. It's the engineers who are actually building the platform who are solving those issues and interacting with the users in Slack. There's no first or second line support, and we see that this creates a bond between the engineers and the users. We also see that other users dip in and help each other out, which is great. We have 48 contributors to our source code repo, and we've promoted that anyone within the company can suggest a modification simply by creating a PR. We see that that happens quite often.

So this is a little bit about where we are today, but what did we start from? This picture might be familiar to anyone working in the ML space. We won't dwell here too long, but in short, at the beginning, we had almost non-existent support for our data scientists. This led to a fairly scattered technology landscape, and we had individuals either working alone or within small groups, often solving the same problems, such as access to data sources. Without a common development process, we risked having a disjointed approach to machine learning.

The benefit is that you really can tailor it for business. You're very close to your users, at least you should be. You really understand the company and what teams are trying to achieve. Then you can tailor your solution, your platform, for the company that you're working with. Ultimately, the success of your users is the success of your platform. That's why it's a two-way street of knowledge sharing and learning. Because when you introduce, let's say, in our example, a Python package, a default structure for a GitHub repository, even if data scientists didn't like it and really liked the notebook server, well, now they're exposed to a Python package and they think about how they structure and design their code.

The learnings, well, the biggest one has already been mentioned by Steve, but integration is a much bigger effort than installation. You can pip install, helm install, kubectl apply, whatever, or double-click install a tool. It works fine, maybe, but integrating everything together within the platform and then integrating the platform externally within the company is where most of the work is. The devil is in the details. The documentation might say one thing, but in reality, it's very different. You need to treat your platform as a product because that's what it is. You have users that you need to tailor for. You need to listen to them. You definitely should not implement everything that they ask for, but if there is a large number of users that need a certain feature, that's probably a good indicator that that's what your platform is missing and that's what you should focus on. Keep it modular and extensible. All right, I'm out of time. Thank you very much. I hope you enjoyed it.