AMA with Neal Lathia: data science career tracks, shipping ML models to production, and Monzo ML stack

For the Ask-Me-Anything series, we invite ML practitioners to share their experiences with the Evidently Community. 

This time, we talked with Neal Lathia. Neal is the Staff ML Engineer at Monzo Bank, where he focuses on building Monzo’s ML platform. Formerly, he was the Associate Director of Machine Learning at Monzo. Neal also leads an open-source project modelstore. It is a Python library allowing to version, export, and save ML models to a filesystem or a cloud storage provider.

We chatted about career paths of an ML Engineer, building and expanding ML teams, Monzo’s ML stack, and 2023 ML trends. 

‍Sounds interesting? Read on for the recap of the AMA with Neal.

Career in ML engineering

You've recently taken the position of Staff ML Engineer at Monzo. Many engineers face this career choice between becoming a team leader and manager VS progressing as an individual contributor in an expert role. Do you have any advice or learnings to share?

For those who don't know, I went from being an Individual contributor (IC) to Lead to Manager (starting with 2 reports and growing the team to 11)  to Director and then back to being an individual contributor as a Staff ML Engineer. 

Unfortunately, in many companies, career progression forces people to go into people management: I've interviewed people who wanted to be managers because they see it as advancing their careers instead of because they want to be managers. Luckily, this is not the case at Monzo- my role was always at least partially hands-on, and I had the opportunity to go back to being full-time hands-on after hiring managers to take over my reports.

My overall recommendation: both the IC track and the People leader track are closer than people often think. Once you reach a certain level, everything is about leadership. You express that leadership through your code, your documents, or your meetings :)
‍

ML moves so fast that one needs to be very close to the metal to be up-to-date. With that said, how can ML managers balance being good managers and staying relevant in an environment where new libraries, advancements, and ideas appear every day?

My motto here is that a manager is good if they can retain empathy for their direct reports. That doesn't mean they have to be ten steps ahead of everyone, technically. But they should know what it feels like to do the actual work.

Many ML people are excited about the new thing, so when I was a manager, I always encouraged my team to bring the new thing to me. 

And finally, as a manager, you may have a wide view of the team (i.e., what everyone struggles with, what causes incidents), and so you have a great view of what tools may help many people!
‍

Unfortunately, there are few opportunities for Junior machine learning engineers (in France). Do you have any advice on how I can improve my profile?

Some off-the-cuff advice: 

1. Don't call yourself "junior" because that may put  off recruiters.  
2. In your CV, don't focus on "projects" or the ability to code things - try and frame things in terms of impact. Don’t feel obliged to jam your CV full of keywords.
3. The world is pretty global in the post-pandemic era, and many companies are hiring within their time zone! You should hopefully be able to look for opportunities beyond France.

Building ML teams

Could you tell a bit about the ML team at Monzo? How many people do you have onboard? How is it structured? Do you have any learning to share about team composition and setup?

Of course! I joined Monzo in 2018, and there was no ML team (there was nobody with the words “machine learning”  in their job title in the company). The two data scientists there had shipped two models, and they were pretty good.

The "team" today is, depending on how you count, just short of 20 people. I've put the word "team" in quotes because we no longer operate as a centralized team. We are embedded into cross-functional squads, and now we have various disciplines involved in building machine learning systems: Machine Learning, Engineering, Data Science, and Decision Science.

We've had a fairly steep growth curve, going from 2 people in 2019 to ~20 people in 2023. I've reviewed this progress annually. Here are the blog posts: Machine Learning at Monzo in 2020, 2021, and 2022. 
‍

In your opinion, what would be an "optimal" ML team composition (if there is such) for a small company that does quite a lot of ML, for example, an ML startup?

What kind of company are we talking about? Is ML an obvious thing for the company to do? Here’s a recent blog post about whether AI comes naturally to a company or not. 

To be honest, I don't think there is an optimal team structure, even though data team structures get many blog posts online. 

My experience is that a static team structure may work but will eventually break. And being responsive to that is a greater superpower than being committed to a structure. 

Shipping models to production

Do you remember your first time shipping a model to production (or managing this process)? What advice would you give to yourself back then? Maybe a couple of DOs and DON'Ts? 

Oh, absolutely! I remember it with a mixture of fear and joy. 

With hindsight, as my friend here:

DO 

• Get the model into production. 
• Make sure it works (test it manually).
• Focus on the value it brings to the business.

DON'T 

• Focus on inference speed.
• Overcomplicate your life (you need it to work, not to be perfect).
• Do all the things associated with "mature" ML systems.

Perhaps, the interesting question is, at what stage of ML maturity you would start measuring or caring about drift. To be honest, it came a bit later in the game for us. We zeroed in on business value first. 

Monzo ML stack

What do you consider when making a Build vs. Buy decision regarding MLOps? I know you've built a lot of stuff internally in Monzo. Did you come to regret some of those decisions? 

I regret nothing!

On a more serious note, Build vs. Buy is a very nuanced topic, and I don't think it is ever as straightforward as "build" OR "buy."

For example, if you "buy" a system, you may need to build an integration with it and educate your team on how to use it. So even when buying something, you need to build (integration or knowledge).

Equally, when you build something, you may be spending time that is better spent on delivering business value directly.

In the case of Monzo, a lot of the build decisions were facilitated by it being very easy to build new systems on our secure, resilient, and easy-to-use platform.
‍

Monzo deals with sensitive data. Have you looked into WebML, ML on the Edge, and similar trends, or more concretely, Tensorflow.js to provide ML models without user data leaving their devices? 

Good question! In Monzo's case, most of the sensitive data is not created on or by the device (the app), so it's best to work with it in the backend infrastructure. 

For example, all transactional data comes to us via the Mastercard network. So if we were to do on-device ML, we'd have to push data to the device. 
‍

How do you do ML model monitoring at Monzo? 

Monzo is (luckily) a very data-literate company with a long history of getting deep insights from its analytics/data stack. Details here. 

My personal view when it comes to tooling is that if we can use existing tooling to achieve the same result, then it's "cheaper" (from a maintenance and knowledge perspective) than adopting a new tool.

So right now, we are monitoring models using Looker (performance, alignment with business outcomes, offline experiment results, and even features!) and Grafana (production service health). 

Some other teams in the company are doing related stuff in a more manual way (Notebooks), but their needs are usually very custom.

modelstore

How did you come up with the Model Store idea? Which were the most important factors in reaching the actual first version?

The public modelstore library is the result of me being involved in building something like that 3-4 times over the last few years. 

And so when Monzo gave all staff a week off during the pandemic, I decided to build an open-source version of it. 

The core problem has always been that people train models and then need to save them — either to get them back or serve from them in production — and the existing tooling like MLflow requires setting up a tracking server (which I really did not want to do 👎)
‍

What are the differences between modelstore and MLflow in functionality and design principles?

The biggest difference (and what prompted me to create modelstore) is that I wanted all of the functionality in MLflow without needing to run or maintain a tracking server. 

I don't think modelstore has (even close) to parity with MLflow. Still, the guiding principle for me is that a user should be able to pip install it and start working, minimizing as much time spent on infrastructure as possible.

ML tools and trends

Do you keep tabs on all the new ML tools? How do you decide which tools are worth giving a try? How would the adoption process of a new tool look like for your team?

Good question. It is difficult to do with the current pace of ML tooling. 

I'd break it down into a combination of things:

1. For a new problem I'm looking to tackle, I look for related tools.
2. I'm subscribed to a ton of newsletters. I don't read them all in detail, but some tools get cross-posted on many of them, which piques my interest.
3. Inside Monzo, we sometimes allocate (time-boxed!) time to explore new tools. For example, we did two weeks of exploration with AutoGluon a while back.
   ‍

Which technologies, libraries, or trends in the ML (or even data in general) space do you find the most exciting in 2023?

I'm fascinated by how excited everyone is about ChatGPT and excited when I see people using it for things that are not text-related. 

I was also looking into other backend engineering-related platforms and have been hacking with encore.dev recently. 
‍

Where do you see the future of Large Language Models at companies such as Monzo? Will we all be working with an API from OpenAI or similar and doing some prompt engineering for our use case, or will they become like CNNs, RNNs, and any other architecture that is run internally within the cloud instances of the company and the company has a fine grain control of the model?

I don't have a definitive answer here because this is still an active conversation topic inside Monzo. 

But my non-Monzo-specific view is that the biggest challenge with LLMs today is no longer about ML. It's about building valuable products.

I'm excited about what will come :)
‍

What's your opinion about Python alternatives for ML (mainly Golang, Rust, and Julia)? Will they replace Python in the long run?

My bias — I write Python and Go at Monzo. I've written a lot of Java in the past, and beyond that, we're back in my uni days. 

We looked into replacing all of our Python with Go for ML inference at Monzo, and we concluded that doing all that work was not worth our time (compared to shipping more models that have value).

Currently, Python is dominating the ecosystem, so as a practitioner rather than a researcher, I don't see it getting displaced. 

A good example of that is that Rust is now being used under the hood of Python libraries rather than replacing Python outright. But, long term, who knows!

The decision we've made for now is to do 99% of the work in Go — because that is the main language at Monzo — and so our feature store is in Go, part of our model registry is in Go, and we implement some transformations in Go. The Python services mostly receive data, pass it into a predict() function, and return the result.

* The discussion was lightly edited for better readability.

[fs-toc-omit] Join 2500+ data scientists and ML engineers

Jump to our Discord community to get support, contribute, and chat about AI products.

Join Discord community ⟶