Tech
Kyrylo Kozlov
Apr 13, 2022
Scala at Wolt – Technical Decisions (Part II)
Behind the scenes of Wolt we have an engineering team of almost 300 people building the tech powering our products, services and platform. Scala is one the core technologies we use for backend services. Our Scala community is currently over 25 people and the plan is to grow this number further this year.
This is the second part of our story about Scala at Wolt. Let’s remind the contents of the series:
In Part I we discussed:
Why we use Scala at Wolt
How we grow as a Scala organization
How we think about the growth and development of our engineers
In Part II you’ll find out more about:
How technical decisions are made
How we approach technical debt
An overview of the frameworks we use
Part II — Technical Decisions
After we have introduced our core principles and how we make judgments about things we do. Let’s talk a bit more about practicalities and how we build up from what was described in the “Part I – OurScala Organization”.
How do we approach technical debt?
As any fast-growing and young company, we have some technical debt. Focusing on shipping features fast often produces non-ideal architecture and solutions. This kind of approach is often required during the early stages, but when it comes to scaling and exponential growth — it can bring some bottlenecks. Saying this, at Wolt we still have a few monolith systems and an architecture around this.
So what do we do with this? Divide and conquer. We’re building microservices and event-driven architecture. Step-by-step: split logic, monitor, extract services, switch to newly extracted services, and again monitor and scale.
At the moment we have a project that we consider technical debt. It’s using the Play web framework and it has notable complexity in the database level logic. Historically this was caused by active features development and a rather small team of Scala superheroes. This project is still evolving and plays an important role in the overall architecture. And yes, we prefer adding new features to the new microservices (ZIO/Cats + Http4s), but if it’s not possible from the scratch or takes too long, often we use a combined approach:
First, we launch the feature as an MVP on top of the existing database structure and project
In parallel, we start a new microservice with an API that serves the feature
During the work on the MVP, we design it in such a way that it will be easy to switch traffic from the old database and project to a new one
We launch an MVP — business wins and we’re delivering value to the customer
In the end, we launch a new service — both business and architecture win!
We’re also actively slicing this technical debt project on the database and business logic level, isolating data and extracting new APIs, extracting then microservices and in the end, we’re isolating this data on the database level.
We make decisions together
Here it makes sense to first describe how our Product Development organization is set up. To build our products, we’re divided into groups that are divided into cross-functional, independent and autonomous teams.
Groups own a core area of Wolt and form a team of teams focusing on key customer groups. An example would be our Courier Group, which builds the tools and automations used by our Couriers partners to make Wolt useful, efficient, reliable and delightful to use for them. Teams focus on a key customer or product area within that customer group. So for instance, within the Courier Group we have various teams, for instance Courier Onboarding team focusing on making the courier partners’ onboarding as smooth as possible, and Courier Delivery Experience team focusing on the details of their experience when delivering orders to customers.
We make decisions as a team and group of teams. We’re free to choose the technology stack and tools and make architectural decisions together. We don’t have separate architecture guilds or architecture castes. Responsibilities are divided at the group level — a pool of engineers is responsible for a pool of services.
On group level: Discussion on architecture happens on the group level. We have regular syncs around the state of the architecture. Our Staff Engineers play the role of communication leaders that keep track of what’s happening inside of each group and facilitate conversation within groups.
On team level: Each team is responsible for their focused area, some companies call these “feature teams”. Teams deliver the code, infrastructure, product features, and overall experience around the core tasks they have — for example deliver the best experience for our courier partners (which includes mobile app experience and all related backend services).
We act locally, but think globally. This means that every team is responsible for their part of the business area. Teams drive features that are “local” to their business area responsibilities. At the same time, your role isn’t limited to the team level. You may pick a task to improve the overall architecture of the services inside of the group or pick “global” architecture challenges for Wolt. Thinking “globally” helps us improve our developer experience at Wolt.
Read more about our philosophy of building an Impact Driven Tech Organization by our CTO at Wolt Niilo Säämänen. Trust me, it’s a great read!
A glance into the frameworks we use
Here we come to a bit more technical details about the work as a Scala developer at Wolt. As mentioned before, our teams choose the tech they use. So anyone can suggest a new technology.
The current set of the most used frameworks:
We put using PlayFramework on hold, but why?
Lightbend has stopped active support of the PlayFramework: “We are working to form a community-led organization that would take responsibility for the future development of Play Framework outside of Lightbend”. Read more in the Lightbend article “On the future of the Play Framework”
And the future, for now, isn’t that bright and it’s a bit unclear as the activity around the project has dropped significantly during the last few years.
Where are we moving to?
At the same time, there has been a rise of a few major branches in backend services development in a Scala world, which are Cats + Http4s led by Typelevel and ZIO led by Ziverge. Pure functional effect systems and newly created libraries for async and concurrent programming with rich data type abstractions.
Why do we use Cats and Zio and so on?
Effects type systems give proper control over the execution of the async and concurrent computations. Rich data structures and types allow to describe problems precisely and follow main functional programming principles, which makes programs more predictable and safe.
We have people that believe that it’s the future of Scala and they’re ready to drive these changes. Of course, these technologies have a learning curve, but anything new does. And Wolt actively works on helping our developers master their skills and knowledge here.
Freedom of tech that we use? How do we decide about libraries or frameworks?
Even when we say that the Play Framework is on hold, we have teams that prefer starting or continuing projects in Play. As we’ve already mentioned before, teams own their technology choices.
What about Databases & storage?
The database selection is rather standard and includes, but isn’t limited to Postgres, Redis, Mongo, DynamoDB, S3, etc. The decision about the database technology again depends on the project needs and the team’s decisions.
What about Scala3?
We do have experiments with small projects, but it isn’t yet fully adopted. The plan is to implement new services in modern frameworks that support the shift to Scala3, but in the end, it’s a team’s decision and responsibility to push such changes.
And finally, a few words on architecture. Most of our services are REST/HTTP services, but we’re adopting Event-based architecture as well where it’s necessary. Also, we’re running our infrastructure on top of the AWS services and Kubernetes ecosystem.
Final words…
Scala at Wolt is an actively growing and evolving community. Tech companies have various paths in adopting technologies and building tech communities and this is ours, so far. We’re constantly reviewing, iterating and improving our approach to best build our tech communities within Wolt for an enjoyable experience for our engineers.
