Picnic 10 years: 2018 — Hello, Germany

With Picnic turning 10 this year, we are dedicating 10 blog posts, one for each year, to highlight our achievements and challenges along that journey of that year. We kicked off the series with our CTO Daniel Gebler sharing 10 lessons learned in that time and subsequently dove into our approach to building our milkman routes and how we accomplished to scale our analytics and master data at Picnic.

In 2018, we were already well on our way to establishing ourselves in our home market, the Netherlands, and the opening of our third fulfilment center in Diemen early that year set the stage for launching in Amsterdam (coincidentally, this also allowed many of our colleagues to experience their first order moment). At the same time we were also looking beyond the border and setting our sights on a different kind of expansion — Hallo Deutschland, wir kommen! (Fun fact, with me being German, this would also allow my family to use the services I helped develop!)

Copy, paste, rinse, repeat … or not?

Early in the process, we decided to treat this second market as an independent deployment — significant variances in the assortment, easier compliance with the rules and regulations in each country and large differences in the integration with our suppliers (ordering, stock, etc.) helped us reach this conclusion quickly.

Nevertheless just repeating what we did in the Netherlands would not get us near our goal of launching in the most western province of Germany, North Rhine-Westphalia. While our codebase supported the needs of the Dutch market we had to tackle several challenges to be able to grow beyond it:

• Our Product Information Management (PIM) system was very basic and relied on close cooperation between the assortment managers, price managers, promotions managers and our developers — acceptable if all of them are in the same location but a roadblock for having multiple teams (with slightly different requirements!) operating across several locations.
• Our Shopping App supported at that point the primary Dutch online payment method iDEAL, as well as direct debit. As the former is not known in Germany, we had to support the inclusion of more suitable payment options, as well as start building up an international payment infrastructure supporting payment processing, fraud prevention, and collection.
• Postal codes — the Dutch system permitted us to use a simple lookup table — each postcode and house number combination is unique and identifiable. In Germany, more effort was needed — the number of people sharing one postcode can be as little as a few dozen but is closer to 10,000 on average.
• Our deployment at that point was based on Java deployments operated in Tomcat on Elastic Beanstalk and required significant manual input for new releases — unless we wanted to essentially double the amount of people working on deployments, we would need to be more efficient about it.

From “better than a spreadsheet” to a scalable PIM platform

As mentioned above, our PIM solution at this point was barebones. We were able to process product and price updates from our suppliers automatically, provide that data to other services and maintain a category tree in the solution but that was already the extent of it.

Setting the need for a solution supporting multiple markets aside, the options for extending our data model were very limited (as each product was represented as one row in a table). Validation was done on field type level but did not support more complex use cases.

Many challenges could be solved with collaborative effort between commercial, operational and tech teams, but required tight interaction and relied heavily on these teams interacting with each other on a daily basis. As Germany would be operating as an independent unit, we would not be able to replicate this way of working lest we would also duplicate all teams — an option that was not on the table for us.

Instead, we focused on integrating a new solution and landed on Salesforce. While not really a known quantity for product information management, the platform offered us many advantages:

• The object oriented data model allowed us to separate and encapsulate the complexity of our use cases much better and express relationships between various types of information circulating in Picnic — incoming raw data, the operational representations of products and the commercial information shown to customers could be cleanly separated and linked to each other.
• The more robust user model allowed us to better align access with the respective roles and separate out responsibilities, providing a more stable and secure environment. By modeling each market as a separate organisation, we could also ensure no unintentional cross-overs occurred between the markets.
• One of the biggest advantages of moving to Salesforce was the possibility for users to adapt and change aspects of the platform itself independently without requiring assistance from the tech team. This allowed operators to update functionality, add data validations and implement market specific differences, as long as they maintained a common API.

Once we had settled on our system of choice for managing the data, there remained two “trivial” problems — how to get data into Salesforce in the first place, as well as distributing the output to the various services consuming aspects of the master data set.

For the former challenge we introduced a new data processing pipeline that transformed the data into a common input format and used a custom client to push that data into Salesforce through the Bulk API.

The output side was a bit more tricky — we wanted to retain control over the Salesforce API usage patterns and therefore did not want callers to directly interact with it to preempt security considerations as well as maintain API calling limits. Instead, we settled on an intermediate caching solution that would be able to process the various objects from Salesforce and allowed consumers to define views on these objects. Using a publish mechanism triggered from within Salesforce, we could then load the latest version of the data objects, verify their format against the expected schemas and finally transform them to the views that would be distributed to clients which get a notification in the form of a new version event on our message bus.

This newly developed service, PIM cache, gave us the needed flexibility to support the use cases for master data in Picnic while at the same time limiting the actual Salesforce API interaction to a minimum.

Extending our payment infrastructure operate on an international level

When starting in the Netherlands in 2015, one of the more clear cut choices we made was to focus our payment options on iDEAL. The system has widespread acceptance, is supported by virtually any bank in the Netherlands and allows for a safe and reliable way to pay for your online purchases at checkout. The only real challenge was keeping the number of transactions per customer order at a minimum, for example when we were refunding deposits on bottles that had been returned with us.

For frequent customers, we additionally wanted to provide more convenience, as iDEAL payments require to execute the full flow at checkout. For that purpose, we also introduced the option to pay through SEPA transaction (direct debit) — in that case the delivery would be charged to the customer’s account once all actions had been reconciled into the final amount due.

Unfortunately for us, this would not be as easy once we’d cross the border to the east. Germany at that point (and arguably also to this day, although this has moved significantly during Covid) was more focused on cash payments, with online transactions primarily being done through Credit Cards. A no-brainer option like was available in the Netherlands did not exist. Furthermore we had not yet generalized our payment routing, so the choice of payment was fixed to a binary one — if it is not iDEAL, it must be SEPA.

The first order of business was finding a suitable equivalent to iDEAL — a method that allowed for secure payment at checkout and was available to most of our customers. We settled in the end on SOFORT which was not as widely adopted but had a very similar flow for the customer.

In order to offer this additional option, we had to rethink our checkout from the ground up. Until this point, we handled most of the transaction volume with our internal tooling but it became apparent to us that this field was very complex — every new market would bring its own payment approaches and supporting all these varying methods came with their own requirements, legally and technically.

We therefore decided to fundamentally change the purpose of our payment module — instead of handling the full payment flow it would route the customer to one of our partnered payment providers that offered the chosen option. While still in its infancy in 2018, we have since onboarded additional payment providers, started to offer new options to pay for your groceries and added additional functionality to prevent fraud as well as handle payment issues.

Moving from postal code lookups to generalized geospatial search

Sometimes, solutions to complex challenges can be very simple. We greatly benefited from such an occurrence when launching in the Netherlands, as we could rely on the very granular postal code system (on average, one postal code comprises eight addresses), which allowed us to define our initial delivery areas as a collection of postal codes and have a straightforward matching logic in place for determining access to the delivery slots we offered.

Unfortunately, the same principle would not apply in our next market — in fact, most countries use a more coarse definition of postal codes. We therefore needed a robust solution that we can apply anywhere without the need to adjust it to local needs.

For us, this meant looking at geospatial coordinates. Instead of just maintaining a list of valid postcodes we instead introduced a new service backed by PostGIS that allowed us to define regions by drawing polygons and determine for any given set of coordinates if they would fall into a region or not.Initially we used this service to draw out delivery areas to determine if a customer is eligible for deliveries. Over time, we could further harness the flexibility that such a lookup provided us and added additional use cases for regions or points of interest, for example functionality to alert runners to areas where caution is advised.

Next to having the regions themselves we also needed actual coordinates to check against. Lucky for us, this was a solved challenge in the Netherlands, as the postal code database we were using already contained that additional information. For other markets, we needed to rely on a third party to provide us coordinates by address and found a suitable option in Google’s Maps offering. To reduce the amount of queries to the Maps API, we elected to model each of our sources as a provider for geocoding requests and continued to use the postal codes database for the Netherlands and the Maps solution for Germany.

Very soon in our testing we also noticed that the result of such geocoding lookups was not always to our expectations, in some cases (e.g. in a very new neighbourhood) we would get no results, in others the coordinates would be off by an amount significant enough to impact our planning if we would use them. This led to the introduction of our third provider that allowed for manual modification of the coordinates returned — putting this provider as our first choice allowed us to override the aforementioned edge cases and allow for accurate deliveries.

The last piece of the puzzle was the extension of the customer UI. Until that point, we had no need to ask for the street name and needed to add that component to our registration flow. This would not stay the only addition or modification to the registration, as over the years we added more comfort functions to allow our customers an easier and more accurate way to set their delivery destination.

Leveraging Kubernetes to provide a stable platform for growth

When we first started working on Picnic in 2015, we knew from the outset that we wanted to fully leverage the evergrowing performance and flexibility of cloud infrastructure. We very quickly settled on AWS due to prior experience with that provider already existing in the team and started to build away.

Initially, we adopted Elastic Beanstalk (EB) as it gave us an easy way to run our Java workloads and added some Opsworks deployments to cover more specific solutions for running RabbitMQ in a clustered environment and operating our logging stack. For the latter, we soon found solutions more suitable to us but by 2016 EB was still our main deployment platform.

Looking forward to our expected use cases and seeing the increasing adoption rate of Docker (our first containers were by then already running in EB) we wanted to provide a good foundation for our deployments and started an evaluation of the various orchestrators, looking among others at Docker Swarm, Nomad and Kubernetes.

In the end, we went with Kubernetes, a choice that is serving us well to this day, although we have adopted EKS in the meantime.

Ready for launch

In hindsight, we were very ambitious with all the changes that we had to make as well as the ones that we wanted to make. Many of the lessons we learned in Germany we could later apply when starting Picnic in France. That launch was made significantly easier by being able to use all the generalisations we had to apply for Germany and, from a technical perspective, allowed us to primarily focus on the integration with our supplier instead of having to deal with major rewrites all over the code base.

Lucky for us we could depend on our team to deliver for Germany. We actually started operations under our Sprinter code name a few months before the official go-live, delivering mostly to internal customers. That allowed us to identify gaps in our tech approach (e.g. introducing the aforementioned manual address override) as well as bootstrap our operation on the ground, getting our employees and their equipment ready.

On March 18th 2018, early in the morning around 1 a.m. we cut over to our Picnic URLs, switched on access to the German version of the app and were officially live and ready to welcome the first customers in Nordrhein-Westfalen which would be our base of operations until our further expansion in 2023.

Picnic 10 years: 2018 — Hello, Germany was originally published in Picnic Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.