Unlocking Success: API-first Design at Etsy

Unlocking Success: API-first Design at Etsy

Table of Contents:

1. Introduction
2. The Challenges with the Existing Architecture
3. The Need for an API-First Approach
4. Introducing API Framework Version 3
5. The Benefits of Decoupling the Endpoints
6. Server-Side Composition of Resources
7. Bringing Concurrency into PHP with an Event Loop
8. The Structure of the API Infrastructure
9. Optimizing Performance with Caching and HHVM
10. The Evolution of the API Framework at Etsy
11. Lessons Learned and Future Developments

The Evolution of the API Framework at Etsy

The architecture at Etsy, a global marketplace for handmade and vintage goods, faced several challenges over time. As more users with different devices wanted to use the platform, duplicate logic became a problem, impacting consistency. Additionally, performance suffered as the user base grew, and complex requests strained the backend. To address these issues, Etsy adopted an API-first approach, transforming their architecture.

Version 3 of the API framework was developed to decouple the endpoints from the framework. By minimizing the endpoints' responsibilities to declaring routes, input expectations, and endpoint implementation, the framework effectively blurred these responsibilities together. The framework also allowed for server-side composition of resources, enabling the aggregation of requests and leveraging a PHP client to perform additional HTTP requests for concurrency.

One significant challenge was introducing concurrency into PHP, a single-threaded environment. While forking or making additional HTTP requests seemed like plausible solutions, a better approach was found with the introduction of an event loop. This event loop controlled the scheduling of parallel requests by checking for non-blocking CoMIT handles.

The API infrastructure at Etsy followed a tree-like structure, mirroring the structure of data on the website. Each HTTP request represented a node, and endpoints were responsible for aggregating data from other endpoints. To improve performance, caching was implemented using Varnish, and the performance gain from Facebook's HHVM was a catalyst for further performance improvements.

The API framework at Etsy evolved organically, with developers adapting it according to their needs. The system's flexibility allowed for continuous deployment and frequent updates. While versioning was not a significant concern due to tight control over both ends of the API, caching and careful capacity planning were essential.

As the API framework matured, it powered almost the entirety of etsy.com, making it easier to share new web features across platforms. The next steps for Etsy's API transformation involve shifting some of the complexity control back to the client, potentially compiling a more knowledgeable PHP client.

In conclusion, the API-first approach transformed the architecture at Etsy, addressing challenges related to duplicate logic, performance, and complex requests. The evolution of the API framework led to improved scalability, performance, and developer flexibility, making it a vital component of Etsy's success.

Keywords: API framework, architecture, Etsy, challenges, duplicate logic, performance, complex requests, API-first, decoupling, server-side composition, concurrency, event loop, infrastructure, caching, HHVM, continuous deployment, versioning, flexibility, scalability, developer flexibility.