![Page 1: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/1.jpg)
Automatic Discovery of Service Metadata
Martina Iglesias Fernández
May 2017
![Page 2: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/2.jpg)
Spotify scale
![Page 3: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/3.jpg)
Spotify Users
100M Active Users
50MPaid Subscribers 60 Countries
![Page 4: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/4.jpg)
Spotify tech
● +1000 tech employees
● +120 teams
● Microservices architecture (scale and work
independently)
![Page 5: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/5.jpg)
Spotify teams: squads
● Cross functional team
● Full, end to end ownership of features
● Autonomous
![Page 6: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/6.jpg)
![Page 7: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/7.jpg)
Artist page
Number of streams
![Page 8: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/8.jpg)
Artist page
Artist Discography
![Page 9: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/9.jpg)
Artist page
Artist Playlists
![Page 10: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/10.jpg)
Artist page
Merchandise
![Page 11: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/11.jpg)
Example: aggregating service
Artist
Discography Playcount Playlists Merch
![Page 12: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/12.jpg)
Spotify infrastructure
● ~1000 services
● The number of services grows as we add new
features
![Page 13: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/13.jpg)
Spotify officesStockholm
BostonNew York
San Francisco
Gothenburg
![Page 14: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/14.jpg)
Previous situation: lack of the big picture
● Microservices are easy to understand in isolation
but it becomes difficult to see the big picture
● Some people knew what was out there
![Page 15: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/15.jpg)
We all agree that we need some kind of
documentation
![Page 16: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/16.jpg)
Previous situation: doc is spread out
● Each team had doc. in different places
● README.mk
● Markdown files in doc/
● Wiki
● Link to a document somewhere else
![Page 17: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/17.jpg)
![Page 18: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/18.jpg)
System Z
![Page 19: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/19.jpg)
System Z
● Angular based web application
● Internal tool
● Catalogue of all our systems and its parts
● Very well integrated with our apollo services
● Easy to discover and access
![Page 20: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/20.jpg)
System Z model● Components and Systems
● Components can be different things:
○ Libraries○ Data pipelines○ Views in the client○ Data store○ Service
![Page 21: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/21.jpg)
![Page 22: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/22.jpg)
![Page 23: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/23.jpg)
![Page 24: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/24.jpg)
![Page 25: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/25.jpg)
![Page 26: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/26.jpg)
artist
playcount
discography
merch
playlist
loadbalancer
![Page 27: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/27.jpg)
System Map
● Generated from runtime and declared
dependencies
● Uses graphviz
![Page 28: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/28.jpg)
How does System Z work
![Page 29: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/29.jpg)
![Page 30: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/30.jpg)
DNS
System Z overview
Register artist serviceService-info.yaml
Component: artistDNS: _artist._httpOWNER: kiwi
SYSTEM Z(backend)
artist-1.spotify.com
Lookup machines
HTTP req to /meta/0/
![Page 31: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/31.jpg)
Apollo and apollo-meta
![Page 32: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/32.jpg)
Apollo
● Java libraries for writing microservices
● Open Source
● https://github.com/spotify/apollo
● In production since 2014
![Page 33: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/33.jpg)
APOLLO SERVICE
MODULES
OKHTTP-CLIENTJETTY-HTTP-SERVER
APOLLO-CORE(manages lifecycle)APOLLO-API APOLLO-META
SERVICE LOGIC
interacts
uses
Apollo service overview
![Page 34: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/34.jpg)
Apollo-meta
● Metadata module
● Open Source
● https://github.com/spotify/apollo-meta
● Exposes endpoints with metadata of the service
● Runtime generated - source of truth
![Page 35: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/35.jpg)
Example: Creating a route
Route.async("GET", "/v1/artist/<id>",
request-> getArtist(request))
.withDocString("Get the artist page for a specific id.")
)
![Page 36: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/36.jpg)
Endpoints exposed by apollo meta
1. Instance information2. Configuration3. Endpoints4. Call Information
![Page 37: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/37.jpg)
1. Apollo-meta: instance info
● Collects information about the service: build
version and uptime
● Useful to get the full picture when making rolling
upgrades.
![Page 38: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/38.jpg)
1. Apollo-meta: instance info
curl http://artist-a1/_meta/0/info =>
{ "result": { "buildVersion": "artist 2.1-SNAPSHOT", "containerVersion": "apollo-standalone 1.1.0", "systemVersion": "java 1.8.0_60", "serviceUptime": 303577.347 }}
![Page 39: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/39.jpg)
2. Apollo-meta: configuration
● The current loaded config of the service, possibly
filtered
![Page 40: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/40.jpg)
2. Apollo-meta: config
curl http://artist-a1/_meta/0/config =>{ result: { http: { server: { port: 8080 } } }}
![Page 41: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/41.jpg)
3. Apollo-meta: endpoints
● Lists the endpoints of the service
![Page 42: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/42.jpg)
3. Apollo-meta: endpoints
curl http://artist-a1/_meta/0/endpoints =>{ methodName: "/v1/artist/<id>[GET]", uri: "/v1/artist/<id>" method: [ "GET" ], docstring: "Get the artist page for a specific id.", queryParameters: []
},...
![Page 43: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/43.jpg)
4. Apollo-meta: call info
● Lists services that make incoming requests
● Lists all other services we make requests to
![Page 44: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/44.jpg)
4. Apollo-meta: call info
curl http://artist-a1/_meta/0/calls =>
incoming: { loadbalancer: { endpoints: [ uri: "/v1/artist/<id>", method: ["GET"], queryParameters: [catalogue, locale] ], }, },..
![Page 45: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/45.jpg)
4. Apollo-meta: call info
... outgoing: { discography: [], playcount: [], playlist: [], merch: [] } }}
![Page 46: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/46.jpg)
Apollo meta <-> System Z
● System Z calls these endpoints
● Displays a merged version of all the data
![Page 47: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/47.jpg)
What other things can we do with System Z?
![Page 48: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/48.jpg)
Manage deployments
![Page 49: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/49.jpg)
![Page 50: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/50.jpg)
Provision hardware
![Page 51: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/51.jpg)
Discover other useful services
![Page 52: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/52.jpg)
Dependency tracking
![Page 53: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/53.jpg)
![Page 54: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/54.jpg)
Ownership and alerting
![Page 55: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/55.jpg)
![Page 56: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/56.jpg)
Conclusions
![Page 57: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/57.jpg)
● Quicker access to relevant information
● Know immediately where to go when solving an
incident
● Quick way to check who is on-call for a service
● Less interruptions
● Less boring work
Situation now
![Page 58: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/58.jpg)
● Think about growth and scaling
● Automate documentation generation
● Put all the information related in one, easy to
access, place
● All related links in one place
Learnings
![Page 59: Automatic Discovery - MicroCPH · Automatic Discovery of Service Metadata Martina Iglesias Fernández martina@spotify.com May 2017. Spotify scale. Spotify Users 100M Active Users](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed3e53754a8a36a2d475a31/html5/thumbnails/59.jpg)
● apollo
● swagger.io
● raml.org (jax-rs)
Documentation generators