Build Great Networked APIs with Swift, OpenAPI, and gRPCTim Burks, Google

Swift Cloud Workshop No. 2

September 30, 2017

What do we do?

WE WRITE CODE

What do we want?

QUALTIY!

Protocol Buffersa language-neutral, platform-neutral, extensible mechanism for serializing structured data.

Interface Builder for Data

message Person {

string name = 1;

int32 id = 2;

string email = 3;

message PhoneNumber {

string number = 1;

}

repeated PhoneNumber phone = 4;

}

Interface Builder: Developers specify their interfaces using a special tool, tooling compiles and integrates that into their apps.

Protocol Buffers: Developers specify their data structures using a special language, tooling compiles and integrates that into their apps.

Google API Discovery Format

Discovery Format Code Generators

github.com/google/google-api-go-client

github.com/google/google-api-python-client

Idiosyncracies and Lessons

Toolkit and the next generation of Google APIs

So far, so good.

REST (Discovery Format)

● 232 API descriptions at https://www.googleapis.com/discovery/v1/apis● 10 generators listed at https://developers.google.com/api-client-library/

gRPC (Toolkit)

● 30+ gRPC-based APIs● 7 target languages

OpenAPI 3.0

Industry standard format for describing for REST APIs

Originally designed for documentation, now with many other applications: API Authoring, Validation, Documentation, Analysis, Search, Testing, Mocking, Management, Code Generation

Consensus

OpenAPI-based code generators

● swagger-codegen (Open source, Smartbear + community)○ 70+ targets○ First commit: July 6, 2011○ Used by Lyft and Square to generate SDKs

● AutoRest (Open source, Microsoft)● oas-nodegen (Open source, Capital One)● APIMatic (Proprietary, APIMatic)

more?

Problems we’ve seen so far in open source code generators

● Missing or weak build system integration.● Invalid service addresses (error in OpenAPI description).● No auth.● No documentation (not even a README!).● No samples.● No test harness.● No assurance that the called service even works.● Ugly generated code.

○ Machine-generated operation names.○ Machine-generated struct names.

● Unwanted dependencies.

swagger-codegen often isn’t used “out of the box”:

“Generating client libraries involves customizing the provided language-specific templates…

The amount of modification each template needs varies by language and we’re looking forward to working with the Swagger Codegen community to share our refinements.” Val Polouchkine, Lyft

“...Swagger Codegen is a pretty active project. If you don’t check in your templates, things are gonna break unexpectedly because Swagger Codegen just uses the latest and greatest templates that are out there. So if you don’t sort of manually make sure that those things work, you’re gonna have an issue there.” Tristan Sokol, Square

Code generation pipeline

API Description GeneratableAPI Description

Language-specific model

GeneratedAPI support

code

render idiomatic API support code according

to user preferences (re. build systems,

dependencies)

verify model, define structures and

entry points,name everything

filter language-specific reserved words,

(optionally) define file structure for generated code

typical monolithiccode generation pipeline

API Description(OpenAPI)

GeneratableAPI Description(internal data

structures)

Language-specific model(internal data

structures)

GeneratedAPI support

code

All-in-one repo and package

Problems with monolithic code generators

● Long build times: changing one target requires rebuilding everything.● Long test times: new builds must be tested for every target language.● For stability, teams may prefer to archive their own generator builds.● Forks will abound.● Quality is uneven.● Versioning is hard.● Complexity and potentially unfamiliar build systems deter contributors.

protoccode generation pipeline

API Description(.proto)

GeneratableAPI Description

(binary FileDescriptors)

Language-specific model (internal data

structures)

GeneratedAPI support

code

language target-specific plugins

protoc

protoc plugin definition

Why does protoc have a plug-in architecture?

● Fast build times: changing one target only requires rebuilding its plugin.● Fast test times: new builds need only be tested for the affected targets.● For stability, teams can archive their own protoc and plugin builds.● New plugins can abound.● Separately-maintained plugins can offer different maturity levels.● Separately-maintained plugins can be appropriately versioned.● Separately-maintained plugins can be in languages that contributors

prefer.

What’s the catch?

Plugins require a well-defined interchange format.

Fortunately, we have two great tools for that.

gnostic

gnostic processed and verified protobuf representation of

OpenAPI description

protoc + pluginsOpenAPI

.proto

reusable data structures and reader for protobuf OpenAPI descriptions

gnostic apps and plugins

OpenAPIdescription

gnostic-generator

OpenAPI.proto and

compiler code

OpenAPIJSON

schema

Kubernetes OpenAPI: .json vs .pb

Format Size Deserialization time Download time (at 80 Mbps)

Json 1653 KB >500 ms 165.3 ms

Proto binary 914 KB 9.3 ms 91.4 ms

Proto binary compressed 96 KB 13.5 ms 1.3 ms

Source: mehdy@google.com

gnosticcode generation pipeline

API Description(OpenAPI v3)

GeneratableAPI Description

(Normalized/Annotated

OpenAPI v3?)

Language-specific model(Normalized/

Annotated OpenAPI v3?)

GeneratedAPI support

code

target-specificplugins

gnostic +linter

let anyone write code generation plugins...

● in their own repositories● with their own versioning● in whatever implementation language they choose

API code generation is a community problem that needs community-based solutions...

Code generators should belong to their communities.

gRPC

@grpcio

RPC: Use Cases

Direct RPCs:Microservices

RPCs to access APIs

Google APIs

OSS APIs

MobileWeb

DesktopRPCs

Datacenters Cloud

Service 1

Service 2

Service 3

Service 4Containers

@grpcio

What is gRPC?● HTTP/2 based RPC framework developed by Google● Open, Multiplatform, Secure, Performant

Multiplatform

● Idiomatic APIs in popular languages (C++, Go, Java, Python, Node.js, C#, Ruby, PHP)● Supports mobile devices (Android Java, iOS Obj-C, Swift)● Linux, Windows, Mac OS X● (web browser support in development)

OpenSource

● developed fully in open on GitHub: https://github.com/grpc/

@grpcio

● Builds on Apple’s swift-protobuf and grpc-core.● Includes:

○ gRPC framework (C and Swift components)○ generated code surface○ protoc plugin for code generation

● Full-service gRPC:○ All four gRPC API styles are supported.○ gRPC framework supports both clients and servers.○ Plugin generates client and server code in separate files.○ Testing on MacOS and Ubuntu.

● Audiences: client and server developers, inside/outside Google

gRPC for Swift (https://github.com/grpc/grpc-swift)

@grpcio

package echo;

service Echo {

// Immediately returns an echo of a request.

rpc Get(EchoRequest) returns (EchoResponse) {}

// Splits a request into words and returns each word in a stream of messages.

rpc Expand(EchoRequest) returns (stream EchoResponse) {}

// Collects a stream of messages and returns them concatenated when the caller closes.

rpc Collect(stream EchoRequest) returns (EchoResponse) {}

// Streams back messages as they are received in an input stream.

rpc Update(stream EchoRequest) returns (stream EchoResponse) {}

}

message EchoRequest {

// The text of a message to be echoed.

string text = 1;

}

message EchoResponse {

// The text of an echo response.

string text = 1;

}

gRPC Swift sample service

@grpcio

/// To build a server, implement a class that conforms to this protocol.

public protocol Echo_EchoProvider {

func get(request : Echo_EchoRequest, session : Echo_EchoGetSession) throws -> Echo_EchoResponse

func expand(request : Echo_EchoRequest, session : Echo_EchoExpandSession) throws

func collect(session : Echo_EchoCollectSession) throws

func update(session : Echo_EchoUpdateSession) throws

}

gRPC Swift server protocol

@grpcio

// get returns requests as they were received.

func get(request : Echo_EchoRequest, session : Echo_EchoGetSession) throws -> Echo_EchoResponse {

return Echo_EchoResponse(text:"Swift echo get: " + request.text)

}

...

// update streams back messages as they are received in an input stream.

func update(session : Echo_EchoUpdateSession) throws -> Void {

while true {

do {

let request = try session.Receive()

try session.Send(Echo_EchoResponse(text:"Swift echo update: \(request.text)"))

} catch Echo_EchoServerError.endOfStream {

break

}

}

try session.Close()

}

}

gRPC Swift server sample

@grpcio

gRPC Swift unary client sample // Unary

if client == "get" {

var requestMessage = Echo_EchoRequest(text:message)

let responseMessage = try service.get(requestMessage) // blocking

print("get received: " + responseMessage.text)

}

@grpcio

gRPC Swift bidirectional streaming client sample (1/2) // Bidirectional streaming

if client == "update" {

let sem = DispatchSemaphore(value: 0)

let updateCall = try service.update() // blocking

DispatchQueue.global().async {

while true {

do {

let responseMessage = try updateCall.Receive() // blocking

print("Received: \(responseMessage.text)")

} catch Echo_EchoClientError.endOfStream {

sem.signal()

break

}

}

}

...

@grpcio

gRPC Swift bidirectional streaming client sample (2/2)...

let parts = message.components(separatedBy:" ")

for part in parts {

let requestMessage = Echo_EchoRequest(text:part)

try updateCall.Send(requestMessage)

sleep(1)

}

try updateCall.CloseSend()

// Wait for the call to complete.

sem.wait()

}

gRPC-Swift TODO

● Build system integration○ Package Manager○ Cocoapods?○ Carthage?

● gRPC interoperability tests● Samples that wrap Google APIs

○ Google Cloud Speech API○ Google Datastore API

github.com/grpc/grpc-swift/issues

More gRPC Information

Website: http://grpc.io

Sources: https://github.com/grpc/grpc

Mailing list: https://groups.google.com/d/forum/grpc-io

Ecosystem: https://github.com/grpc-ecosystem

github.com/googleapis/gnosticplugins for OpenAPI-based code generation

github.com/grpc/grpc-swiftFast streaming APIs in Swift

github.com/google/auth-library-swiftOAuth support for Google Cloud

http://twitter.com/timburkstimburks@google.com