streaming microservices: contracts & compatibility gwen ... · streaming microservices:...
TRANSCRIPT
![Page 1: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/1.jpg)
1
Streaming Microservices: Contracts & Compatibility
Gwen ShapiraConfluent Inc.
![Page 2: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/2.jpg)
2
APIs are contractsbetween services
Profile service
Quote service
{user_id: 53, address: “2 Elm st”}
{user_id: 53, quote: 580}
![Page 3: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/3.jpg)
3
But not all servicesTalk to each other directly
Profile service
Quoteservice
{user_id: 53, address: “2 Elm st.”}
{user_id: 53, quote: 580}
![Page 4: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/4.jpg)
4
And naturally…
Profile service
Quoteservice
Streamprocessing
Profile database
{user_id: 53, address: “2 Elm st.”}
![Page 5: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/5.jpg)
5
… and then you have a streaming platform
Producer Consumer
Streaming Applications
Connectors ConnectorsApache Kafka
![Page 6: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/6.jpg)
6
Schema are APIs.
![Page 7: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/7.jpg)
7
It isn’t just about the services
Softwareengineering Teams &
Culture
Data & Metadata
![Page 8: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/8.jpg)
8
Lack of Schema can tightly couple teams and services
2001 2001 Citrus Heights-Sunrise BlvdCitrus_Hghts 60670001 3400293 34 SACSacramento SV Sacramento Valley SACSacramento County APCD SMA8 Sacramento
Metropolitan Area CA 6920 Sacramento 28 692013588 7400 Sunrise Blvd 95610 38 41 5638.6988889 121 16 15.98999977 -121.27111110 4284781 650345 52
![Page 9: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/9.jpg)
9
Schemas are about how teams work together
Bookingservice
{user_id: 53, timestamp: 1497842472}
create table (use_id number,timestamp number)
new Date(timestamp)
Attributionservice
Booking DB
![Page 10: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/10.jpg)
10
Bookingservice
{user_id: 53, timestamp: “June 28, 2017 4:00pm”}
Attributionservice
Booking DB
![Page 11: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/11.jpg)
11
Moving fast and breaking things
Bookingservice
{user_id: 53, timestamp: “June 28, 2017 4:00pm”}
create table (use_id number,timestamp number)
new Date(timestamp)
Attributionservice
Booking DB
![Page 12: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/12.jpg)
12
Back in my day… It was never a problem.
![Page 13: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/13.jpg)
13
And then it was.
![Page 14: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/14.jpg)
14
Moving data around since 1997Missing my Schema since 2012.Apache Kafka PMC
Tweeting a lot @gwenshap
![Page 15: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/15.jpg)
15
Existing solutions
![Page 16: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/16.jpg)
16
Existing solutions
“It is a communication problem”
“We need to improve our process”
“We need to document everything and get stakeholder approval”
![Page 17: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/17.jpg)
17
Schema are APIs.We need specificationsWe need to make changes to themWe need to detect breaking changesWe need versionsWe need tools
![Page 18: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/18.jpg)
18
Imagine a world where engineers can find the data they need and use it safely.
Its easy if you try
![Page 19: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/19.jpg)
19
There are benefits to doing this well
Bookingservice
Bookings
Profile updates
RoomGift
service
loyaltyservice
Room gift requests
![Page 20: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/20.jpg)
20
Sometimes, magic happens
Bookingservice
Profile updates
RoomGift
serviceloyaltyservice
Room gift requests
Bookings
New! Beach promo
![Page 21: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/21.jpg)
21
… but most days I’m happy if the data pipelines are humming and nothing breaks.
![Page 22: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/22.jpg)
22
![Page 23: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/23.jpg)
23
Forward compatibility:
![Page 24: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/24.jpg)
24
Forward & Backward compatibility:
![Page 25: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/25.jpg)
25
Compatibility Rules
Avro JSON
Forward Compatibility Can add fields
Can delete optional fields
(nullable / default)
Can add fields
Backward Compatibility Can delete fields
Can add optional fields
Can delete fields
Full Compatibility Can only modify optional fields Nothing is safe
![Page 26: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/26.jpg)
26
It is confusing. So it is tempting to simplify
“Never change anything”
“Adding fields is ok. Deleting is not”
”Everything is always optional except for the primary key”
![Page 27: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/27.jpg)
27
Enter Schema Registry
![Page 28: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/28.jpg)
28
Schema Registries Everywhere
![Page 29: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/29.jpg)
29
What do Schema Registries do?
1. Store schemas – put/get2. Link one or more schema to each event3. Java client that fetches & caches schemas4. Enforcement of compatibility rules5. Graphical browser
![Page 30: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/30.jpg)
30
Make those contracts binding
SerializationException
![Page 31: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/31.jpg)
31
Responsibility is slightly distributed
Producer
Serializer
Schema Registry
![Page 32: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/32.jpg)
32
Producers contain Serializers
1. Define the serializers:props.put("key.serializer", ”org.apache.kafka.serializers.StringSerializer");props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");props.put("schema.registry.url", schemaUrl);…producer<String, LogLine> producer = new KafkaProducer<String, LogLine>(props);
2. Create a record:ProducerRecord<String, LogLine> record = new ProducerRecord<String, LogLine>(topic, event.getIp().toString(), event);
3. Send the record:producer.send(record);
![Page 33: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/33.jpg)
33
Serializers cache schemas, register new schema … and serialize
serialize(topic, isKey, object):subject = getSubjectName(topic, isKey)schema = getSchema(record)schemaIdMap = schemaCache.get(subject)if (schemaIdMap.containsKey(schema):
id = schemaIdMap.get(schema)else
id = registerAndGetId(subject, schema)schemaIdMap.put(schema, id)output = MAGIC_BYTE + id + avroWriter(schema, object)
![Page 34: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/34.jpg)
34
Schema Registry caches schemas and validates compatibility
register(schema, subject):if (schemaIsNewToSubject):
prevSchema = getPrevSchema(subject)level = getCompatibilityLevel(subject)if (level == FULL):
validator = new SchemaValidatorBuilder().mutualReadStrategy().validateLatest()
if (validator.isCompatible(schema, prevSchema))register
elsethrow
…
![Page 35: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/35.jpg)
35
![Page 36: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/36.jpg)
36
Maven Plugin – because we prefer to catch problems in CI/CD
http://docs.confluent.io/current/schema-registry/docs/maven-plugin.html
• schema-registry:download
• schema-registry:test-compatibility
• schema-registry:register
![Page 37: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/37.jpg)
37
So the flow is…
Dev Nightly build / merge Prod
Test Registry
Prod Registry
Test
Dev or MockRegistry
![Page 38: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/38.jpg)
38
What if…. I NEED to break compatibility?
Customer_v1
Customer_v2
Translator
![Page 39: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/39.jpg)
39
I have this stream processing job…
Nodes can willmodify the schema
![Page 40: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/40.jpg)
40
Tracking services for fun and profit
![Page 41: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/41.jpg)
41
Schema discovery for fun and profit
![Page 42: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/42.jpg)
42
Can we enforce compliance better?
![Page 43: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/43.jpg)
43
Speaking of headers…
![Page 44: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/44.jpg)
44
And really, as an old school DBAI miss my constraints
![Page 45: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/45.jpg)
45
Why should Avro users have all the fun?
![Page 46: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/46.jpg)
46
Summary!
1. Schema are APIs for event-driven services
2.Which means compatibility is critical
3. Use Schema Registry from Dev to Prod
4.Schema Registry is in Confluent Open Source
![Page 47: Streaming Microservices: Contracts & Compatibility Gwen ... · Streaming Microservices: Contracts & Compatibility Gwen Shapira Confluent Inc. 2 APIs are contracts between services](https://reader033.vdocuments.site/reader033/viewer/2022042223/5ec9661ad52f7232643f34b2/html5/thumbnails/47.jpg)
47
Thank You!