Microservices .NET Core

About us

Piotr Gankiewicz Dariusz Pawlukiewicz

Microsoft MVP, Bottega IT Minds trainer, Software Engineer, IT Consultant, Open Source contributor.

Microsoft MVP, Bottega IT Minds trainer, Software Engineer, IT Consultant, Open Source contributor.

Distributed systemsIntroduction

Monolith vs microservicesusers, orders,

deliveries, discounts, etc.

API Gateway

Domain

DAL

DB

APIDomain

DAL

DB

APIDomain

DAL

DB

APIDomain

DAL

DB

APIDomain

DAL

DB

Users

Orders

Discounts

Deliveries

Monolith vs microservices

RepositorySingle

Monolith Microservices

Multiple

TechnologySingle Multiple

DomainWhole Partial

DeploymentSingle Multiple

SPOFYES NO

MaintenanceEasy Difficult

CommunicationProcess

Monolith Microservices

Network

IntegrationSynchronous Asynchronous

ConsistencyImmediate Eventual

TestabilityEasy Difficult

ScalabilityVertical Horizontal

Applicability~90% ~10%

Systems division

Monolith

Modular monolith

Distributed monolith

Microservices

Distribution

Modularity

API

Services

Domain

Users

API

CRUD

Orders

API

Port

Adapter

Deliveries

API

Domain

DAL

Discounts

Modular monolith

DB

DAL

DB

Domain

DB DB

DTO DTO DTO

Summary● There are no precise measures of microservices (e.g. 2 weeks to rewrite it)

● Services should be autonomous (repository, code, contracts, CI & CD)

● Independence and scalability are one of the core advantages of microservices

● Polyglot programming - beware of that freedom due to maintainability

● Network partitioning is one of the key challenges in distributed systems

● Data consistency is mostly the business decision

● Distributed transactions, testing - a set of whole new problems/challenges

● Infrastructure and DevOps are critical when working with microservices

● Modular monolith - good starting point before separating microservices

What is Pacco?

● The exclusive package delivery system

● Users might send parcels containing guns, money, jewelry etc.

● Delivery is handled by the selected courier at a given date

● There’s a limited number of couriers (availability archetype)

● Customers with the higher priority (e.g. VIP) may expropriate other ones

● The availability service will be responsible for the resource reservations

Clean architecture

Clean architecture

Core

● Focuses on modelling (proper OOP) of the business entities and their behaviours.

● Free from all of the infrastructural implementation details (DB, mailing, authentication etc.) - instead, defines the abstractions to be implemented in the other layers.

● An area of the “pure functions”, where behaviour of particular objects is fully deterministic and depends only on the input parameters.

Aggregates (Entities + VO)

Exceptions

Repositories (abstractions)

Domain events

Clean architecture

App

● Modelling the particular use cases (e.g. with the usage of CQRS).

● Makes use of e.g. implemented repositories and domain models in order to orchestrate application logic.

● May define its own abstractions (ports).

● Does not contain business logic - it does belong to the domain layer.

Exceptions

Integration events

DTO

CQRS

Clean architecture

Infra

DB client

Mailing integration

Message broker

etc...

● Contains adapters (implementation of the ports).

● Cross-cutting concerns - persistence, logging, security etc.

● Might be highly modularized (e.g. separate project per infrastructural concern)

Summary

● Adjust the architecture to the application, not the other way around● Avoid over-engineering (DDD + CQRS + ES for everything)● Central layer (Core) should not have any dependencies● Aggregate defines transactional boundaries and object consistency● Keep the business logic in the domain layer● Do not allow the domain leakage (beyond the application layer boundaries)● Violation of the business logic or invariants should throw an exception● Domain events might help with decoupling the responsibilities● Business use cases should be kept in an application layer● Use the proper abstractions (ports) to define the possible dependencies● Persistence, communication etc. - adapters are part of the infrastructure

CQRSCommand Query Responsibility Segregation

Command Query Separation

Method

Query Command

Returns the data

Doesn’t mutate state

Doesn’t return data

Mutates the state

IdempotentMicro-level of

code separation

For example IEnumerable<T>

Micro-level of code separation

Macro-level of code separation (architecture)

MethodObject

Command Query Responsibility Segregation

Query Command

Returns the data

Doesn’t mutate state

Doesn’t return data

Mutates the state

Idempotent

WRITE SIDE

API GATEWAY

READ SIDE

WRITE DB READ DB

Events

Command Query

Event Sourcing

ID AccountID Balance LastUpdate

1 1287654 1000.00 12-12-2012 21:38

2 0986778 23.89 12-12-2012 12:00

AccountCreated

Balance Changed

Balance Changed

CQRS

ES

Required Not required

Summary

● CQS - divide the methods to the ones writing and fetching the data

● Ensure the idempotency of the operations retrieving the data (HTTP GET)

● Changing the system state (side-effects), should not return the data

● CQRS - handling the commands and queries (separate objects, SRP)

● Model might be splitted into 2 separate ones: write (domain), read (DTO)

● CQRS does not require Event Sourcing

● Unique ID generation might be accomplished with GUID, Snowflake ID etc.

Message exchange (direct)

PaymentsService

DeliveriesService

OrdersService

DeliveryStarted

OrderCreated

OrderCreated

PaymentCompleted

Integration via events (EDA)

Message broker

PaymentsService

OrderCreated

DeliveriesService

OrdersService

OrderCreated OrderCreated

Message brokerIntroduction to RabbitMQ.

RabbitMQ

● An open message broker, supporting e.g. AMQP● Started in 2007, implemented in Erlang language● Supported by most of the technologies (.NET, Java, Python, PHP, JS etc.)● Messages can be exchanged using the binary serialization● Exchange is a set of routing rules, that redirects the messages to queue● Routing is based on the so-called routing key, depending on the exchange● There are 4 network topologies: direct, topic, fanout, headers● Messages might contain additional metadata and headers (like in HTTP)● Queues might be durable, and the broker can persist the data e.g. in DB

Exchange types in RabbitMQ

Exchange

Polish_Queue

General_Queue

routing_key: queue.1.pl

binding: queue.*.pl

binding: queue.#

Message delivery mode

Message broker

OrderCreated

DeliveriesService

OrdersService

OrderCreated

ACK

● At most once delivery - message might get lost.● At least once delivery - message might be sent again.● MessageId - the unique message identifier, sent along with each message.

Exactly-once processing

Message broker

CreateOrder

DeliveriesService

MessageHandler

CreateOrder

UniqueMessages

CreateOrder

Entities Messages

EXISTS messageId

ACK

OrderCreated

Outbox pattern

Message broker

OrderCreated

DeliveriesService

MessageHandler

OrderCreated

ACK

Entities Outbox

INSERT order

OutboxProcessor

OrderCreated

OrderCreated

Summary

● Message exchange is being handled by the infrastructure component● Message broker allows to isolate the microservices from each other● Regardless of microservices being offline - messages will be delivered● All of the instances of particular microservices do share a common queue● Flexible serialization schema allow sending any type of data● Message count * size might slow down the queue● There are 4 network topologies (“topic” is the default one)● Message persistence reflects on the overall performance due to I/O● “At least once delivery” requires the proper handling of messages● Outbox pattern gives us a certainty of messages being always sent

API GatewaySynchronous vs. asynchronous gateway.

API Gateway (synchronous)

API

RabbitMQ

OrdersService

HTTPCreateOrder

DeliveriesService

OrderCreated

HTTP 201

OrderCreated

API Gateway (asynchronous)

APIHTTP 202

RabbitMQ

OrdersService

DeliveriesServiceOrderCreated

OrderCreated

CreateOrder

CreateOrder

CorrelationContext as message metadata

routing_key: queue.1.pl,correlationContext: { “id”: “d037e1e4-127e-41ae-a75c-7deeea266e2b”, “userId” : “0b9f0485-33fc-4d0a-871e-2e2b83a7ee6c”, “resource” : “orders” ….}

● CorrelationId - common ID binding the messages

● Created in API Gateway as an immutable object

● Contains the data that might be useful in particular services (cross-cutting concerns), yet we don’t want to put them directly into contracts definitions

● Is part of the flow, meaning that it’s never changed during the whole messages chain

Informing end-user about operation state

API RabbitMQ

OrdersService

Request-ID: 234e1da7781d421282733eebd671c84a

CreateOrder

OperationsService

OrderCreated

WS: Operation Updated

Request-ID: 234e1da7781d421282733eebd671c84aStatus: PendingRequest-ID: 234e1da7781d421282733eebd671c84aStatus: Completed

Common CorrelationId

HTTP 202

Summary

● API Gateway acts as a facade and an entrypoint to the overall system● Gateway is usually aware about the internal microservices ● API is responsible for the things like:

○ Establishing a common communication protocol○ Proper traffic redirection (e.g. typical HTTP request-response)○ Retry and other patterns making our solution resilient○ Load balancing○ Data transformation and aggregation

● Redirecting an incoming HTTP request to the queue implies asynchronicity● Asynchronous requests should have a reply channel (e.g. WebSockets)

An issue with services communication via HTTP

Service A Service B

{“host” : “locahost”,“port” : 5001,“scheme” : “http”}

appsettings.json

GET http://localhost:5001/values

dynamic* const

{“host” : “app.my”,“port” : 443,“scheme” : “https”}

var response = _client.GetAsync<Response>(“http://localhost:5001/values”)

appsettings.prod.json

Service DiscoveryWith the usage of Consul and Fabio.

Service Discovery with Consul

Service A Service B{ “name” : “service-b”, “ID” : “uniqueId”, “address” : “http://localhost”, “port”: 5001}

service-b

GET http://localhost:5001/values

Service Discovery with Consul and Fabio

Service A

Service B

{ “name” : “service-b”, “ID” : “uniqueId”, “address” : “http://localhost”, “port”: 5001, “tags”: [“url-prefix-values”]}

GET http://localhost:5001/values

values | http://localhost:5001/values

GET http://localhost:9999/values

Routing table

Summary

● HTTP communication results in the temporal coupling● The data is always up to date (which is not always true for async events)● Microservices needs to be aware of their existence (and endpoints)● Retry, circuit breaker - such patterns might make our apps more resilient● In a dynamic environment, we want to use the service names / DNS● Service registry keeps track of the existing microservices instances● Service discovery allows to find available instances of microservices● Load balancing might work either in server side or client side mode● Orchestration engines have their own registries and load balancers.

Distributed business transactionsIn the world of microservices.

Distributed transaction

API

Vehicles service

Hotels service

Flights serviceConfirmation, only if all of the steps succeed.

Otherwise, drop the reservation

and release resources.

Two phase commit (2PC)

Coordinator

Vehicles service

Hotels service

Flights service

Lock the resources at a given time.

2PC - unhappy path

Coordinator

Vehicles service

Hotels service

Flights service

At least one service didn’t send the confirmation.

Release the resources.

2PC - happy path

Coordinator

Vehicles service

Hotels service

Flights service

All of the services sent confirmations. Commit

the transaction.

Two phase commit (2PC)

● Easy to understand● Centralized business process manager● Synchronous communication

● Requires the additional code● Overall processing time depends on the slowest node● Does lock the resources

Event choreography

API

Vehiclesservice

Hotels service

Flights service

Send compensating messages

BookFlight (C)

FlightBooked (E)

HotelBooked (E)CancelHotel (C)

CancelFlight (C)

Vehicles service

Event choreography

● Easy to understand● Does not require an additional code● Fits into the Event-Driven Architecture

● No centralized place to manage the distributed process● Might result in “spaghetti” dependencies● Difficult to test

StateNo state

Saga Process Manager

Has current state

DataNo data Has the data

ScopeCompensation mechanism

Process management

Saga pattern

API

Vehiclesservice

Hotels service

Flights service

BookTrip (C)

FlightBooked (E)

HotelBooked (E)

Vehicles service

Saga

BookHotel (C)

BookVehicle (C) BookVehicleRejected (E)

Saga

Send compensating messages

CancelHotel (C)

BookFlight (C)CancelFlight (C)

Saga pattern

● Easy to understand● Centralized business process manager● Fits into the Event-Driven Architecture● “Tangible” transaction (logs)

TestsTesting the distributed applications.

Contract Testing (CDC) with PACT

Service A(consumer)

Service B(provider)

Service B(MOCK)

{ id: 1, Name: “Product”, Size: “huge”}

{ consumer: “serviceA”, provider: “serviceB”, interactions:[{ description: “GET to retrieve product”, request : { method: “GET”, path: “/parcels/1” }, response: { status: 200, body: {

id: 1, Name: “Product”, Size: “huge” } }}] }

http:localhost:9222/parcels/1

~pacts/serviceA-serviceB.jsonOn the consumer’s side the tests always

pass.

http:localhost:5003/parcels/1

On the provider’s side the tests will pass only if the

response has 100% match with the

expectations defined in the pact

ObservabilityLogging, monitoring and distributed tracing.

MetricsMonitoring the services with Prometheus and Grafana.

Logs vs. Metrics

LOGS

REACTIVE

TEXT

“DRILL DOWN”

METRICS

PROACTIVE

NUMERIC

WHOLE APP/ENV

“Processed a message with id: 1” { "value": 51683328, "name": "Memory", "unit": "bytes" }

How to efficiently monitor an app?

DEFINE THE METRICS

QUERY THE METRICS

VISUALIZE THE QUERIES

How to efficiently monitor an app?

APP

http://host:port/metricshttp_requests_total{method="post",code="400"} 3

Summary

● Centralized logs are a good starting point for system diagnostics. They give precise information about the processes taking place in a particular service.

● Metrics are countable information. They can help monitor the entire application and prevent developers from crashing the system (e.g. by informing about leaking memory in advance).

● Prometheus enables simple structuring of metrics using labels.

● PromQL is the language used to create metric projections.

Distributed tracingWith OpenTracing and Jaeger.

Fundamentals of distributed tracing

SPAN - UNIT OF WORK

START STOP

CHILD

START STOP

FOLLOWS

START STOP

Structure of Span object

CONTEXT

TRACE ID

SPAN ID

BAGGAGE

PROCESS NAME

TAG

LOG

TAG TAG

LOG

LOG

How to efficiently trace an app?

DEFINE THE SPANS

QUERY THE SPANS

VISUALIZE THE QUERIES

How does Jaeger work?

https://www.jaegertracing.io/docs/1.9/architecture/

Summary

● Span is a logical unit used for distributed tracing. It is a kind of Unit of Work, which has time-defined boundaries and additional metadata (logs, tags, baggage).

● Spans can be in one of two relationships: CHILD and FOLLOWS FROM.

● Open Tracing provides a full mechanism of span instrumentation within our application. For span reporting, a tracer is used, which consists of a sampler (with sampling heuristics selected) and a reporter (by default, reporting after UDP).

● Jaeger provides the complete infrastructure needed for effective span analysis.

SecurityHardening the microservices.

Secure configuration

Configuration Store

Service B

Service C

Service A

Decrypt service X appsettings on the fly.

ConfigureApp()

ConfigureApp()

ConfigureApp() Override appsettings

Override appsettings

Override appsettings

Dynamic credentials

Credentials Store

Service AGenerateCredentials()

[connectionString]

Database

OpenConnection(connectionString)

ExtendLease()

Lease the dynamic credentials with expiry.

Allow opening the database connection.

PKI certificates rotation

Certificate Authority

Service AIssueCertificate()

[X.509 certificate]

GetData(header: X.509 RawData())

IssueCertificate()

Issue the revokable X.509 certificate.

Service B

Validate the certificate contents and expiry.

Authentication JWT

IdentityService

Service A

Issue the expirable JSON Web Token.

SignIn()

[JWT]

GetData(header: Authorization: Bearer [JWT])

Validate JWT (issuer, expiry, signature etc.)

DevOpsBuilding, deploying and orchestrating applications.

What is Service Mesh?

k8s cluster

Ordersservice

PaymentsserviceSidecar Sidecar

pod pod