mesos http api - events.static.linuxfound.org · non-standard framework api post...
TRANSCRIPT
![Page 1: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/1.jpg)
M E S O S H T T P A P I
@ v i n o d k o n e @ i j i m e n e
![Page 2: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/2.jpg)
Mesos 1.0 IS COMING
![Page 3: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/3.jpg)
M E S O S A P I S
S c h e d u l e r M a s t e r S l a v e E x e c u t o r
Scheduler API Executor API
Internal API
Operator API
![Page 4: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/4.jpg)
![Page 5: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/5.jpg)
D E P E N D E N C E O N N A T I V E L I B R A R Y
S c h e d u l e r
M a s t e rJ a v a L i b r a r y
N a t i v e L i b r a r y
mesos.jar libmesos.so
Hard to debug
Not portable
![Page 6: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/6.jpg)
N O N - S TA N D A R D F R A M E W O R K A P I
POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1
User-Agent: libprocess/[email protected]:8081
Libprocess-From: [email protected]:8081
Connection: Keep-Alive
Host: Transfer-Encoding: chunked
![Page 7: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/7.jpg)
N E T W O R K I N G C O N S T R A I N T S
S c h e d u l e r M a s t e r
Firewall
Containers :( Network partitions :(
![Page 8: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/8.jpg)
L A C K O F A P I V E R S I O N I N G
GET /metrics/snapshot
Version
GET /state.json “version” : 0.23.0
JSON
![Page 9: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/9.jpg)
P I TA F O R M E S O S D E V S
Lot of boiler plate to add new calls/events
Forced upgrade dependencies
![Page 10: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/10.jpg)
– H A R V E Y D E N T
“You either die a hero… or live long enough to see yourself become the villain”
replaced by a better API
![Page 11: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/11.jpg)
1 . 0 G O A L S
Consistent APIs
Versioning
![Page 12: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/12.jpg)
N E W H T T P A P I
• Standard HTTP 1.1
• Versioned !
• Well documented
![Page 13: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/13.jpg)
N E W M E S O S A P I S
Endpoint API Hosted by
/api/v1/scheduler Scheduler API Master
/api/v1/executor Executor API Slave
/api/v1/internal Internal API Master
/api/v1/admin Operator API Master / Slave
![Page 14: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/14.jpg)
S C H E D U L E R H T T P A P I
• Based on Calls and Events
• Scheduler opens connections to the master
• A persistent connection to receive events
• One (or more) connection(s) to send calls
![Page 15: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/15.jpg)
R A T I O N A L E
Simplicity
Off-the-shelf HTTP client libraries
No native dependencies
![Page 16: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/16.jpg)
R A T I O N A L E
Upgradability
Familiar to existing APIs
killTask() Call.Kill
![Page 17: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/17.jpg)
R A T I O N A L E
Extensibility
Easy to add support for new features
![Page 18: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/18.jpg)
C A L L S Old APIS U B S C R I B E start()T E A R D O W N stop()
A C C E P T acceptOffers()D E C L I N E declineOffer()
R E V I V E reviveOffers()K I L L killTask()
S H U T D O W N * Shutdown executor *A C K N O W L E D G E acknowledgeStatusUpdate()
R E C O N C I L E reconcileTasks()M E S S A G E sendFrameworkMessage()R E Q U E S T requestResources()
![Page 19: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/19.jpg)
E V E N T S Old API
S U B S C R I B E D registered() / reregistered()
O F F E R S resourceOffers()
R E S C I N D offerRescinded()
U P D AT E statusUpdate()
M E S S A G E frameworkMessage()
FA I L U R E slaveLost() / executorLost()
E R R O R error()
H E A R T B E AT * Periodic heartbeats *
![Page 20: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/20.jpg)
P R O T O C O L
• Every call is a HTTP POST request
• application/json or application/x-protobuf
• SUBSCRIBE call results in a “200 OK” streaming response
• Record-IO formatted events
• chunked encoding
• All successful non-SUBSCRIBE calls result in “202 Accepted”
![Page 21: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/21.jpg)
S U B S C R I P T I O N R E Q U E S T
POST /api/v1/scheduler HTTP/1.1
Host: masterhost:5050 Content-Type: application/json Accept: application/json Connection: close
{ “type” : “SUBSCRIBE”,
“subscribe” : { “framework_info” : { “user” : “foo”, “name” : “Example HTTP Framework” },
“force” : true } }
![Page 22: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/22.jpg)
S U B S C R I P T I O N R E S P O N S E
HTTP/1.1 200 OK
Content-Type: application/json Transfer-Encoding: chunked
<Event Length> { “type” : “SUBSCRIBED”, “subscribed” : { “framework_id” : {“value”:“12220-3440-12532-2345”}, “heartbeat_interval_seconds” : 15 } } <more events>
![Page 23: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/23.jpg)
K I L L
POST /api/v1/scheduler HTTP/1.1
Host: masterhost:5050 Content-Type: application/son Accept: application/json Connection: close
{ “framework_id” : {“value” : “12220-3440-12532-2345”},
“type” : “KILL”,
“kill” : { “task_id” : {“value” : “12220-3440-12532-my-task”}, “slave_id” : {“value” : “12220-3440-12532-S123345”} } }
Response: HTTP/1.1 202 Accepted
![Page 24: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/24.jpg)
D I S C O N N E C T I O N S & PA R T I T I O N S
• Master tracks the persistent subscription connection
• Reconnect within failover timeout
• Subscribe.force : Failover
• Periodic HEARTBEATs sent by master
![Page 25: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/25.jpg)
V E R S I O N I N G
Explicit Simple
Avoid version explosion
/api/v1/scheduler
/api/v1/executor/api/v1/admin/api/v1/internal
![Page 26: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/26.jpg)
A P I V E R S I O N V S R E L E A S E V E R S I O N
• API version == Major release version
• v1 API supported by 1.0.0, 1.4.0, 1.20.0
• vN API released in N-1 release version
• vN API considered stable in the last N-1 release
• Version bumping
• Major/API version bumped for backwards incompatible changes (> yearly)
• Minor version bumped regularly (4-8 weeks)
![Page 27: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/27.jpg)
R O A D T O 1 . 0
v1 scheduler API
v1 executor API
0.24.0
0.25.0
v1 operator API
1.0v1 internal API
![Page 28: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/28.jpg)
D E M O
![Page 29: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/29.jpg)
![Page 30: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/30.jpg)
R A T I O N A L E
Simplicity
![Page 31: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/31.jpg)
R A T I O N A L E
Simplicity
Upgradability
![Page 32: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/32.jpg)
R A T I O N A L E
Simplicity
Upgradability
Extensibility
![Page 33: Mesos HTTP API - events.static.linuxfound.org · NON-STANDARD FRAMEWORK API POST /master/mesos.internal.LaunchTasksMessage HTTP/1.1 User-Agent: libprocess/scheduler-1234-23-23342@127.0.0.1:8081](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec565a2d68084568c75fdb0/html5/thumbnails/33.jpg)
A C K N O W L E D G E M E N T S