![Page 1: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/1.jpg)
Middleware Support for RDMA-based Data Transfer in Cloud Computing
Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi
Department of Electrical and Computer Engineering
Stony Brook University
![Page 2: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/2.jpg)
Outline
Introduction and Background Middleware Design and RFTP application Experimental Results Conclusion
![Page 3: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/3.jpg)
Outline
Introduction and Background Overview RDMA Semantics
Middleware Design and RFTP application Experimental Results Conclusion
![Page 4: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/4.jpg)
Today’s Data-intensive Applications
Explosion of data, and massive data processing Scalable storage systems Ultra-high speed network for data transfer: 40/100Gbps
networks Reliable Transfer (error checking and recovery) at
40/100G speed, burden on processing power
![Page 5: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/5.jpg)
ANI Ultra-high Speed Network
![Page 6: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/6.jpg)
End-to-End 40/100G Networking
100G APPS 100G APPS
FTP 100FTP 100
40/100G NIC 40/100G NIC
40/100 GbpsBackbone40/100 GbpsBackbone
100 G APPS 100 G APPS
FTP 100FTP 100
40/100G NIC 40/100G NIC
End-to-End Networking at 40/100 Gbits/sEnd-to-End Networking at 40/100 Gbits/s
Our project and its role
![Page 7: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/7.jpg)
Protocol Offload and Hardware Acceleration TCP/IP Offload Engine (TOE) Protocol Offload Engine (POE) Remote Directory Memory Access (RDMA)
Kernel by pass Zero-copy
![Page 8: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/8.jpg)
Applications over different RDMA implementations
![Page 9: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/9.jpg)
RDMA Semantics
Channel Semantic – SEND/RECV Two-side operation Both data source and data sink are involved. The sink pre-
posts a list of buffers into receive queue.
Memory Semantic – RDMA WRITE/RDMA READ One-side operation Credit-based. The sink advertises its available registered
memory to the source for RDMA_WRITE operation.
We use RDMA WRITE operation to deliver user payload(128KB ~ 4MB per block), while use SEND/RECV to exchange control messages( ~2KB).
![Page 10: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/10.jpg)
Outline
Introduction and Backgroud Middleware Design and RFTP application
Middleware Layer Middleware Software Architecture Asynchronous Communication Events design RFTP Modules RDMA extension to standard FTP protocol
Experimental Results Conclusion
![Page 11: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/11.jpg)
Middleware Layer
InfiniBand RoCE iWARP
IB Verbslibibverbs
RDMA CMlibrdmacm
ApplicationApplication
BufferManagement
ConnectionManagement
EventDispatch/Join
TaskScheduling
Middleware
OFED
Hardware
![Page 12: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/12.jpg)
Middleware – Multi-threaded ArchitectureThreadsData Structure
CQQP-1 QP-2 QP-n
Data Block List
Receive Control Message List
Send Control Message List
Remote MR Info List
application
system
Queue Pair List
Memory
Sender
CE dispatcher
CE slave-n
...
CE slave-2
CE slave-1
Logger
Hardware
HCA
1
234
![Page 13: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/13.jpg)
Communication Events
Session ID negotiation Each data transfer task will be assigned a unique session ID
Number of data connection negotiation Establish several parallel connections
Memory region credit request and response The source issues request of Memory regions’ information The sink feedbacks several credit according to buffer status
Block completion notification The source issues a notification to the sink which block’s data
is ready
![Page 14: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/14.jpg)
Parallel and Pipelined Data Transfer
Explore parallelism of RDMA operations Multiple active data streams Each stream uses a pipelined execution
Out-of-order blocks Reorder Deliver in-order blocks to application
![Page 15: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/15.jpg)
RDMA-enabled FTP - RFTP
RDMA Middleware
FTP …
Disk I/O Module
InfiniBand iWARP RoCE
Verbs Communication manager
SSD Magnetic
Disk Driver
API
API
Hardware
OperatingSystem
Middleware
Application
Buffer Manage
I/O Scheduling
Connection Manage
Event Dispatch
Task Scheduling
Direct I/O
API
![Page 16: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/16.jpg)
RDMA extension to standard FTP protocol
![Page 17: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/17.jpg)
Outline
Introduction and Backgroud Middleware Design and RFTP application Experimental Results
Testbed Setup LAN results MAN results
Conclusion
![Page 18: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/18.jpg)
Testbed Setup - LAN
10Gbps
40Gbps
40Gbps
![Page 19: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/19.jpg)
Testbed Setup - MAN
40Gbps RoCE linkRTT = 3.6ms
![Page 20: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/20.jpg)
LAN – Bandwidth and CPU Usage Comparison
![Page 21: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/21.jpg)
LAN – Bandwidth and CPU Usage Comparison
![Page 22: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/22.jpg)
MAN – RFTP evaluation
![Page 23: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/23.jpg)
Outline
Introduction and Background Middleware Design and RFTP application Experimental Results Conclusion
![Page 24: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/24.jpg)
Conclusion
Data-intensive application in cloud computing require efficient data transfer protocols to fully utilize the capacity of advanced network infrastructure
Designed and implemented a RDMA-based middleware layer
Developed a FTP application based on this middleware layer
Tested the performance of our design and implementation on both LAN and long-haul MAN links
![Page 25: Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical](https://reader036.vdocuments.site/reader036/viewer/2022070308/551c37b855034693488b4669/html5/thumbnails/25.jpg)
Thank you