ecs overview and architecture ... 8 ecs overview and architecture | h14071.16 the design of ecs is...

Download ECS Overview and Architecture ... 8 ECS Overview and Architecture | h14071.16 The design of ECS is optimized

If you can't read please download the document

Post on 30-Aug-2020

0 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • h14071.17

    Technical White Paper

    ECS Overview and Architecture

    Abstract This document provides a technical overview and design of the Dell EMC™

    ECS™ software-defined cloud-scale object storage platform.

    November 2020

  • Revisions

    2 ECS Overview and Architecture | h14071.17

    Revisions

    Date Description

    December 2015 Initial release

    May 2016 Updated for 2.2.1

    September 2016 Updated for 3.0

    August 2017 Updated for 3.1

    March 2018 Updated for 3.2

    September 2018 Updated for Gen3 Hardware

    February 2019 Updated for 3.3

    September 2019 Updated for 3.4

    February 2020 Updated ECSDOC-628 changes

    May 2020 Updated for 3.5

    November 2020 Updated for 3.6

    Acknowledgements

    This paper was produced by the following:

    Author: Zhu, Jarvis

    The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this

    publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any

    software described in this publication requires an applicable software license.

    Copyright © 2015–2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its

    subsidiaries. Other trademarks may be trademarks of their respective owners. [11/30/2020] [Technical White Paper] [h14071.17]

    mailto:jarvis_zhu@dell.com

  • Table of contents

    3 ECS Overview and Architecture | h14071.17

    Table of contents

    Revisions............................................................................................................................................................................. 2

    Acknowledgements ............................................................................................................................................................. 2

    Table of contents ................................................................................................................................................................ 3

    Executive summary ............................................................................................................................................................. 5

    1 Introduction ................................................................................................................................................................... 6

    1.1 Audience ............................................................................................................................................................. 6

    1.2 Scope .................................................................................................................................................................. 6

    2 Value of ECS ................................................................................................................................................................ 7

    3 Architecture .................................................................................................................................................................. 9

    3.1 Overview ............................................................................................................................................................. 9

    3.2 ECS portal and provisioning services ............................................................................................................... 10

    3.3 Data services .................................................................................................................................................... 12

    3.3.1 Object ............................................................................................................................................................... 12

    3.3.2 HDFS ................................................................................................................................................................ 13

    3.3.3 NFS ................................................................................................................................................................... 16

    3.3.4 Connectors and gateways ................................................................................................................................ 16

    3.4 Storage engine ................................................................................................................................................. 17

    3.4.1 Storage services ............................................................................................................................................... 17

    3.4.2 Data .................................................................................................................................................................. 17

    3.4.3 Data management ............................................................................................................................................ 19

    3.4.4 Data flow ........................................................................................................................................................... 21

    3.4.5 Write optimizations for file size ......................................................................................................................... 22

    3.4.6 Space reclamation ............................................................................................................................................ 22

    3.4.7 SSD metadata caching ..................................................................................................................................... 23

    3.4.8 Cloud DVR ........................................................................................................................................................ 24

    3.5 Fabric ................................................................................................................................................................ 24

    3.5.1 Node agent ....................................................................................................................................................... 24

    3.5.2 Lifecycle manager............................................................................................................................................. 25

    3.5.3 Registry ............................................................................................................................................................. 25

    3.5.4 Event library ...................................................................................................................................................... 25

    3.5.5 Hardware manager ........................................................................................................................................... 25

    3.6 Infrastructure ..................................................................................................................................................... 25

    3.6.1 Docker .............................................................................................................................................................. 25

    4 Appliance hardware models ....................................................................................................................................... 27

  • Table of contents

    4 ECS Overview and Architecture | h14071.17

    4.1 EX-series .......................................................................................................................................................... 27

    4.2 Appliance networking ....................................................................................................................................... 29

    4.2.1 S5148F - front-end public switches .................................................................................................................. 29

    4.2.2 S5148F - back-end private switches ................................................................................................................ 30

    4.2.3 S5248F - front-end public switches .................................................................................................................. 31

    4.2.4 S5248F - back-end private switches ................................................................................................................ 31

    4.2.5 S5232 – aggregation switch ............................................................................................................................. 32

    5 Network separation .................................................................................................................................................... 33

    6 Security ....................................................................................................................................................................... 34

    6.1 Authentication ................................................................................................................................................... 34

    6.2 Data services authentication .....................................................................................