tiering in glusterfs: hardware config considerations

TIERING IN GLUSTERFS:

Hardware Config Considerations

Veda Shankar, Red Hat

April 2016

AGENDA •  Introduction •  Basics: Concepts •  Basics: Data Placement •  Basics: Accessibility and Deployment •  Tiering Architecture •  Tiering Commands •  Tiering Volume Settings •  Tiering Performance •  Tiering Demo

Distributed Volume

Two-way Distributed Replicated Volume

Three-way Distributed Replicated Volume

Dispersed Volume (Erasure Coded)

Distribute-ReplicateVolume(Primary)

Distribute-ReplicateVolume(Primary)

Tiering Architecture

Benefits of Tiering

Tiering Commands

A8achthe:ertothevolumebyexecu:ngthefollowingcommand:# gluster volume tier test-volume attach replica 2 server1:/gluster/ssd1 server2:/gluster/ssd2 Usethesetcommandtosetthevarious:ervolumeconfigura:onop:ons:# gluster volume set test-volume <option name> <value>Thestatuscommanddisplaysthe:eringvolumeinforma:on.# gluster volume tier test-volume status !Todetachthehot:erfromthevolume,firstini:atethedetachstartcommandandthenmonitortomakesurethatthedetachprocesshascompletedsuccessfully.Finallyusethecommitop:ontocompletethedetachprocess.# gluster volume tier test-volume detach start !# gluster volume tier test-volume status !# gluster volume tier test-volume commit !

Tiering Volume Options Op#on Descrip#on Allowed

ValuesDefaultValue

cluster.read-freq-threshold Specifiesthenumberofreads,inapromo:on/demo:oncycle,thatwouldmarkafileHOTforpromo:on.AnyfilethathasreadhitslessthanthisvaluewillbeconsideredasCOLDandwillbedemoted.

0-20 0

cluster.write-freq-threshold

Specifiesthenumberofwrites,inapromo:on/demo:oncycle,thatwouldmarkafileHOTforpromo:on.AnyfilethathaswritehitslessthanthisvaluewillbeconsideredasCOLDandwillbedemoted.

0-20 0

cluster.:er-promote-frequency Specifieshowfrequentlythe:erdaemonmustcheckforfilestopromote. 1–172800seconds

120seconds

cluster.:er-demote-frequency

Specifieshowfrequentlythe:erdaemonmustcheckforfilestodemote.

1–172800seconds

3600seconds

cluster.:er-mode Ifsettocachemode,promotesordemotesfilesbasedonwhetherthecacheisfullornot,asspecifiedwithwatermarks.Ifsettotestmode,periodicallydemotesorpromotesfilesautoma:callybasedonaccess.

Test|cache cache

TieringVolumeOptions Op#on Descrip#on Allowed

ValuesDefaultValue

cluster.:er-max-mb SpecifiesthemaximumnumberofMBthatmaybemigratedinanydirec:onfromeachnodeinagivencycle.

1–100000(100GB)

4000MB

cluster.:er-max-files

Specifiesthemaximumnumberoffilesthatmaybemigratedinanydirec:onfromeachnodeinagivencycle.

1–100000files

10000

cluster.watermark-hi Upperpercentagewatermarkforpromo:on.Ifhot:erfillsabovethispercentage,nopromo:onwillhappenanddemo:onwillhappenwithhighprobability.

1–99% 90%

cluster.watermark-low

Lowerpercentagewatermark.Ifhot:erislessfullthanthis,promo:onwillhappenanddemo:onwillnothappen.Ifgreaterthanthis,promo:on/demo:onwillhappenataprobabilityrela:vetohowfullthehot:eris.

1–99%

75%

QCT Storage Servers

Test Configuration

TwoQuantaPlexT21P-4U/DualNodeservers(fourGlusterfsnodes)wereconfigured,eachnodewitha40GbEinterface,withthetestworkloaddrivenby16clientnodes.Clientshavea10GbEinterface.

Test Configuration

SixQuantaGridD51PH-1Userverweretestedwithclusterandpublic10GbEnetworks.Thetestworkloadwasdrivenby16clientnodes.

DEMO

THANK YOU!

bit.ly/glustertestdrive

Test drive: