Dovecot IMAP Server

http://www.dovecot.org/

Date: July, 2009

RackspaceEmail

•  DovecotisusedtoserveIMAPforoveramillionpaidmailboxes(MSExchangealsoavailable)

•  Usersassignedtospecificbackendservers– WithproprietaryreplicaEonsoFware

•  DovecotIMAP/POP3proxiesinfront– AlsoforExchangeIMAP/POP3users

•  AmazonS3for(encrypted)backups•  Moreaboutclusteringlater..

TheTalk

•  Dovecotfeatures•  IMAP&Dovecotperformance

•  dboxmailboxformat

•  Clustering

Dovecot

Pictures from Wikipedia, by Cyril Thomas and Carcharoth

History

•  DovecotdesignwasstartedaroundJune2002•  FirstreleasewasJuly2002•  Late2003aredesignstarted•  v1.0.0releasedApril13th2007•  v1.1.0releasedJune21st2008•  v1.2.0releasedJuly1st2009•  v2.0betashopefullythisyear

Features

•  OFenhasbeZerperformancethancompeEEon.– OpEmizedforminimizingdiskI/O(index/cachefiles)– HosEngmyownmailson10yearsoldSparchelps

•  Highlyconfigurablefordifferentenvironments–  StandardmboxandMaildirwithtransparentindexing(externalmailboxmodificaEonsareok)

–  dbox:Dovecot’shigh‐performancemailboxformat– Manydifferentwaysofclustering

–  ExtremelyflexibleauthenEcaEon•  Pos`ixandEximsupportDovecotforSMTPAUTH

Features•  Admin‐friendly/self‐healing

– Allerrorsarelogged– Understandableerrormessages

– Improvedconstantly(toreducemyemailload)

– Detected(index)corrupEongetsfixedautomaEcally

•  file_dotlock_create(/home/Emo/Maildir/dovecot‐uidlist)failed:Permissiondenied(euid=1000(Emo)egid=1000(Emo)missing+xperm:

/home/Emo)

•  chown(/home/Emo/Maildir/.box,‐1,0(root))failed:OperaEonnotpermiZed(egid=1000(Emo),groupbasedon/home/Emo/Maildir)

v1.2NewFeatures

•  Virtualmailboxes(searchviews)– ”Allunreademailsinallmailboxes”

– Allmessagesinallmailboxes(exceptTrash)•  VirtualPOP3INBOX•  Forsearchingmessagesfromallmailboxes•  gmail‐likeconversaEonviews

•  Userscansharemailboxestoeachothers–  IMAPACLcommands

•  ModificaEonsequences(CONDSTORE)– CustomcodewanEngquicksync?(e.g.backups)

AuthenEcaEon

•  PasswordanduserdatabaseseparaEon– Passdbforverifyinguser’spassword– Userdbforlookinguphowtoaccessmailbox

•  Supportforalmosteverything:SQL,LDAP,PAM,checkpasswordscripts,etc.– Everythingisconfigurable(e.g.fullSQLqueries)– SupportsmulEpledbs(e.g.system+virtualusers)

•  Authmechanisms:PLAIN,CRAM‐MD5,DIGEST‐MD5,Kerberos,OTP,etc.

•  Passwordschemes:Plaintext,CRYPT,MD5,SHA1,SHA256,SSHA,SSHA256,etc.

AuthenEcaEonCache

•  Passdbanduserdblookupscanbecached•  PasswordchangesareautomaEcallydetected:Ifauthisunsuccessful,andpreviousauthwasa)  successful:douncachedpassdblookupb)  usuccessful:faillogin

•  NegaEvecachingcanbedisabled– Userdoesn’texistcaching– Passwordfailures(v1.2+)

•  Avoidsaneedforimapproxywithwebmails?

IMAPProtocol

•  Baseprotocoliscomplex–difficulttoimplementitcorrectly(bothclient&server)

•  Flexible–manydifferentwaystoimplementaclient(online&offlineclients)

•  Extensible–therearealotofextensions– Clientsrarelysupportmorethansomebasicextensions,suchasIDLE.

– Thunderbirdv3addssupportforseveralnewextensions,suchasCONDSTORE.

ImapTestIMAPServerTester

•  WriZenoriginallyforDovecotstresstesEng–  Foundalotofcrashes,hangsandmailboxcorrupEononotherIMAPserversaswell

•  TestsIMAPservercompliancewithscriptedtestsanddynamicrandomstresstesEng.

•  DovecotiscurrentlytheonlyIMAPserverthatfullypassesallofImapTesttests.•  PandaIMAPispracEcallytheretoo

•  Mostotherserversfailinmanydifferentways.•  hZp://imapwiki.org/ImapTest

OfflineIMAPClients

•  Typicallydownloadnewlyseenmessages’bodiesonceandcachethemlocally

•  OFencanbeconfiguredtodownloadimmediatelyvs.downloadwhenreading

•  Someuseserversidesearches(Thunderbird)andsomedon’t(Outlook–ifsomemessageshaven’tbeendownloaded,thosearen’tsearched)

•  Usuallyalsofetchmessages’metadataonce(headers,receiveddate)

•  Server‐sidecachingmayhelp,butnotthatmuch–  It’sextradiskI/O‐>morelikelyjusthurts

OnlineIMAPClients

•  WebmailsoFenkeepaskingforthesameinformaEonoverandoverandoveragain

•  Pineandsomewebmailscachewhatthey’vealreadyseen,butnotpermanently

•  MuZ(withoutlocalcache)andsomeothersfetchallmessages’metadataeveryEmewhenopeningamailbox

•  Cachingisveryuseful,butdifferentclientswantdifferentmetadata

IMAPServerPerformance

•  Difficulttobenchmark•  Dependsalotonclients:Whetherclientsusealocalcachemakesahugedifference.– Onlinevs.offlineclients

•  Whatdatatoindex/cache?•  SPECmail2009addssupportforIMAP

– EmulatesdifferentIMAPclients.Clientamountsareconfigurable.

– TheonlybenchmarkgivingrealisEcresults.

DovecotCacheFile

•  dovecot.index.cachefiles•  ThemainreasonforDovecot’sgoodperformance•  Dynamic:cachesonlywhatclientswant.

–  Specificmessageheaders(From:,Subject:,etc),– MIMEstructureinformaEon,–  Sent/receiveddate,etc.

•  Cachingdecisionsforeachfield:“no”,“temporary”,“permanent”

•  UnusedfieldsdroppedaFeramonth.•  Cacheddataneverchanges(IMAPguarantees)•  Cachefilegets“compressed”onceinawhile•  OFenabout10‐20%ofmailboxsize

DovecotIndexFiles

•  dovecot.indexcontainsmessages’metadata–  IMAPUniqueIDnumber(UID)idenEfiesmessages

– Flags(\Seen,\Answered,keywords,etc.)– Extensiondata:mboxfileoffsets,cachefileoffsets,modseqnumber(v1.2CONDSTORE),etc.

•  Lazilycreated/updatedsincev1.1– dovecot.index.loghasallthelatestchanges.dovecot.indexisupdatedaFer8kBofnewdatahasbeenwriZentothe.log

DovecotIndexFiles

•  dovecot.index.logisamailboxtransacEonlog– Somewhatsimilartodatabases’transacEonlogsorfilesystemjournals.

– Containsallchangestobedonetodovecot.index.•  dovecot.indexisreadtomemoryonceandthenupdatedfromdovecot.index.log– VeryefficientwithNFS/clusteredfilesystems!– Veryefficienttofindoutwhatchangesanothersessionhaddone!

Plugins

•  DovecotpluginscanhookintoalmostanythingandmodifyDovecot’sbehavior.SomeexisEngfeaturesimplementedasplugins:– AccessControlLists– Quota–  Fulltextsearchindexes–  Readingcompressedmbox/maildirfiles

•  CanaddnewIMAPcommands•  Implementnewmailstoragebackends(virtual,SQL,IMAPproxying)

MailboxFormats

•  mbox– Onemailbox=onefile

•  Slowtodeleteoldmessages

•  Maildir– Onefile=onemessage

•  Fasttodeletemessages•  Slow(er)toreadthroughallmessages•  Filereadorderaffectsperformance,even2xormore!

•  Single‐dboxandmulE‐dbox– Dovecot’sextensibleandhigh‐peformancemailboxformats

Single‐dbox•  AvailableinDovecotv1.1andlater•  MainadvantageoverMaildir:filenamesdon’tchange.

•  Directorylayoutlookslike:– mailboxes/INBOX/dbox‐Mails/

• dbox.index–dboxindex(removedinv2.0)• dovecot.index*‐Dovecot’sindexfiles• u.123‐MessagedataforIMAPUID123• u.125‐MessagedataforIMAPUID125

– mailboxes/Trash/dbox‐Mails/– mailboxes/Trash/temp/dbox‐Mails/

Single‐dbox

•  PrimarymetadatastorageisDovecot’sindexfiles•  MetadatabackupswriZenaboutonceadaytodboxfiles‐>losingindexeswon’tloseallflags

•  AutomaEcallyfixes/rebuildsbroken/lostindexes

•  Future:Dovecotv2.0nolongerwritesflagstodboxfiles.Itcreatesseparateindexfilebackupsinstead.

dboxFileFormat•  Fileheader

–  Messageheadersize

–  FilecreaEondata•  Messageheader(extensible)

–  Messagesize

•  Messagebody•  Messagemetadata(extensible)

–  Message’sgloballyUniqueID(GUID)

–  Receiveandsavedate/Eme–  Message’s”virtualsize”

–  etc.•  [mulE‐dbox:Nextmessage…]

Single‐dbox:MaildirMigraEon

•  SuperfastmigraEonfromMaildir:– RenamesMaildir/cur/todbox‐Mails/

– MovesotherusefulMaildirfilestoo

•  NewmailswillbesavedusingnaEvedboxformat

•  Oldmailsgetconvertedtodboxlaterwhenuserchangesoldmails’flags.– MailsmightstayasMaildirforalongEme

Single‐dbox:AlternaEveStorage

•  Usersrarelyaccesstheiroldmails•  Lowerperformancestorageischeaper‐>Moveoldmailstolowperformancestorage

•  dboxsupports”alternaEvepath”se~ng:Ifadboxfileisn’tfoundfromprimarypath,it’slookedupfromalternaEvepath.– mail_locaEon=dbox:~/dbox:ALT=/slow/%u/dbox

•  Future:Supportforcloudstorage(likeCloudFiles/S3)?

MulE‐dbox

•  AvailableinupcomingDovecotv2.0•  MulEplemessagesinasinglefile•  MulEplefilesinasinglemailbox

–  Filesareabout2MB(configurable)–  Canberotatedeveryndays(forincrementalbackups)–  Largerfiles‐>lessfragmentaEon,butdeleEonslower– DelayedionicednightlydeleEons

•  Triesveryhardtopreserveasmuchdataaspossibleincaseof(filesystem)corrupEon.

•  Crashorpowerlosscan’tcorruptorlosedata

dboxFuture

•  SingleinstanceaZachmentstorage•  Abstractoutfilesystemaccessandimplement

– RegularPOSIXI/O– AsyncI/O– CloudstorageI/O

•  MakeDovecotdomoreparallelprocessingtogetgoodperformancefor(highlatency)cloudstorageandtogetfulladvantageofasyncI/O.

DovecotClustering

•  Twodifferentwaystodoit:•  Globallysharedfilesystem

– ManyIMAPservers,eachabletohandleanyuser

– NFS,clusterfilesystems

•  Sharding– Eachuser’sdatamirroredin2‐3servers

–  IMAPproxyforwardsuserstocorrectserver(s)– DRBD,proprietaryclusteringsoFware/hardware

Clustering:TwoTypesofData

•  Messagedata– ExisEngmessages(files)don’tchange– Userstypicallyreadmessagesonce‐>messageisreadfromdiskonlyonce(orfewEmes)

– Latencyhurts,butnotbadly(infutureevenless)•  Indexdata

– Constantlookups:”Hasmailboxchanged?”– Latencyisverybadforperformance– ExisEngfileschangeconstantly‐>cachingtrouble!

•  Differentstoragesformessages/index?

Clustering:NFS

•  NFSserverisoFensinglepointoffailure– Performanceproblemsaffecteveryone.Mightbedifficulttodiagnose/fix.

– Example:NFSlockingbroke‐>restarted‐>Dovecotbecameunusablyslow

•  Cachingproblems,especiallywithindexfiles– mail_nfs_*se~ngstrytosolvethese

•  Indexfilesonlocaldiskhelpsperformance

•  hZp://wiki.dovecot.org/NFS

Clustering:NFS

•  SEckyserversforusers=onlyoneservermodifiesauser’smailbox–  IMAPproxylooksupdesEnaEonserverfromdb– Avoidscachingproblems–  Ifmaildeliveryupdatesindexes,mustbedonebythesameserverasIMAP.•  EachserverreceivesmailswithSMTP/LMTP

– Storingindexesonlocaldiskshelpsperformance•  Ifservergoesdown,reindexingmaybeslow•  DRBDhybrid?

Clustering:ClusterFS

•  DovecotknowntoworkwithGFS,OCFS2,..•  LesscachingproblemsthanwithNFS

– PerformancesEllbeZerwhenuseraccessesonlysingleserver(beZercaching,lesslockwaits)

•  Performance?– Manysmallfilesarebad?

Clustering:Sharding

•  TypicallyinacEve/passiveserverpairs:•  DedicatedacEveandpassiveservers

– Wastesservers

•  Crossedpairs– EachserverisacEveforonesetofusersandpassiveforanothersetofusers

– Serverfailuredoublesthepassive’sload•  DovecotIMAP/POP3proxyclusterinfront

Clustering:Sharding

•  Distributeindividualusers(notenEredomains)todifferentservers– Reducesloadspikes

•  UsestaEsEcstoautomaEcallydistributeheavyuserstodifferentservers– v1.2canexportverydetailedstaEsEcsviaplugin– v2.0’supcomingdsyncuElity

Clustering:DRBD

•  FilesystemcorrupEongetsreplicated•  SynchronousreplicaEon

– Nomaillossonfailures

– Tooslowforcross‐datacenter(?)•  AsynchronousreplicaEon

– Somedatalossonfailure

•  3servers:SyncreplicaEonforin‐datacenterandasyncforcross‐datacenterbackup?

ClusteringFuture:TheCloud

•  Savemessagedataincheapcloudstorage– TypicallysimpleAPIstoaccessfiles

•  dboxdesignedforthis– Typicallyhigherlatency

•  DovecotneedstodomoreworkwhilewaiEng

•  Indexdatakeptprimarilyinmemory– Mustbeverylowlatency‐>directcommunicaEonbetweenserversthataccessthesamemailbox

– Permanent(backup)storagemaysEllbeincloud

•  Result:mulE‐masterreplicaEon

Dovecotv2.0

•  Somenewfeaturesalreadyimplemented:– Redesignedmasterprocess

•  Easytoaddexternalservices,e.g.ManageSieve

– RedesignedconfiguraEon•  Local/remoteIP/mask‐specificconfiguraEon

–  SSLcerts•  Allowchangingconfigdatasource(e.g.SQL?)

– LMTPserver– dsync:Realiablyandefficientlysynctwomailboxes(e.g.viaSSH)

Dovecotv2.0

•  Featuresnotyetimplemented,buthopefullywillbebytheendofthisyear:–  Indexfileimprovements

•  Nolocking(withatomicappends)

•  SmallchecksumsallaroundfordetecEngcorrupEon•  Ingeneralmakethecodesimplerandmorerobust

– MulE‐masterreplicaEon•  dboxcloudstorage(forsomeexisEngcloudAPI(s)?)

•  Indexsharing/replicaEonbetweenservers

QuesEons?