Directory Delegations in NFSv4

CITIUniversity of MichiganAnn Arbor

Brian Wickman

page2

NFS version 4

NFSv4 introduces state! OPEN and CLOSE integrated locking file delegations

Delegation Paradigm Server-driven optimizations visible to the

client but transparent to applications. More client responsibility; but better

performance!

page3

NFSv4 File Delegations

Server-driven optimization: Server has final authority over delegating ability to clients

Transparent to applications: No special application-level tooling required, otherwise delegation utility would be undermined.

Better performance: Local locking! Smart caching! Less latency! Less traffic!

page4

NFS Directory Delegations

Want to delegate some portion of directory state to client

What to delegate? Potential state

Filename to filehandle bindings (DNLC) File attributes Anything else?

page5

How to Enable Directory Delegations? Introduce pair of ops, OPENDIR/CLOSEDIR?

Better mirroring of file delegation semantics Same interoperability concerns as delegating v4.0

servers speaking with v2/v3 clients. Introduce single-op?

GIVE_ME_DELEGATION_PRETTY_PLEASE? Half as many ops to implement. Works, too.

How information-rich? Notifications? Asynchronous, synchronous, whaaaa?

page6

draft-ietf-nfsv4-directory-delegation-01.txt

Saadia’s proposal New op (GET_DIR_DELEGATION) New callback (CB_NOTIFY)

Delegations and notifications, all in one Read-only, as specified Information rich, fine-grained Lots of new functionality to code,

especially for server implementer

page7

draft-wickman-nfsv4-directory-delegations-00.txt

My proposal New op (REQUEST_DELEGATION) No new callbacks

No notifications, existing data structures unchanged

Read and write delegations specified Minimal information, coarse-grained Implementation more straightforward?

page8

Write directory delegations

Server doesn’t need to change its behavior, relies more on client implementer

If the client is clever it can Issue directory updates asynchronously Perform whole-tree write delegation on new

directories Effectively own file write delegations on all

files within a write delegated directory

page9

Write delegations and STATEIDs

Overlooked in my draft, need errata Directory ops such as READDIR or MKDIR

do not reference a delegation stateid Need PUT_STATEID function and “current”

STATEID so that we can modify directories on which we hold delegations

Discussed at Connectathon…?

page10

Evaluating the proposals

For starters, how do you define better? My take: how many ops can you service

locally and at what extra cost, given common workloads?

So far, I’ve investigated this for read-only directory delegations and for two common(?) workloads.

page11

Steps to Evaluating Delegation Behavior

Passively capture traces from NFSv3 deployments with thousands of users (v4 deployments, anyone?)

Build an NFS (v3) metadata simulator Instrument extra information into traces (e.g.,

OPEN, CLOSE, REQUEST_DELEGATION) Reconstruct filesystem image on the fly Observe delegation behavior! Measure what you see

page12

Traces Used

From Dan Ellard’s PhD research CAMPUS trace

Read dominated e-mail workload Good test of notification behavior? (taking pulse of mail

spool) Digital UNIX server, one week, late October 2001, 242M

v3 ops EECS trace

Write-dominated research workload, lots of small files NetApp filer, one week, late October 2001, 41M v3 ops

page13

Simple use of Directory Delegations

For fun, I emulated the v3 cache algorithm using the REQUEST_DELEGATION op When first caching directory entries,

REQUEST_DELEGATION If clock expires and no revocation, do nothing If revocation occurs, issue all LOOKUPs to

server, synchronously until clock expiration Reacquire delegation at clock expiration

page14

Simple use of Directory Delegations

Use liberal server policy for granting directory delegations (i.e., always-grant)

In both traces, >90%** of LOOKUPs are serviced locally and guaranteed consistent

In both traces, >95%** of directory cache validation traffic is eliminated

** - with unbounded lookup caches

page15

What about with notifications?

I also implemented the same algorithm but with GET_DIR_DELEGATION

We can service a bunch of extra LOOKUPs (practically all) because the server notifies us when our lookup cache changes and what is entailed in each change, rather than revoke

How much can we service and at what cost?

page16

Not much

CAMPUS EECS

Extra LOOKUPs serviced locally

162K 14K

Percentage of total LOOKUPs

2.1% 0.4%

Notifications received 9M 164K

Ratio of Notifications to extra LOOKUPs

56x 12x

page17

FILE_ATTR notifications

If FILE_ATTR notifications are added, the ratio goes from 56x to 500x for CAMPUS

Millions of DIR_ATTR notifications to service tens of thousands of LOOKUPs

Tens of millions of FILE_ATTR notifications (67M) to service millions of GETATTRs (2.3M)

page18

Reducing Notification Traffic

Can we win if using asynchronous notifications? If nothing else, we can reduce traffic. Because the client can specify a notification

window, the server has the opportunity to eliminate self-nullifying updates (e.g., ADD fh; REMOVE fh) within that window.

Asynchronous notifications would still mean less LOOKUP consistency than no notifications at all, as long as revocations are synchronous

page19

Reducing Notification TrafficRatio of Generated to Delivered Notifications*

05

1015202530354045

0.1

0.31 3 10 30 10030010003000

Notification Window (s)

Rati

o

CAMPUS EECS

page20

Another Drawback of Notifications

How many notifications refer to directory objects whose data or metadata we are actively accessing?

CAMPUS: 4% EECS: <1%

Are we taking sips from a fire-hose? Also, knowing the existence of a change may be

sufficient more often than knowing the change itself.

page21

Summary

A simple directory delegation scheme without notifications gives immediate improvements in latency and caching, in the read-only case

Notifications have the tendency to create lots of information that is otherwise unused by the client

Still much more work to be done…

page22

TODO

Start looking at v4 traces. Of course, I need v4 traces, especially Windows v4 traces Anyone want to help?

Simulating write directory delegations Looking at other lookup cache maintenance

algorithms (instead of just clock-based) Adding more variables to the simulator (cache

eviction, for example) Implementing directory delegations for real

page23

Questions?