extensible file allocation table (exfat) ahtcc v1.1

Extensible File Allocation Table (exFAT)

Zoran Iliev 2010

History

• 2006, on Windows Hardware Engineering Conference (WinHEC), Microsoft presented “Personal Storage: Opportunities and challenges for pocket-sized storage devices in the Windows world”.

• Same Conference Vishal Ghotge - Program Manager, Core File Services, Microsoft Corporation presents exFAT

The design goals of exFat

• Be as similar to FAT32 as possible

• Overcome FAT32’s limitations

• Design for the future

• Make it extensible

• Scale to large volume sizes

• Manage large files

• Designed to optimize for flash media

• exFAT – A simple, extensible, rich filesystem for next generation devices and flash media.

Some Key Features Of exFAT

• Free space bitmap Identifies free and available clusters on the volume allowing Faster look-up for free space allocation, Faster deletes

• Support for large files File size field is 8 bytes, Theoretical maximum file size of 264 bytes

• Large cluster size The on-disk structure limit for cluster size is 2255,

Implementation limit is 32 MB Source: download.microsoft.com/download/5/b/9/5b97017b.../sto072_wh06.ppt

Some Key Features Of exFAT • Extensibility for the future

- Directory structure is flexible and extensible - Support for more than 216 (65,536) files in a single directory - Timestamps can be in UTC rather than only local time (starting with Vista SP2). - Easy to add new features and can add some features without breaking compatibility

• Name hashes - 2 byte checksum of up-cased file name, allows for faster checks of filenames

• OEM specific parameters and types - Feature to store OEM specific volume information

• Support for access control lists (not supported yet )

• Support for Transaction-Safe FAT File System (TFAT) (optionally WinCE activated function)

exFAT vs FAT 32

• exFAT is not backward compatible with FAT32

• There are no “short names” in exFAT

Disk Space Overhead

Default cluster sizes for exFAT

VOLUME Structure

BOOT Regions – Main and Backup

Boot Sector – On Disk Hex View


JumpBoot (0, 3)

Contains the jump instruction for CPUs common in personal computers, which, when executed, "jumps" the CPU to execute the boot-strapping instructions in the BootCode field


File System Name (3, 8)

The FileSystemName field contains the name of the file system on the volume, "EXFAT“ in ASCII characters, which includes three trailing white spaces.


Must Be Zero (11, 53)

The MustBeZero field directly corresponds with the range of bytes the packed BIOS parameter block consumes on FAT12/16/32 volumes. The valid value for this field is 0, which helps to prevent FAT12/16/32 implementations from mistakenly mounting an exFAT volume.


Partition Offset (64, 8) 02 00 = 8192

The PartitionOffset field describes the media-relative sector offset of the partition which hosts the given exFAT volume. This field aids boot-strapping from the volume using extended INT 13h** on personal computers. **Extended Int13h functions address sectors using Logical Block Addressing


Volume Length (72, 8) F4 40 00= 16007168 * 512= 8195670016 ≈ 7.6GB

The VolumeLength field describes the size of the given exFAT volume in sectors.


FAT Offset (80, 4) 08 00 = 2048

The FatOffset field describes the volume-relative sector offset of the First FAT.


FAT Length (84, 4) 07 C0 = 1984

The FatLength field describes the length, in sectors, of each FAT table (the volume may contain up to two FATs, not implemented yet).


Cluster Heap Offset (88, 4) 10 00 = 4096

The ClusterHeapOffset field describes the volume-relative sector offset of the Cluster Heap. This field enables implementations to align the Cluster Heap to the characteristics of the underlying storage media and it is the Physical Sector Location of the Bitmap.


CLUSTER COUNT (92, 4) 03 D0 C0 = 250048

The ClusterCount field describes the number of clusters the Cluster Heap contains. The value of the ClusterCount field determines the minimum size of a FAT.


First Cluster of Root Directory (96, 4) 04 = 4

The FirstClusterOfRootDirectory field contains the cluster index of the first cluster of the root directory. The FirstClusterOfRootDirectory field contains the cluster index of the first cluster of the root directory.


Volume Serial Number (100, 4)

The Volume Serial Number field contains a unique serial number.


File System Revision (100, 4) 1.0

The high-order byte is the major revision number and the low-order byte is the minor revision number. For example, if the high-order byte contains the value 01h and if the low-order byte contains the value 05h, then the FileSystemRevision field describes the revision number 1.05.


Volume FLAGS (106, 2)

The VolumeFlags field contains flags which indicate the status of various file system structures on the exFAT volume.


Bytes Per Sector Shift (108, 1) 29 = 512 Bytes per Sector

The BytesPerSectorShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.


Sectors Per Cluster Shift (109, 1) 26 = 64 Sectors per Cluster

The Sectors PerClusterShift field describes the bytes per sector expressed as binary logarithm log2(N), where N is the number of bytes per sector.


Number of Fats (110, 1)

The NumberOfFats field describes the number of FATs and Allocation Bitmaps the volume contains. Can be 1 or 2.


Drive Select (111, 1)

The DriveSelect field contains the extended INT 13h drive number, which aids boot-strapping from this volume using extended INT 13h on personal computers.


Percent In Use (112, 1) 0% - no files written

The PercentInUse field describes the percentage of clusters in the Cluster Heap which are allocated. FFh-indicates the percentage of allocated clusters in the Cluster Heap is not available; 00 – indicates backup of the Main Boot region.


Percent In Use (112, 1) 9% in use


Reserved (113, 7)


Boot Code (120, 390)

The BootCode field contains boot-strapping instructions.


Boot Signature (510, 2)

Boot Sector – Cont’d

• First 9 sectors (0 to 8) are marked with the x55 AA signature

• Sector 11 contains Boot Checksum, 4 bytes value is repeated from the beginning to the end of the sector. This value, generated during format and is copied in the backup VBR, is a checksum of the other areas of the boot region, not including the volume flags or the percent-in-use fields.

• Sectors 12-23 containing a complete backup of the first 12 sectors of the volume

Sector 10

Sector 11 Boot Checksum

Sector 12

FAT Region

• A FAT describes cluster chains in the Cluster Heap. A cluster chain is a series of clusters which provides space for recording the contents of files, directories, and other file system structures.

• A FAT represents a cluster chain as a singly-linked list of clusters indices.

• With the exception of the first two entries, every entry in a FAT represents exactly one cluster and each entry is 4 bytes in size.

FAT Region

Data Region

• The Data region contains the Cluster Heap, which provides managed space for file system structures, directories, and files.

• The Cluster Heap's structure is very simple; each consecutive series of sectors describes one cluster, as the SectorsPerClusterShift field defines.

• Importantly, the first cluster of the Cluster Heap has index two, which directly corresponds to the index of Fat Entry 2.

Allocation Bitmap

• In an exFAT volume, an Allocation Bitmap maintains the record of the allocation state of all clusters.

• This is a significant difference from exFAT's predecessors (FAT12, FAT16, and FAT32), in which a FAT maintained a record of the allocation state of all clusters in the Cluster Heap.

Allocation Bitmap

• An Allocation Bitmap records the allocation state of the clusters in the Cluster Heap.

• Each bit in an Allocation Bitmap indicates whether its corresponding cluster is available for allocation or not.

• An Allocation Bitmap represents clusters from lowest to highest index.

• For historical reasons, the first cluster has index 2. • Note: the first bit in the bitmap is the lowest-order

bit of the first byte.

Directory Structure

• The exFAT file system uses a directory tree approach to manage the file system structures and files which exist in the Cluster Heap.

• Directories have a one-to-many relationship between parent and child in the directory tree.

• The directory to which the “FirstClusterOfRootDirectory” (96, 4) field refers is the root of the directory tree.

• All other directories descend from the root directory in a singly-linked fashion. Each directory consists of a series of directory entries.

Example: Calculating the first sector of the Root entry

Directory Structure

• One or more directory entries combine into a directory entry set which describes something of interest, such as a file system structure, sub-directory, or file.

Volume Label Directory Entry

• The Volume Label is a Unicode string which enables end users to distinguish their storage volumes.

• In the exFAT file system, the Volume Label exists as a critical primary directory entry in the root directory .


Entry Type (0, 1)


• The CharacterCount field contains the length of the Unicode string the VolumeLabel field contains.

Character Count (1, 1)


• The VolumeLabel field contains a Unicode string, which is the user-friendly name of the volume.

• Maximum 11 characters Volume Label Name.

Volume Label (2, 22)


Reserved (24, 8)


• If the volume was not given a label, the first value for the first entry will be: 0x03 instead of 0x83.

• Incidentally, renaming of the Volume name changes the inUse flag to zero to the previous Volume Label Directory Entry and creates new Volume Label Directory Entry. Nothing else is changed within the previous Volume Label Directory Entry.

Allocation Bitmap Directory Entry

• In the exFAT file system, a FAT does not describe allocation state of clusters; rather, an Allocation Bitmap does.

• Allocation Bitmap exist in the Cluster Heap and have corresponding critical primary directory entries in the root directory.

• The “NumberOfFats” field determines the number of valid Allocation Bitmap directory entries in the root directory.


• If the “NumberOfFats” field contains the value 1, then the only valid number of Allocation Bitmap directory entries is 1.

• If the NumberOfFats field contains the value 2, then the only valid number of Allocation Bitmap directory entries is 2.


Entry Type (0, 1)


Bitmap Flags (1, 1)

0, which means the given directory entry describes the First Allocation Bitmap 1, which means the given directory entry describes the Second Allocation Bitmap and is possible only when “NumberOfFats” contains the value 2


Reserved (2, 18)


First Cluster (20, 4)

This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Allocation Bitmap.


Data Length (24, 4) 7A 18 = 31256 = 30.5 Kb

Up-Case Table Directory Entry

• The Up-case Table defines the conversion from lower-case to upper-case characters.

• This is important due to the File Name directory entry using Unicode characters and the exFAT file system being case insensitive and case preserving.

• The Up-case Table exists in the Cluster Heap and has a corresponding critical primary directory entry in the root directory.


• Entry Type 82

Entry Type (0, 1)


Reserved (1, 3)


• The “TableChecksum” field contains the checksum of the Up-case Table (which the “FirstCluster” and “DataLength” fields describe).

• Implementations shall verify the contents of this field are valid prior to using the Up-case Table.

Table Checksum (4,4)


Reserved (8, 12)


• This field contains the index of the first cluster of the cluster chain, as the FAT describes, which hosts the Up-case Table.



Data Length (20, 4) 16 CC = 5836 = 5.699 Kb

Volume Guid Directory Entry

• The Volume GUID directory entry contains a GUID which enables implementations to uniquely and programmatically distinguish volumes.

• The Volume GUID exists as a benign primary directory entry in the root directory.

File Directory Entry

• File directory entries describe files and directories. They are critical primary directory entries and any directory may contain zero or more File directory entries.

• For a File directory entry to be valid, at most one Stream Extension Directory entry and at least one File Name Directory entry immediately follow the File directory entry.


• Entry Type 85

Entry Type (0, 1)


• The “SecondaryCount” field describes the number of secondary directory entries which immediately follow the given primary directory entry.

• These secondary directory entries, along with the given primary directory entry, comprise the directory entry set.

Secondary Count (1, 1)


• The “SetChecksum” field contains the checksum of all directory entries in the given directory entry set.

• The checksum excludes this field

Set Checksum (2, 2)


File Attributes (4, 2) 20h = 32 (25)-> Archive


Reserved 1 (6, 2)


Create Timestamp 1 (8, 4)

• In combination, the “CreateTimestamp” and “CreateTime10msIncrement” fields describe the date and time the given file/directory was created.

• These two fields conform to the definitions of the Timestamp and 10msIncrement fields.

Create 10ms Increment1 (20, 1)


Timestamp Field Structure :

• 0 – 4 Seconds divided by 2 (0 = 0 sec., 29 = 58 sec.)

• 5 – 10 Minutes 0 – 59

• 11 – 15 Hours 0 – 23

• 16 – 20 Days 1 – 31

• 21 – 24 Months (01 = January 02 = February etc. )

• 25 – 31 Years since 1980


10ms Increment Fields Structure :

• The valid range of values for these fields is:

– At least 0, which represents 0 milliseconds

– At most 199, which represents 1990 milliseconds

File Directory Entry Create Timestamp 1 (8, 4)

25/02/2010 16:39:54 Create 10ms Increment1 (20, 1)

50h = 80 (* 10) = 800ms

File Directory Entry Last Modified Timestamp 1 (12, 4)

25/02/2010 16:20:52 Last Modified 10ms Increment1 (21, 1)

00 ??


Time Zone Created (22, 1)

Time Zone Accessed (24, 1)

Time Zone Modified (23, 1)

Time Zone Calculation

• There is one byte for each Time Stamp (Created, Modified and Accessed).

• The bit assignment is: - 1 bit reserved (the most significant bit) - the other 7 bits are representing a signed integer that is recording the number of 15 minutes intervals.

Time Zone Calculation

A8h = 10101000 -> 101100 = 40 X 15 = 600 min

=> Time Zone +10 (Canberra, Sydney)


Reserved 2 (25, 7)

Stream Extension Directory Entry

• The Stream Extension directory entry is a critical secondary directory entry in File directory entry sets.

• This directory entry is valid only if it immediately follows the File directory entry.


• Entry Type C0

Entry Type (0, 1)


Secondary Count (1, 1)


Reserved 1 (2, 1)


• The NameLength field contains the length of the Unicode string the subsequent File Name directory entries collectively contain.

Name Length 1 (3, 1)


The “NameHash” field contains a 2-byte hash of the up-cased file name. This enables implementations to perform a quick comparison when searching for a file by name. Importantly, it provides a sure verification of a mismatch. Implementations shall verify all “NameHash” matches with a comparison of the up-cased file name.

Name Hash 1 (4, 2)


Reserved 2 (6, 2)


• The “ValidDataLength” field describes how far into the data stream user data has been written.

• Implementations shall update this field as they write data further out into the data stream.

Valid Data Length (8, 8)


Reserved3 (16, 4)


Data Length (28, 4)

File Name Directory Entry

• File Name directory entries are critical secondary directory entries in File directory entry sets.

• The valid number of File Name directory entries in a File directory entry set is “NameLength”/15, rounded up to the nearest integer.

• File Name directory entries are valid only if they immediately follow the Stream Extension directory entry as a consecutive series.

• File Name directory entries combine to form the file name for the File directory entry set.


• Entry Type C1

Entry Type (0, 1)


General Secondary Flags Field (1, 1)


File Name(2, 30)


• The file names "." and ".." have the special meaning of "this directory" and "containing directory", respectively. Implementations shall not record either file name in the FileName field.

• However, implementations may generate these two file names in directory listings to refer to the directory being listed and the containing directory.

File Creation

By Examining the FAT entries we can find the allocated clusters.

We know that the First cluster is 78h, so we go to the beginning of the FAT, at sector 128, which is defined in the boot sector:

• So, if we do the math, we have:

• First Fragment: 4999-119 = 4880

• Second Fragment: 7135-5607 = 1528

• FILE SIZE: = 6408 CLUSTERS

Deleted Files

Directory Entry after file deletion: • As we already mentioned, The “InUSE” bit describes

whether the given directory entry in use or not. The valid values for this field are:

• 0, which means the given directory entry is not in use; this means the given structure actually is an unused directory entry

• 1, which means the given directory entry is in use; this means the given structure is a regular directory entry

• Accordingly, when a file or directory is deleted, the flag changes to “0”, indicating that this directory entry is free or unused directory entry.

Deleted Files

FAT after file deletion

• No change until re-used for the next fragmented file.

BITMAP after file deletion

• The bits that were showing “in use (1)” are immediately changed to “not in use (0)”

Recovery of Contiguous files in exFAT

Questions?

• Special Thank you to my colleague and friend Steve Ilett.

• Thank you all for attending this session!

Bibliography

• Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Contiguous Allocation in an Extensible File System, US Patent Application Publication, Pub. No.: US 2009/0164539 A1.

• Ravisankar V. Pudipeddi and Vishal Ghotge and Ravinder Thind 2009: Quick Filename Lookup Using Name Hash, US Patent Application Publication, Pub. No.: US 2009/0164440 A1.

• Jeff Hamm, Extended FAT File System, 2009 • Article ID: 955704 - Last Review: September 29, 2009 - Revision:

3.1: Description of the exFAT file system driver update package, http://support.microsoft.com/kb/955704, accessed February 2009.

• Raymond Chen 2006: Windows Confidential, A Brief and Incomplete History of FAT32, TechNet Magazine.

• Vishal Ghotge, (Program Manager, Core File Services Microsoft Corporation), exFAT (PPT), Win HEC 2006

http://support.microsoft.com/kb/955704

extensible file allocation table (exfat) ahtcc v1.1

Documents

binary logarithm

patent application

disk hex viewpercent

firstclusterofrootdir

time zone

file system

relative sector

primary directory