storage and file systems data timeline and recovery · 2019-11-04 · ntfs partition 2 t deleted...

4. 11. 2019

2

PowerShell brief

250MB

3221225472 / 3GB

0x1C6EE88F

'{0:X8}' -f 477030543

Storage and disk formats

4. 11. 2019

3

Storage

JBoD (just a bunch of disks)

• single physical disk

disk array + RAID

• usually redundancy

• many disks virtualized as several independent disks

slices, RAIDs, virtual disks, ...

Windows: LDM, Storage Spaces

addressing

• cylinder - head - sector (CHS)

• logical block address (LBA)

Disk array 1 Disk array 2

RA

ID m

n

RA

ID x

y

Disk arrays

RA

ID a

b

4. 11. 2019

4

RA

ID m

n

RA

ID x

y

Disk arrays

RA

ID a

bslice

disk 2

slice

disk 1

disk 3

disk 4

Windows file systems

FAT

FAT32

ExFAT

CDFS

UDF

NTFS

ReFS

gwmi Win32_Volume

4. 11. 2019

5

Windows and Hyper-V virtual hard disks

VHD

• <= 2 TB

• 512 B sector

VHDX

• newer format

• <= 64 TB

• 512 B or 4096 B sector

• TRIM/UNMAP

partition driver, format, NTFS, ReFS reclaiming and zeroing

freed storage

Disk formats

diskette or USB flash by default

• single partition boot sector

MBR

• 4 partitions, <= 2 TB

• BIOS boot

GPT

• 128 partitions, <= 18446744073709551616 sectors

• UEFI boot

MBR + LDM

GPT + LDM

4. 11. 2019

6

GPT disk

Disk formats

MBR disk

Partition 1

BOOTMBR

Partition 2

BOOT

Partition 1

BOOT

MBR

protec

tive

Partition 2

BOOTGPT

diskette, flash, CD/DVD

Partition 1

BOOT

Master Boot Record (MBR)

0th sector

4 byte random signature

4 primary partitions• 32bit = 4 bytes = offset = start at most at 2 TB

• 32bit = 4 bytes = length = max 2 TB length

bootstrap assembler code

sig 55AA

active partition (0x80)

standard data partition NTFS, ReFS (0x07)• even the System Reserved is a normal partition formatted with NTFS

standard data partition FAT32 (0x0C)

extending partition (0x0F)

GPT protective partition (0xEE)

LDM protective partition (0x42)

4. 11. 2019

7

MBR boot with the old BIOS

MBR disk

active partitionBOOTMBR

OS partition any FSBOOT

boot code

boot code bootmgr

ntldr

BCD

boot.ini

C:\Windows\winload.exe

C:\Windows\ntoskrnl.exe

BIOS

Globally Unique Identifier (GUID)

128bit =16 bytes

randomly generated, not registered

example:{185656d3-32af-43ee-a28c-281e0efd2201}

D3-56-56-18-AF-32-EE-43-A2-8C-28-1E-0E-FD-22-01

LDAP, PKI, GPT, volume, shadow copy, ...

4. 11. 2019

8

GUID in PowerShell

$ng = [Guid]::NewGuid()

$ng

[BitConverter]::ToString($ng.ToByteArray())

$rxGUID = '[\dA-Fa-f]{8}(?:-[\dA-Fa-f]{4}){3}-[\dA-Fa-f]{12}'

$ng.ToString('B') -match "\A\{$rxGUID\}\Z«

# from byte[] sequentially as in memory

New-Object guid @(,([byte[]] (0x33, 0x20, 0xA8, ...)))

GUID Partition Table (GPT)

0th sector protective MBR partition type 0xEE• signature + 2 TB

1th sector GPT

128 partitions• 64bit = 8 byte = start offset in LBA => 8.5 MLD TB

• 64bit = 8 byte = end offset in LBA => 8.5 MLD TB

{EBD0A0A2-B9E5-4433-87C0-68B6B72699C7}• Microsoft Basic Data Partition (BDP)

• normal formatted data FAT/NTFS

{E3C9E316-0B5C-4DB8-817D-F92DF00215AE}• Microsoft Reserved Partition (MSR)

• space reservation, no format, will be replaced later if necessary

• invisible from diskmgmt.msc, visible from diskpart

{DE94BBA4-06D1-4D40-A16A-BFD50179D6AC}• Microsoft WinRE Recovery Partition

• bootable Windows Recovery Environment

{C12A7328-F81F-11D2-BA4B-00A0C93EC93B}• EFI System Partition

• boot-loader on UEFI firmware platforms

• must be FAT32 formatted

• cannot assign letter

4. 11. 2019

9

GPT boot with the new UEFI firmware

GPT disk

EFI system partition FAT32BOOTGPT

OS partition any FSBOOT

bootmgr

BCD

C:\Windows\winload.exe

UEFI

protec

tive

MBR

signature

checks with

Secure Boot

FS and boot sector notes

(ex)FAT(32)• 1x boot sector

• 1x directory (names)

• 2x FAT (chained allocations) sector offset of FAT1 in field called reserved sectors

• volume signature

• volume name <= 10 chars ASCII

• dirty bit

NTFS• 2x boot sector (represented as $boot in MFT)

• 2x MFT (names + allocated data runs)

• volume signature

• volume name in $volume

• dirty bit in $volume

• permissions in $secure

• transaction log $logfile boot recovery

• free clusters in $bitmap

• USN journal in $extend

4. 11. 2019

10

7 7

Cluster vs. sector

1 2 3 4 5 6 7 87 8

legitimate data

NTFS offset 32bit

maximum cluster size 64kB

=>

2^32 * 64kB = 256 TB volume

NTFS offset 32bit

default cluster size 4kB

=>

2^32 * 4kB = 16 TB volume

Windows Mount Manager

HKLM\System\MountedDevices

• no Last Known Good backup

letter ~ partition mapping

• MBR - disk signature + partition start [B]

• GPT - partition GUID

• removable USB flash, CD, DVD - PNP device ID

4. 11. 2019

11

Disk, volume and partition serial numbers found and used

in system

Disk hardware serial numbergwmi Win32_DiskDrive | select SerialNumber

• not guaranteed to be unique

• not guaranteed to be present at all

Volume serial numberdir

gwmi Win32_Volume

MBR• disk signature 4B

gwmi Win32_DiskDrive | select Signature

GUID• disk GUID not used

• partition GUIDs

Flush file system volume caches

sync tool from sysinternals

disk.sys

FS

IO manager

kernel

cache

manager

write

always async

thread until 1809

flash drives

user

4. 11. 2019

12

disk

Deleted partition example

(UNMAP does not hit yet)

partition 1

visible

partition 3

visible

NTFS

partition 2

deleted

MB

R o

r G

PT

boot

boot

back

MFT

MFT2

data

data

data

disk

visible overlapping partition


(UNMAP hits during the red format)

partition 1

visible

partition 3

visible

NTFS

partition 2

deleted

MB

R o

r G

PT

boot

boot

back

MFT

MFT2

data

data

data

boot

back

boot

MFT

MFT2

data

data

data

data

4. 11. 2019

13

disk

visible overlapping partition


partition 1

visible

partition 3

visible

NTFS

partition 2

deleted

MB

R o

r G

PT

boot

boot

back

MFT

MFT2

data

data

data

boot

back

boot

MFT

MFT2

data

data

data

data

Dynamic Disks aka Logical Disk Manager (LDM)

on both MBR and GPT disks

on MBR disks

• a single protective MBR partition type 0x42

• last 7 MB of the disk contains LDM tables in un-partitioned

space

on GPT disks

• LDM Metadata Partition (1 MB)

{5808C8AA-7E8F-42E0-85D2-E1E90434CFB3}

• Microsoft Reserved Partition

• LDM Data Partition (the rest of the disk)

{AF9B60A0-1431-4F62-BC68-3311714A69AD}

4. 11. 2019

14

Storage Spaces

Lower-level implementation of software RAID

Windows Server 2012+

File systems

4. 11. 2019

15

NTFS notes

timestamps

Get-Item c:\theFile.txt | select *time | fl *

fsutil behavior query disablelastaccess

resident vs. non-resident data

• whole allocation in MFT

• allocation in MFT in the form of data runs

• TRIM/UNMAP on supporting storage

Common application write handling

original.docx

savedIntoTemp~.do~

backupedUpOriginal~.do~

Application

original.docx

read

write

rename

rename

delete

4. 11. 2019

16

Files in PowerShell

$file = New-Object System.IO.FileStream 'size.txt', 'Create',

'Write', 'ReadWrite'

$file.SetLength(800)

$file.Seek(0, 'Begin')

$bytes = [System.Text.Encoding]::ASCII.GetBytes('some text')

$file.Write($bytes, 0, $bytes.Length)

$file.Close()

# Make it smaller without zeroing the data manually

$file = New-Object System.IO.FileStream 'size.txt', 'Open',

'Write', 'ReadWrite'

$file.SetLength(100)

7 7

Slack space - statically hidden data

1 2 3 4 5 6 7 87 8

legitimate data

never read/copied by FS

properties of slack-space:

on TRIM/UNMAP zeroed by storage when cluster released

file length shortening inside a cluster may leave content

other hiding possibilities:

non-allocated space (may contain some old data)

falsely marked bad clusters

inter-partition space (may contain some old data)

zeroed on write operations

deleted file portions on

nonUNMAP media

4. 11. 2019

17

NTFS deleted files

MFT FILE record marked as free

• reused as soon as possible

resident data remains until FILE reused

non-resident data

• allocation remains in MFT

• marked as free in $BitMap

• TRIM/UNMAP immediately

(ex)FAT(32) notes

no internal record change date

• only CreationTime, LastAccessTime, LastWriteTime

deleting from Directory with E5

• short names always created

• long names

if present split on multiple entries

E5 does not overwrite the first letter

chained allocation in FAT

4. 11. 2019

18

(ex)FAT(32) chained allocation

next is 38theFile.txt

next is 39

next is 40

next is 41

next is 42

next is 2085

next is 2086

next is 2087

next is 2088

last

37

(ex)FAT(32) chained allocation

next is 38E5heFile.txt

next is 39

next is 40

last

next is 42

next is 2085

next is 2086

next is 2087

next is 2088

last

37

next is 37

4. 11. 2019

19

NTFS special objects

hard link• more names for a single "file object"

• used for short names if generated

junction (mount point)• special "file object" reparse point containing a link reference as data

• cannot point to any other system than local volumes

• R2L processed by the local system when accessed remotely

symbolic (soft) link• special "file object" containing reparse point information

• can point to any other system namespace object including SMB etc.

• R2L parsed by the remote system when accessed remotelyfsutil behavior query symlinkevaluation

fsutil behavior set symlinkevaluation R2L:1

fsutil behavior set symlinkevaluation R2R:1

sparse files• zeroed sections are not allocated and have 0 bits in $bitmap

• summing file sizes may yield more than is volume size

• NTFS is guaranteed to never return the real data regardless (non)sparse nature

alternative data streams

NTFS in CMD

fsutil hardlink list c:\windows\system32\kernel32.dll

fsutil reparsepoint query "%userprofile%\application data"

mklink /D c:\symlink c:\windows

fsutil reparsepoint query c:\symlink

mklink /J c:\junction c:\windows

fsutil reparsepoint query c:\junction

echo some visible text > alternate.txt

echo some hidden text > alternate.txt:HiddenData

more < alternate.txt:HiddenData

sysinternals\streams.exe

fsutil file createnew sp.txt 0x100000000

fsutil sparse setflag sp.txt 1

# Note: only marks an existing data range as sparse

fsutil sparse setrange sp.txt 0 0x100000000

4. 11. 2019

20

Junctions vs. symbolic links

local systemremote system

c:\junction

c:\windows

c:\symlink

C:\Windows

\\system\c$\junction

\\system\c$\symlink

C:\Windows

Symbolic links resolved locally in kernel

local systemremote system

c:\symlink

c:\windows

c:\UNCsymlink

\\data\Doc

\\system\c$\junction

\\system\c$\UNCsymlink

c:\windows

4. 11. 2019

21

volume D:\NTFS "FILE"

FILE record

timestamps

attributes

name1.txt folder1

name2.txt folder2

name3.txt folder3

security descriptor

data DEFAULT

data NAMED1

data NAMED2

volume C:\

volume D:\

symlink/junction

FILE record

timestamps

NTFS "FILE"

FILE record

timestamps

attributes

name1.txt folder1

name2.txt folder2

name3.txt folder3

security descriptor

data DEFAULT

data NAMED1

data NAMED2

security descriptor

4. 11. 2019

22

Start process from NTFS ADS

$exePath = 'T:\sevecek\TestWinApp-40-any.exe'

$hiddenExe = 'C:\temp\calc.txt'

$hiddenExeADS = '{0}:EXEfile' -f $hiddenExe

if (Test-Path $hiddenExe) { Remove-Item $hiddenExe -Force }

New-Item $hiddenExe -ItemType File -Force | Out-Null

$sourceBytes = [System.IO.File]::ReadAllBytes($exePath)

Set-Content -Path $hiddenExe -Value 'Calculate something' -Encoding

UTF8 -Force

Set-Content -Path $hiddenExeADS -Value $sourceBytes -Encoding byte

-Force

(gwmi -List Win32_Process).Create($hiddenExeADS)

Folder with data ADS

$theFolder = 'C:\TEMP\folderWithADS'

$theFolderADS = '{0}:SECRET' -f $theFolder

if (Test-Path $theFolder) { Remove-Item $theFolder -Recurse

-Force }

New-Item $theFolder -ItemType Directory -Force | Out-Null

Set-Content $theFolderADS -Value 'Hello' -Encoding utf8

4. 11. 2019

23

Start process from folder with ADS

$exePath = 'T:\sevecek\TestWinApp-40-any.exe'

$theFolder = 'C:\TEMP\folderWithADS'

$theFolderADS = '{0}:SECRET' -f $theFolder

if (Test-Path $theFolder) { Remove-Item $theFolder -Recurse -Force }

New-Item $theFolder -ItemType Directory -Force | Out-Null

$sourceBytes = [System.IO.File]::ReadAllBytes($exePath)

Set-Content -Path $theFolderADS -Value $sourceBytes -Encoding byte -Force

(gwmi -List Win32_Process).Create($theFolderADS)

NTFS transaction log

low-level metadata changes

• no file name usually

circular logging

• short time only until commit

4. 11. 2019

24

NTFS change journal

USN change journal

• file ref#

• enabled on system volumes by default

• file names

• growths with new allocations, older records left on

nonTRIM/UNMAP media

fsutil usn queryjournal c:\

fsutil usn enumdata

fsutil file queryfilenamebyid c:\ 0x...

(ex)FAT(32) notes

short names vs. long names

• 8.3

• short names "deleted" with 0xE5, long name remains if

present

• long name trick - 0x0F volume label, system, R/O, hidden

no transaction log

no journal

longer file name life than NTFS

4. 11. 2019

25

Mounted VHDs or ISOs or volume mount points

NTFS junction

when removed, may remain as deleted FILE records

in MFT

Volume Shadow Copy (VSS)

Copy-on-write partition block backup

System Volume Information• can be file imaged to another computer

• can be on another volume or even disk

• machine independent

• stores copies up to a limit or free disk space

• automatic copy deletion when necessary

Used by• backup software, Windows Server Backup (wbadmin)

• System Restore, System Protection Windows Installer, updates, ...

Some files excluded• HKLM:\System\CCS\Control\BackupRestore\FilesNotToSnapshot

• DisableUserProfileExpansion

4. 11. 2019

26

VSS in CMD

vssadmin

diskshadow

list shadows all

set context persistent

begin backup

add volume C:

add volume E:

create

end backup

expose {guidguid-guid-guid-guid-guidguidguid} X:

delete shadows all

mklink /J C:\mountedVSS

\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy35\

VSS in PowerShell

(gwmi -List Win32_ShadowCopy).Create('C:\', 'ClientAccessible')

gwmi Win32_ShadowStorage |

select *,

@{ n = 'DriveLetter' ; e = { ([wmi] $_.Volume).DriveLetter

} },

@{ n = 'DiffVolumeLetter' ; e = { ([wmi]

$_.DiffVolume).DriveLetter } }

gwmi Win32_ShadowCopy |

select *,

@{ n = 'DriveLetter' ; e = { (gwmi -Query ('SELECT * FROM

Win32_Volume WHERE DeviceId = "{0}"' -f $_.VolumeName.Replace('\',

'\\'))).DriveLetter } }

4. 11. 2019

27

DefineDosDevice() kernel symbolic link

$src = @'

namespace Sevecek {

using System.Runtime.InteropServices;

public class Win32Api {

[DllImport("kernel32.dll", SetLastError = true, PreserveSig = true,

CharSet = CharSet.Auto)]

public static extern bool DefineDosDevice(uint dwFlags, string

lpDeviceName, string lpTargetPath);

}

}

'@

if (-not ('Sevecek.Win32Api' -as [Type])) { Add-Type $src }

[Sevecek.Win32Api]::DefineDosDevice(0, 'S:',

'\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy38')

System Namespace

Named kernel objects• managed by a kernel driver .SYS

SysInternals WINOBJ

\\?\GLOBALROOT\• \GLOBAL??\C:\test.txt => \Device\HarddiskVolume3\test.txt

• \GLOBAL??\Volume{GUIDGUID-GUID-GUID-GUID-GUIDGUIDGUID}\test.txt => \Device\HarddiskVolume4\test.txt

• \GLOBAL??\UNC\gps-data\Doc\test.txt = \Device\MUP\gps-data\Doc\test.txt

• \SystemRoot\windowsupdate.log

• \Device\CdRom1

• \GLOBAL??\PhysicalDrive1 = \Device\Harddisk1\DR1

• \GLOBAL??\COM1 = \Device\Serial1

4. 11. 2019

28

Volume{GUID}

GPT partitions

• volatile

• \GLOBAL??\Volume{partition-GUID-from-GPT}

MBR partitions

• volatile

• \GLOBAL??\Volume{MBR-signature-0000-0000-0000-byte-

Offset}

CDROM, USB flash, floppy

• permanent by PnP Device Instance ID

• HKLM\System\MountedDevices

Security ID (SID)

EMPTY• S-1-0-0

BUILTIN• S-1-5-32-544 = Administrators

• S-1-5-32-545 = Users

NT AUTHORITY• S-1-5-18 = SYSTEM

• S-1-5-20 = Network Service

• S-1-5-11 = Authenticated Users

account database SIDs• S-1-5-21-numberA-numberB-numberC-RID

local SAM• HKLM\SAM\SAM\Domains\Account - V=REG_BINARY

• 12B at the end

domain NTDS.DIT Active Directory database

4. 11. 2019

29

SID in PowerShell

'S-1-5-18', 'S-1-5-32-544' | Select @{ n = 'SID' ; e = { $_ } }, @{

n = 'Name' ; e = { (New-Object

System.Security.Principal.SecurityIdentifier

$_).Translate([System.Type]::GetType('System.Security.Principal.NTA

ccount')).Value } }

'Administrators', 'NT AUTHORITY\Network Service' | Select @{ n =

'Name' ; e = { $_ } }, @{ n = 'SID' ; e = { (New-Object

Security.Principal.NTAccount

$_).Translate([Security.Principal.SecurityIdentifier]).Value } }

$rxSID = '[Ss]-1(?:-\d+){1,}'

[regex]::Match('This SID S-1-5-80-3964583643-2633443559-2834438935-

3739664028-1580655619 has been detected', $rxSID).Value

64bit vs. 32bit

System32 vs. SysWow64

SysWow64 vs. SysNative

Program Files vs. Program Files (x86)

• %ProgramFiles% vs. %ProgramFiles(x86)%

HKLM:\Software\Wow6432Node

no easy way to navigate 64bit registry from 32bit process$hklm = [Microsoft.Win32.RegistryKey]::OpenBaseKey('LocalMachine',

'Registry64')

$key64 = $hklm.OpenSubKey('SOFTWARE\XX64')

$key64.GetValue('theValue')

4. 11. 2019

30

64bit vs. 32bit EXE/DLL/SYS/EFI (portable executable)

MZ ... DOS header

0x3C offset PE header 64bit

P E 00 00 D P E 00 00 L

MZ ... DOS header

0x3C offset PE header 32bit

Explorer desktop.ini

False folder names and icons

$Recycle.Bin

%userprofile%

C:\Windows\Assembly

4. 11. 2019

31

Other clues of deleted files and usage

timestamps

executables

• Security event log - Process Tracking

• Prefetcher (SuperFetch, sysmain)

• MRU, bags, MUICache, ...

• Compatibility Assistant

• BAM

documents

• Windows Search (Indexing Service)

• MRU, bags, MUICache, ...

storage and file systems data timeline and recovery · 2019-11-04 · ntfs partition 2 t deleted...

Documents