SQLite Forensics The little database that could

About viaForensics

viaForensics is an innovative digital forensics

and security company providing expert services

to:

• Law Enforcement

• Government Agencies

• Corporations

• Attorneys/Individuals

What’s the problem?

• We want to recover as much data from devices

as possible

• People delete data, mostly the data we want!

• SQLite is a very popular data storage format

• Currently no advanced SQLite recovery tool on

the market (but stay tuned)

What is SQLite?

• SQLite is a widely used, lightweight database

contained in a single cross-platform file used by

developers for structured data storage

• Used in most smart phones (iPhone, Android,

Symbian, webOS)

• Used in major operating systems and

applications (Apple OS X, Google Chrome and

Chrome OS, Firefox)

Why do developers need structured data storage?

• Applications need to store and retrieve data

• In past and today, developers created their own

file formats

• But why reinvent the wheel for basic data

storage?

• SQLite is free, open, high quality and takes care

of the messy details

Core SQLite characteristics (from their FAQ)

• Transactions are atomic, consistent, isolated, and durable (ACID)

even after system crashes and power failures.

• Zero-configuration - no setup or administration needed.

• A complete database is stored in a single cross-platform disk file.

• Small code footprint: 190KiB - 325KiB fully configured

• Cross-platform and easy to port to unsupported systems.

• Sources are in the public domain. Use for any purpose.

• Standalone command-line interface (CLI) client

SQL = Structured Query Language

• SQL is the language used to interact with many

databases, including SQLite

• Basic functions: Create, Read, Update and

Deleted (CRUD)

• Transactions: Start a change and it either

completes in entirety (commit) or not at all

(rollback)

• Very powerful, many variations

SQL – basic commands

• SELECT – queries data from tables or tables – SELECT rowid, address, date, text FROM message;

• INSERT INTO – adds data row to table – INSERT INTO message VALUES (NULL, ‘3128781100’, 1282844546, ‘text message’);

• UPDATE – updates data rows in tables – UPDATE message SET date=1282846291 WHERE rowid=4;

• DELETE – deletes data rows in tables – DELETE FROM message WHERE rowid=4;

• Many good tutorials online

Viewing a SQLite database – command line

• Command line apps

– sqlite3 for full SQLite functions

– sqlite_analyzer for db metadata

• Linux/Mac/Windows versions

• Represents latest version

• Full source code and documentation

• http://www.sqlite.org/download.html

Example sqlite3 session

Run sqlite3 on database file ahoog@linux-wks-001:~/sqlite$ ./sqlite3 iPhone-3G-313-sms.db

SQLite version 3.7.4

Enter ".help" for instructions

Enter SQL statements terminated with a ";"

sqlite>

List tables in database sqlite> .tables

_SqliteDatabaseProperties msg_group

group_member msg_pieces

message

Examine schema (structure) of message database sqlite> .schema message

CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date

INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER,

association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT,

country TEXT, headers BLOB, recipients BLOB, read INTEGER);

Example sqlite3 session - continued

View record “4” in 2 formats sqlite> .headers on

sqlite> SELECT * FROM message WHERE ROWID = 4;

ROWID|address|date|text|flags|replace|svc_center|group_id|association_id|height|UIFlags

|version|subject|country|headers|recipients|read

4|(312) 898-4070|1282844546|Sure is a nice day out |3|0||3|1282844546|0|4|0||us|||1

sqlite> .mode line

sqlite> SELECT * FROM message WHERE ROWID = 4;

ROWID = 4

address = (312) 898-4070

date = 1282844546

text = Sure is a nice day out

flags = 3

replace = 0

svc_center =

group_id = 3

association_id = 1282844546

height = 0

UIFlags = 4

version = 0

subject =

country = us

headers =

recipients =

read = 1

sqlite3_analyzer – very useful in forensic analysis

ahoog@linux-wks-001:~/sqlite$ ./sqlite3_analyzer iPhone-3G-313-sms.db /** Disk-Space Utilization Report For iPhone-3G-313-sms.db

Page size in bytes.................... 2048

Pages in the whole file (measured).... 14

Pages in the whole file (calculated).. 14

Pages that store data................. 13 92.9%

Pages on the freelist (per header).... 0 0.0%

Pages on the freelist (calculated).... 0 0.0%

Pages of auto-vacuum overhead......... 1 7.1%

Number of tables in the database...... 7

Number of indices..................... 4

Number of named indices............... 3

Automatically generated indices....... 1

Size of the file in bytes............. 28672

Bytes of user payload stored.......... 1833 6.4%

*** Page counts for all tables with their indices ********************

MESSAGE............................... 3 21.4%

SQLITE_MASTER......................... 3 21.4%

_SQLITEDATABASEPROPERTIES............. 2 14.3%

MSG_PIECES............................ 2 14.3%

<snip>

Viewing a SQLite database – SQLite Database Browser

• Freeware, public domain, open source visual tool used

to create, design and edit database files compatible with

SQLite

• Windows/Linux/Mac

• Support SQLite 3.x

• Last updated 12/2009

• http://sqlitebrowser.sourceforge.net/

• Many other (free) options listed at:

http://www.sqlite.org/cvstrac/wiki?p=ManagementTools

Viewing a SQLite database – SQLite Database Browser

Viewing a SQLite table – SQLite Database Browser

SQLite – database header format

• The first 100 bytes of the database file comprise the

database file header. • First 5 of 22 fields

Offset Size Description

0 16 The header string: "SQLite format 3\000"

16 2 The database page size in bytes. Must be a power of two between 512 and 32768 inclusive, or the value 1 representing a page size of 65536.

18 1 File format write version. 1 for legacy; 2 for WAL.

19 1 File format read version. 1 for legacy; 2 for WAL.

20 1 Bytes of unused "reserved" space at the end of each page. Usually 0.

SQLite – Organized in Pages

• Database consists of one or more pages, logical units which store

data

• Pages are numbered beginning with 1

• A page is one of the following:

Page type Description

B-Tree page B-Tree pages are part of the tree structures used to store database tables and indexes.

Overflow page Overflow pages are used by particularly large database records that do not fit on a single B-Tree page.

Free page Free pages are pages within the database file that are not being used to store meaningful data. (or so they think!)

Pointer-map page Part of auto-vacuum system

Locking page Tracks when database rows are locked for updating

B+tree and B-Tree formats – on-disk data structure

• Data structure which represents sorted data in a way

that allows for efficient insertion, retrieval and removal of

records

• Optimized for storage devices (vs. in memory) by

minimizing the number of disk accesses.

• In a B+tree, all data is stored in the leaves of the tree

instead of in both the leaves and the intermediate branch

nodes.

• A single B-Tree structure is stored using one or more

database pages. Each page contains a single B-Tree

node.

B+Tree graphical representation

http://www.sqlite.org/fileformat.html#table_btrees

SQLite storage classes and data types

• Only 5 storage classes/data types :

1. NULL: The value is a NULL value.

2. INTEGER: The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes

depending on the magnitude of the value.

3. REAL: The value is a floating point value, stored as an 8-byte IEEE floating

point number.

4. TEXT: The value is a text string, stored using the database encoding (UTF-

8, UTF-16BE or UTF-16LE).

5. BLOB: The value is a blob of data, stored exactly as it was input. Often

used to store binary data

SQLite storage classes – on disk example

• 5 storage classes in hex on disk:

• NULL: 0x00

• INTEGER (4-byte): 0x4c76a782 = 1282844546

• REAL: 0x41B1EC2EC004D9D7 = 300691136.018949

– http://babbage.cs.qc.edu/IEEE-754/64bit.html

• TEXT (ASCII): 0x53757265206973 = Sure is

• BLOB: hard to represent binary here…see Text

Variable Integers – saving space, adding confusion

• A variable-length integer or "varint" uses less space for small positive

values.

• Used in SQLite metadata (row headers, b-tree indexes, etc.)

• A varint is between 1 and 9 bytes in length.

• The varint consists of either zero or more byte which have the high-order bit

set followed by a single byte with the high-order bit clear, or nine bytes,

whichever is shorter. The lower seven bits of each of the first eight bytes

and all 8 bits of the ninth byte are used to reconstruct the 64-bit twos-

complement integer.

• Varints are big-endian: bits taken from the earlier byte of the varint are the

more significant and bits taken from the later bytes.

• http://www.sqlite.org/fileformat.html#varint_format

• Clear? How about an example ->

Variable Integers – example

• Let’s say you find the following hex varint: 0x8CA06F

– Examine each bit, if > 0x80 then not the last byte

– So, we have 3 bytes: 0x8C 0xA0 0x6F (since 0x6F < 0x80 it’s

the last byte). Here’s how to convert:

* MSB: Most significant bit (left most bit)

Original Bytes 0x8C 0xA0 0x6F

Convert to binary 1000 1100 1010 0000 0110 1111

Remove MSB* 000 1100 010 0000 110 1111

Concatenate 000110001000001101111

In hex/decimal Hex: 0x03106F Decimal: 200,815

Freelist / Free page list

• When information is deleted from the database,

pages containing that data are not in active use.

• Unused pages are stored on the freelist and are

reused when additional pages are required.

• Forensic value: “Freelist leaf pages contain no

information. SQLite avoids reading or writing

freelist leaf pages in order to reduce disk I/O.”

Rollback journal

• Created when a database is going to be updated

• The original unmodified content of that page is written

into the rollback journal.

• The rollback journal is always located in the same

directory as the database file and has the same name as

the database file but with the string "-journal" appended

• Excellent source of forensic data if recoverable

• Recoverable on many systems though some are now

writing to tmpfs/RAM disks

Write Ahead Log (WAL)

• New technique just introduced in 3.7.0

• Generally faster and disk I/O is more sequential (which helps us in

advanced recovery)

• All changes to the database are recorded by writing frames into the WAL.

• Transactions commit when a frame is written that contains a commit marker.

• A single WAL can and usually does record multiple transactions.

• Periodically, the content of the WAL is transferred back into the database file

in an operation called a "checkpoint".

• Forensic value: recovery of WAL files

Record Format

• A record contains a header and a body, in that order. The

header:

– begins with a single varint which determines the total number of

bytes in the header. The varint value is the size of the header in

bytes including the size varint itself.

– Following the size varint are one or more additional varints, one

per column. These additional varints are called "serial type"

numbers and determine the datatype of each column

– After the final header varint, the record data immediately follows

– The 2-bytes prior to the start of the header correspond to the

auto-increment integer assigned by the system (also a varint)

Record Format – visual representation

• http://www.sqlite.org/fileformat.html#record_format

Record Format

Header Value Data type and size

0 An SQL NULL value (type SQLITE_NULL). This value consumes zero bytes of space in the record's data area.

1 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 1-byte signed integer.

2 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 2-byte signed integer.

3 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 3-byte signed integer.

4 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 4-byte signed integer.

5 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 6-byte signed integer.

6 An SQL integer value (type SQLITE_INTEGER), stored as an big-endian 8-byte signed integer.

7 An SQL real value (type SQLITE_FLOAT), stored as an 8-byte IEEE floating point value.

8 The literal SQL integer 0 (type SQLITE_INTEGER). The value consumes zero bytes of space in the record's data area. Values of this type are only present in databases with a schema file format (the 32-bit integer at byte offset 44 of the database header) value of 4 or greater. (iOS4 uses this)

9 The literal SQL integer 1 (type SQLITE_INTEGER). The value consumes zero bytes of space in the record's data area. Values of this type are only present in databases with a schema file format (the 32-bit integer at byte offset 44 of the database header) value of 4 or greater. (iOS4 uses this)

10,11 Not used. Reserved for expansion.

bytes * 2 + 12 Even values greater than or equal to 12 are used to signify a blob of data (type SQLITE_BLOB) (n-12)/2 bytes in length, where n is the integer value stored in the record header.

bytes * 2 + 13 Odd values greater than 12 are used to signify a string (type SQLITE_TEXT) (n-13)/2 bytes in length, where n is the integer value stored in the record header.

Recovery from allocated SQLite with strings

ahoog@linux-wks-001:~/sqlite$ strings iPhone-3G-313-sms.db | less

<snip>

msg_group

(314) 267-6611us

(920) 277-1869us

(312) 898-4070us

(312) 401-1679us

(414) 331-5030us

Piece of cake! Can't wait to try em out on Sunday

text/plain

2text_0002.txt

image/jpeg

1IMG_6807.jpg?

Check out mccalister

text/plain

2text_0002.txt

image/jpeg

1IMG_6807.jpg

<snip>

Carving SQLite files

• File header readily identifiable

• Sample scalpel entry:

• Other tools like FTK/EnCase also carve

0000000: 5351 4c69 7465 2066 6f72 6d61 7420 3300 SQLite format 3.

0000010: 0800 0101 0040 2020 0000 00fc 0000 0000 .....@ ........

0000020: 0000 0000 0000 0000 0000 000f 0000 0001 ................

# extension case size header footer

# sensitive

sqlitedb y 819200 SQLite\x20format

Carving SQLite files – OS specific findings

• iOS

– Good recovery of both allocated and “latent”

SQLite files

• Android

– Excellent recovery but high repetition due to

log-structured file system repeating SQLite

header

• Other common file systems

– Good recovery form typical magnetic media

device running FAT, FAT32, NTFS, HFS, etc.

SQLite in Hex (really the only way to look at it)

0002270: 0000 0000 0000 0000 004d 0d12 0029 0445 .........M...).E

0002280: 0101 0001 0401 0101 0011 0000 0128 3331 .............(31

0002290: 3229 2038 3938 2d34 3037 304c 77d8 a257 2) 898-4070Lw..W

00022a0: 696c 6c20 796f 7520 676f 2067 6574 206d ill you go get m

00022b0: 6520 6120 636f 6666 6565 3f03 0003 4c77 e a coffee?...Lw

00022c0: d8a2 0000 0075 7301 3f0c 1200 2904 2901 .....us.?...).).

Name Type Header Converted Body Value / notes

Rowid – actual Varint 0x0d 13 So rowid = 13

Header Size Varint 0x12 18 Length of header is 18 bytes (header size + 17 rows)

Rowid NULL 0x00 0 NULL tells SQLite on insert to determine next auto increment

Address Text 0x29 (41 -13)/2 = 14 (312) 898-4070 [14 chars - covert 0x29 to decimal, calc size]

Date Integer 0x04 4-byte integer 0x4c77d8a2 in decimal is 1282922658 [recognize number format?]

Text Text 0x45 (69 -13)/2 = 28 Will you go get me a coffee?

Flags Integer 0x01 1-byte integer 0x03 = 3

Replace Integer 0x01 1-byte integer 0x00 = 0

Svc_center Text 0x00 NULL No value, not represented in data at all

Group_id Integer 0x01 1-byte integer 0x03 = 3

Association_id Integer 0x04 4-byte integer 0x4c77d8a2 in decimal is 1282922658 [recognize number format?]

Height Integer 0x01 1-byte integer 0x00 = 0

UIFlags Integer 0x01 1-byte integer 0x00 = 0

Version Integer 0x01 1-byte integer 0x00 = 0

Subject Text 0x00 NULL No value, not represented in data at all

Country Text 0x11 (17 – 13)/2 = 2 us

Headers Blob 0x00 NULL No value, not represented in data at all

Recipients Blog 0x00 NULL No value, not represented in data at all

Read Integer 0x01 1-byte integer 1 [Last data byte]

Advanced Technique

• Use well defined SQLite structure to develop a program

to recover SQLite rows

• Row header and data values “decay” over time due to

– Being (partially) re-allocated

– Fragmentation

– Compensated for this with simple probability engine which

determined likelihood sequence of bytes represented header row

we are interested in

• Underlying file system can have great impact, from FAT,

HFSplus (iPhone) and YAFFS2 (Android)

• Look for journal files and WAL data too

Contact Us

Andrew Hoog, CIO ahoog@viaforensics.com

http://viaforensics.com

1000 Lake St, Suite 203

Oak Park, IL 60301

Tel: 312-878-1100 | Fax: 312-268-7281