However, the F2FS file system allows the system to be cleaned. The idea behind this is performance. So that when data is deleted, the system can clean and defragment the live or valid data. Cleaning could be initiated by the user in the device settings, initiated due to a lack of free sections, or part of a regular background cleaning. The cleaning process is divided into three steps (see Table 3.9). First of all, the section to clean must be selected. Different selection strategies are used for this purpose. The selection of the target section is followed by identifying the invalid blocks. For this, the bitmap can inside the SIT is used. The cleanup process ends with the creation of a new checkpoint to free the blocks that have been released in the meantime for reallocation.

#### **Adaptive Logging**

You just read about the two, four, and six multi-head logging options to handle different types of data based on their frequency of writes. The Adaptive logging has to do with where the writes will occur and possibly involve the cleaning process in F2FS. The Log Structured File System used two logging features:

So F2FS is using a dynamic approach with Adaptive Logging. There are two strategies based on the presented circumstances. See Table 3.10 and the illustration (Fig. 3.16) below.

Table 3.9: Cleaning Process: Three Steps [45]

As long as the data medium still has sufficient storage space, the append logging known from LSF is used. In this mode, only clean segments that are not yet occupied are written to. Remember: Sequential writes are always preferable to random writes in flash memory. In Threaded Logging mode, the whole thing is reversed.  So now the remaining gaps are being used. F2FS automatically switches between the two modes depending on the remaining free memory. This is an attempt to achieve the best possible write performance.

#### **Roll-Back Recovery**

F2FS uses checkpoints to ensure integrity in the event of power failures or other disruptions. A Checkpoint uses not only the checkpoint but also both the Node Address Table (NAT) and the Segment Info Table (SIT). There is NAT and SIT journaling and bitmap addressing the valid NAT and SIT within the checkpoint.

Table 3.10: F2FS Adaptive Logging [45]

Fig. 3.16: Threaded Logging is tactical in going after invalid blocks in a segment

Fig. 3.17: F2FS Index Structure (Source: [46])

If there is a disruption of a power issue then F2FS can do a rollback recovery using the most recent valid Checkpoint (see Fig. 3.17). Earlier checkpoints were discussed, and F2FS has two to maintain a stable one. The header and footer identify the stable Checkpoint is the same. There is also a version number if both checkpoint headers and footers match. In this instance, the file system will choose the most recent Checkpoint. F2FS does not always need to rely on the Checkpoint for recovery when *fsync()* is involved. F2FS focuses on the data blocks and direct node blocks (which are marked). The Roll-Forward Recovery Procedure consists of the following steps according to Lee et al. [45]:

### **•** > **Important**

Under Linux/Unix, the C function *fsync()* from the standard library transfers changed buffered data from the working memory to the file on the physical device. The call blocks until the device have reported that the transfer is complete. Of course, the buffer should only be flushed occasionally to avoid stress for the hardware.

### **3.5 Forensic Analysis**

The Flash-Friendly File System (F2FS) is made for removable media and used with some Android devices, or maybe changed over from another Android (Linux) based File System such as EXT. You will find plenty of videos on how to change the File System on an Android device and speed comparisons between F2FS and other file systems. In this section, we want to turn to how to read and acquire F2FS-formatted memory sticks or even SSDs. This section will talk about forensic analysis of the different F2FS formatted devices.

### **3.5.1 F2FS Sample Dataset**

The examples discussed in this chapter with the associated binaries can be found at github.com under the following link: https://github.com/Xamnr/F2FS. For the examples, three different drives were analysed: 1) the memory of a Huawei P9, 2) the content of a USB memory stick, and 3) an SSD. All three volumes were formatted with F2FS. The volumes were filled with text documents, image and video files. Afterwards, some of the files were deleted. The first example comes up with a binary file of a HuaweiP9 Superblock <File:HuaweiP9Superblock> and Checkpoint <File:HuaweiP9Checkpoint.zip>.

Beyond this, for USB flash drive three extracts had been made:

The second example contains a sample of a USB flash drive. A total of 2 folders and four files were added to the drive:

```
<ADDED EXTRACTION>
     |-- folder
     |-- Test
            |-- pngpicture.png
            |-- pngtodelete.png
            |-- textstays
            |-- texttodeletediconderoga
```
The third example contains the formerly added files, but this time some of them had been deleted:

```
<DELETED EXTRACTION>
     |-- folder
            |-- pngpicture.png <moved>
            |-- textstays <copied>
     deleted:
           |-- pngtodelete.png
           |-- textstays
           |-- texttodeletediconderoga
```
The example is completed by three dumps from the NAT, SIT and SSA region of the flash drive.

### **3.5.2 F2FS and Windows**

The main issue for forensics as it relates to F2FS is that Microsoft Windows does not recognize F2FS formatted devices. Windows sees the USB Device itself but does not recognize the partition.

Fortunately, at least forensic tools should recognize the device (as shown below with MSAB's XRY) and be able to do extractions. Best practice is to use a write blocker for the media. Once the drive has been detected, the analysis process with XRY is quite straightforward.

Fig. 3.18: F2FS formatted USB drive connected to a Microsoft Windows 10 computer

Fig. 3.19: MSAB's XRY 9.3 showing the Disk Connection Options

### **3.5.3 Data-Extraction with XRY**

The first step is, of course, to establish a connection to the device and the storage medium. For Android devices it will depend on whether the mobile device is supported by model or by a generic one (see Fig. 3.19). Mobile Device support may not include all options such as Logical and Physical extraction, but rather only one. Remember Micro SD Cards inserted into the mobile device should be removed at some point and done separately.

The forensic tool(s) should be able to image the mobile device and/or the removable media without an issue. Since F2FS is made for Flash memory it could be found on Solid State Drives as well. After we have successfully read in the flash drive, the next step is to analyse the data it stores. For an example, we read the internal memory of a Huawei P9 that was formatted with F2FS. The actual extraction was carried out using the software XAMN. Fig. 3.20 shows the result for of our example drive. Obviously, in addition to some regular files, deleted file artefacts were also found during the scan process. The F2FS formatted Flash drive also had deleted files that were retrieved as well. Remember recovering deleted files relies on a lot of variables, but at least we know it is possible with the F2FS file system.

Fig. 3.20: MSAB's XAMN Showing that the extraction did decode data from the Huawei P9 cell phone Including deleted files

The header of the checkpoint area is initially less interesting from a forensic point of view (see Table 3.12). It only contains information about the number of available segments.

### **3.5.4 Superblock Examination**

After successfully making a forensically sound copy of the medium to be examined, we can begin the actual analysis. Our investigation should also begin here since an F2FS partition starts with the Superblock(SB). In this region, important information about the structure of the rest of the file system can be found.

As examiners we commonly see the file system as raw data with our tools as seen in Fig. 3.21. Can we make sense of this data? In researching F2FS, a resource that describes the Superblock data structure was found 1. This data was used to create Tables 3.11 and 3.12 for the Superblock and the Checkpoint, respectively2.

1 www.programmersought.com/article/49182049693/

2 \* Offset 3204+ Kernel Information and more: Linux version 5.10.0-kali3-amd64<EMAIL_ADDRESS>(gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.1) #1 SMP Debian 5.10.13-1kali1 (2021-02-08)

Fig. 3.21: Viewing the USB Flash Drive's F2FS Superblock with MSAB's XAMN

The Superblock (SB) provides us with important information about the structure of the partition and the exact location of NAT, SIT, SSA and MAIN area. The values given are to be understood as multiples of the block size. In our example, the SIT starts at block no. 1536. The correct address is obtained when we multiply this number by the block size. In our case, the SIT thus starts at byte 6,291,456 (1536\*4096). The total size of the SIT can again be determined via the 4-byte value at offset 56. In our case, this is ℎ𝑥02000000. Since this is a little-endian (LE) value with the least significant bit on the far left, the size is 2. Incidentally, all other header fields are also LE values. The node ID of the root directory can also be determined in the superblock (offset 96).

If your forensic tool supports viewing the raw data then that is a start. Hopefully the tool has options to translate the code such as showing the bit options i.e. 16 or 32, Little or Big Endian, and others such as the GUID. If your tool does not have these options then you can use the HxD Hex Editor/Viewer with the Inspector feature. HxD can be found here mh-nexus.de/ or for English mh-nexus.de/en/.

### **• ! Remember**

HxD is an editor, so work off of a copy of the file.

### **3.5.5 Examine NAT, SIT & SSA with Linux**

Since F2FS was developed specifically for use with Linux or Android, it is obvious to conduct an investigation with this system as well. This section will show the forensic analysis of F2FS with open-source digital forensic tools if you are inclined to use Linux, i.e. Kali, Santoku, or another forensic type, and command line. Then you may

Source: www.programmersought.com/article/49182049693/

Source: www.programmersought.com/article/37962049663

Table 3.11: Super Block (USB Flash Drive) Example Values

want to (write-protected, of course) gather additional system information about the removable media (USB Flash Drive in this case). You could mount the dd(bin) image as well. You will find F2FS Tools in the official Linux kernel GitHub repository 4. Alternatively, you can install the necessary tools via package management. Under Ubuntu, for example, the following command can be used for this purpose:

```
# sudo apt-get install f2fs-tools
```
File system information can be dumped from the device using the F2FS Tools dump function. The primary interest is obtaining data from the NAT, SIT, and SSA. However, we can also use it to obtain a file if we want to. The dump.f2fs shows the on-disk inode information using the inode number and a dump of all SSA and SIT entries to file recognised by dump\_ssa and sump\_sit. On the web page of the

4 http:// git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git

Table 3.12: Checkpoint (USB Flash Drive) Example Values

Fig. 3.22: HxD with color coded selections of Superblock to match table data 3.11

the Linux Kernel Organisation [89] all options can be retrieved, with which the command can be executed (see Table 3.13).

Table 3.13: Usage: dump.f2fs command options

The tool can also be run without special parameters. In this way, information about the size of the disk and the sector size can be determined first of all. A typical output of the dump.f2fs command results without any options where /dev/sdb1 is the USB Flash Drive looks like this:

```
#sudo dump.f2fs /dev/sdb1
Info: [/dev/sdb1] Disk Model: Flash Disk
Info: Segments per section = 1
Info: Sections per zone = 1
Info: sector size = 512
Info: total sectors = 30717952 (14999 MB)
Info: MKFS version
"Linux version 5.10.0-kali3-amd64 ..."
...
Info: superblock features = 0 :
Info: superblock encrypt level = 0, salt =<PHONE_NUMBER>000000000000000
Info: total FS sectors = 30717952 (14999 MB)
Info: CKPT version = 373c4953
Info: checkpoint state = c5 : nat_bits crc compacted_summary unmount
Done: 0.177078 secs
```
### **Node Allocation Table (NAT) Data**

In section 3.3.2 we have already learned about the task and function of the Node Allocation Table. With dump.f2fs we can output the contents of the table for our disk. The output of command sudo dump.f2fs -n 0~-1 /dev/sdb1 for our example is shown in Table 3.14. Remember: All the node blocks are mapped by NAT. Hence, the position of each node is translated by the NAT table. Apparently there are exactly 4 valid direct accounts on the device. In addition to the logical node ID, the block address is also specified. Since the default block size is 4 KB by default, we can thus determine the exact physical on-disk location of the node.

Table 3.14: Example for a Node Allocation Table (NAT)

#### **Show the Segment Info Table (SIT) Data**

The second important data structure besides the NAT is the Segment Information Table. The term segment here means a contiguous lump of disk blocks. Normally 512 continuous Blocks grouped into one segment. As already discussed in section XX, a segment is assigned a sector by default. For our example USB stick, the command would look like this: sudo dump.f2fs -s 0~-1 /dev/sdb1.  The segment summary contains 512 entries, which is the 2MB segment size. A summary entry for a 4KB-sized block in a segment contains the. The first segment with number 0 seems to have 2 valid block. It is a hot data block with directory entries inside (seg\_type=3). In contrast, the second segment contains Data blocks (seg\_type=4).

#### **Look inside the Segment Summary Area (SSA) Data**

A look at the SSA allows us to find out who exactly owns each block. For example, to get an insight into the segment with the number 0, we can use the following command: sudo dump.f2fs -a 0~-1 /dev/sdb1

```
segno: 0, Current Node
[ 0: 3][ 1: 3][ 2: 4][ 3: 5][ 4: 5]
[ 5: 5][ 6: 5][ 7: 5][ 8: 4][ 9: 4]
[10: 5][11: 5][12: 4][13: 4][14: 5]
[15: 5][16: 3][17: 5][18: 3][19: 3]
[20: 4][21: 3][22: 4][23: 3][24: 3]
[25: 0][26: 0][27: 0][28: 0][29: 0]
...
```
As clearly seen, the blocks with the numbers 0 and 1 from segment no 0 both belong to the node with the node ID 3. The third block belongs to the node with the number 4 and so on.

### **Obtain a file by it's node ID**

If you recall earlier the linux command sudo fsck.f2fs -t /dev/sdb1 obtained some data including the files and their node identifiers.

```
```
In order to get the file, again we can to use the linux command line with the F2fs.dump command. Your forensic tools or even Linux should do this for you, but just to reinforce how this F2FS directory works with nodes you could use the command:

```
#sudo dump.f2fs -i 0x6 /dev/sdb1
Info: [/dev/sdb1] Disk Model: Flash Disk
Info: Segments per section = 1
Info: Sections per zone = 1
Info: sector size = 512
Info: total sectors = 30717952 (14999 MB)
...
[print_node_info: 353] Node ID [0x6:6] is inode
i_mode [0x 81a4 : 33188]
i_advise [0x 3 : 3]
i_uid [0x 0 : 0]
i_gid [0x 0 : 0]
i_links [0x 1 : 1]
i_size [0x 3c41f : 246815]
i_blocks [0x 3e : 62]
i_atime [0x606db122 :<PHONE_NUMBER>]
i_atime_nsec [0x24990b38 : 614009656]
i_ctime [0x606db114 :<PHONE_NUMBER>]
i_ctime_nsec [0x2bc016f9 : 734009081]
i_mtime [0x606b7024 :<PHONE_NUMBER>]
i_mtime_nsec [0x ad1c7fa : 181520378]
i_generation [0xd0856627 :<PHONE_NUMBER>]
i_current_depth [0x 0 : 0]
i_xattr_nid [0x 0 : 0]
i_flags [0x 0 : 0]
i_inline [0x 1 : 1]
i_pino [0x 4 : 4]
i_dir_level [0x 0 : 0]
i_namelen [0x e : 14]
i_name [pngpicture.png]
i_ext: fofs:0 blkaddr:ef400 len:3d
i_addr[0x0] [0x ef400 : 979968]
i_addr[0x1] [0x ef401 : 979969]
i_addr[0x2] [0x ef402 : 979970]
i_addr[0x3] [0x ef403 : 979971]
...
i_addr[0x3b] [0x ef43b : 980027]
i_addr[0x3c] [0x ef43c : 980028]
```
3 The Flash-Friendly File System (F2FS) 101

We can thus query important meta-information about the file. In addition to file names (*pngpicture.png*) and size, the MAC timestamps are displayed. The number and concrete address of the blocks and the size of the file are printed out. The file occupies a total of 63 blocks. With a block size of 4096 bytes, this corresponds to 258.048 bytes. The actual size of the file is somewhat smaller, with 246.815 bytes.

### **3.5.6 Carving for artefacts with XAMN**

XAMN is an intuitive tool that helps you find and analyse data faster and easier, we can even find a formerly deleted audio file (Fig. 3.23). For the file content search, all three samples of the dataset were first loaded for analysis with XAMN.

Fig. 3.23: MSAB's XAMN Spotlight showing a deleted audio file

First of all, it can be noted, that the program is able to detect F2FS partitions and their regions. Each region can be called individually and its content can be examined separately.

Looking at the screen capture on the left you should recognize the F2FS Areas. Or more importantly that you are dealing with a Flash-Friendly File System. Both the Superblock and Checkpoint Area are highlighted.

The files themselves and their meta information must be located in the Main Area of the disk. Accordingly , we checked to see if there were any references to the files added or deleted in the examples. A search for the PNG picture file name resulted in two hits (see Fig. 3.24). A simple search with a hex editor should give the same result. But also the names of the image data could be found (see Fig. 3.25). This appears to be the contents of the directory. In addition, a dot '.' and then two dots '..' are recognizable in the dump. Presumably these stand, as usual in Linux for the current directory or for the parent directory entry.

Fig. 3.24: MSAB's XAMN Elements showing the "pngpicture" search results

We are then able to find all four file names together in both the ADDED and DELETED extractions.

Even more, all three samples of the USB flash device could be successfully imported into XAMN. A look at the file tree shows that all file artefacts were found for the three scenarios (1..BASE,2..ADDED,3..DELETED). The correct directory structure could also be reconstructed. In addition to the regular files, the moved or deleted files are displayed as well (Fig. 3.26).

Fig. 3.25: Found all four file names together in the extraction notice the ".." and "." Before the data

Fig. 3.26: File Tree showing the three USB Flash Drive Extractions

### **PNG File Signature Analysis**

In the normal logging, blocks are written to clean unused segments. Thus there is a good chance that blocks occupied by a file are also written together on the volume. In our example, among other things, two .png image files were uploaded to the disk and then deleted again. This raises the question of whether we can recover the file contents using carving? We want to see and know about the ability to carve the PNG Files. Keep in mind the Cleaning ability of F2FS would most likely cause additional changes.

It is often assumed by laymen that once files have been deleted they cannot be recovered - neither in whole nor in part. This (careless) assumption is found especially often with respect to Unix. The possibilities of finding data fragments at different places of the file system or the hard disk can be used in an investigation. Often these individual fragments can be put together to a file with the method called file *Carving* or at least essential information can be extracted. In our example we want to search for the two image files in *png* format. Binary formats of image files often start with a magic number by which we can recognize the file. Referring to Gary Kessler's File Signature website: www.garykessler.net/library/file\_sigs.html, a search for "PNG" resulted in the following hits (Trailer means the end of the file):

Table 3.15: PNG File Signature as shown from Gary Kessler's website

Knowing both the Header and footer gave us a few options. We decided to use a Global Regular Expression (GREP) to see if we could recover the complete file from file header to footer (or trailer). To verify the file header and footer is correct. We can look at the extraction and pick a PNG file and view the raw file data. As you can see below (see Fig. 3.27 and 3.28) both the header and the footer look good.

Fig. 3.27: MSAB's XAMN Source Mode showing the PNG File Header (Hex): 89 50 4E 47 0D 0A 1A 0A

Excellent, now we can carve for the deleted png pictures to see if we can find them. A search for file headers in both the Added data USB and the Deleted USB resulted in 101 png file header hits for both of them. Note the deleted png file carved below. The forensic Analysis Tool used below (see Fig. 3.30 and 3.31) was able to find the deleted files and present them.

A search for Test did reveal a hit with the word Folder and Test (the two folders on the USB Flash Drive). Note the red characters are showing that these are different by comparison with the other extraction or extractions.

Fig. 3.28: MSAB's XAMN Source Mode showing the PNG File Footer (Hex): 49 45 4E 44 AE 42 60 82

Fig. 3.29: MSAB's XAMN Elements showing the PNG file header search and the deleted PNG File

### **3.5.7 Node Allocation Table (NAT) Comparisons**

The changes made to samples 2 (ADDED) and 3 (DELETED) must also have led to a change in the Node Access Table (NAT). Blocks must have become invalid or new blocks must have been occupied on the data carrier as a result of deleting or moving files. For the examination with XAMN we must first open in the NAT area. Then we can compare the two areas with each other. The results are shown in Fig. 3.32 for the ADDED sample and in Fig. 3.33 for the DELETED sample. The only differences in the NAT area was found between the USB Flash Drive, with Data Added and when data was moved/deleted; these are shown in red.

Fig. 3.30: MSAB's XAMN showing the Deleted Text File

Fig. 3.31: MSAB's XAMN showing the Deleted PNG File

Fig. 3.32: USB Flash Drive NAT bytes in red are different when compared with the DELETED sample's NAT

Fig. 3.33: USB Flash Drive NAT bytes in red are different when compared with the ADDED sample's NAT

### **Additional Data Structure**

In addition to the works in this chapter, a F2FS forensic paper [96], in Chinese, was found and the following data structure tables (Table 3.16 and 3.17) are included as they may be deemed useful. For convenience a copy of the the original and the translated version using Google Translate (accuracy cannot be guaranteed) can be found here: http://github.com/Xamnr/F2FS.

Table 3.16: Common file metadata information data structure [96]

Table 3.17: Data structure of catalog file metadata information block [96]

### **3.6 F2FS - Application fields**

One of the largest Android Manufacturers Samsung and the original creator of the Flash-Friendly File System (F2FS) is using F2FS in combination with UFS in some of their devices over using EXT, such as the Galaxy Note 10 and Galaxy Tab S6. Early on Motorola and Google used F2FS. Huawei and ZTE have also used F2FS on some of their devices.

An interesting, albeit dated article that is of interest entitled "Drone Forensic Analysis Using Open Source Forensic Analysis Using Open Source Tools" [94]. Drones use flash media, many have removable media, so it is not a surprise to see the connection. What is interesting is the mounting of the DD image to use the F2FS Tools. Working off a copy of the dd image would be advised. There are certainly a lot of developments in F2FS, including last year with Linux 5.11 and encryption.

### **3.7 Conclusion**

The Flash-Friendly File System (F2FS) has been around for some time and as you can see still may be used. You have seen that it was specific for Flash memory and that this includes some mobile devices. With regards to the Android mobile devices the user may elect to use F2FS over EXT4 if that is an option. Forensic Tools should be able to handle the Flash-Friendly File System, so test them to be sure. The issue is will we find the data that has been deleted. As you saw recovering deleted files is a possibility, however, not a certainty as there are so many variables involved.

**Acknowledgements** Many thanks to Changman Lee, Dongho Sim, Joo-Young Hwang, and Sangyeun Cho, Samsung Electronics Co., Ltd. for their documentation and the 2015 USENIX Conference presentation by Joo-Young Hwang entitled: "F2FS: A New File System for Flash Storage" [45] and to Neil Brown for his 2012 article "An f2fs teardown" [8].

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

 The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Chapter 4 QNX6**

Conrad Meyer

**Abstract** The QNX6 filesystem is present in Smartphones delivered by Blackberry (e.g. Devices that are using Blackberry 10) and modern vehicle infotainment systems that use QNX as their operating system. In 2015 QNX as an OS was used in over 50 million vehicles [6] and can hence be considered as one of the most important operating systems in the automotive world. Today's digital forensics tools don't recover a lot from this filesystem, have difficulties with different block sizes, or even don't support the filesystem at all. So it's crucial for the forensic examiner to understand the principles of this filesystem used. This chapter gives an overview of how the filesystem generally stores the files and metadata to give the examiner the chance to get the most information out of the evidence.

### **4.1 Introduction**

This chapter gives an insight into the different structures and principles of the QNX6 filesystem developed by QNX.The filesystem was first introduced within QNX Neutrino 6.4 real-time operating system, which today is owned and developed by Blackberry. It is a power-safe file system [7] and can withstand a sudden loss of power without corrupting or losing data. This property is especially useful for the forensic examiner, as it can easily happen that evidence (e.g. a vehicle or smartphone) loses its power supply due to a battery pack running empty.

Conrad Meyer

© The Author(s) 2022 109 https://doi.org/10.1007/978-3-030-98467-0\_4 C. Hummert, D. Pawlaszczyk (eds.), *Mobile Forensics – The File Format Handbook*,

Central Office for Information Technology in the Security Sector (ZITiS), Zamdorfer Straße 88, Munich, Bavaria e-mail<EMAIL_ADDRESS>

Table 4.1: Standard Parameters of the QNX6 Filesystem

Table 4.1 shows the standard values that are regularly used when formatting a volume with the QNX6 filesystem. Note, that especially in-car infotainment systems, those values can be different (e.g. larger blocksize). All the addressing inside the filesystem is based on the blocksize, extracted out of the superblock.

The following sections will give the reader an insight into the binary structures of the most important parts of the filesystem, like a superblock or inode and some basic knowledge about the mechanism when files are deleted.

### **4.2 QNX6 Filesystem Structure**

To understand the principle behaviour and main functions of the QNX6 filesystem, the following chapter shows the structure of a volume and how files, directories and metadata are linked. Volumes can be formatted in QNX6 in little-endian or big-endian style. All the examples in the following show a QNX6 Volume formatted with little endianness. Fig. 4.1 shows the main parts of a QNX6 filesystem and their standard size and addresses. The system area contains the Bitmap of the allocated

Fig. 4.1: Layout of a QNX6 filesystem volume

and unallocated Blocks of the Filesystem. Each bit represents a Block. Suppose the volume is formatted in the standard way. In that case, the volume will start with a volume boot record, which contains standard ASCII coded bootloader messages (Fig. 4.2), already giving a hint that the Volume is formatted with QNX.

### **• ! Attention**

Sometimes, on non standard volumes a partition directly starts with the Superblock.

Fig. 4.2: Sector 0 of a QNX6 Partition/Volume

In the following, we will have a closer look at all the structures above. We will follow those structures to construct a file and its metadata out of the filesystem information. The example filesystem is in little-endian mode.

### **4.2.1 Superblock**

The filesystem maintains two Superblocks or global root blocks. One of those blocks, called the working Superblock, manages the modified data, while the other one, the stable Superblock, consists of the original version of all the blocks.Which Superblock is the active one is determined by the 64-bit long serial number. The Superblock with the higher serial is the active one. After all, active write operations are done, and the integrity is checked, the former working superblock becomes the new stable one by updating the serial number (old superblock serial +1).

The superblock contains the global information of the filesystem. Table 4.2 contains the offset address of the main features of the Superblock.

### **• ! Attention**

When used with the standard driver issued by Blackberry and the default settings, you can determine the last access to the filesystem by selecting the stable superblock (highest serial) and checking the access timestamp (assuming that system time is used was valid). However, some non-standard drivers don't touch this timestamp, so for reliable results, you have to test the drivers from the System where the image originated in each case!

The superblock contains three root inodes that point to the main parts of the filesystem. The first array root inode contains the pointers to the inodes that contain the data (files, directories, data). The second one contains the pointers to the bitmap of the allocated blocks, and the third one is the pointers to the long filenames (filenames > 27 utf8 characters, up to 510 characters). The data inside those root inodes is shown in Table 4.3. Those root inodes contain pointers to the corresponding filesystem parts. If the level parameter is zero, the root inode has 16 direct pointers. By adding another level, indirect pointers are added, as shown in Fig. 4.4. Each indirect pointer then points to a block containing inodes or indirect 32-bit pointers, depending on the defined number of levels. The actual data is always at the lowest level of the tree. Given the value of blocks that such a tree can address is 16 \* (block size in bytes / 4) 𝑙𝑒𝑣𝑒𝑙 So, for example, with a level value of 2, and a block size of 1024 bytes, already 1,048,576 blocks can be addressed.

Fig. 4.3: An example of a QNX6 superblock.

Fig. 4.4: Illustration of inode levels, here a level value of 3

### **4.2.2 Bitmap**

The Bitmap block is used to determine whether a block in the filesystem is used or not. Each bit in the bitmap represents a block. A value of 0 means the Block is unused, 1 means that the Block is allocated. If the volume size is smaller than the bits available in the Bitmap Block, the unused bits are stuffed with ones. The bitmap incorporates two parts. First, system area 1 is split into two halves, where the upper half is used by superblock 1, and the lower half is used by superblock 2. This bitmap area contains the bitmap, inode and indirect addressing blocks of those structures. Second, the bitmap of the blocks that are not used for the filesystem structure (bitmap and inodes). The preallocation of the first system area block leads to the effect that each superblock always works on its own filesystem structure, and to the point that there is always a non-corrupted structure, even in the case of a sudden power loss (a superblock is just becoming the stable one, if all write operations are done, see sect. 4.2.1).

Fig. 4.5 depicts the end of the used space of the bitmap pointed to in the example superblock from Fig. 4.3. The bitmap comprises two blocks, starting at 0x3000, and the volume contains a total of 0xC7F8 blocks. In Fig 4.5, the stuffing of the unused space with ones therefore starts at 0x48FF: Bitmap starting address: 0x3000 + number of blocks 0xC8f8 divided by 8 (each Block represented by 1 bit).

Fig. 4.5: An example of a QNX6 Bitmap

### **4.2.3 Inode**

On the lowest level of the root inode tree, in the "leaves", the direct inode data is found. Depending on the level defined, also those inodes can address other indirect inode addressing blocks. An inode contains a vast amount of data useful for the forensic examiner, e.g. permissions, access time, change time, and modification time. Table 4.4 shows the offsets and the size of the various parameters in an inode.

Table 4.4: Structure of an inode

Table 4.5: inode status byte

As QNX OS is in line with the POSIX standards; also the timestamps are. The epoch is the standard POSIX (or UNIX) epoch, the 01.01.1970, 00:00 UTC. From that epoch, the timestamps are counted in seconds. The modified timestamp (mtime) is the time of the last write operation on this specific file. The access timestamp (atime) tells the examiner the time the file was last read. The change timestamp (ctime) is changed when the permissions of a file are changed. So ctime can be changed without a change in atime. The timestamp ftime is not fully referenced in the POSIX standard. Like in many other filesystems, it is the timestamp when the file was created. The inode 1 always contains the root directory, and inode counting starts with 1.

### **• ! Attention**

When it comes to timestamps, the forensic expert has to pay attention to the reliability of the timestamps given. This is especially true for QNX6. Not all timestamps are actualised on some systems, as with QNX with the standard QNX6 file-system driver. Whenever possible, tests with the system you are examining should be performed (e.g. changing permissions, modifying files, etc.)!

Fig. 4.6: An example of a QNX6 Inode.

### **4.2.4 Directories**

Inodes with the status 0x3 point to a directory file system object that contains subdirectories and file entries with names shorter than 27 UTF-8 characters. An entry starts with the inode number of that entry, where you can find the metadata like timestamps and the pointers to the Data or other directories, followed by a name length field and the actual name. A directory always contains a "." and a ".." entry. The "." entry contains the inode number of the directory inode, and the ".." entry contains the inode number of the parent directory inode. In the example Fig. 4.7, those entries are both pointing to the same inode number because the directory shown is the root directory.

Table 4.6: Directory entry

Fig. 4.7: An example of a QNX6 directory. Here, the root directory is shown.

A long directory entry has a different structure (Table 4.7). It includes the Inode, in which the timestamps and pointers to the data are. Furthermore, the long filenames inode Number, where the entry's name is found, is noted in this structure. An example of a long filename/directory entry is displayed in Fig. 4.8.

Table 4.7: Long Directory entry

Fig. 4.8: An example of a QNX6 inode entry of a long filename

### **4.2.5 Long Filenames Inode**

If a file or directories length is longer than 27 UTF-8 characters, the name is stored in the long filenames node. Long filenames Inodes start counting with zero. The structure is shown in Table 4.8, an example is Fig. 4.9.

### **4.3 Example: Construction of a file**

To understand how a file can be retrieved from the filesystem data, we will manually find the file /usr/fileformathandbook.ascii with its content and metadata by using the

Fig. 4.9: An example QNX6 long filenames entry

filesystem information. We will begin the reconstruction from the root directory. As already mentioned in the previous chapter, inode 1 contains the root directory. From there, we will start finding the file in the filesystem structure. The first step is to determine the valid stable superblock by the serial number. The superblocks inode root block is shown in Fig. 4.10

Fig. 4.10: Inode Root block used in the file reconstruction example

The root block tree has one level, meaning that we go on with the indirect inode block in the next step. The formula can easily calculate the physical address of those blocks:

### 𝑏𝑙𝑜𝑐𝑘𝑎𝑑𝑑𝑟𝑒𝑠𝑠 = 𝑏𝑙𝑜𝑐𝑘𝑛𝑢𝑚𝑏𝑒𝑟 ∗ 𝑏𝑙𝑜𝑐𝑘 𝑠𝑖𝑧𝑒 + 𝑜 𝑓 𝑓 𝑠𝑒𝑡

On standard QNX6 Volumes, the offset is the superblock size + the offset of the beginning of the superblock. Thus, the first indirect inode block is located at 0xCD \* 0x1000 + 0x3000 = 0xD0000, where 0xCD is the block number, 0x1000 the blocksize and 0x3000 the global offset due to the superblock with size 0x1000 and start at 0x2000. From the indirect inode (Fig. 4.11), we can retrieve the number 0x03, and by this, the address of the first inode block, which is located at 0x6000.

The first inode in this block is the root inode. If we take the first block pointer, 0x7F10, of this inode, we get the address of the root directory: 0x7F13000. This root directory, Fig. 4.13 is already familiar to us, as the second version of it is shown in Fig. 4.7, but this time, it is the root directory maintained by the first superblock. In the root directory, we take the inode number for the /usr directory, 0x08. With this number, we go back to the first Inode Block, where the inode 8 is located at 0x6380 (0x6000, where inode 1 is located plus 7 \* 0x80 offset, for the preceding inodes). From that inode (Fig. 4.14) we can then calculate the /usr directory offset in the way we already did for the root directory. The /usr directory is defined at block 0x7F72

#### Fig. 4.11: Indirect inode block

Fig. 4.12: inode 1 which contains the pointers to the root diretory

Fig. 4.13: Root Directory

which is at offset 0x7F75000. Here we see now our filename and the corresponding inode Number, where the metadata and pointer to the file content is.

We see that the *fileformathandbook.ascii* file has the inode number 0x258. Knowing this, we have to find the offset where this inode is defined. With a block size of 0x1000 and an inode size of 0x80, each inode block contains 0x20 inodes, so the inode we are looking for is the 24th inode in inode block number 19. Going back to Fig. 4.11, the 19 inode block is at physical block 0xE0, calculated address 0xE3000

Fig. 4.14: Inode 8, which has the pointer to the /usr directory in our example

Fig. 4.15: /usr directory with the entry of the file we are looking for

+ 0xB80 (24th inode in Block). In this inode, depicted in Fig. 4.16 we find all the relevant filesystem metadata for this file and the pointers to the filesystem content.

Following now the pointers to the content, beginning with 0x19D, we can retrieve the file block by block (Fig. 4.17).

After demonstrating the retrieval of the example file from the file system data, it is easy to understand the next section, which shows the possibilities to reconstruct deleted files.

### **4.4 Deleted Files**

There are some possibilities to recover deleted files in a QNX6 Volume, depending, when the file or directory was deleted and what happened with the filesystem in the meanwhile. Deleting an entry (directory or file) in QNX6 means that the Status in

Fig. 4.17: Content of our example file

an Inode switches to "deleted" (see Table 4.5) and that the entries inode number is deleted from the directory as shown in Fig. 4.18. By this, it is not possible to recover a file by its name, because there is no link anymore between the filename and the inode containing the metadata and the pointers to the file content. If a directory is updated after a file was deleted (e.g. a new file is added), the filesystem driver moves the directory to another block. The filename is "lost" from the regular filesystem directory tree. Also, the blocks, which contain the content of the files are set to unused in the bitmap, which means, they are free to be overwritten by other data. Knowing this, there are still some possibilities to recover files, with and without their respective names.

Fig. 4.18: Directory entry before (bottom) and after (top) deletion

The first possibility, if the file was just deleted recently, it may still be present in the non-active filesystem structure of the second superblock. If this is the case, the file can normally be fully recovered, even with its content (still, it is possible that the content is not original).

Second, you can parse the inodes to recover files with their metadata without the associated filename. This fact is quite problematic because the Blocks do not necessarily still contain the files original data.

In conclusion, we see that the reconstruction of files is sometimes possible. However, compared to some other filesystems (e.g. NTFS), there is a smaller possibility to recover deleted files from the filesystem information. In some special cases where you can prove the integrity of a file in another way (e.g. some packed/zipped files), it is still helpful to take advantage of the inode structure and the possibility to put together fragmented files from the pointers inside the inode.

### **4.5 Forensic Tools supporting QNX6 filesystems**

The Linux kernel includes a read-only driver for QNX6 (and QNX4) file systems. Also, some mobile forensic tools like UFED physical analyzer support this file system to a certain degree. Until today, those tools just support volumes formatted with the standard values shown in Table 4.1. Lately, there have been some projects in the Autopsy / Sleuthkit community to support QNX6, but until today, none of the projects has come to an end.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

 The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Part II Mobile File Formats**

File format analysis examines one specific file. An App or a program typically interprets the data contained in a file. Files can contain user data as well as configuration data, caches or any data. One aim of file format analysis is to recover corrupted files or restore deleted entries from files that only mark entries as deleted but do not overwrite the deleted data.

In this part of the book, the general design of five common file formats in mobile devices is described, and different analysis techniques are presented. This part abstractly approaches the topic and is not limited to how a specific tool analyzes a file format.

File formats are used by Apps or programs and provide mechanisms to store data in a structured way. File formats can organize metadata as well as and data such that the specific App can use this. The described file formats in this part are typically used in mobile devices.

### **Chapter 5 SQLite**

Dirk Pawlaszczyk

**Abstract** SQLite is, without doubt, the most widely used database system worldwide at the moment. The single file database system is used, among other things, in operating systems for cell phones, such as Android, iOS or Symbian OS. On a typical smartphone, we usually find several hundred SQLite databases used by a wide variety of apps. Due to its widespread use, the database format is of particular importance in mobile forensics. It is not uncommon for the suspect to try to cover his tracks by deleting database content. Recovering deleted records from a database presents a special challenge. In this chapter, the on-disk database format of the SQLite database system is highlighted. Therefore, we take a closer look at the database header as well as record structure on a binary level. We first examine the structure of the data. Recovery options for erased records are discussed as well. Special attention is paid to the slack areas within the database: unallocated space, Freelist as well as free blocks. In this context, we discuss basic techniques for carving and acquisition of deleted data artefacts. Despite the main database format and recovery options, temporary file types like write-ahead logs and rollback journals are analyzed as well.

### **5.1 Introduction**

A large amount of data is being stored and processed in relational databases. The most widely used database system in the world is undoubtedly SQLite since it is the default solution for the Android and iOS operating systems. So it is not surprising, that web browsers, messenger services and mobile applications employ the free and serverless database solution as their storage format of choice [61],[60]. At the moment, there are more than a trillion SQLite instances in active use [81]. In the vast majority of criminal investigations involving information technology, one task is to make information stored in such databases accessible. Evidence acquisition for

University of Applied Sciences (Hochschule Mittweida), Technikumplatz 17, 09648 Mittweida, Germany, e-mail<EMAIL_ADDRESS>
databases is traditionally made with SQL, a powerful query language. Also, SQLite supports most of the SQL language commands. In this way, the data can be accessed with one of the freely available viewers. Unfortunately, this form of analysis usually does not allow access to deleted records or temporary data content such as recently added but not committed entries. This creates the need for alternative ways to analyze such databases forensically.

### **5.2 The SQLite File Structure**

SQLite is a single-file database engine, i.e., all tables are managed in only one file on disk. There is no intermediary server process; an application has to communicate with first, for storing data. It does not work this way. Instead, the database can be integrated directly into an application. Therefore, it provides a library and an easy to use programming interface. This fact has significantly contributed to the current spread and popularity of the program. We will discuss the basic structure of a database before turning to the details of carving for data records.

Fig. 5.1: Schematic structure of a SQLite database

Like most structured binary formats, the database file starts with a header part [80]. Its size is exactly 100 bytes. Beyond this, the database file is divided into pages of equal size. The file size is thus always a multiple of the page size. A page number uniquely identifies a single page, whereas the first page has the number one. The default page size usually is 4096 bytes. However, it can be adjusted if necessary to a minimum of 512 bytes and a maximum of 64KB [32]. Of course, the header is part of the first page. In a relational database system, all data is stored in tables. This is also the case with SQLite. In turn, a table is distributed over one or more pages of the database on the binary level (see Fig. 5.1). Each data page again contains one or more records, for precisely one table. To access and acquire all records of a particular table, we must first determine which pages of the database are associated with this table. This information can again be taken from the first page of the database. Besides the header string, this page contains one more piece of information - the database schema. Necessary information such as the root page numbers, column names, and column types of the tables are stored here, in a data structure called *SQLite\_Master Table*. We will discuss the details of this table in sect. 5.2.3. To represent a table and its pages, SQLite uses a balanced tree data structure (B+tree) under the hood. In a B+tree, the raw data elements are stored exclusively in the leaf nodes, while the inner nodes contain only links. Since the maximum size of a page is limited from above, we can gain more space for links or branches in the inner nodes by moving the leaves' data records. Moreover, this limits the height of the tree. Since data elements are normally accessed via the tree's root, a lower height reduces the number of nodes to be traversed. Many relational database systems manage their records in this way.

Table 5.1: SQLite page types and byte flags

A page with links to other pages only is called a *b-tree interior* page [80]. The record nodes are saved in *table b-tree leaf pages*. Beyond this, a table can have multiple indexes. An index contains links to normal table records to speed up searching and sorting by specific fields. Whenever we create an index, SQLite creates a B-tree structure to hold the index data as well. Similar to normal tables we can distinguish between *index b-tree interior pages* as well as *index b-tree leaf pages*. When a data record is too large for a single data leaf page, the excessive bytes are spilt onto socalled *overflow pages*. Several overflow pages are filled at once to store large amounts of data such as Binary Large OBjects (BLOBs). Together all overflow pages for one record form a linked list. To capture all the data associated with a record, we need to read all the pages. The payload for an record and the preceding pointer are combined to form a cell.

Despite the five data page types, SQLite knows three more page classes. A database file might contain one or more pages that are not in active use. Whenever the last record is deleted from a page, this page is released. The freed page will be reused when new pages are required and filled with new table contents. In the meantime, all unallocated pages are stored in a so-called freelist (sect. 5.3). These freelist pages are of particular forensic value since most of the removed content can be found here.

A further not yet discussed page type are so-called *pointer maps*. A pointer map has the function of not losing track when pages are moved from one position in the database file to another. This page type is created whenever the database is reorganized or cleaned up. A pointer map provides a lookup table to quickly determine page types and their parents. However, this page type exists only in autovacuum databases. The *locking page* is the last page type in SQLite. The first page of this page class starts at byte offset 2 30 (1,073,741,824) and always remains unused. Conversely, this means that a locking page only appears when the database size is more extensive the 1 GB. Since it is empty, it has only a technical, but no forensic value and is therefore not considered further.

We can usually determine the type of page by looking at the page's first byte. The flag-byte at offset 0 indicates the page class. Table 5.4 lists all the page types discussed so far. However, not every database will include all of these types. With the page size and type information at hand, an investigator can walk through the database and identify all areas of interest.

### **5.2.1 The Database Header**

Every forensic investigation starts with analysing the file header. The header contains important information that will help us to carve for deleted records. The fields of the header have a precisely defined size and position (see Fig. 5.2). The individual (multi-byte) fields are encoded as big endian (BE) values.

Fig. 5.2: The SQLite Database Header Format and fields

We will discuss the fields below and evaluate them in terms of their respective value for a forensic investigation [80]:

All remaining header bytes are reserved for future expansion. Consequently, we can ignore them.

### **•** > **Important**

As can be seen from what has been said, various header fields must be read and analyzed as the first step of every examination. Thus, the page size (offset 16) and the number of pages (offset 28) must always be determined, since we need to know the structure and size of the database. In order to interpret the strings correctly, the encoding must also be examined (offset 58). A look at the freelist entries at offset 32 and 36 tells us whether unused pages in the database exist. If we do not find any references to free pages, it may be an auto vacuum database (offset 64). Using the flags for transaction management at offset 18 or 19, we can also find out which additional SQLite files may exist. This is of particular interest because these files can also contain records of former transactions. Thus, old states of the database have been overwritten in the meantime could be made visible again. The header's remaining information is more technical and is, therefore, less interesting for the investigator.

### **5.2.2 Storage Classes, Serial Types and Varint-Encoding**

In order to understand the binary format of records we first need to clarify what data types SQLite knows at the binary level and how they are encoded. Like most other databases, SQLite uses strict typing. Therefore, each value stored is mapped to one of the five storage classes (Table 5.2). The word storage class is just another term for a data type. However, the latter is more commonly used in connection with programming languages. SQLite supports storage classes for integers (INTEGER), floating-point numbers (REAL), strings (TEXT), binary objects (BLOB), and other numeric data such as dates (NUMERIC). The storage class thus determines how the binary data is to be interpreted. Conceptually, each column of a table is assigned with a specific affinity. The affinity denotes the preferred storage class for a column. The data type of a column defines what value the column can hold. However, the SQL standard knows several data type names for one SQLite storage class. For example, there exist more than ten different integer data types in SQL. For texts, there exist seven different types. Accordingly, each data type is mapped to exactly one storage class.

A second essential aspect is a length occupied by a cell value. An integer, for example, will consume a length between zero and a maximum of 8 Bytes. A floatingpoint number is mapped to a 64-bit field. A text can have an arbitrary length. SQLite uses the so-called serial types to map storage class and length. In simplified terms, this type is a number. The concrete value of the number provides information about the length of a cell value. At the same time, the storage class can be derived from the numerical value. Table 5.3 lists all possible serial types. For serial types 0, 8, 9, the value is zero bytes in length. The serial type is used whenever the type and length of a cell must be determined. Usually, each table row has a corresponding header that summarizes the serial bytes for each column. As a rule, a serial type occupies exactly one byte. Especially with texts or BLOBs, this principle is sometimes deviated from as soon as the numerical value's length exceeds 127. In this case, additional bytes may be added to map the serial type.

Table 5.2: Mapping from SQL types to SQLite storage classes [80]

SQLite uses a particular encoding for storing serial types. The representation form used is a variable-length integer (varint). SQLite version 3 uses this simple byteoriented encoding where each byte contains 7 bits of the integer being encoded. The most significant bit (MSB) is a flag bit, indicating more bytes to follow. Since most integers in a database have relatively small values, we can keep memory consumption low this way. Storing with a fixed-length integer will mostly generate unnecessarily many null bytes. Instead, SQLite uses a static Huffman encoding of 64-bit twos-complement integers that needs less space for small positive values. The serial type varints for large strings and BLOBs might extend up to nine-byte varints. The following illustration should once again make clear the storage principle of varint-values:

Since texts have a variable size, and the length calculation is performed by a formula. A numerical value above 12 or 13 can only occur with texts or BLOBs. An odd value will be correspondingly for texts. On the other hand, if the value is even, then it is the BLOB storage class. For example, to store the word *Test*, the value 21(0x15) - 2 \* text length + 13 - is stored as the length specification. A JPEG file with, let us say, the length of 109 Bytes would be encoded with the serial type number 230 since 𝑁 ∗ 2 + 12 is what we need to calculate for a binary object. However, since we cannot map this value with 7 bits, we have to add a second byte for the varint:

> decimal: 230 = 128 + 64 + 32 + 4 + 2 binary: 1110 0110 varint: 1000 0001 and 0110 0110 (2-Byte: 1X.. 0X..)

Thus, we must first calculate the respective length specification each time we need to know the exact length of a table cell. The serial values 8 and 9 are noteworthy features. They can be used to map the two values 0 or 1. An extra content byte is not necessary in this case. With the information presented, we are now able to decode the cells of a table row.

### **5.2.3 Decoding The SQLite\_Master Table**

A database schema is a set of data definitions that define the structural design of a database. As already explained, the schema, or the master table, resides on the database's first page, just behind the header. Technically, it is a regular table [85]. Table 5.4 shows all columns and their meaning for the master table. The schema table contains all database objects in the database and the statement used to create each object. With the schema table's help, all table names, the corresponding column names and data types can be determined. Each table entry is opened by two additional fields: the *rowid* and the payload (see Fig. 5.3). Both values are only visible on the

Table 5.3: Serial Type Codes Of The Record Format [80]

binary level. Any row of the master table and therefore every database object is assigned to a unique, non-NULL, signed 64-bit integer - the *rowid*. This value is used as the access key for the data in the underlying B-tree. On the binary level, each table row starts with a rowid number greater than null. Most tables in a typical SQLite database schema are rowid tables. A *rowid table* is defined as any table in an SQLite schema that is not a virtual table and is not a WITHOUT ROWID table. The rowid is not part of the table definition. A payload field that stores the length of the record follows directly after the rowid.

Table 5.4: Structure of the sqlite\_master table [85]

Interestingly, we can find descriptions for tables that have already been removed. If an object in the database is erased, the schema table's corresponding record is marked as removed. If a table is dropped, the rowid value for the line in question is set to 0x0000. The entry that is no longer needed is only overwritten when a new database object is added. In the meantime, the entry is still accessible. Figure 5.3 shows an example of a deleted entry for a table in hex mode. The table header and all columns of the record are intact. Only the rowid value at Offset 3935 has been wiped with zero bytes.

In the example below, the signature 0x7461626C65 represents the object type of a table. The table name, i.e. "users", directly follows the type column. However, we must parse and analyse the corresponding SQL statement from the fifth column to get all column names and the corresponding type information.

Fig. 5.3: Record of a dropped table from the sqlite\_master (example)

By analyzing the SQL statement, a storage class can be derived for each table column. For the five columns of the table <users>, the following columns can be identified: INT, TEXT, TEXT, INT, REAL. This type of vector can be considered as a kind of fingerprint. Sometimes, a found record could be recovered, but it is not clear to which table it belongs. With the help of the table's signature derived in this way, an assignment can still be made, even for a deleted record. Of course, this rule is not always 100% accurate. It is not excluded that two tables have the same signature. However, it can help us make an educated guess, which will be correct in most cases.

### **5.2.4 Page Structure**

All records are stored on pages. Approaching the data of a table requires a leaf page scan. To access the data, we must understand the structure of a page. Each page starts with a header, with a total size of 8 bytes in the case of a data leaf page (see Fig. 5.4). All header bytes are big-endian values. The header starts with the page type at offset 0. In the case of a leaf page, the page starts with the value 0x0D. It can be classified from the other pages by reading this value. The 2-byte value at offset 1 marks the beginning of the first free block on the page. A free block is created whenever a record is deleted from the database. All free blocks are organized as a linked list, whereas the first two bytes of the free block point to the offset of the following free block within the list. If the free block is the last on the chain, this value is zero. If we want to identify deleted records, our search should start right here in the free block list [80].

Fig. 5.4: Fields of a b-tree leaf page header

Another essential value is located directly behind the free block field at offset 3. The 16-bit twos-complement integer field is called *number of cells*. Its value indicates how many active cells exist within the current page. In SQLite, the serial type header and the values of a particular table row are combined into a structure called "cell". So if we want to access a record, we need to locate the matching cell. Fortunately, all cell offsets are stored in an array directly after the page header. Hence, to read a regular record of a table, we need to iterate through the cell pointer field.

Fig. 5.5: Structure of a regular data leaf page (permanent and temporary)

The next header field at offset 5 provides the start-offset of the content area. A b-tree leaf page is divided into regions (see Fig. 5.5). The cell content area is always located at the bottom of the page. The header and the cell pointer array are always located at the beginning of the page. Between them resides the unallocated space. As the content area grows from the highest memory address towards the lower address, overlapping the two mentioned regions is prevented. The concept is thus similar to the management of heap and stack areas within memory management. The last value in the header denotes the number of fragmented bytes. A free block requires at least 4 bytes of space. Areas between 1 to 3 bytes form a fragment and thus cannot hold any data records.

Figure 5.6 shows an example of the header of a page on a binary level. In addition to 15 cells, we can also find at least one free block of offset 3620(0x0E24). The content area in this example starts at 0x0DEC. The cell pointer array is highlighted in yellow. Interestingly, we can find five more cell pointers shown in red. The value of the surplus cell offsets corresponds to the start offset of the cell content area. From this, we can conclude that apparently, five other records must have existed on the page in the past. Nevertheless, they have been deleted in the meantime. Thus, in addition to the 15 regular records, there should be five more deleted records on the page. However, the deletion turned the cells into free blocks. So, to find and restore them, we need to examine each element of the free block list.

Fig. 5.6: Sample header and cell content array for a data leaf page

It is not always possible to find all deleted records by checking the free blocks. If a record is deleted that resides directly at the unallocated area, the offset value for the cell content area start is moved up in the direction to a higher address. Of course, this address denotes the cell pointer offset of the next regular record. The data set is thus moved to the unallocated area by changing the border. We must consider this case in our search since this record will never appear in the free block list.

However, it gets even worse. If a complete page is deleted, SQLite typically wipes the first 4 Bytes of the header with zeros. So, in this case, the offset for the first free block is erased. Thus, we do not know where precisely the list begins. What does this, in turn, mean for our search for hidden records? The best way to approach our search for slack areas is to use the exclusion principle. *Slack space* is the leftover storage that exists on a page when records do not need all the space which has been allocated. Slack areas are always created when records are deleted. Hence, the total amount of slack space can thus be calculated as shown in the equation below.

```
slack space with (possible) deleted content = page content
                       - header (8 bytes)
                       - N times 2-Byte cell pointer
                       - fragmented bytes
                       - N times cell
```
If we exclude the regular, well-known areas of the page, we automatically access the slack areas. Only the areas determined in this way can contain deleted data artefacts. In any case, we must always consider the unallocated space and the free block list when searching within the page. Fortunately, leaf pages are always structured the same. However, there is a second type of leaf page, the index leaf page. In its structure, this page corresponds to a regular data leaf page, except for one difference. The index leaf page starts with the value 0x0A at offset 0. However, what has been said so far also remains valid for the second type of page.

### **5.2.5 Recovering Data Records**

Now that we know the location of the records, we can start reading them. This information can be derived from the cell offset array (see the last section). Every cell has the same structure (see Fig. 5.7). The cell header opens with a payload value. It indicates the total size of the cell in bytes. This value does not include the cell header itself. Normally, the payload field is followed by the rowid (see sect. 5.2.3). As already explained, the pseudo-column is usually generated automatically by SQLite. It is used to enable efficient access via the table tree. However, not all records have a rowid. For example, index records are created without this field. If the option "WITHOUT ROWID" is part of the CREATE TABLE statement, this field is also missing. Thus, the cell header has a minimum size of 1 byte for a mandatory payload value. The values in the cell header and all other header fields are varint values without a fixed size. So to read a record, we always have to read value by value. Skipping or omitting bytes is not possible because the fields do not have a fixed offset. The actual cell starts again with a header. This time, it is the header of the data record.

The *header size* field indicates how many bytes the header contains. Its value includes the actual header size byte. The individual serial types follow immediately. Column by column, we must first determine the storage class and space for each table cell. The header is followed directly by the actual data record. Since we operate on a binary level, the exact length of each field to be read and the data types can only be determined via the serial bytes in the header. However, it might be challenging to determine the exact beginning or end of the column cell values without this information. An intact header is, therefore, an essential prerequisite for successful data recovery.

### **•** > **Information**

The recovery of deleted data depends on the data management policy used. This, of course, differs from application to application. We can distinguish three cases:

In the case of a data record that has been deleted, it sometimes happens that the cell header and parts of the record header are replaced with new information [59]. These new data fields cover the free block's length in bytes and the address of the following free block. Since both pieces of information are mapped to a 16-bit fixed-length integer, a total of four bytes of the respective cell are overwritten. In total, we can discern six situations when dealing with a deleted record (see Table 5.5).

Many records are deleted without being marked or overwritten. As explained earlier, some records are deleted by merely moving the cell content area's border upwards. Thus, the records slip into the unallocated area of the page. When clearing the browser cache, for example, almost all entries are removed from a caching table. Instead of first marking each record as deleted, the links to the affected pages are deleted from the table tree. Anything else would be a time-consuming process. Instead, the page as a whole is skipped. In both cases, however, the deleted records remain intact. Complete reconstruction is, therefore, possible. Sometimes a record is removed from the middle of the content area of an active page. In this case, the record

Fig. 5.7: Schematic structure of a data leaf cell

Table 5.5: Recovery Situations

is converted to a free block. Thus, the beginning is overwritten, at least partially. The previously occupied space will be released for reallocation. This, in turn, can result in different cases that influence the recoverability of the data record. Sometimes only the payload got wiped. In another case, the payload field, together with rowid, may be overwritten. We can mostly do without this field information. As long as the rest of the cell record remains intact, we can read the required column lengths and types and correctly interpret the data. Even a wiped header length field should not be a big problem. This field only holds the total length of the header. It can be reconstructed by summing the individual serial lengths. It gets tricky when columns are also overwritten. Without a valid column type and length specification for our first column, we cannot reconstruct the remaining columns correctly. However, the first column of a table is often an ID column with a numerical value. Knowing the length of the first column of a regular record on the same page can indirectly infer the first column's length for our destroyed record. Unfortunately, this rule does not work in every case. For example, if the first column contains a text with variable length, we will most likely not restore the record correctly. If more than one serial type has been overwritten, reconstruction seems unlikely. We then have too many possible lengths to consider. Strictly speaking, the number of possible lengths for a column grows exponentially with the number of overwritten length or type information in the header.

Figure 5.8 shows the content area of a data leaf page. There are a total of three records on the page. The cells are located at the end of the page. Remember, the cell content area always grows from higher towards the lower address. The record in the middle is deleted. The records before and after it are intact. Cell header, record header and all data are unaltered. Even without knowledge of the table, it can be deduced from the serial types alone that it is a table with apparently two columns. The first column can store integers (serial types 0𝑥02 resp. 0𝑥03). The second column is a string since the value is odd and greater than 13 (see sect. 5.2.2).

We can see that the second of the three data cells have been deleted because the first 4 bytes of the data set have been overwritten with the free block identifier. The identifier is 0𝑥0000000𝐶. The first two bytes have the value 0𝑥0000. From this, we can conclude that it is the last free block within the page. The second half of the identifier tells us something about the length of the free block. It is exactly 12 bytes (0𝑥000𝐶). The free block is outlined in red in the illustration. As we can see, the actual

Fig. 5.8: Example data page with three records (one is wiped)

data fields of the deleted record are still intact. However, the PPL-field, ROWID, header length byte, and the first column's serial type are no longer accessible. The serial type of the second column is not wiped. From the length specification of the free block and the knowledge about the length of the second column, we can infer the length of the first column in this case. Accordingly, the first column of our data set can only be 2 bytes in size:

Thus, we can recover deleted content in many cases, even when parts of the header have been overwritten.

### **5.3 Accessing The Freelist**

As soon as the last record on a page is deleted, it is transferred to the free list. At the same time, the link within the table tree is removed. From now, the page cannot be accessed from an active table. However, it can be assigned to a new table at any time. Meanwhile, the content of the page is still accessible. Usually, it is not wiped or replaced with random values. The pages are just sitting on the free list, waiting to be used again. Like the slack areas in the standard database pages, these unused pages may contain forensically exciting values such as chat protocols, short messages, or web pages visited [61].

The freelist is a simple linked list consisting of *trunk pages* 5.9. Each trunk page initially contains a 4-byte integer pointer referencing to the next trunk page in the list

Fig. 5.9: Schematic principle of a freepage trunk list

[80]. A zero-byte value means that this is the last trunk page in the list, and the list ends here. The second 4-byte value in a trunk page contains the number of leaf page offsets. To analyse all the list pages, we must first visit each trunk page and query the offsets stored. Nevertheless, where do we have to start our search for freelist treasures?

The starting address for the freelist can be calculated very easily [59]. We must first determine the start offset of the first trunk page from the header at offset 32 of the database. Second, we need the page size. The latter can also be determined from the header. From these two values, we can calculate the actual offset of the first trunk page:

$$\text{\"offset\" } \text{of } \text{\" 1st\" } \text{\"truk\" } \text{\"page = \"(\text{\"trunk page\" } \text{number} - 1)} \text{ \* \" page\" \"size.}$$

A trunk page consists of an array of 4-byte big-endian integers. As pointed out, the first 4 bytes of the trunk page header references the next trunk page within the list. The next, a four-byte big-endian integer holds the length of the leaf pointer array of the current page. With these two pieces of information at hand, we can quickly iterate over the array's entries.

The basic algorithm is shown in Listing 1. An example of a trunk page will illustrate what has been said so far (see Fig. 5.10). In addition to the reference to the next trunk page at offset 0, the number of page pointers to follow is visible (offset 4). The offset of the first free page can be found directly behind the two header integers at Offset 8. The second pointer is exactly 4 bytes behind. In the example, there are a total of 555 entries on the TrunkList page. The data size is therefore 8 + 555 \* 4 = 2228 bytes. Thus, all unused pages can be found and accessed with linear time complexity with the described algorithm.

Fig. 5.10: start of a freelist trunk page (example)

### **5.4 More Artefacts**

As explained earlier, SQLite manages all records in a single database file. However, access management, transaction handling, and integrity protection are performed with the help of primarily additional temporary files [84]. Despite the main database file, SQLite uses nine distinct types of temporary files (see Fig. 5.11). Below we will take a look at the other file types of SQLite. The focus is on searching for records no longer in the regular database but can still be found in one of those files.

### **5.4.1 Temporary File Types**

SQLite creates several temporary files when managing the database. A *transient database*, for example, is a temporarily created file when the database is reorganized.

### **Algorithm 1** Freelist Page Recovery

Data pages that are no longer required are removed. The whole process is comparable to the defragmentation of a hard disk. Pages are joined together, and gaps are closed. Then, the temporary file's content is copied back into the original database file, and the temporary file is deleted. However, this file type is generated only for databases for which the VACUUM property is activated. Since the database copy is deleted immediately afterwards, it is not easy to locate it on the disk. However, it might be possible to find old page versions of the database on the medium through carving. From time to time, SQLite makes use of *transient indices*. Each index is therefore stored in a separate temporary file. For example, if the ORDER-BY or GROUP-BY clause is used in an SQL statement, a corresponding index file is created to manage the intermediate results. The index is automatically deleted at the end of the statement that uses it.

In the case of complex SQL statements, partial queries are sometimes stored in a temporary file. In SQLite, this method is called "materializing" the subquery. This is the case, for example, with large SQL INNER JOIN statements. The query optimizer decides for which query a separate swap file is created.

Database users can create a temporary table using the "CREATE TEMP TABLE" command. Since this unique table is created only for a particular database connection and is not visible to other database users, it is swapped out to a separate file. Again, the temporary database file used to store temporary tables is removed automatically when the database connection is closed. When SQLite performs a transaction with multiple statements, a *Statement Journal File* can be used to undo individual steps. Assume that by executing a statement, 100 rows of a table are modified. After half of the records have been modified, the execution must be aborted due to an error. The rows of the database that have been modified so far are written back with the statement journal's help. All five of the temporary file formats discussed can contain data or temporary results of the database transactions. However, these data are highly volatile. In most cases, the temporarily stored results are already deleted when the statement is finished. Thus, it is not very likely for an investigator to come into contact with such artefacts. We will, therefore, not consider them further.

There are four remaining file types in SQLite. Unlike the formats discussed so far, these are files that are often encountered when examining a database. These files are *Rollback Journals*, *Write-ahead Logs*, *Shared-Memory Files* as well as *Super Journals*. They can usually be found in the same directory as the actual database file. Admittedly, the data stored in it is also classified only temporary within the official documentation of SQLite. However, the data stored in them is updated or overwritten much less frequently. We almost always find one of these file types. For this reason, these are also listed under the heading *other permanent files* in Fig. 5.11. Thus, the chance to acquire data from these files is much more likely. However, in some cases, the use of one file format excludes the use of the second. For example, the shared memory file and write-ahead log are usually found together. In contrast, the rollback journal is only found in a directory if the first-mentioned files are absent. Of the file formats mentioned above, super journals are relatively rare. The files are created only in transactions where multiple databases are updated simultaneously in an atomic transaction. Accordingly, without a super-journal in place, transaction commit on

Fig. 5.11: The SQLite file types (permanent and temporary)

a multi-database transaction would be atomic for each database individually, but it would not be atomic across all databases. Due to the relatively low usage level, we will not take a closer look at this file format. Instead, we will focus on the two remaining journal formats. Besides availability and confidentiality, data integrity forms a central goal of every database system. SQLite is no exception. SQLite maintains its integrity by using journals and transactions. Below we will examine the two integrity protection techniques offered by SQLite in more detail: Write-ahead logs and Rollback Journals [80],[32].

### **5.4.2 Rollback Journals**

The idea behind the rollbacks is simple: If a database gets into an inconsistent state due to write access, it is reset to the last valid state. To implement atomic commit and rollback capabilities, SQLite offers a file called *rollback journal*. Rollback refers to resetting the individual processing steps of a database transaction [79]. The system is thus wholly returned to the state before the start of the transaction. In the case of SQLite, a copy is first created for all database pages possibly affected by the transaction and stored in the rollback journal. If something goes wrong during transaction processing, the database can always be reset to the last valid state if required. Note: SQLite permanently stores the entire page in the journal file, even if the transaction modifies only a single record.

A journal file is usually created when a new transaction is started and deleted after the transaction is completed. Although this is the default behaviour, in many cases, there is a deviation from this approach. For example, if the application developer activates the *exclusive locking mode* for a database, then the rollback journal is not immediately deleted. An application can enable the exclusive locking mode by using the following pragma-statement:

```
PRAGMA locking_mode=EXCLUSIVE;
```
In this case, the journal file may be truncated, or the file's header may be wiped with zero bytes. Which behaviour of this occurs depends on the SQLite version used. However, the file is preserved in any case as long as the locking mode is activated. Fortunately, many applications that use rollback journals for transaction safety operate in this mode, reducing unnecessary IO operations. The same behaviour as is seen in EXCLUSIVE locking mode can also be reached by setting the *journal mode pragma* to PERSIST instead of DELETE which is the default behaviour in SQLite:

```
PRAGMA journal_mode=PERSIST;
```
No matter which of the two modes is activated, an investigator can restore the old execution states of the database. In this way, data records that may have been deleted in the meantime can be made visible again.

### **• ! Attention**

The rollback journal file is always located in the same directory as the actual database. One can quickly identify the journal by the file name: It has the same name as the database but with the extension **"-journal"**. Thus, the name of a journal file is precisely eight characters longer than the original name of the database [84].

A rollback journal is a binary format. Just like the main database file, it contains a small header. The header has a fixed size of a maximum of 28 bytes. The individual header fields and their meanings are shown in Table 5.6. Next to the Magic Header String, information about the total number of database pages stored in the journal. The header also records the original size of the database file. So if a change causes the database file to grow, we will still know the original size of the database. Unfortunately, the fields carried in the header are usually automatically overwritten after a COMMIT and wiped with null bytes. Thus, we will rarely be able to recover useful information from it. However, the header is usually preserved if a transaction cannot be completed due to a power down.

The journal file has a preset page size. The value can be determined via the offset 20 in the header. Even if this value can no longer be determined due to wiping, there is a way out. The default value of the first sector is 512. The remaining space of the first journal page is filled with zero bytes. Since the default page size is 512 bytes, the header is thus always followed by a padding area of zero bytes. After the header and padding area, zero or more page records will follow. Such a record contains a copy of precisely one database page. Additionally, each record is introduced by a one-field header. Only with this value, SQLite can reset the correct page in the database in case of a rollback. On offset four, the original content of the database

#### Table 5.6: Rollback Journal Header Format

page follows. The journal page record ends again with a 4-byte big-endian value. It holds the checksum for this page. The value is used to guard against incomplete write operations.

Table 5.7: Rollback Journal Page Record Format [80]

Since the header is always reset for each new transaction, the page records directly following the header are always the most current. However, journal records of past transactions can still be stored in the same journal. For example, suppose a transaction changed ten database pages. The following transaction only rewrote five pages. In that case, the database subsequently contains the database's state before the last transaction plus five more pages from the previous. The following example shows the beginning of the second journal page of a rollback file:

```
0xBAC4E54E -> Checksum of the 1st journal records
0x0000000B -> page 11 in the database (start of the 2nd journal)
0x0D000000 -> start of a data leaf page (snapshot)
```
The start of 2nd journal record can be calculated as follows:

The example shows the end of the first journal page and the beginning of the second journal frame. While the green highlighted value at offset 0x1204 still belongs to the first journal page, the value at offset 0x1208 already initiates the next journal record. Generically, the address of each journal could be determined as follows:

```
Record𝑠𝑡𝑎𝑟 𝑡(N+1) = size of 1st sector + N × (page size + 8)
```
However, how can we determine whether the database's journal page belongs to the last transaction or is not perhaps older? A different random nonce is used each time a transaction is started to minimize the risk that unwritten sectors might by chance contain data from the same page that was a part of prior journals. The last nonce is a 4 Byte integer value and can be found at offset 12 in the journal header. By changing the nonce for each transaction, stale data will still generate an incorrect checksum. Since the entire page is always saved from the database, we can restore the actual data described in section 5.2.5.

### **5.4.3 Write-Ahead Logs**

As pointed out in the last section, a copy of the data page to be changed is first created before writing directly into the database file in a classic rollback journal [86]. Version 3.7.0 of the SQLite database engine introduced an alternative concept for transaction management [84]. With *write-ahead logs (WAL)*, this procedure is reversed. The content of the original database file is not changed. Instead, every change is appended into a separate WAL file. It works like a roll-forward journal. All changes are first written to the WAL file. Even a COMMIT does not automatically update the database file [79]. If, for example, other reading database connections exist simultaneously, they can operate as usual on the original unaltered data. Meanwhile, a concurrently running write process stores its changes into the WAL file. Moving the WAL file transactions back into the database is called a *checkpoint*. Usually, SQLite does a checkpoint automatically. If the WAL size reaches a threshold size of 1000 pages, a checkpoint is triggered by default. As soon as we examine a database that works in WAL mode, we must also analyse the included WAL archive. Simultaneously, this also means that we may have different versions of the same database page in the main database and the WAL file. As long as no checkpoint has been carried out, the WAL file exclusively contains the latest changes. The database is, therefore, still in an old state. If we look at both files together, we can get a consistent view [86].

To access the content of a WAL file, all we have to do is open the corresponding database file. When opening a WAL mode database, the WAL file's content is automatically transferred back to the database. In other words, a checkpoint is executed. However, this procedure is usually not recommended for various reasons. With this approach, old artefacts that are evidentially valuable to the investigator could be overwritten and thus lost. Moreover, we would be violating a fundamental rule of any forensic investigation: Never change the evidence.

### **•** > **Important**

It is best not to work with a standard database viewer when evaluating a database in WAL mode. Even by opening the database, one risks losing old data due to checkpointing.

But how should we proceed then? One possibility is the use of a special forensic database browser. An example would be the FQLite1 browser. This program reads the database and the WAL file separately. Since access is read-only, all data is preserved.

### **• ! Attention**

A particular database will use either a rollback journal or a write-ahead log. It is not possible to use both at the same time. The write-ahead log is always located in the same directory as the actual database. One can quickly identify the journal by the file name: It has the same name as the database but with the extension **"-wal"**.

Let us now turn to the actual structure of the file. The WAL file starts with a header. Zero or more so-called WAL-frames follow it. Just as with the rollback journal, a frame represents the altered content of exactly one page of the database. The file header has a size of exactly 32 bytes. It starts with a 4 byte long Magic Number (see Table 5.8). At offset 4 follows the file format version. Again, this is a 4-byte unsigned integer value. The size of one page of the database is stored at offset 8. Using the field *checkpoint sequence number* at offset 12, we can again determine how many checkpoints have already been executed since their creation.

The last fields of the header form two salt values and two checksum values. Using these fields, we can determine which frames belong to the current checkpoint and

1 https://github.com/pawlaszczyk/fqlite

have not yet been transferred to the database. Figure 5.12 shows an example of the header of a WAL archive in FQLite.

Each WAL frame also starts with a header [84]. The structure of the header with its fields is shown in Table 5.8. The header consists of exactly six big-endian values, each with a size of 4 bytes. The first object is the page number this frame is assigned. Using the page number, we can identify the place in the database where the change takes effect. The value at offset four can be used to determine whether a COMMIT was performed. A value other than 0 is a so-called *commit frame*. Let us remember that a COMMIT does not automatically update the database. Like the header of the WAL file, each frame header ends with two salt values and two checksums. The four big-endian 32-bit unsigned integer values are located from Offset 8 to 24.

Fig. 5.12: View on a WhatsApp-DB WAL-Header with FQLite Carving Tool

A WAL archive always grows from the beginning. It can cause frames from different checkpoints to appear in the same file, whereas current ones are always at the file's beginning. Fortunately, we can use the mentioned salt values to determine relatively quickly whether the frame under consideration is valid or whether it belongs to an older state already transferred to the database. Whether a frame is valid can be determined as follows [86]:

If a checkpoint was executed successfully, the WAL file is reset afterwards. In this case, the salt values are overwritten. The value of salt-1 is incremented, while a

Table 5.9: WAL Frame Header Format [86]

new random value is assigned to salt-2. Previously valid frames are automatically discarded due to this procedure. However, the previous frames usually remain in the archive due to the I/O- operations when the file is truncated. Thus, there is an excellent chance to make past states of database pages visible again with the help of the WAL file.

Let us take a look at how write-ahead logs work. Figure 5.13 shows the frames list of a WAL file. Below the header field for the Salt-1 value, several frames are shown. All frames with matching salt values belong to the same checkpoint. The first seven frames thus form a unit. The remaining frames are part of an older checkpoint. As we can see, the salt in the header matches the salt in the first unit. Accordingly, the pages have not yet been transferred to the database. In other words, the WAL file contains the latest version of page 2,4,6,18. The pages within the database are out of date. The next checkpoint is usually executed when opening the database, and these data records are transferred to the database. Since WAL files always work at a page level, the complete database page is updated. Remember, the salt value changes for each checkpoint. Thus the Salt-1 field in the header is discarded afterwards.

Interestingly, page 6 has been updated three times. When a checkpoint occurs, each page will be written back to the database in the same order written to the WAL file. Pages are written from the start of the WAL file. Accordingly, the update order would be 2,6,4,12,6,18,6. This allows a timeline to be created, starting with the first to the last update step.

### **5.5 Conclusions**

The SQLite database format has great importance in the field of mobile forensics. In this chapter, we have therefore tried to take a look behind the scenes. As quickly became apparent, the file format of SQLite has some similarities to a classic file system, where files are usually stored in blocks. Instead of blocks or clusters, data content in SQLite is managed in pages. As has been shown, even records are often recoverable after they have been deleted. Analogous to a file system, these are usually

Fig. 5.13: Frame list of a WAL file (example)

not wiped but merely marked as deleted. However, we do not manage files but data sets.

We further identified different slack spaces of an SQLite database. Besides free blocks and the unallocated space, we can find deleted records, especially in the freelist area of the database. The carving techniques discussed within this chapter can help make these data sets visible again in many cases.

Of the temporary file-formats considered, rollback journals and the WAL files are of particular interest to the investigator, as they may contain old or previously altered data. However, special care must be taken when acquiring data from these files. Thus, the data stored in a WAL file can be reconstructed manually or with specialized forensic tools. Using an ordinary SQLite reader, on the other hand, can lead to the loss of data.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

 The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Chapter 6 Property Lists**

Christian Hummert and Georgina Louise Humphries

**Abstract** Property List files (\*.plist) are a widely used data storage format used by Apple software. Most of the system properties are stored in plists, but also, many apps store their configuration in plist-files. The data held within Property is regularly of high evidential value for forensic analysts, so understanding the format is essential for the forensic investigation of Apple mobile devices and computers. Not all of today's digital forensics tools recover plists properly. Especially for carved or damaged plists, the support is insufficient. So the forensic examiner must understand the principles of this file format. This chapter gives an overview of the plist structure to give the examiner the knowledge to get the most information out of the evidence possible.

### **6.1 Introduction**

Property List files (\*.plist) are one of the widely used data storage formats used by Apple software . Most of the system properties are stored in plists (many of them are located in /Library/Preferences/), but many apps store their configuration in plist-files. Therefore, property lists can be found in various places on Apple systems. They sometimes can even be found on devices other than Apple operating systems (especially if other Apple software like Safari or iTunes is installed). The data held within Property is regularly of high evidential value for forensic analysts, so understanding the format is essential for the forensic investigation of Apple mobile devices and computers.

Georgina Louise Humphries

Christian Hummert

Agentur für Innvation in der Cybersicherheit, Halle, Germany, e-mail: hummert@cyberagentur. de

Politihøgskolen - Norwegian Police University College Department of Postgraduate Studies, Oslo, Norway, e-mail<EMAIL_ADDRESS>
Property lists offer a structured and efficient way to represent and persist hierarchies of objects to disk. They are the standard way to save and load data between the internal representation within objects in Objective-C or Swift programs and disk files. The standard objects of the Cocoa framework have built-in methods to deal with plist files.

The first plists were developed with the NeXTSTEP operating system. NeXT-STEP is a discontinued operating system that merges the Mach kernel and the BSD kernel. It was developed by NeXT, a company founded by Steve Jobs in 1985, and released in 1989. NeXTSTEP had a text-based, human-readable format for plists, serialized to ASCII in a syntax somewhat like a programming language. Apple replaced the format with an XML-based format and also introduced the binary plist format. Since Mac OS X 10.7, in addition, JSON notation can be read and written. So, there are four different property list formats[9]:

The formats except the binary plist have the advantage of being human-readable. In contrast, the binary plist offers the most efficient representation on disk and fast serialization/deserialization. OS X offers the plutil utility (introduced in OS X 10.2) to check the syntax of property lists or convert a property list file from one format to another. It also supports converting plists to Objective-C or Swift object literals. Another tool that comes with OS X is PlistBuddy (it can be found at /usr/libexec). PlistBuddy allows to merge plists or edit their content.

Property Lists in XML or JSON notation can be easily edited and evaluated in any desired text editor. Apples IDE Xcode also contains a hierarchical viewer and editor for binary and XML plists. In addition, Apple offers an Apple Script interface to create, edit and write property lists (since OS X 10.5). Due to the human-readable notation of NeXTSTEP, XML and JSON property lists, they are not an obstacle for forensic investigates. Therefore, this chapter concentrates on the binary plists (bplist) format and will afterwards describe some of the interfaces to plist files.

### **6.2 Binary plist Structure**

Apple disclosed the structure of the binary property list format; it is documented in the comments of the Apple-provided open-source CFBinaryPList.c1 and declarations of the ForFoundationOnly.h2. Every binary plist file comprises four sections: a header, an object table, an offset table and a trailer (compare table 6.1).

Each bplist file begins with an 6-byte header, containing the magic bplist (Hex: 0x62706C697374). The header is followed by a 2-bye version. The most common

1 https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c

2 https://opensource.apple.com/source/CF/CF-550/ForFoundationOnly.h

Table 6.1: Structure of a bplist file.

version on Apple devices is 00, but there are at least two other versions of binary property lists, too; bplist15 or bplist16 occur. Unlike for bplist00, there is no documentation for either format. The bplist15 format appears to be internal to CoreFoundation. The bplist16 format is internal to Foundation, too, and is used almost exclusively in Objective-C remoting over XPC. The format of bplist16 is similar, but not compatible with bplist00, noting the following differences: Files in format version 16 do not have a trailer, and the items start directly at the head of the property list, right after the bplist16 magic, and are packed (not aligned). In addition, in bplist16 there are more data types available.[47]

The bplist file ends with a 32-byte long trailer. The structure of the trailer is shown in table 6.2. The bytes 0 to 4 of the trailer are unused. Byte 5 contains the sort version. Byte 6 stores the information of the size in byte of each offset entry in the offset table. Similarly, byte 7 stores the information of the size of each object reference in a container. At offset 0x8, there is an 8-byte entry that saves the number of objects that are encoded inside the object table. The following 8 bytes save the offset of the first offset in the offset table (usually zero). The last 8 bytes of the trailer denotes the start of the offset table, counting from the start of the bplist.

Table 6.2: Structure of the bplist trailer.

The second section in every bplist file is the object table. The object table contains all the data objects of the plist. All object types are identified by a single byte, also called a marker (compare Table 6.3). This byte encodes the type of an object and the size of the data.

Table 6.3: Format of object types.

The marker is the binary representation of a single byte. All other objects can be uniquely identified by the marker byte's 4 most significant bits (MSB). At the same time, the least significant bits (LSB) of the marker byte denotes sizing information. If the object size is small enough, the size is encoded immediately in the 4 right-most bits, and then the actual data values follow. If the object size is larger, the LSB matches 0xF (1111), denoting that the next bytes encode size information before the actual value bytes.

The size is encoded as follows: The MSB equals 0x1 (0001), and the LSB contains a value 𝑥. The size will be stored in the following 2 𝑥 bytes in big-endian.

For example: let us assume the object table contains the sequence 0x5F 10 19. The first 0x5F is converted into its binary representation 0101 1111. The MSB is 0101, so the object denotes a string. The LSB matches 1111, so the size is encoded in the next byte. The next byte is 0x10 = 0001 0000. The MSB equals 0x1, and the LSB shows that 2 0 = 1 byte follows, which stores the size of the string. The next byte is 0x19 resulting in a 25-byte long string.

Markers corresponding to objects such as ints, real numbers, strings are immediately followed by a multibyte sequence representing their actual values. This is not always the case, though. In the case of object containers, such as arrays and dictionaries, the marker byte is followed by object references that are simply offset to the offset table. The length of this offsets is determined in the bplist trailer, and are counted from the beginning of the offset table. Therefore, a container element is just a reference that points back to a position in the offset table, which points back to the object table and specifically to a marker corresponding to the individual object. This technique flat-maps the actual multi-level hierarchy and allows all objects to have fixed sizes.

The third section in bplists contains offsets to the object table and guides to the actual values of objects. The size of each offset is defined in the file trailer. All offsets are calculated from the beginning of the file (not the end of the header). The number of offsets stored in the offset table is also given in the trailer.[40]

### **6.3 Example**

Given is the following plist (Table 6.4) from a MacBook Pro:

Table 6.4: Example plist (object table colored in blue, offset table colored in red, trailer colored in yellow).

62 70 6C 69 73 74 30 30 D2 01 02 03 04 5E 42 61 b p l i s t 0 0 " ˆ B a 74 65 72 79 48 69 73 74 6F 72 79 5F 10 13 54 t t e r y H i s t o r y \_ T 6F 74 61 6C 4E 75 6D 62 65 72 6F 66 45 76 65 6E o t a l N u m b e r O f E v e n 73 09 10 0A 08 0D 1C 32 33 00 00 00 00 00 00 t s - 2 3 01 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00 35 5

The given bplist is version 00 as the header states. To analyze the bplist in a first step the trailer is marked (here yellow) the trailer comprises the last 32 bytes of the file. Now the trailer can be decoded (result in Table 6.5):

Table 6.5: Decoded example bplist trailer.

Now, it is clear that the bplist contains five objects in the object table, and the offset table starts at 0x35, whereas the first object-offset starts at 0x35 + 0x00, and each offset has the size of one single byte. In consequence, the offsets from the offset table are 0x08, 0x0D, 0x1C, 0x32 and 0x33, which leads to the following four objects from the object table:

The first object starts with 0xD2, which is 1101 0010 in binary. The MSB (1101) shows that the object-type is a dictionary. The LSB (0010 = 2) shows that the dictionary has two entries. The data 0x01 02 03 04 has to be interpreted as object references that are simply offsets to the offset table. Before interpreting the dictionary as an object container, the other four entries should be decoded.

The second object starts with 0x5E, which is 0101 1110 in binary. The MSB (0101) shows that the object type is a string. The LSB (1110 = 14) shows that the string is 14 chars long. The content of the string is "BatteryHistory".

The third object starts with 0x5F, which is 0101 1111 in binary. The MSAB (0101) shows that the object is another string. The LSB (1111) shows that the string is longer than 14 chars, and the size is encoded in the following bytes. The next byte is 0x10, which is 0001 0000 in binary. The MSB (0001) is defined as 0001, and the LSB (0000) shows that the following 2 0 = 1 bytes encode the length of the string size. So the next byte has to be decoded. It is 0x13, which states that the string has a length of 0x13 = 19 chars. The content of the string is "TotalNumberOfEvents".

The fourth object starts with 0x09, which is 0000 1001 in binary. The MSB (0000) indicates the object as bool, and the LSB (1001) indicates the content as "true".

The fifth object starts with 0x10, which is 0001 0000 in binary. The MSB (0001) indicates the object as an integer. The LSB indicates that the integer is 2 0 = 1 byte long. The content of the integer is 0x0A which is 10. Now the dictionary (the first object) can be decoded. The dictionary has two entries: the objects at offset 0x01 and 0x02 in the offset table, which are the two strings. The first entry is connected to the object on offset 0x03, which is the boolean. The second entry is connected to the object on offset 0x04, which is the integer. That gives the following result (Table 6.6):

Table 6.6: Decoded example bplist.

Fig. 6.1 shows the same plist file decoded with Apples XCode IDE and confirms the correct decoding.

#### 6 Property Lists 163

Fig. 6.1: Illustration of the elements of a block group.

### **6.4 Forensic Tools Supporting plists**

There is quite a bunch of tools supporting the decoding of plist files. Most of the tools support binary plists as well as XML property lists. As the property lists are in an Apple format, the macOS universe gives the best support. If the given property list file is in XML format, it can be edited in any text editor. On the other hand, if the given property list file is in the binary format, it can be converted to XML first by running on the macOS shell:

plutil -convert xml1 file.plist

If an XML property list should be converted back this is possible with:

```
plutil -convert binary1 file.plist
```
A more convenient way to edit and browse plist files is to install the Apple development platform Xcode. The suite includes a graphical editor which is easy to use (compare Fig. 6.2).

For old versions (Xcode 4.2 and earlier) there was a separate application for editing property lists (/Developer/Applications/Utilities/Property List Editor.app/).

Nevertheless, despite the Apple world, there are plenty of alternatives. One free plist parser for binary property lists is binplist 3 a parser module written in Python. For forensic use, it is possible to create an instance of the BinaryPlist class and then call the Parse() method, with a file-like object as an argument.

```
with open("myfile.plist", "rb") as fd:
    bplist = BinaryPlist(fd)
    top_level_object = bplist.Parse(fd)
```
The *Parse()* method returns the top-level object, just as readPlist. Once parsed, BinaryPlist. is\_corrupt can be checked to recognize whether the plist had corrupt data. This allows to maybe decode the corrupted data manually and gather the maximum information on the plist even when they are corrupt.

3 https://github.com/google/binplist

Fig. 6.2: View of an example plist in the Xcode editor.

Another Python module dealing with binary property lists is ccl-bplist4. A C-library to handle plists is libplist5.

Forensic Suites on Windows create a different picture. The MSAB Forensic suite6 offers property list decoding as well as the Cellebrite7 suite. Oxygen Forensics8 offers the Oxygen Forensic Plist Viewer that automatically unpacks one plist file and displays its content as a folder tree (compare Fig. 6.3). The entries can be converted to different encodings. In addition, the binary content of cells, such as images, video and sound, can be decoded and visualized. The Forensic Toolkit (FTK)9 does support plist decoding, whereas the Encase Forensic Software10 offers a plugin script to decode these files.

4 https://code.google.com/archive/p/ccl-bplist/

5 https://github.com/JonathanBeck/libplist

6 https://www.msab.com/

7 https://www.cellebrite.com/

8 www.oxygen-forensic.com

9 https://accessdata.com/products-services/forensic-toolkit-ftk

10 https://security.opentext.com/encase-forensic

#### 6 Property Lists 165

Fig. 6.3: View of an example plist in the Oxygen Forensic Plist Viewer.

### **6.5 Conclusions**

Property Lists have great importance in the field of mobile forensics. In this chapter, a look behind the scenes was given. There are four different types of property lists. The chapter concentrated on the binary property list format because it is not intuitive to decode and has a certain prevalence. The binary property list format contains a header, a trailer and an object table. The type of the single objects can be found in a look-up table.

There are plenty of tools that support the decoding of plist files. But in a forensic manner, information from corrupted files (especially parts of files carved from the unallocated space) must be revealed. To fulfill this challenge, deep knowledge about file structures is crucial.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

 The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Chapter 7 Java Serialization**

Dirk Pawlaszczyk

**Abstract** Java Serialization is a popular technique for storing object states in the Java programming language. In the field of mobile forensics, we come across such artefacts. App developers very often resort to this technique to make their application state persistent. Serialization is also used when transferring data over a network between two Java applications using Remote Method Invocation(RMI). In the past, there have been recurring security issues associated with this technology. Despite its importance for forensic casework, one can hardly find any literature on this topic. In this chapter, we give an insight into the binary format. For this purpose, special features of the format are presented using an example. In addition to the actual protocol structure, basic steps for acquiring such data and analyzing it will be discussed. Practical hints for searching serials are given. Finally, the security issues are addressed.

### **7.1 Introduction**

Among app developers, the Java programming language has been the first choice for many years. The popularity of the language can be attributed to its simple syntax and compelling framework. As with any object-oriented language, the execution state of the program is managed through objects. From time to time, an application needs to back up its current state to disk. Of course, it is possible to store important data in a database such as SQLite. However, this usually requires object-relational mapping to be performed first. From the beginning, Java offers an alternative for persistent writing of objects: the so-called Java Object Serialization (JOS) [93]. Java's standard serialization seems to be a good choice, especially for app developers who want to store objects' current execution state. By serialization, we understand the ability to

Faculty for Applied Computer Science

University of Applied Sciences (Hochschule Mittweida), Technikumplatz 17, 09648 Mittweida, Germany, e-mail<EMAIL_ADDRESS>
convert an object in the application's main memory into a format that allows the object to be written to a file or transported over a network connection.

Since many apps rely on this format by default to store their program data, investigators are of particular interest. This chapter will take a closer look behind the scenes at the Standard Serialization concept in Java. We pay special attention to the binary format and explore how to analyze this file type.

### **7.2 Object Serialization in Java**

### **7.2.1 Serialization Techniques in Java**

Under Java SE, objects can be automatically mapped and stored persistently using various approaches [91], [54]:

All three options are already built into Java by default. The standard object serialization creates a binary format and is very strongly oriented towards Java. Other systems cannot do much with the data. XML is convenient as a format because other systems can process it. Another compact binary format that also allows interoperability is Protocol Buffers 1 from Google. The company uses it internally when different applications are to exchange data.

Finally, objects can also be stored in relational databases called object-relational mapping (OR mapping). This technique is very sophisticated because the object

1 http://code.google.com/p/protobuf/

models and tables are quite different. The Java SE does not offer any support for OR mapping, but it can be done with additional frameworks, such as the JPA (Java Persistence API).

### **7.2.2 Serialization by Example**

The traditional way from an object to persistent storage is via Java's serialization mechanism [57][54]. JOS is the technology we want to deal with in the following. The standard serialization offers a simple possibility to make objects persistent and to reconstruct them later. The object state (no static ones!) is written into a byte stream (serialization). From this, it can be reconstructed to an object again later (deserialization). The object state is written into a serial data stream of 0 and 1. Java provides two special classes for this purpose: *ObjectOutputStream* and *ObjectInputStream* with a *writeObject()* respectively *readObject()*-method. Both classes can be found in the *java.io* package of the Java standard class library 2. To save an object's state, we must pass the object reference as a parameter to the writeObject()-method. In the Java ecosystem, the applications programmers are encouraged to use serialization almost everywhere.

Fig. 7.1: Java Object Serialization (JOS) - Concept

Serialization concept can show its strengths, especially in communication between different Java processes distributed over a network. We serialize some of the objects, send them to another process for processing, serialize the transformed object and send

2 https://docs.oracle.com/javase/7/docs/api/java/io/ObjectOutputStream.html

it back. To illustrate this, we will discuss a small example program. In the following, we want to make a class *SaveMe* serializable. For this, we need the following code:

Listing 7.1: Class definition of a class to serialize

```
i m p o rt j a v a . i o .  In order for objects to be Serialized, the classes must implement the *Serializable* interface. This interface thus serves as a marker to indicate that the class can be Serialized.

### **• ! Attention**

When serializing an object, only its attributes are stored. Methods or program code remains in the *.class* file.

Java is assigning a serial number to each object of the class it writes to a stream. This serial number is then used to re-create the class when it is reread. If two variables contain references to the same object and we write the objects to a file and later read them from the file, then the two objects that are read will again be references to the same object. All attributes of an object can be made persistent in this way. However, there are two exceptions. Attributes defined with the prefixed key *transient* are not Serialized. This identifier was explicitly introduced to exclude an attribute from Serialization. It can be helpful, for example, if confidential or volatile data should not be saved. The second exception is class attributes preceded by the keyword *static*. With one exception, such attributes shared by all objects of a class are not Serialized. In our example, there are no transient attributes and only one static class attribute. Since the serialVersionUID property is defined as static and thus should not be stored. In this case, however, an exception is made. In Java, serialVersionUID is like version control, ensuring that both Serialized and deSerialized objects use the compatible class. For example, if an object is saved into a stream with serialVersionUID=1L, when we convert the stream back to an object, we must use the same serialVersionUID=1L. Otherwise, an InvalidClassException is thrown.

If we create a *SaveMe* object 𝑜1 and call 𝑤𝑟𝑖𝑡𝑒𝑂𝑏 𝑗𝑒𝑐𝑡(𝑜1), the ObjectOutput-Stream pushes the variable assignments (here 𝑥, 𝑑 and 𝑠) into the data stream.  With a few lines, the state of our object 𝑜1 can be saved to a file. In the example shown, the serialized object is written to a file. To send the object over the network, we have to create an object of the Socket class and start writing to the output stream of this class:

Listing 7.3: Output to a socket connection instead of a file

```
. . .
S o c k et c o n n e c t i o n = new S o c k et ( hostName , p o rtN um be r ) ;
Oj b e ct O ut p ut St r e a m o o s = new
      O bj e ct O ut p ut St r e a m ( c o n n e c t i o n . g et O ut p ut St r e a m ( ) , t r u e ) ;
o o s . w r i t e O b j e c t ( o1 ) ;
```
We only need to adjust two lines in our program, and we are ready to go. It could not be simpler. However, this form of serialization also has disadvantages. Standard serialization works according to the principle: Everything reached from the object graph enters the data stream serialized. Suppose the object graph is extensive, the time for serialization and the data volume increase. Unlike other persistence concepts, it is not possible to write only the changes. For example, if only one attribute value has changed in an extended object list, the entire list must be rewritten. This is not efficient. However, let us focus on analyzing the binary format.

The output file <saved.ser> from the above example has a size of 80 bytes. The content of the file can be seen in Fig. 7.2. In addition to the actual attribute values, information about data type and class type is also stored in the file. Fig. 7.3 offers a high-level look at the serialization algorithm for this example. In the next section,

Fig. 7.2: Hex view of the serialized object 𝑜1

we will take a closer look a the serialized format of the object and see what each byte represents.

Fig. 7.3: Outline of the Serialization steps for class *SaveMe*

### **7.3 Java Object Serialization Protocol Revealed**

As already discussed, Java's object Serialization creates a binary stream. Unlike JavaBeans persistence, for example, it is not readable by humans. Fortunately, the format is well documented. Oracle provides corresponding documentation on its website in which details of the *Object Serialization Stream Protocol* are presented [56]. The specification defines context-free grammar for the stream format. It gives a good insight into the Serialization process. The stream rules formulated in it are used directly in the Serialization of an object. In addition, a look at the source code of the *ObjectOutputStream* class reveals a lot about concrete implementation. Fig. 7.4 shows the first part of the grammar using a syntax diagram. It defines a set of production rules (<𝑅𝑛>). These rules can then be used directly to generate an object data stream.

Fig. 7.4: Syntax-diagram for the Java Object Serialization Protocol (Header Detail)

The syntax diagram for this is thus as follows: Definitions of symbols are followed by a ":". We first distinguish between the terminal and non-terminal symbols. The latter can be recognised by the fact that they consist of a literal enclosed in double quotes. Constant values are enclosed in an oval. A rectangle marks non-terminals. A sequence of values is represented as a series of symbols on the same line. The individual values can be found precisely in this order in the stream. A definition consists of zero or more alternative values. Alternatives are indicated by a branch (exclusive OR).

Let us now turn to the concrete meaning of the rules. Each object stream initially consists of a magic number, the version number and the actual content specification (𝑅1). A magic number opens the data stream. The 2-byte integer value resides on offset 0. On offset two, the stream version field follows. According to the internal specification, this is assigned to value *0x0005* (𝑅3). Thus, a Java object stream can be detected with the help of the header signature 0XACED0005 (𝑅2 + 𝑅3). This information is beneficial when carving to serial format files on disk. The actual content directly follows the header in the data stream.

The *contents* field is defined recursively (see 𝑅4). Thus it can hold multiple content objects. A *content* object is first divided into an object description and a data block with the concrete attribute values (𝑅5). Thus, valid values for a content element are *objects, classes, arrays, strings, enumerations, exceptions* (𝑅6). In this way, all elements of a class and its objects can be described. Within the byte stream, limiter symbols indicate the type, start and end of particular elements. These terminal constants (TC) are shown in Table 7.1.

Fig. 7.5: Syntax-diagram for the 'newClass'production rule

Table 7.1: Stream Terminal Constants (TC)

Fig. 7.5 describes the syntax definition for a class. Besides the class name, the *SerializationUUID* and *ClassDescInfo* elements are of particular importance (see 𝑅10). Note: A corresponding class description must first be placed in the data stream for each object to be Serialized. It contains information about attribute names and data types. If the class has been derived from a special super-class, the class must, of course, also be Serialized. Since attributes of the super-class are also inherited in the deriving class, we must also capture them. If the super-class, in turn, has a parent class, then this must also be described. Inheritance relationships thus significantly increase the data stream. Fortunately, Java supports only single inheritance.

Particular attention should be paid to the symbol TC\_CLASSDEC. It is used to show the start of a new class definition. The byte TC\_OBJECT (0x73) represents the start of an object. A data block (TC\_BLOCKDATA) is in turn initiated by the byte value 0x77. We need to search the serial stream for these symbols to make the data visible. The constant *baseWireHandle* is of particular importance. Each Serialized element is assigned to such a handle. The first Serialized element contains the handle 0x7E0000, the next object is the trade 0x7E00001 and so on. In this way, for example, an object can reference its class.

Table 7.2: Type Codes / Stream Symbols for Primitive Types

Table 7.2 shows the identifiers used in the stream for the eight built-in data types in Java. We can now start decoding the first part of our sample file <saved.ser> with what we have discussed so far (see below). As discussed above, the binary stream is opened by the magic number (0xACED) and the stream version (0x0005). The next byte indicates that this is an object that follows (0x73). The following byte introduces the class identifier (0x72).

### **•** > **Important**