Skip to content

Home

Jump to bottom
Razvan Deaconescu edited this page Dec 25, 2017 · 5 revisions

Welcome to the iExtractor wiki!

iExtractor is a collection of tools and scripts to automate data extraction from iOS firmware files (i.e. IPSW - iPhone Software - files). It runs on macOS and partially on Linux (certain tools and features only work on macOS).

iExtractor may be used by people interested in iOS internals to automate the processing of data for multiple iOS versions or simply to learn more about IPSW files, kernelcaches and sandbox profiles. By using a single command you will go through all steps from downloading the public Apple-provided IPSW file, to unpacking the kernel, to reversing the sandbox profiles. You can customize the scripts or build your own to process multiple IPSW files or to get a particular piece of information.

A tool similar to iExtractor is ipsw_decrypt.py. ipsw_decrypt.py is a Python script that decrypts all encrypted information inside an IPSW file. iExtractor also extracts kernel extensions, reverses sandbox profiles and extracts system dynamic libraries from the dyld shared cache.

Check the README file for information about installing and using iExtractor. Check the sections below for detailed information.

Overview of Steps

iExtractor requires the IPSW download URL and the decryption keys (for iOS <= 9) to do the downloading, unpacking, extraction and processing of data. For a given firmware, it follows the steps below:

  1. Download the IPSW file from a publicly available link provided by Apple.
  2. Unpack the contents of the IPSW file.
  3. Decrypt the disk image file for iOS <= 9.
  4. If running on Linux, convert the disk image file (.dmg) to a filesystem image file (.img).
  5. With the image file mounted, copy the sandboxd file storing the sandbox profiles for iOS <= 8.
  6. With the image file mounted, extract the system dynamic library files from the dyld shared cache.
  7. With the image file mounted, pack the entire filesystem contents for "offline" use.
  8. Decrypt the raw kernelcache file for iOS <= 9 into a kernelcache dump file.
  9. Unpack the kernelcache dump file into the actual kernelcache.
  10. Extract the sandbox operations from the kernelcache.
  11. Extract the sandbox kernel extension from the kernelcache.
  12. Extract the binary sandbox profiles from the kernel sandbox extension (iOS >= 9) or from sandboxd (iOS <= 8).
  13. Reverse the binary sandbox profiles into their original human readable form.

Configuration

The firmware-metadata/ folder in the repository stores metadata for each firmware; each metadata file inside the folder is named after the firmware ID (e.g. iPhone5,1_9.3_13E237). Inside each metadata file we store the download URL and the decryption keys; for example:

$ cat firmware-metadata/iPhone5,1_9.3_13E237
firmware_keys_url: https://www.theiphonewiki.com/wiki/Eagle_13E237_(iPhone5,1)
firmware_download_url: http://appldnld.apple.com/iOS9.3/031-55518-20160328-92D092CC-F29A-11E5-A1E1-DC0EF816D560/iPhone5,1_9.3_13E237_Restore.ipsw
rootfs_key: 2a66fd6377af8f60d5e300ac3aa8d9c44a1c0dee94579ad3f8a26515debbf381bb971ae8
kernelcache_iv: 17026ef62f8da01dc0f4e7d2b8b54d4c
kernelcache_key: 4ca5ec9e8dc9c5f77412c699fe724a5bfe101a79e1f2134e53bbb990e62a8c96

Keys are listed on The iPhone Wiki. The download URL is available on ipsw.me and, for most versions, on The iPhone Wiki.

The config file defines the IPSW_STORE and OUT_STORE variables. This where the downloaded IPSW files and the extracted files are stored, respectively. The downloaded IPSW files are stored directly in the IPSW_STORE folder. Extracted files are stored in a subfolder of the OUT_STORE named after the firmware ID.

Input and Output Data

The listing below shows the hierarchy of the IPSW_STORE folder (in/) and the OUT_STORE folder (out/):

.
|-- in/
|   |-- iPhone5,1_9.3_13E237_Restore.ipsw
|   `-- iPhone_4.0_64bit_11.1.2_15B202_Restore.ipsw
`-- out/
    |-- iPhone5,1_9.3_13E237/
    |   |-- 058-25481-332.dmg
    |   |-- 058-25512-331.dmg
    |   |-- 058-25517-331.dmg
    |   |-- BuildManifest.plist
    |   |-- Firmware/
    |   |-- Restore.plist
    |   |-- com.apple.security.sandbox.kext
    |   |-- decrypted.dmg
    |   |-- dyld_shared_cache/
    |   |-- fs.tar.gz
    |   |-- kernelcache.decrypted
    |   |-- kernelcache.mach.arm
    |   |-- kernelcache.release.n41
    |   |-- reversed_profiles/
    |   |-- sandbox_bundle
    |   `-- sb_ops
    `-- iPhone_4.0_64bit_11.1.2_15B202/
        |-- 058-84556-102.dmg
        |-- 058-84705-099.dmg
        |-- 058-85048-102.dmg
        |-- BuildManifest.plist
        |-- Firmware/
        |-- Restore.plist
        |-- com.apple.security.sandbox.kext
        |-- com.apple.security.sandbox.kext.ARM64.48E3D31D-C7FB-300C-9BDC-1C578EE2AA5B
        |-- decrypted.dmg
        |-- dyld_shared_cache/
        |-- fs.tar.gz
        |-- kernelcache.decrypted
        |-- kernelcache.mach.arm
        |-- kernelcache.release.iphone6
        |-- kernelcache.release.iphone8b
        |-- reversed_profiles/
        |-- sandbox_bundle
        `-- sb_ops

Let's do a walkthrough of the files and folders in the IPSW_STORE and OUT_STORE listed above, in the order they are downloaded, unpacked, extracted and processed by iExtractor, mirroring the overview of steps above:

  1. The iPhone5,1_9.3_13E237_Restore.ipsw and iPhone_4.0_64bit_11.1.2_15B202_Restore.ipsw files in the in/ subfolder are the firmware files. They are downloaded from a publicly available link provided by Apple.
  2. The contents of the IPSW files are unpacked in the out/iPhone5,1_9.3_13E237/ and out/iPhone_4.0_64bit_11.1.2_15B202/ folders. Contents include the 058....dmg files, the BuildManifest.plist and Restore.plist files, the Firmware/ folder the the kernelcache.release... files.
  3. The largest .dmg file is the root disk image file. For iOS <= 9 it is encrypted. The decrypted disk image file (or, if not encrypted, a link to the original file) is decrypted.dmg.
  4. If iExtractor were to be run on Linux, there would have been a rootfs.img file converted from decrypted.dmg. macOS is able to directly attach and mount .dmg files, while Linux requires .img files.
  5. For iOS <= 8, a sandboxd file would have been present. The output folders are for iOS 9.3 and iOS 11.1.2 and the sandboxd file is absent.
  6. The dyld_shared_cache/ folder stores the system library files extracted from the dyld shared cache.
  7. The fs.tar.gz file is an archive of the entire filesystem contents.
  8. The kernelcache.decrypted file stores the decrypted kernelcache dump of one of the kernelcache.release.... files. For iOS >= 10 the kernelcache release file is not encrypted so the kernelcache.decrypted file is a link to it.
  9. The kernelcache.mach.arm file is the actual kernelcache file unpacked from the kernelcache.decrypted file.
  10. The sb_ops file stores the sandbox operations, a list of strings required for reversing the sandbox profiles.
  11. The com.apple.security.sandbox.kext file stores the kernel sandbox extension.
  12. The sandbox_bundle file (or sandbox_profiles/ folder for iOS <= 8) stores the binary sandbox profiles.
  13. The reversed_profiles/ folder stores the reversed sandbox profiles in human readable form.

Tools of The Trade

iExtractor uses existing tools for all the steps of processing IPSW files. These are non-interactive command line tools that enable iExtractor to use a single script to process one or multiple IPSW files. Some of them are part of the installed environment while others are part of the repository (directly or as submodules) in the tools/ subfolder. Mirroring the steps above, the tools employed are:

  1. wget downloads the IPSW file from the publicly available link provided by Apple.
  2. unzip unpacks the IPSW file (a ZIP file) into the output folder.
  3. vfdecrypt decrypts the largest .dmg file into decrypted.dmg for iOS <= 9 using the community provided keys from The iPhone Wiki.
  4. dmg2img converts decrypted.dmg into rootfs.img.
  5. hdutil on macOS and mount and umount on Linux mount and unmount the decrypted.dmg and rootfs.img files respectively. cp copies sandboxd from the mount point to the output folder.
  6. (only on macOS) After mounting the image file, dsc_extractor extracts the system dynamic library files from the System/Library/Caches/com.apple.dyld/ path from the mount point to dyld_shared_cache/.
  7. After mounting the image file, tar packs the filesystem contents in fs.tar.gz.
  8. xpwntool decrypts kernelcache.release.... to kernelcache.decrypted for iOS <= 9.
  9. lzssdec unpacks kernelcache.decrypted to kernelcache.mach.arm.
  10. (only on macOS) extract_sbops extracts the sandbox operations in the sb_ops file from kernelcache.mach.arm.
  11. joker extracts com.apple.security.sandbox.kext from kernelcache.mach.arm.
  12. extract_sbprofiles extracts binary sandbox profiles from sandboxd in sandbox_profiles/ for iOS <= 8 (only on macOS); dd inside a custom script extracts sandbox_bundle from com.apple.security.sandbox.kext for iOS >= 9.
  13. SandBlaster reverses binary sandbox profiles from sandbox_profiles/ or sandbox_bundle to reversed_profiles/.

Behind the Scenes Walkthrough

Let's get a glimpse of how the above tools are actually used. Below is a set of commands that have been run manually to go through all steps. iExtractor automates the commands below and hides the particularities of iOS versions. Commands have been run on a macOS 10.13.2 system; paths to commands and files are particular to the environment. The input file is an iOS 8 IPSW (firmare ID iPad2,1_8.4.1_12H321), with the the download URL and keys listed in the corresponding firmware metadata file. Commands mirror the steps above.

# Download IPSW file.
cd in/
wget http://appldnld.apple.com/ios8.4.1/031-31231-20150812-751BEF2C-3C8F-11E5-94D1-BB1A3A53DB92/iPad2,1_8.4.1_12H321_Restore.ipsw
# Unpack IPSW file.
cd ../out/
mkdir iPad2,1_8.4.1_12H321
cd iPad2,1_8.4.1_12H321
unzip -qq ../../in/iPad2,1_8.4.1_12H321_Restore.ipsw
# Decrypt .dmg file.
../../../iExtractor.git/tools/vfdecrypt/vfdecrypt -i 058-24482-024.dmg -o decrypted.dmg -k d33bfc10cc397ae008d2e56685acb172b74f0eccbe652ecd52e234ab12705980b6483b60
# Do not convert .dmg to .img (not required on macOS).
# Attach and mount disk image.
sudo mkdir -p /mnt/ios/iPad2,1_8.4.1_12H321
sudo hdiutil attach -noverify -mountpoint /mnt/ios/iPad2,1_8.4.1_12H321 decrypted.dmg
# Copy sandboxd.
cp /mnt/ios/iPad2,1_8.4.1_12H321/usr/libexec/sandboxd .
# Extract dyld shared cache.
mkdir dyld_shared_cache
../../../iExtractor.git/tools/dyld/dsc_extractor /mnt/ios/iPad2,1_8.4.1_12H321/System/Library/Caches/com.apple.dyld/dyld_shared_cache_armv7 dyld_shared_cache/
# Pack filesystem.
sudo tar -C /mnt/ios/iPad2,1_8.4.1_12H321 -czf fs.tar.gz .
# Unmount and dettach disk image.
sudo hdiutil detach /dev/disk2
# Decrypt kernelcache.
../../../iExtractor.git/tools/xpwn/builddir/ipsw-patch/xpwntool kernelcache.release.k93 kernelcache.decrypted -k 6dea51edc4c6f6c205e25ca1d9af5b9ee7167ed5685a937ad892e7fcbfb3e615 -iv 38d12734c845d1dcdb84846caf619da7 -decrypt
# Unpack kernelcache.
../../../iExtractor.git/bin/get_lzss_section_offset.py kernelcache.decrypted # command output is 448 (offset)
../../../iExtractor.git/tools/lzssdec/lzssdec -o 448 < kernelcache.decrypted > kernelcache.mach.arm
# Extract sandbox operations.
../../../iExtractor.git/tools/sandblaster/tools/sandbox_toolkit/extract_sbops/extract_sbops kernelcache.mach.arm > sb_ops
# Extract kernel sandbox extension.
JOKER_DIR=. ../../../iExtractor.git/tools/joker/joker.universal -K com.apple.security.sandbox kernelcache.mach.arm
# Extract sandbox profiles.
mkdir sandbox_profiles
cd sandbox_profiles/
../../../../iExtractor.git/tools/sandblaster/tools/sandbox_toolkit/extract_sbprofiles/extract_sbprofiles ../sandboxd
cd ..
# Reverse sandbox profiles.
mkdir reversed_profiles
cd ../../../iExtractor.git/tools/sandblaster/reverse-sandbox/
for i in ../../../../store/out/iPad2,1_8.4.1_12H321/sandbox_profiles/*; do python reverse_sandbox.py -r 8.4.1 -o ../../../../store/out/iPad2,1_8.4.1_12H321/sb_ops -d ../../../../store/out/iPad2,1_8.4.1_12H321/reversed_profiles/ "$i"; done

iExtractor Internals

To make it easier to automate and to hide specifics of iOS versions, iExtractor uses two layers of scripts on top of which wrapper scripts are built.

The lower layer scripts are located in the bin/ subfolder. They take care of iOS version specifics and call the required tools in the tools/ subfolder. For example, bin/decrypt_fs decrypts the disk image file if a key is available (i.e. if it is encrypted), otherwise it just creates a link; if it runs on Linux it also converts the decrypted disk image file (decrypted.dmg) to a filesystem image file (rootfs.img). There is a bin/common configuration script that's being sourced by all higher layer scripts.

The higher layer scripts are located in the scripts/ subfolder. All these scripts are passed a single argument: the firmware ID. They will parse firmware ID information from the corresponding file in the firmware-metadata/ subfolder, construct parameters and call lower layer scripts in the bin/ subfolder or existing tools. There is a scripts/common configuration script that's being sourced by all higher layer scripts.

Wrapper scripts are custom scripts created by the user to call higher layer scripts according to need. Sample scripts are part of the repository: scripts named with run_.... A wrapper script is passed the firmware ID as a parameter and calls corresponding higher layer scripts. For example the run_all wrapper script calls all higher layer scripts, while the run_sandblaster wrapper script only calls SandBlaster-specific higher layer scripts.

In short, wrapper scripts in the root of the repository call higher layer scripts in the scripts/ subfolder which in turn call lower layer scripts in the bin/ subfolder.

You will usually run a wrapper script, but you can run a higher layer or lower layer script as well. All will work with files in the folders defined by the IPSW_STORE and OUT_STORE variables in the config file.

Running a wrapper script is as simple as passing it the firmware ID as a parameter:

./run_all iPhone5,1_9.3_13E237
./run_sandblaster iPhone5,1_9.3_13E237

Output files will be stored in the OUT_STORE/iPhone5,1_9.3_13E237/ folder.

Either for testing, debugging or to do a specific action, you can directly call a higher layer script or a lower layer script. Running a higher layer script requires scripts/ to be the working directory; similarly, running a lower layer script requires bin/ to be the working directory. For example, if you wanted to decrypt the disk image for a given firmware you could run one of the two commands below (a higher layer and a lower layer script):

cd scripts/
./decrypt_fs iPhone5,1_9.3_13E237
cd bin/
./decrypt_fs ../../store/out/iPhone5,1_9.3_13E237/058-25512-331.dmg 2a66fd6377af8f60d5e300ac3aa8d9c44a1c0dee94579ad3f8a26515debbf381bb971ae8

Similarly to running a wrapper script, the decrypted.dmg output file will be stored in the OUT_STORE/iPhone5,1_9.3_13E237/ folder.

Higher layer scripts do sanity checks for input and output files. If output files already exists, the higher layer script will prompt if you want to overwrite them. This is why wrapper scripts use a construct such as the one below, to prevent the likely useless action of overwriting existing output files with the same content. Wrapper scripts can, of course, be updated to force the overwrite.

echo "* Extract dynamic library cache"
yes N | ./extract_dyld_shared_cache "$firmware_id"

Debugging

In case of issues you can turn on debugging by altering the DEBUG variable in the config file and setting it to 1. That causes lower and higher layer scripts to print out detailed information about what they are doing including printing out commands they run. Debug messages use the [debug] prefix. For example, running the decrypt_fs lower layer script in the bin/ subfolder with debugging enabled renders the output below:

$ ./decrypt_fs ../../store/out/iPhone5,1_9.3_13E237/058-25512-331.dmg 2a66fd6377af8f60d5e300ac3aa8d9c44a1c0dee94579ad3f8a26515debbf381bb971ae8
[debug] Decrypting dmg file ../../store/out/iPhone5,1_9.3_13E237/058-25512-331.dmg to ../../store/out/iPhone5,1_9.3_13E237/decrypted.dmg ...
[debug] ../tools/vfdecrypt/vfdecrypt -k2a66fd6377af8f60d5e300ac3aa8d9c44a1c0dee94579ad3f8a26515debbf381bb971ae8 -i../../store/out/iPhone5,1_9.3_13E237/058-25512-331.dmg -o../../store/out/iPhone5,1_9.3_13E237/decrypted.dmg

Extending iExtractor

If you only want certain actions to automate you can create a custom wrapper script using existing run_... scripts as starting point. Then you can use the for_each_firmware script to run the custom wrapper script on each firmware.

To add support for a new firmware, create a new file in the firmware-metadata/ using existing files as starting point. Extract the download firmare ID and download URL from ipsw.me and decryption keys for iOS <= 9 from The iPhone Wiki.

If you want a new action as part of iExtractor (e.g. extracting only Mach-O files from the filesystem, extracting all kernel sandbox extensions, extracting the launchd executable etc.) you need to create a higher layer script in the scripts/ subfolder and, if required, a lower layer script in the bin/ subfolder using existing scripts as starting points.

Supported Platforms and iOS Versions

iExtractor runs on firmware files from iOS 7 onward. The sandbox profile reverser doesn't work for iOS <= 6. All other components should likely work for earlier iOS versions as well.

iExtractor run on macOS. It partially runs on Linux except for:

  • extracting the dyld shared cache: dsc_extractor only works on macOS
  • extracting the sandbox operations: extract_sbops only works on macOS
  • extracting sandbox profiles for iOS <= 8: extract_sbprofiles only works on macOS
  • mounting root filesystem image files for iOS >= 10.3: since iOS 10.3 the filesystem is Apple File System (APFS) for which there is currently (December 2017) no support on Linux
Clone this wiki locally