WARNING

I have no responsibility whatsoever if this guideline causes any harm to your device. The intention of these posts are solely as personal notes for myself. Follow them at your own risk.

WARNING

Through these steps I will unlock the phone’s bootloader, erasing all data. This includes the DRM keys stored in the Trim Area (TA) partition. I’ll attempt backing them up but, as of today, there is no way of restoring them to the previous state nor knowing if the actual backup is usable at all.

Without these DRM keys, several audio and video proprietary functionality provided by Sony won’t be available including some camera post-processing features, color gamut profiles, white balance, noise reduction, X-Reality Video Enhancement, DSEE HX, ClearAudio+, and Widevine L1 support for HD Netflix.

Appendixes

Previously, we downgraded Sony’s Stock firmware, backed up the Trim Area, installed LineageOS, and, finally, bring back Sony’s stock camera app.

The installation had some bumps, so here a list of the things I could comment about …

Installing TWRP

I wanted to install the TWRP recovery tool just because I have experience with it and like it better. I did and used it successfully in the first installation of the LineageOS that I did. However, the LineageOS installation also installed the Lineage Recovery so I lost TWRP and, funnily enough, the steps I followed to install TWRP won’t work any more. Hence, I’m stack with the Lineage Recovery and that’s why I explain how to directly install Lineage Recovery in the previous post.

For the curious, these are the steps I followed.

First, we need to download the Unofficial TWRP recovery and extract it in a folder. After this I rebooted the device in fastboot mode via software. This is important. Don’t do it using the hard keys.

Once in fastboot mode:

root$ fastboot --disable-verity --disable-verification flash vbmeta vbmeta.img
[...]
root$ fastboot boot twrp-apollo.img

This will boot temporarily into TWRP. Now, we need to do the actual flash of TWRP. First, we need to copy the image:

root$ adb push twrp-apollo.img /tmp

And, from TWRP: Advanced menu -> Install recovery ramdisk -> Choose /tmp/twrp-apollo.img. After finishing installing TWRP, we can reboot into Recovery using the newly installed TWRP.

From there, you can also wipe the system and data and install LineageOS, Open GApps and Magisk. Interestingly, TWRP provides the possibility of pushing these files and installing without using adb sideload. This is interesting because it will make also easier to install a customized Open GApps package using the gapps-config file.

Notice that, more often than not, when booting into TWRP recovery the touchscreen wouldn’t work. This is a random behavior and rebooting again into Recovery may fix it in the next try.

Installing the Android System Webview from the Open GApps

As mentioned, I wanted to customize Open GApps to install the Android System Webview.

I did.

However, choosing this will only install a Google WebView Stub. What this means is that you won’t really have a WebView and, hence, when booting in the system, anything which makes use of a WebView will just crash (including registering into Google for using Google Play).

In other words, you need to install the real package but you won’t have an easy way of doing that. You will have to manually download the real APK from some 3rd party site and install it, for example, with adb install.

In the end I just reinstalled without it after reading that the WebView used by LineageOS is based in Chromium. So far so good.

Migration from my older phone

In order to move all my data from my Xiaomi Redmi 2 running LineageOS 14.1 I used the OAndBackupX application. It needs root rights but, fortunately, I had them in both systems and devices.

Some comments:

• Make sure of using the exact same version of OAndBackupX in both devices.
• The app suggests to use encryption when creating the backups. I didn’t use it since it was doing things more complicated.
• I only did back ups of the user applications. First I also did back ups of the special cases (user accounts, for example) and after restoring, that caused me a headache with NextCloud and DavX⁵. Of course, jumping from 14.1 to 17.1 could cause these kind of troubles.

I initially used a SD Card to move the back ups. I wouldn’t recommend it now. If you have limited memory just create the back ups in steps and move them in batches using adb. With both devices plugged with USB cables and in developer mode:

root$ adb devices
List of devices attached
origin device
destination device

root$ adb -s origin root
restarting adbd as root
root$ adb -s destination root
restarting adbd as root
root$ # Create some backups in the origin device
root$ adb -s origin pull /storage/emulated/0/OABX .
[...]
root$ adb -s destination push OABX/* /storage/emulated/0/OABX/
[...]
root$ # Restore the backups in the destination device
root$ adb -s origin shell
origin:/ $ rm -rf /storage/emulated/0/OABX/*
origin:/ $ exit
root$ adb -s destination shell
destination:/ $ rm -rf /storage/emulated/0/OABX/*
destination:/ $ exit
root$ # Create some more backups in the origin device and repeat the process

Of course, no need to delete the back ups if you don’t need to.

WARNING

I have no responsibility whatsoever if this guideline causes any harm to your device. The intention of these posts are solely as personal notes for myself. Follow them at your own risk.

WARNING

Through these steps I will unlock the phone’s bootloader, erasing all data. This includes the DRM keys stored in the Trim Area (TA) partition. I’ll attempt backing them up but, as of today, there is no way of restoring them to the previous state nor knowing if the actual backup is usable at all.

Without these DRM keys, several audio and video proprietary functionality provided by Sony won’t be available including some camera post-processing features, color gamut profiles, white balance, noise reduction, X-Reality Video Enhancement, DSEE HX, ClearAudio+, and Widevine L1 support for HD Netflix.

Upgrading to latest stock firmware

In the previous posts we have downgraded the stock firmware from Sony and backed up the Trim Area (TA) partition.

Since the guideline to install LineageOS mandates that we have the latest stock firmware from Sony running in the phone, I upgraded now from the downgraded exploitable version.

Using the built-in updater doesn’t seem to work any more (?!) so I had to flash the latest stock firmware in a similar fashion as I did for downgrading.

I already downloaded the firmware and it is already properly extracted so, in this case, I won’t need to use flashtool, only newflasher.

These would be steps to follow once I connected the phone to the USB cable in flashable mode:

root$ rm "H8324_Customized NOBA_1313-6167_52.1.A.3.49_R4C"/*ta
root$ rm "H8324_Customized NOBA_1313-6167_52.1.A.3.49_R4C"/boot/*ta
root$ cp -a newflasher.x64 "H8324_Customized NOBA_1313-6167_52.1.A.3.49_R4C"
root$ cd "H8324_Customized NOBA_1313-6167_52.1.A.3.49_R4C"
root$ chmod +x newflasher.x64
root$ ./newflasher.x64

[...]

Reboot mode at the end of flashing:
typa 'a' for reboot to android, type 'f' for reboot to fastboot, type 's' for reboot to same mode, type 'p' for poweroff, and press ENTER.
a

[...]

Optional step! Type 'y' and press ENTER if you want dump trim area, or type 'n' and press ENTER to skip.
Do in mind this doesn't dump drm key since sake authentifiction is need for that! But it is recommend to have dump in case hard brick!
n

[...]

Recommended step to skip this! Type 'y' and press ENTER if you want flash persist partition, or type 'n' and press ENTER to skip.
More info https://forum.xda-developers.com/xperia-xz1-compact/help/android-attest-key-lost-bootloader-t3829945
n

[...]

Device is put now out of flash mode.
Sent command: Sync
Waiting sync to finish…
……………… done
Sent command: continue.
Done.
Closing device.

Notice the questions and the answers. After a while, the phone will complete its reboot and we will be able to verify that the running firmware is the one flashed.

As explained in the previous posts, enable once again developer mode and USB debugging in the phone. For the next steps, enable also OEM unlocking in the developer options.

Installing LineageOS

We’ll follow the official guide.

First, we’ll unlock the bootloader. This will finally wipe out the TA partition losing the DRM keys. This is a point of no return.

I checked that my phone’s bootloader can be unlocked. I opened the phone application and dialed *#*#7378423#*#*. The service menu is now open. Go to Service info -> Configuration and checked that the Rooting status: states Bootloader unlock allowed: Yes. Also, noted down the IMEI.

Now, I connected the device to my PC with the USB cable and continued to set it in flashable mode:

root$ adb reboot bootloader
root$ fastboot devices
[...] fastboot
root$ fastboot oem unlock 0x<insert your unlock code>
…
OKAY [ 16.947s]
finished. total time: 16.947s

Unplug and start the phone. As explained in the previous posts, enable once again developer mode and USB debugging in the phone.

At this point, I decided not to keep following the LineageOS guideline since it explains how to use the Lineage Recovery. Instead, I used the TWRP recovery tool just because I have experience with it and like it better. However, I don’t recommend it as I’ve explained in the Appendixes.

Therefore, I explain here the same steps than the official guide. Download the latest Lineage Recovery and LineageOS installation package and connect the device to your PC with the USB cable and continued to set it in flashable mode:

root$ adb reboot bootloader
root$ fastboot devices
[...] fastboot
root$ fastboot flash boot <lineage_recovery>.img

Power off the device and now turn it on into Recovery mode by pressing Volume Down + Power.

Now, being booted into the Recovery mode, I downloaded a pre-install copy-partitions-20200903_1329.zip tool and selected Apply Update -> Apply from ADB.

root$ adb sideload copy-partitions-20200903_1329.zip

Once finished, reboot again into Recovery mode: Go back -> Advanced -> Reboot to recovery. Once back, Factory reset -> Format data/factory reset. Once finished, let’s install LineageOS: Go back -> Apply Update -> Apply from ADB.

root$ adb sideload lineage-17.1-<date>-xz2c-signed.zip

Now, before booting into the system, we want to add some more stuff: Open GApps and Magisk.

Magisk is a suite for customizing Android. Most importantly, it provides root access to the device, which I wanted to have in order to create backups of the installed applications and restore them, among other things.

Open GApps will provided us the core functionality provided by Google for Android. Most importantly, it will provide us Google Play. It is critical to install this before booting for the first time into the system. Otherwise, we would have to repeat the Factory reset step and wipe out all our personal data before attempting to install it.

Open GApps provides different size packages. The recommended for LineageOS are the pico or nano, but nothing bigger. Since I’m a troublemaker, I also wanted to customize the package to include the Android System Webview and remove even further packages from the pico and nano packages.

Hence, I downloaded the stock package for the ARM64 platform and the 10.0 Android version.

For reasons that I explain in the Appendixes, I finally didn’t install the Android System Webview but did remove some packages any way. The way of customizing a package is through a gapps-config file. However, to use this method the installation cannot be done through adb sideload and, unfortunately, the Lineage Recovery only offers this way of installing packages into the system.

Luckily enough, the Open GApps package is no more than a zip file so I could embed my options directly into the installer. So, after installing the LineageOS system, I rebooted again into Recovery mode: Go back -> Advanced -> Reboot to recovery and, back in Recovery, let’s install Open GApps: Apply Update -> Apply from ADB.

root$ mkdir my_gapps
root$ cd my_gapps
root$ unzip ../open_gapps-arm64-10.0-stock-<date>.zip
root$ cat installer.sh # Added the following lines in bold

[...]

echo "Include" > "$TMP/my_config.txt"
echo "" >> "$TMP/my_config.txt"
echo "CalSync" >> "$TMP/my_config.txt"
echo "DialerFramework" >> "$TMP/my_config.txt"
echo "GoogleTTS" >> "$TMP/my_config.txt"
echo "PackageInstallerGoogle" >> "$TMP/my_config.txt"
echo "BatteryUsage" >> "$TMP/my_config.txt"
echo "Speech" >> "$TMP/my_config.txt"
echo "#GooglePay" >> "$TMP/my_config.txt"
echo "Translate" >> "$TMP/my_config.txt"

# Locate gapps-config (if used)
for i in "$TMP/aroma/.gapps-config"\
"$TMP/my_config.txt"\
"$zip_folder/.gapps-config"\

[...]

root$ zip -0 -r ../my_gapps *
root$ cd ..
root$ adb sideload my_gapps.zip

And, now, let’s install Magisk: Apply from ADB.

root$ adb sideload Magisk-v20.4.zip

Once finished we can finally Go back -> Reboot system now.

Congratulations, your Sony Xperia XZ2 Compact Dual is now running LineageOS 17.1!!!

I have to say that, so far, I’m quite happy with the phone. It is a huge improvement for me, coming from a Xiaomi Redmi 2.

However, the camera has lost some enhanced functionality so let’s continue to bring back Sony’s stock camera app.

WARNING

I have no responsibility whatsoever if this guideline causes any harm to your device. The intention of these posts are solely as personal notes for myself. Follow them at your own risk.

WARNING

Through these steps I will unlock the phone’s bootloader, erasing all data. This includes the DRM keys stored in the Trim Area (TA) partition. I’ll attempt backing them up but, as of today, there is no way of restoring them to the previous state nor knowing if the actual backup is usable at all.

Without these DRM keys, several audio and video proprietary functionality provided by Sony won’t be available including some camera post-processing features, color gamut profiles, white balance, noise reduction, X-Reality Video Enhancement, DSEE HX, ClearAudio+, and Widevine L1 support for HD Netflix.

Backup the TA partition

As explained in the previous post, enable developer mode in the phone.

Following this guide, download the latest Magisk release. At the time of writing this it’s v20.4.

Download the tama-mroot.zip with the needed exploit.

Push both archives into the phone (you may need to give consent in a pop up dialog in the phone):

root$ adb push tama-mroot/tama-mroot.zip Magisk/Magisk-v20.4.zip /data/local/tmp
tama-mroot/tama-mroot.zip: 1 file pushed. 0.5 MB/s (21355 bytes in 0.039s)
Magisk/Magisk-v20.4.zip: 1 file pushed. 32.0 MB/s (5942417 bytes in 0.177s)
2 files pushed. 25.4 MB/s (5963772 bytes in 0.224s)

Get into the phone and follow the steps to get a root shell:

root$ adb shell
H8324:/ $ cd /data/local/tmp
H8324:/data/local/tmp $ unzip tama-mroot.zip
Archive: tama-mroot.zip
inflating: magisk-start.sh
inflating: magisk-setup.sh
inflating: tama-mroot
H8324:/data/local/tmp $ chmod 755 tama-mroot magisk-setup.sh magisk-start.sh
H8324:/data/local/tmp $ ./magisk-setup.sh

[...]

H8324:/data/local/tmp $ cd /data/local/tmp
H8324:/data/local/tmp $ ./tama-mroot

[...]

root_by_cve-2020-0041:/data/local/tmp # ./magisk-start.sh -1

[...]

root_by_cve-2020-0041:/data/local/tmp # ./magisk-start.sh -2

[...]

root_by_cve-2020-0041:/data/local/tmp # ./magisk-start.sh -3

[...]

We can verify now that we have really root privileges:

root_by_cve-2020-0041:/data/local/tmp # id
uid=0(root) gid=0(root) groups=0(root),1004(input),1007(log),1011(adb),1015(sdcard_rw),1028(sdcard_r),3001(net_bt_admin),3002(net_bt),3003(inet),3006(net_bw_stats),3009(readproc),3011(uhid) context=u:r:magisk:s0
root_by_cve-2020-0041:/data/local/tmp # uname -a
Linux localhost 4.9.186-perf+ #1 SMP PREEMPT Fri Jan 17 01:22:05 2020 aarch64

Hence, let’s go ahead and back up the TA partition:

root_by_cve-2020-0041:/data/local/tmp # dd if=/dev/block/bootdevice/by-name/TA of=TA-locked.img
4096+0 records in
4096+0 records out
2097152 bytes (2.0 M) copied, 0.039839 s, 50 M/s
root_by_cve-2020-0041:/data/local/tmp # chown shell:shell TA-locked.img
root_by_cve-2020-0041:/data/local/tmp # sync
root_by_cve-2020-0041:/data/local/tmp # sync

Now, from another terminal in the computer, pull the created backup:

root$ adb pull /data/local/tmp/TA-locked.img

That’s it, we have finished backing up the TA partition!

Now, we can continue to install the LineageOS system.

WARNING

I have no responsibility whatsoever if this guideline causes any harm to your device. The intention of these posts are solely as personal notes for myself. Follow them at your own risk.

WARNING

Through these steps I will unlock the phone’s bootloader, erasing all data. This includes the DRM keys stored in the Trim Area (TA) partition. I’ll attempt backing them up but, as of today, there is no way of restoring them to the previous state nor knowing if the actual backup is usable at all.

Without these DRM keys, several audio and video proprietary functionality provided by Sony won’t be available including some camera post-processing features, color gamut profiles, white balance, noise reduction, X-Reality Video Enhancement, DSEE HX, ClearAudio+, and Widevine L1 support for HD Netflix.

Why

I want to have a phone running Free Software with the possibility of using all the major apps out there. That means, unfortunately, some kind of Android flavor. I also want a small phone and, since I’m a Spaniard living in Finland, I’d like that my phone has dual SIM capability. Until now I was using a Xiaomi Redmi 2 but, after years of usage, I cannot squeeze it any more. I’d also love to have a decent camera, if possible.

After quite some research, my conclusion is that the only real alternative that ticks (most of) the boxes is the Sony Xperia XZ2 Compact Dual.

Preconditions

I’ll be doing all the stems in a Debian Buster GNU/Linux distribution on a x86_64 platform.

Downgrading the stock firmware

If you have read the warning above, you now know that installing LineageOS implies unlocking the bootloader. In turn, this means wiping the TA partition and losing the DRM keys, which will cause several advanced audio and video features to get lost forever.

We want to keep the hope that, at least, we would be able to restore this functionality. This means that we want to backup the TA partition. Doing that without actually unlocking the bootloader is not really possible … unless you exploit a security bug.

The Sony firmwares build 52.1.A.0.618 contains one such kernel bug. Therefore, first thing, downgrading the current firmware in the device to use the one affected.

As a previous step, I downloaded the firmware that was running in my device at the time: 8324_Customized NOBA_1313-6167_52.1.A.3.49_R4C

For downloading the up to date firmware I used the XperiFirm program:

root$ apt install mono-complete
root$ cert-sync /etc/ssl/certs/ca-certificates.crt
root$ certmgr -ssl -m https://software.sonymobile.com
(Entered Y twice when asked)
$ mono XperiFirm.exe

Then, I selected the Xperia XZ2 Compact Apollo: H8324 dual phone and downloaded the Nordic Combined available firmware.

Now, I downloaded and extracted the exploitable H8324-52.1.A.0.618 firmware for the Xperia XZ2 Compact Dual and followed this guide to install it into the phone without unlocking the bootloader.

We want to use the Flashtool to create a bundle with the parts of the firmware we are going to use.

I downloaded the latest version of the tool (0.9.29.0 at the time) and tried to launch it:

root$ DISPLAY=:0.0 ./FlashTool
Running as root.
Error: dl failure on line 603
Error: failed /home/tanty/personal/sony_xperia_xz2c/flashtool/FlashTool/x10flasher_native/jre/lib/server/libjvm.so, because /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/tant
y/personal/sony_xperia_xz2c/flashtool/FlashTool/x10flasher_native/jre/lib/server/libjvm.so)

Bad news … it seems this version was compiled against a different C toolchain? Let’s use an older version then. 0.9.27.0 it is.

root$ DISPLAY=:0.0 ./FlashTool
Running as root.
Used java home : /usr
03/057/2020 16:57:14 - INFO - <- This level is successfully initialized
(Flashtool:8027): GLib-CRITICAL **: 16:57:14.971: g_base64_encode_step: assertion 'in != NULL' failed
(Flashtool:8027): GLib-CRITICAL **: 16:57:14.971: g_base64_encode_step: assertion 'in != NULL' failed
...

Voilá!

In the UI, we select Tools -> Bundles -> FILESET Decrypt. In the popping up Window we browse to the folder containing the exploitable firmware: H8324_Customized FR_1313-2469_52.1.A.0.618_R2C. There, we choose the 3 zip files that it contains and move it from the left side to the right side in the window and click Convert.

Once it has finished, we want to skip creating the bundle by just pressing Cancel in the next window. We have finished using flastool.

In the exploitable firmware directory we now have a new folder called decrypted. We want to delete some files there:

root$ rm decrypted/*ta
root$ rm decrypted/boot/*ta

Before we proceed to flash, we need to add some rules to the udev system in our computer so it can identify the Xperia phone:

root$ echo 'SUBSYSTEM=="usb", ACTION=="add", ATTRS{idVendor}=="0fce", ATTRS{idProduct}=="*", MODE="0777"' > /etc/udev/rules.d/51-sony.rules
root$ service udev restart

Now, we want to use another tool to flash these files into the phone: newflasher. I downloaded the latest v36 version, extracted, placed the executable in the decrypted folder from the previous step:

root$ mv newflasher.x64 decrypted
root$ cd decrypted
root$ chmod +x newflasher.x64

Last thing is setting the phone into flashmode.

First, let’s enable developer mode in the phone: Settings -> About phone, and hit several times the Build number field until it states that you have become a developer. Then, go to Settings -> System -> Advanced -> Developer options and switch on the USB debugging option.

Now, power down the phone. Once off, press and hold Volume Down and connect the USB cable that is connected to the computer in the other end. The led of the phone turns on and stays on. It is now into flashmode. Let’s flash:

root$ ./newflasher.x64

[...]

Reboot mode at the end of flashing:
typa 'a' for reboot to android, type 'f' for reboot to fastboot, type 's' for reboot to same mode, type 'p' for poweroff, and press ENTER.
a

[...]

Optional step! Type 'y' and press ENTER if you want dump trim area, or type 'n' and press ENTER to skip.
Do in mind this doesn't dump drm key since sake authentifiction is need for that! But it is recommend to have dump in case hard brick!
n

[...]

Recommended step to skip this! Type 'y' and press ENTER if you want flash persist partition, or type 'n' and press ENTER to skip.
More info https://forum.xda-developers.com/xperia-xz1-compact/help/android-attest-key-lost-bootloader-t3829945
n

[...]

Device is put now out of flash mode.
Sent command: Sync
Waiting sync to finish…
……………… done
Sent command: continue.
Done.
Closing device.

Notice the questions and the answers. After a while, the phone will complete its reboot and we will be able to verify that the running firmware is the one flashed.

That all by now, next step: backing up the TA partition!