Cross-exec host storage virt for non-native targets

。。。where we try to debug a raspbian distro boot by chrooting and emulating binaries on the target root filesystem inside a sdcard

This post seeks to find answers to these thoughts:

  1. How to debug a boot problem on raspbian (ARM rpi3) from a x86_64 host?
  2. Can’t I just use chroot and get diagnostics?
  3. What are the differences between the default bind mount vs udisks/udisksctl?
  4. What about if I wanted to use the smartphone SDcard reader and mount the filesystem from there?
  5. What are Object device trees
  6. Wait, this was build from Buildroot, by inspecting the rootfs. How does buildroot works?
  7. How does wpantund DHCP dynamic IP assignment works?
  8. How does LCDs and displays works on raspberry pi? How to configure a LCD that was not detected?
  9. Can I mount the target filesystem on termux from the smartphone sdcard reader?
  10. How does chroot works?
  11. How does binfmt_misc works?
  12. What is the role of qemu-user-static?
  13. What about fakeroot?
  14. How startalpine/AlpineOnTermux does it
  15. Proot: storage virtualization in userspace
  16. What is the relation of FUSE and fusermount with all of this?
  17. What if I want to do something like this from a Kotlin/Java app for Android?
  18. Extra: binfmt_misc and wine interop
  19. “if Proot offers usermode qemu as equivalence to binfmt_misc (alongside chroot and bind mount), what is the difference between usermode qemu to the qemu-user-static which uses binfmt_misc?”

[17/05/2024 02:52]

TIL binfmt_misc that is used by qemu-user-static can be leveraged to chroot into a mounted root filesystem with a different architecture than your current cpu… I wonder how the wine passing echo argument is different from just running it directly. Proot that is used on Termux’s startalpine also implements it alongside other utils like chroot and bind mount :o

It has now been two years since I saved the proot wiki page to read later, and finally with the time to go deeper, my first question was “how did they adapted a kernel module to userspace?”… I could be insanely wrong but anyway, his reminds me of virtual device drivers and slirp4netns, which in my head does something similar to network namespaces.

These thoughts come into mind when trying to learn raspberry pi.

1. How to debug a boot problem on raspbian (ARM) from a x86_64 host?

So I recently borrowed a rpi3 (Raspberry Pi 3 Model B v1.2 - Raspberry Pi 2015; CPU: Broadcom BCM2837RIFBG) to try and install a MQTT server. It recognizes a generic video projector, but not a TFT LCD which I got from the same person. When displaying the boot process, at startup the wpantund[20] DHCP enters into loop and that’s it.

My first approach was to use a sdcard reader and mount the filesystem using a bind mount:

; udisksctl mount -b /dev/sdc2

2. Can’t I just use chroot and get diagnostics?

The second step is to use chroot to change the root directory for the one mounted. This don’t work out of the box because chroot don’t handle well different architectures than the one you are currently using. This is actually because execve(2), see:

; file bin/busybox
bin/busybox: setuid ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 4.14.0, stripped
; ./bin/busybox
-bash: ./bin/busybox: cannot execute binary file: Exec format error

For that, there is qemu-user-static that handles “static user mode emulation” of various targets on QEMU.

sudo pacman -S qemu-user-static qemu-user-static-binfmt

By installing qemu-user-static, it populates the directory usr/bin/ with static emulators, but it is the qemu-user-static-binfmt package the one that makes it possible for you to run the binary without directly invoking qemu. The file magic/signature (in this case, any binary from the target storage unit, the sdcard) is read by binfmt_misc, which redirects the exec syscall to use the specific ISA emulator as interpreter. As the code snippet below shows, the first attempt to run busybox is executed with only the qemu-user-static installed. The second attempt have both packages installed.

; ./bin/busybox
-bash: ./bin/busybox: cannot execute binary file: Exec format error
;
; sudo pacman -S qemu-user-static-binfmt
;
; ./bin/busybox
qemu-arm-static: Could not open '/lib/ld-linux-armhf.so.3': No such file or directory
;

Now, you need to copy the qemu-<ARCH>-static to the storage. Other way to do this would be with a bind mount, but it seems overkill.

;
; sudo cp /usr/bin/qemu-arm-static ./usr/bin/
;
; ls -l ./usr/bin/qemu-arm-static
-rwxr-xr-x 1 root root 4033176 mai 24 08:18 ./usr/bin/qemu-arm-static
;
; file ./usr/bin/qemu-arm-static
./usr/bin/qemu-arm-static: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=98747a65af24f822274e0d8f1e5d49f58a632c51, for GNU/Linux 4.4.0, stripped
;

In comparison, check out how the busybox binary shows up in the output of file:

; file ./bin/busybox
./bin/busybox: setuid ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 4.14.0, stripped

As you can see by the output of file, the qemu-arm-static binary is a PIE/PIC, which stands for “Position Independent Executable/Code”. It means that every time you run the file it gets loaded into a different memory address, where you cannot hardcode values such as function addresses and gadget locations without finding out where they are.

Basically, PIC/PIE for respectivelly shared libraries and binaries allows code to be loaded into random memory locations, introducing an additional layer of protection against attacks or overall exploit mitigations, by making it required by ASLR. In the case of a program compiled without PIC/PIE, its data and text sections cannot be relocated, but ASLR can still be applied to the heap, stack and other dynamic libraries like libc.

The relation between PIC/PIE and ASLR is the mitigation: in attacks like ROP [26], the attacker must know or guess the absolute addresses of the ROP gadgets in the memory of the victim process. One defense is making the addresses hard to guess. This is the idea behind the Address Space Layout Randomization (ASLR for short) mitigation: by loading program segments at random addresses, the attacker cannot know the address without bruce-forcing, which could be impractical. This is not an ultimate solution though, just a mitigation. [24]

Building PIC/PIE binaries also requires that all of their dependencies are also built as Position Independent, and errors like the one below means the identified object file has not been built as Position Independent. [25]:

/[...]/ld: /path/to/object/file: relocation R_X86_64_32S against symbol
`symbol name' can not be used when making a PIE object;
recompile with -fPIE

Next step, the full script. To the chroot work, you need to mount some devices from host in the chroot target. The script below shows it, just do the inverse to unmount.

udisksctl mount -b /dev/sdc2
# /run/media/$USER/random_number/
cd /run/media/$USER/random_number/

# on target rootfs, bind local rootfs directories in the
# chroot target rootfs
for f in 'dev' 'dev/pts' 'sys' 'proc' 'run'; do
   sudo mount --bind /$f /run/media/$USER/random_number/$f;
done

PS: when finished, unmount the directories doing the reverse:

for f in 'dev' 'dev/pts' 'sys' 'proc' 'run'; do
    sudo umount -b /run/media/$USER/random_number/$f;
done

Now chroot into the bind mounted target. Here I invoke the busybox POSIX shell /bin/sh

sudo chroot . /bin/sh

3. What are the differences between the default bind mount vs udisks/udisksctl?

fusermount vs udisks2 vs bind mounts Basically udisksctl uses fusermount. Other than this,

Basically, there’s udev which have daemon that acts as an interface with the kernel through userspace handlers, which detects kernel generated events and adjusts permission levels for non-privileged users.

It means you can mount a peripheral device from userspace, and this device can be a storage device. So by using bind(2) you need root, but udisks2 leverages the udev daemon and that’s what makes it possible.

Bind mounts are available since Linux 2.4 with the MS_BIND flag in the system function (mount(2)). It makes a file or directory subtree visible at another point within the single directory hierarchy. Bind mounts may cross filesystem boundaries and span chroot(2) jails, which is the difference between them and soft links

4. What about if I wanted to use the smartphone SDcard reader and mount the filesystem from there?

You would have to interact with Object Device Trees, which are hardware descriptions in a declarative file (similar to YAML), but

5. What are Device Trees?

device tree blob (dtb), device tree object (dto) or object device tree (odt) are hardware descriptions in declarative files (similar to YAML) used by firmwares and embedded kernels such as Android and Raspberry Pi. Android uses it too to recognize devices.

They are, as the name implies, Tree Data Structures. The nodes describe system devices, and each node has key-value pairs describing characteristics of the device being represented. A Device Tree Blob (dtb) is the binary representation of this data.

Device Trees Overlay modifies dynamically the Device Tree, allowing addition, removal or other modifications at runtime. On raspberry pi, they make possible to support Hardware on Top (HAT). It also is good to make changes without modifying the base dtb.

Lingo:

6. Wait, this was build from Buildroot, by inspecting the rootfs. How does buildroot works?

Buildroot is a distro generator for Linux and other operating systems, targeting embedded hosts. This usually means cross-compilation, that is, to compile for a different Instruction Set Architecture other than your host machine, which comes with a toolchain acoordingly.

Buildroot’s interface is based in ncurses with menuconfig like the kernel or busybox. udev

7. How does wpantund DHCP dynamic IP assignment works?

8. How does LCD display works on raspberry pi? How to configure a LCD that was not detected?

9. Can I mount the target filesystem on termux from the smartphone sdcard reader?

10. How does chroot works

chroot works by finding

11. How does binfmt_misc works

12. What is the role of qemu-user-static?

13. What about fakeroot?

From the manpage: “[…] fakeroot is a regular, non-setuid program. It does not enhance a user’s privileges, or decrease the system’s security.”. “[…] fakeroot was specifically written to enable users to create Debian GNU/Linux packages (in the deb(5) format) without giving them root privileges. […]”. “[…] fakeroot works by replacing the file manipulation library functions (chmod(2), stat(2) etc.) by ones that simulate the effect the real library functions would have had, had the user really been root. These wrapper functions are in a shared library /usr/lib//libfakeroot-.so or similar location on your platform. The shared object is loaded through the LD_PRELOAD mechanism of the dynamic loader. (See ld.so(8)) […]”

For example, in LFS and BLFS, packages are usually manually installed, but you can also use fakeroot for this task, which can act as a package manager to facillitate system maintenance. Besides, it doesn’t need any dependency because it is statically linked. The overall process of package managing with fakeroot consists in these basic steps:

14. How startalpine/AlpineOnTermux does it

Like other Termux distros, the startalpine project uses fakeroot to fake… a root… superuser. Seriously, it combines fakeroot with the LD_PRELOAD trick and Proot that leverages qemu-user-static and binfmt_misc.

15. Proot: storage virtualization in userspace

Proot is one of those softwares that makes it possible any madness you come up out of the pocket. Although it is awesome, is not magic: it leverages binfmt_misc, qemu-user-static and chroot

16. What is the relation of FUSE and fusermount with all of this?

17. What if I want to do something like this from a Kotlin/Java app for Android?

18. Extra: binfmt_misc and wine interop

19. “if Proot offers usermode qemu as equivalence to binfmt_misc (alongside chroot and bind mount), what is the difference between usermode qemu to the qemu-user-static which uses binfmt_misc?”

References

[16/05/2024 20:17]

; [17/05/2024 03:04]

“if Proot offers usermode qemu as equivalence to binfmt_misc (alongside chroot and bind mount), what is the difference between usermode qemu to the qemu-user-static which uses binfmt_misc?”

; [17/05/2024 03:09]