[BUNDLE] Add script to bundle x86 libraries (#2380)

This manages both x86_64 and x86 libraries. It helps with improving emulation in all of the Box projects.
2025-05-07 15:36:41 +08:00 · 2025-02-17 13:32:12 -07:00 · 2025-02-17 13:32:12 -07:00 · c27b659bce
commit c27b659bce
parent 049cbd3783
5 changed files with 212 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -73,3 +73,17 @@ backup/

 # macOS
 .DS_Store
+
+# Archives
+*.7z
+*.deb
+*.eopkg
+*.gz
+*.rar
+*.rpm
+*.tar*
+*.xbps
+*.xz
+*.zip
+*.zst
+*.zstd
--- a/README.md
+++ b/README.md
@ -34,6 +34,7 @@ Box64 offers environment variables to control its behavior. For details, see [Us
 ## 🚀 Compiling and Installation

 - [Compilation Instructions](https://github.com/ptitSeb/box64/blob/main/docs/COMPILE.md)
+- [Bundle x86 Libraries](https://github.com/ptitSeb/box64/blob/main/docs/BUNDLE-X86-LIBS.md)
 - [Wine Installation for Box64](https://github.com/ptitSeb/box64/blob/main/docs/X64WINE.md)

 ---
--- a/box64-bundle-x86-libs.csv
+++ b/box64-bundle-x86-libs.csv
@ -0,0 +1,38 @@
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/bluez-libs-5.72-2.el9.i686.rpm,af1aba8e6fa5ccf33032d58a3acc59368c07d532293396db205fb9a4bc1241ae
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/bluez-libs-5.72-2.el9.x86_64.rpm,49c273813b4616240acdae8dae893192658c7541482f9bbb2c46dfe19b61877d
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/fribidi-1.0.10-6.el9.2.i686.rpm,75f808f0f5f93aceb4e477aa5f302edbed07a605a0644f0ea929d4ea609e36a0
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/fribidi-1.0.10-6.el9.2.x86_64.rpm,b4763ca410f8479b2775efb55ba09e6722948af7526180a56309e3e090c27010
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/gmp-6.2.0-13.el9.i686.rpm,fe352f7d197ebdf2c92bbd6e548f9f6b7ac9e69e7a9d6c9861a6274482e79d73
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/gmp-6.2.0-13.el9.x86_64.rpm,babe0b1385e723476a216d5f4b72453e031c3d683b19344786c2bddf7203cb04
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/gnutls-3.8.3-4.el9_4.i686.rpm,64cf0b0136d3f9b80b07ce2d4b8ce6a4ab86617e8ef9baf560bdfa5edef825ea
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/gnutls-3.8.3-4.el9_4.x86_64.rpm,16c48c504fe62f8e19cac7c6442d3e08bba487f610c2f533116dac32357cda86
+https://archive.debian.org/debian/pool/main/c/curl/libcurl3-gnutls_7.74.0-1.3+deb11u7~bpo11+1_amd64.deb,d5ae55e3a9838d12358530cd03218cc0d18127a84e9cfc73bdf79dacd40b25e4
+https://archive.debian.org/debian/pool/main/c/curl/libcurl3-gnutls_7.74.0-1.3+deb11u7~bpo11+1_i386.deb,625e5189304d4d3bfb27cb821e257816c5ce761b1696b5da5ca4fa1035c48463
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libgomp-11.5.0-2.el9.alma.1.i686.rpm,686ac26afe2073a3607f26fd1592a8ccd9b78f6eec6e872528634b43c4e04dd7
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libgomp-11.5.0-2.el9.alma.1.x86_64.rpm,9a2bdc6807b22ffb6294b30dcc13e20198abe5facaeefcf1bfacec3e1013e3c2
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libICE-1.0.10-8.el9.i686.rpm,d9c44bb34c990fdcc6987feefacfc9e1e7997ca25f1e0a59c9adc9c16f2266d1
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libICE-1.0.10-8.el9.x86_64.rpm,d0b867203384942985c3c4ad6fc9ec0000fcc9965408451a6cdef8df06a3833c
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libidn2-2.3.0-7.el9.i686.rpm,4458e2d416b5e6714ec664c4f88791eee7d1e707026a5854d948a6159911465b
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libidn2-2.3.0-7.el9.x86_64.rpm,c3ff43237c05258ee0dd77c28540fc8d0d824f1567b115ddfd555b78d2ee0e7a
+https://cdn.getsol.us/repo/shannon/libj/libjpeg-turbo/libjpeg-turbo-32bit-3.0.3-21-1-x86_64.eopkg,5624990281f6520026aa2252c711f02d4916099aeac63accc83bd2135b9d3220
+https://cdn.getsol.us/repo/shannon/libj/libjpeg-turbo/libjpeg-turbo-3.0.3-21-1-x86_64.eopkg,c92f19d446b4c2450306f13b1a908f2552400da09301996fe349fee243d964d2
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libogg-1.3.4-6.el9.i686.rpm,9e4ff6fa0251c5357ae8f648b3e396d0043d5b3f2fb28d23325f5bd592c67aee
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libogg-1.3.4-6.el9.x86_64.rpm,83b4b333e335654674032d1cda4d7deb328b5a409f95e3952c3dc482b684fd8f
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libpng-1.6.37-12.el9.i686.rpm,66bf967e84f0d8e7a0059f2b7dda33482ff5e4fe088d0e3bc86dc54e6c7c97be
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libpng-1.6.37-12.el9.x86_64.rpm,ebb3e927dfafa0b199e00e2bef4c10c85fc82a70c359faf9aeb599eda70bfa5e
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libSM-1.2.3-10.el9.i686.rpm,ec68b3def828ba442248f3cff495f42c4af3dd58f38c8daad07dfddc109d999a
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libSM-1.2.3-10.el9.x86_64.rpm,d00318843a50138829b470a97b8db1a00723b10bca23c16a8386d9ab670e4c64
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libtasn1-4.16.0-8.el9_1.i686.rpm,9cbd11277e89bd0d4a80900bee01ca71b0c582c9cb5744f5035135b2ed907c85
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libtasn1-4.16.0-8.el9_1.x86_64.rpm,2dd3a1ad2785df642cb03dfed6dfbeec929ff8b14e52e61272ca7e89f86ba3cd
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libtheora-1.1.1-31.el9.i686.rpm,2155c688b516ab359d0c47be6d73e51d3c65f199ed5591f7bb7ce6d5d9799977
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libtheora-1.1.1-31.el9.x86_64.rpm,25ab4a8ad997b450b4a34a777f5a0678eea72ca1665944a7a461ac63de4b25bf
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libunistring-0.9.10-15.el9.i686.rpm,792600a879fbd94ce4bf249713b0cb3a166c147cfad397e8a15e6b4f01c75997
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/libunistring-0.9.10-15.el9.x86_64.rpm,d2f86fbee729a4900a433a6c52a3cb95d9b52b1d79c9e719ee3ef509813eb6d9
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libvorbis-1.3.7-5.el9.i686.rpm,e38c5185fea611330ea253db020315dd564507bf95c48a81dbe7db186ce4214b
+https://repo.almalinux.org/almalinux/9.5/AppStream/x86_64/os/Packages/libvorbis-1.3.7-5.el9.x86_64.rpm,3c340337a073975636d198a282503f32a54bf85fb5345522a2c22e530b8d6c9e
+https://repo.almalinux.org/almalinux/8.10/BaseOS/x86_64/os/Packages/nettle-3.4.1-7.el8.i686.rpm,13bc8758355c3ec885f414e0e86efad29a7fe82c0999cdb3754d666949c06cc5
+https://repo.almalinux.org/almalinux/8.10/BaseOS/x86_64/os/Packages/nettle-3.4.1-7.el8.x86_64.rpm,fb4aaa22b2e2f2fc0c451b87ac464e8746d240397d64e79f6bfd38f4e9128a5a
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/nettle-3.9.1-1.el9.i686.rpm,e5610fe3b25cfca1616440e1a8a7489fd37b0852f182fdda87ab8e196d981970
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/nettle-3.9.1-1.el9.x86_64.rpm,9786b6610d17aa495fb640ba6d046bf73704915b82d12e6d8339090ff4891de5
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/p11-kit-0.25.3-3.el9_5.i686.rpm,abe4ab924b405c236eaf3fa067fed50ac66d15d0ef42c782a745788c1a9c5b91
+https://repo.almalinux.org/almalinux/9.5/BaseOS/x86_64/os/Packages/p11-kit-0.25.3-3.el9_5.x86_64.rpm,0e1bd97e697a9fa3b4e508b945cf352b73b06e4e8218c14ac5fb912b0022c646
--- a/box64-bundle-x86-libs.sh
+++ b/box64-bundle-x86-libs.sh
@ -0,0 +1,93 @@
+#!/bin/bash
+
+set -x
+
+extract_pkg_auto() {
+    # Example x86_64 package name: libcurl3-gnutls_7.74.0-1.3+deb11u7~bpo11+1_amd64.deb
+    # Example x86 package name: libcurl3-gnutls_7.74.0-1.3+deb11u7~bpo11+1_i386.deb
+    if echo "${1}" | grep -q -P ".deb$"; then
+        echo "DEB (Debian) package detected."
+        ar x "$(basename "${1}")"
+        tar --extract --file data.tar*
+    # Example x86_64 package name: curl-gnutls-8.1.2-44-1-x86_64.eopkg
+    # Example x86 package name: curl-gnutls-32bit-8.1.2-44-1-x86_64.eopkg
+    elif echo "${1}" | grep -q -P ".eopkg$"; then
+        echo "EOPKG (Solus Linux) package detected."
+        unzip -o "$(basename "${1}")"
+        tar --extract --file install.tar.xz
+    # Example x86_64 package name: bluez-libs-5.63-1.el8.x86_64.rpm
+    # Example x86 package name: bluez-libs-5.63-1.el8.i686.rpm
+    elif echo "${1}" | grep -q -P ".rpm$"; then
+        echo "RPM (Fedora) detected."
+        rpm2cpio "$(basename "${1}")" | cpio -idmv
+    # Example x86_64 package name: libjpeg-turbo-3.0.1_1.x86_64.xbps
+    # Example x86 package name: libjpeg-turbo-32bit-3.0.1_1.x86_64.xbps
+    elif echo "${1}" | grep -q -P ".xbps$"; then
+        echo "XBPS (Void Linux) package detected."
+        tar --extract --file "$(basename "${1}")"
+    else
+        echo "Unknown package."
+        echo "Unsupported package type."
+        exit 1
+    fi
+}
+
+box64_dir=$(pwd)
+dir_tmp_local="$(mktemp --directory)"
+cd "${dir_tmp_local}"
+
+for line in $(cat "${box64_dir}/box64-bundle-x86-libs.csv");
+    do pkg_url="$(echo "${line}" | cut -d, -f1)"
+    pkg_checksum_expected="$(echo "${line}" | cut -d, -f2)"
+    pkg_name="$(basename "${pkg_url}")"
+    curl --silent --location --remote-name "${pkg_url}"
+    pkg_checksum_actual="$(sha256sum ${pkg_name} | awk '{print $1}')"
+    if [[ "${pkg_checksum_expected}" -ne "${pkg_checksum_actual}" ]];
+        then echo "Invalid checksum for ${pkg_name}"
+        echo "Expected: ${pkg_checksum_expected}"
+        echo "Actual: ${pkg_checksum_actual}"
+        exit 1
+    fi
+    extract_pkg_auto "${pkg_name}"
+done
+
+# Start preperation of bundles.
+mkdir -p \
+  "${dir_tmp_local}/bundle-libs/usr/lib/box64-i386-linux-gnu" \
+  "${dir_tmp_local}/bundle-libs/usr/lib/box64-x86_64-linux-gnu" \
+  "${dir_tmp_local}/bundle-pkgs" \
+
+# Copy Debian family libraries.
+cp --archive "${dir_tmp_local}"/usr/lib/i386-linux-gnu/*.so* "${dir_tmp_local}/bundle-libs/usr/lib/box64-i386-linux-gnu"
+cp --archive "${dir_tmp_local}"/usr/lib/x86_64-linux-gnu/*.so* "${dir_tmp_local}/bundle-libs/usr/lib/box64-x86_64-linux-gnu"
+# Copy Fedora family libraries.
+cp --archive "${dir_tmp_local}"/usr/lib/*.so* "${dir_tmp_local}/bundle-libs/usr/lib/box64-i386-linux-gnu"
+cp --archive "${dir_tmp_local}"/usr/lib64/*.so* "${dir_tmp_local}/bundle-libs/usr/lib/box64-x86_64-linux-gnu"
+# Copy Solus and Void Linux family libraries.
+cp --archive "${dir_tmp_local}"/usr/lib32/*.so* "${dir_tmp_local}/bundle-libs/usr/lib/box64-i386-linux-gnu"
+
+# Create "<LIBRARY_FILE>.so" symlinks.
+for dir_lib in \
+  "${dir_tmp_local}/bundle-libs/usr/lib/box64-i386-linux-gnu" \
+  "${dir_tmp_local}/bundle-libs/usr/lib/box64-x86_64-linux-gnu"
+    # If two or more libraries of the same name exist, this will symlink the oldest version.
+    # Pipe this into a reverse sort `| sort -r` to symlink the newest version.
+    do for lib in $(ls -1 "${dir_lib}")
+        # Example output: libbluetooth.so
+        do lib_base="$(echo "${lib}" | cut -d. -f1,2)"
+        if ! ls "${dir_lib}/${lib_base}" 2> /dev/null
+            then ln -s "${lib}" "${dir_lib}/${lib_base}"
+        fi
+    done
+done
+
+mv "${dir_tmp_local}"/*.deb "${dir_tmp_local}/bundle-pkgs/"
+mv "${dir_tmp_local}"/*.eopkg "${dir_tmp_local}/bundle-pkgs/"
+mv "${dir_tmp_local}"/*.rpm "${dir_tmp_local}/bundle-pkgs/"
+
+tar --directory "${dir_tmp_local}/bundle-libs" --create --file "${box64_dir}/box64-bundle-x86-libs.tar.gz" .
+tar --directory "${dir_tmp_local}/bundle-pkgs" --create --file "${box64_dir}/box64-bundle-x86-pkgs.tar.gz" .
+
+# Clean up.
+cd -
+rm -r -f "${dir_tmp_local}"
--- a/docs/BUNDLE-X86-LIBS.md
+++ b/docs/BUNDLE-X86-LIBS.md
@ -0,0 +1,66 @@
+# Bundling x86 Libraries
+
+The script `box64-bundle-x86-libs.sh` is provided to download, extract, and re-bundle x86_64 and x86 libraries for use with Box64/Box32 and Box86. These libraries improve Box emulation. This bundle provides a much smaller size compared to using a full root file system of an operating system.
+
+## Usage
+
+### Administrator
+
+Install required dependencies for the bundle script.
+
+- Debian
+    ```
+    sudo apt-get update
+    sudo apt-get install coreutils curl binutils rpm2cpio tar zstd
+    ```
+
+Run `box64-bundle-x86-libs.sh`. This will create two archives:
+- `box64-bundle-x86-libs.tar.gz` = All of the extracted library files in the directory structure of `usr/lib/box64-i386-linux-gnu` and `usr/lib/box64-x86_64-linux-gnu`.
+- `box64-bundle-x86-pkgs.tar.gz` = All of the Linux distribution packages used to extract the library files. This is only created for preservation purposes and is otherwise unused.
+
+Extract the library archive. Box will automatically search these paths for library files to emulate.
+
+```
+sudo tar --extract --file box64-bundle-x86-libs.tar.gz --directory /
+```
+
+### Developer
+
+Add new packages to the `box64-bundle-x86-libs.csv` file in the format of `<PACKAGE_URL>,<SHA256_CHECKSUM>`.
+
+Supported packages:
+- DEB (Debian)
+- EOPKG (Solus)
+- RPM (Fedora)
+- XBPS (Void Linux)
+
+Preference of operating system packages that provide a library:
+1. RPMs from AlmaLinux 9 provide 10 years of updates.
+2. DEBs from Debian 11 provide 5 years of updates.
+3. EOPKG from Solus provide old variants of obscure 32-bit libraries.
+4. XBPS from Void Linux provide updated variants of obscure 32-bit libraries.
+5. Other (whereever a library is packaged)
+
+For finding package names that contian a specific library file, use the [pkgs.org](https://pkgs.org/) website. Otherwise, use one of these package manager commands on a x86_64 Linux distribution:
+- DEB
+    - `apt-file <LIBRARY_FILE>`
+- EOPKG
+    - Does not support file search. Use pkgs.org instead.
+- RPM
+    - `dnf provides <LIBRARY_FILE>`
+- XBPS
+    - `xbps-query --repository --ownedby <LIBRARY_FILE>`
+
+Once the package name is found, find the exact URL for downloading it.
+- DEB = Explore "main" and "contrib". Search for packages with the name "deb11" in it. Use the "amd64.deb" and "i386.deb" packages.
+    - https://archive.debian.org/debian/pool/
+- EOPKG = Use the "`<VERSION>-<RELEASE>-x86_64.eopkg`" and "`32bit-<VERSION>-<RELEASE>-x86_64.eopkg`" packages.
+    - https://cdn.getsol.us/repo/shannon/
+- RPM = Navigate to the "`<MAJOR>.<MINOR>`" version directory first. Then explore "AppStream" and "BaseOS". Use the "x86_64.rpm" and "i686.rpm" packages.
+    - https://repo.almalinux.org/almalinux/
+- XBPS = There are too many files in the repository to display on a web browser. Use pkgs.org instead to find the exact URL for downloading it.
+    - https://repo-default.voidlinux.org/current/
+
+Download the package and then find the checksum with the `sha256sum` command.
+
+Major Linux distributions such as AlmaLinux and Debian provide archives of packages. Rolling distributions such as Arch Linux, Solus, and Void Linux do not. In those cases, use the [Internet Archive](https://web.archive.org/) to archive any rolling release package used by Box by using the "Save Page Now" feature. If the error "Save Page Now browser crashed" appears, it can be safely ignored. The archive will still have been saved.