Add a new test that checks that no tasks between do_fetch (exclusive)
and do_build (inclusive) are allowed to use the network, with rare
exceptions.
The only exception currently is build-appliance-image's do_image task,
as that currently usese pip to install the required Toaster dependencies.
Note that this will mean layers that have Go-based recipes will fail
unless they're using the gomod fetcher and have a complete list of
modules in the SRC_URI.
(From OE-Core rev: e95b3bd194e294412bc0419c9c74abfc2f37406f)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rearrange the class so that the check is a separate function, to make it
neater to add further tests in the future.
(From OE-Core rev: 36658a113811de98249bc1e1b79cfadd405e5391)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We call the installer script but if it fails the user doesn't see the
reason, which makes resolving the problem hard. Capture both stdout and
stderr as text, and display it to the user when it fails.
(From OE-Core rev: c67fbe17ce063350181c28f99662004737d8b770)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than reporting each invalid PACKAGECONFIG with a separate error
message, report them all with one error message.
(From OE-Core rev: bf9366583f53fe2498d7aa9192ebfe6562887cf3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes sure invalid PACKAGECONFIGs are reported also for recipes
that have no do_configure task, e.g., packagegroups.
(From OE-Core rev: d3325c384a7df54c564cae093659cf7b692629f2)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is needed to build nodejs from meta-oe. Check this early to avoid
an error later in the build.
Fixes [YOCTO #15804]
(From OE-Core rev: 19004950ad5691ef51f7fcf946558bd0de561403)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can be use to check for other flags.
(From OE-Core rev: 42d012e8414736d89618422fb6f9bce9e4a2a391)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Ubuntu 20.04 is heading to EoL and that we're having other issues
on that platform needing buildtools anyway, we're about to lose the
last python 3.8 platform we were supporting.
Bump the minimum version to 3.9 since there are many developers wanting
access to newer python features.
This likely won't be seen since the bitbake minimum has also been
increased but helps for documentation purposes.
(From OE-Core rev: c3db241c6e97420ca0705d1535a7cdd0c273b761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change DEBIAN_MIRROR to point at the canonical server, deb.debian.org.
This is a CDN-backed server using network magic to load balance across
the planet so there's no need to set a slew of regional mirrors.
Also add a more recent snapshot.debian.org from the beginning of 2025.
(From OE-Core rev: 3d95d45836accd29916dd8cb9bfe624d63d6c202)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In rare cases BUILDNAME can seemingly be None outside of heartbeat events
which leads to UnboundLocalErrors as bsdir and taskdir aren't defined.
Skip the code in these cases rather than generate tracebacks which cause
bitbake server to exit.
(From OE-Core rev: 0f74d804ba0daf7e8bd6481597740b9d89821414)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This prevented package_qa sstate from being reusable unless host uid/gid
values would match exactly (and they unfortunately do on the yocto autobuilder
worker machines which all share a 'pokybuild' user).
I noticed this when testing CDN sstate reuse, which otherwise works well.
(From OE-Core rev: 6ea8b4b10b0549c858427a8411bf2a4cd5c0eb7b)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some modules (like Perl's MakeMaker) do not support non ASCII
characters in build folder's path, this would cause build failures
of software that does not support non ASCII.
A sanity check is added to warn the user.
Fixes [YOCTO #15764]
(From OE-Core rev: 2b3be97a0d0d60d026786a4465b24b6f6752ba32)
Signed-off-by: Sofiane HAMAM <sofiane.hamam@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't have many cross recipes that use PACKAGECONFIG but gdb-cross does,
so correctly remap dependencies for that case allowing the gdb recipe to be
simplified.
(From OE-Core rev: c5bb7976f4d6e8559b4b87e4c3f39135dbb40ef8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
opkg accepts both - and _ in its options but use the help text
preferred format. From https://github.com/yoctoproject/poky/pull/14
from Philipp-Alexander Blum <blum@gessler.de>.
(From OE-Core rev: 4bea64347407dfb1eb46dacffd05af3bd200b419)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
meta-oe has switched from p7zip to 7zip.
p7zip recipe does not exist anymore and p7zip is provided and rprovided
by 7zip recipe.
Use real provider instead of replaced one.
(From OE-Core rev: 5aa516bfa295d5be919459dfe45f452cdec45e81)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recipes are much more readable with whitespace around the assignment operators.
Fix various assignments in OE-Core to show this is definitely the preferred
formatting.
(From OE-Core rev: 30ea609d3357fb3de911f2f6a5e6856c151b976a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users reported issues caused by missing the right libstdc++-version-dev.
A new function 'check_cpp_toolchain' added in sanity.bbclass to test linking libstdc++
[YOCTO #15712]
(From OE-Core rev: 611c1a26212dfbfe8d0640d9fefe5df49f7b69b8)
Signed-off-by: Christos Gavros <gavrosc@yahoo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea of the base class dependency is to say "yes, I need a C cross compiler"
and this was never meant to be gcc specific. Looking at the codebase, whilst we
code triplets into this, it does overcomplicate things as there are only ever
limited, "target", "sdk" and the class extended versions like mutlilib.
After much thought, we can simplify this to virtual/cross-cc and virtual/nativesdk-cross-cc.
This lets us remove the "gcc" specific element as well as removing the over
complicated triplet usage.
At the same time, change the much less widely used "g++" variant to "c++" for
similar reasons and remove the triplet from virtual/XXX-binutils too.
Backwards compatibility mappings could be left but are just going to confuse
things in future so we'll just require users to update.
This simplification, whilst disruptive for any toolchain focused layers, will
make improved toolchain selection in the future much easier.
Since we no longer have overlapping variables, some code for that can just
be removed. The class extension code does need to start remapping some variables
but not the crosssdk target recipe names.
This patch is in two pieces, this one handles the renaming with the functional
changes separate in a second for easier review even if this breaks bisection.
(From OE-Core rev: 4ccc3bc8266c327bcc18c9a3faf7536210dfb9f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git commit hashes for the kernel checkout are not reproducible under
certain conditions:
- If the git repository is initialized on an archive (rather than a
git), the initial git commit not only has the current user name set,
it also uses the current system time as committer and author date.
This will affect the initial git hash and thus all subsequent ones.
- The patches applied by the kern-tools have a valid author and date.
However, their committer again depends on the user building the BSP.
This is an issue, for example, if one compiles a kernel with
CONFIG_LOCALVERSION_AUTO enabled where the commit hash lands into the
kernel and thus the package version. This not only makes the package
version non-reproducible, but also leads to version mismatches between
kernel modules built against a fresh kernel checkout and the kernel
retrieved from the sstate cache.
The class uses 'check_git_config' from utils.bbclass, but this only sets
the git user and only if none existed before. Thus it doesn't really
help here.
Since in Git the committer information can be set only from the
environment variables GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, and
GIT_COMMITTER_DATE, we introduce a helper function to set those and
apply the author settings in the same way.
As values simply use PATCH_GIT_USER_NAME, PATCH_GIT_USER_EMAIL (from
patch.bbclass) and SOURCE_DATE_EPOCH.
For convenience, put the new helper 'reproducible_git_committer_author'
into utils.bbclass next to 'check_git_config' so others can use it, too.
Using this helper in kernel-yocto.bbclass makes the committer and author
date/name/email for the initial commit reproducible, as well as the
committer name/email for the patches applied with kern-tools.
For debugging purpose, allow disabling the reproducibility features by
setting KERNEL_DEBUG_TIMESTAMPS to "1".
Suggested-by: Felix Klöckner <F.Kloeckner@weinmann-emt.de>
(From OE-Core rev: aab4517b4649917abd519ea85a20fd9d51bf3d99)
Signed-off-by: Enrico Jörns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when the do_package_write_rpm task is executed
when bitbake linux-yocto:
warning: %clamp_mtime_to_source_date_epoch is deprecated, please use %build_mtime_policy
After RPM is upgraded to 4.20.0, the %clamp_mtime_to_source_date_epoch macro is
now deprecated in favor of %build_mtime_policy set to clamp_to_source_date_epoch.
(From OE-Core rev: fa9b9e599bbf3ac73c7a9b1d15ccb0e0bbbe4c11)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When TMPDIR has length 410, bitbake uninative-tarball will fail at
do_populate_sdk with error: Failed to open backup database:
"TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite"
sqlite3 has a hardcode max pathname limit 512, refer [1], and there is a
check, refer [2], pathname + 8 > max path name limit(512) is not
allowed. So the pathname length need to <= 504. While, length of
"TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite"
is 505, so error happened.
We had met similar issue [3], and refer the comments, upstream reject
the change of the hardcode length, see [4][5].
This patch try to improve and workaround this issue by lower max TMPDIR
length to 400, at least for distro name length <=13, the build can
success.
[1] https://github.com/sqlite/sqlite/blob/master/src/os_unix.c#L180
[2] https://github.com/sqlite/sqlite/blob/master/src/pager.c#L4777
[3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=12374
[4] https://sqlite.org/cgi/src/tktview/c060923a5422590b3734
[5] https://sqlite.org/forum/forumpost/0b1b8b5116
(From OE-Core rev: da1c0998bdaf5def7ae94f44a33b9a505edc99c1)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that value "0" can turn them off.
(From OE-Core rev: 1a2afcd0773c5ec5946d1dfc27bde585c52a5724)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of iterating every datastore value by hand to find tasks, use
the new bb.build.listtasks() function (bitbake 185c4b)
(From OE-Core rev: 466c3ed0d01bc70caa29d5eb8bb99f7d0e6e710c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this release, as expected, deprecated openpgp support
has been fully removed and replaced with sequoia:
https://github.com/rpm-software-management/rpm-sequoia/
Alas, it's written in rust, and has no recipe, so for now
signing rpm packages has to be disabled.
Remove package name parameter to %prep in source archiving, as it's
been dropped in the new rpm, and wasn't needed to begin with.
Drop 0001-perl-disable-auto-reqs.patch
(files removed upstream).
(From OE-Core rev: 8c15b4577d5e554cc2dd5adfb88b816894b05a9a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During handle PACKAGECONFIG, DEPENDS/RDEPENDS/RRECOMMENDS is appended by
calling function appendVar according to the PACKAGECONFIG setting:
PACKAGECONFIG[f1] = "\
--with-f1, \
--without-f1, \
build-deps-for-f1, \
runtime-deps-for-f1, \
runtime-recommends-for-f1, \
packageconfig-conflicts-for-f1"
[snip of base.bbclass]
pkgconfigflags = d.getVarFlags("PACKAGECONFIG") or {}
if pkgconfigflags:
...
appendVar('DEPENDS', extradeps)
appendVar('RDEPENDS:${PN}', extrardeps)
appendVar('RRECOMMENDS:${PN}', extrarrecs)
...
[snip of base.bbclass]
After following commit in meta-oe,
[fe3ba7ac5c cryptsetup: demote udev runtime dependency to recommendation]
'udev lvm2' changed from RDEPENDS to RRECOMMENDS, but RRECOMMENDS is not
handled in this process, and causes following error:
$bitbake lib32-cryptsetup
ERROR: QA Issue: lib32-cryptsetup package lib32-cryptsetup - suspicious values 'udev lvm2' in RRECOMMENDS [multilib]
In layer openembedded-core, we do have very few recipes that have
runtime-recommends-for-f1 in PACKAGECONFIG setting, but f1 is not
enabled in PACKAGECONFIGS, so not report error when multilib is enabled.
(From OE-Core rev: 2f539a1318a80d3ad5d0d43a9ba4070465e402af)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The check to see if a provider of a given package is listed first
checks for an exact match of the provider name. However, if this match
existed, but didn't match in the task dependencies, it would not
continue to look for other providers of package. This would manifest if
one (non-virtual) recipe package RPROVIDES the name of a package
produced by another recipe.
Fix this, and also clean up the code to make it more readable by using a
function to check if a runtime dependency is in the task dependencies.
In addition, if no provider is found, list all the possible providers
instead of the last one that was looked at.
(From OE-Core rev: f13de6ab616eb1e38960a2296111febe2a9f4a28)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two files are the only ones that are left in meta/lib.
They logically belong to meta/lib/oe, so move them there.
(From OE-Core rev: c65dd0e3e463d6072b9364ac74e1fef0d998068f)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
insane.bbclass now defines CHECKLAYER_REQUIRED_TESTS list with required
QA checks that are becoming mandatory for Yocto Project Compatible layers.
Update yocto-check-layer.bbclass in order to catch when packages from such
layers try to skip any of the required QA checks.
(From OE-Core rev: 9c3ba88628853b20fb4c98c99cf3fe8349024016)
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like to start requiring some set of checks are enabled for Yocto Project
Compatible Status and to pass yocto-check-layer. Start by splitting ERROR_QA
into two sets, the ones we think can be required and the ones we know have
challenges to implement (e.g. with prebuilt binaries).
To change the required list, the YP TSC would need to approve.
(From OE-Core rev: ce2e42ace2d15fb6745437cf0a7f07d28398ca12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TL;DR version:
with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:
webkitgtk goes from 78 seconds to 37 seconds
glibc-locale goes from 399 seconds to 58 seconds (!)
The long version:
rpm uses multithreading for two purposes:
- spawning compressors (which are nowadays themselves
multi-threaded, so the feature is not as useful as it once
was)
- parallel file classification
While the former behaves well on massively parallel CPUs
(it was written and verified here :), the latter was then added
by upstream and only benchmarked on their very old, slow laptop,
apparently:
41f0e214f2
On anything more capable it starts showing pathologic behavior,
presumably from spawning massive amount of very short-lived threads,
and then having to synchronize them. For example classifying glibc-locale
takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads
Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads
I have found that the optimal amount is actually four: this also
means that only four compressors are running at a time, but
as they're themselves using threads, and typical recipes are dominated
by just two or three large packages, this does not affect overall
completion time.
(From OE-Core rev: 286d456e71ee2730c197ce394d6be2c7eeced18d)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that),
while 19 is the highest and slowest.
It's not clear why 19 was picked to begin with, possibly
I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66
This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.
The rpm size increases from 175648k to 234860k. I think it's
a worthy default tradeoff.
(From OE-Core rev: f1c7d76361425c85d68ad1f61be1e7ff05df4bab)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As sstate_archive_package just calls tar, writing the function in shell
is actually more complex and opaque than the equivalent python.
Don't check for zstd vs pzstd, we have pzstd in HOSTTOOLS so it will
always be available.
(From OE-Core rev: c9ac5d9d94f254292cf3cafdf273dd6b61d3baa7)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was changing the timestamps of the files in the do_package output,
particularly the files added for debug sources. This was to do two things:
a) make do_package sstate more reproducible
b) ensure better hash equivalence matching
Unfortuately the debug source files are hardlinks into the source tree for
efficiency so touching these, touches a lot of files in ${B} and ${S}. This
causes unpredictable effects if compile is run again for example, or could
cause compiling in the install task.
The hash equivalence matching is of key importance but we can mimic that
using clamping of the file timestamps in the depsig output used to generate
the hashes.
This patch drops the global timestamp clamping, instead allowing the files
to retain their creation timestamps into sstate. This makes do_package sstate
slightly less reproducibile. We could clamp the sstate timestamps but that
would lead to two different sets of timestamps depending on whether the
data came from sstate or not. I'd prefer to have consistent code behaviour,
rather than differing behavhour depending on whether data came from sstate
or not.
If we wanted to have reproducibiliy and fix the "corruption" of S/B and have
consistent codepaths, the only other option would be two copies of the
sources, which could end up huge and seems the least desireable option.
This patch therefore drops the timestamp clamping in the sstate files
and tweaks the depsig data generation to clamp the timestamps for do_package
instead since this seems the best compromise.
I validated that rpm/deb/ipk files still generate correctly as before.
(From OE-Core rev: 475759fdab7200488b2a568b2ba1aa31a456d113)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.
The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since it was added as
a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.
(From OE-Core rev: 1816c5a5e724a7f907c9afe4a47b6699ea081ace)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations
(From OE-Core rev: 86eb409e3c1b30110869ec5a0027ae2d48bbfe7f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves several of the functions in license.bbclass to be library code
New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)
(From OE-Core rev: 0333e04e353991260c5f67a72f80f3ab9dcf526a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you "bitbake XXX" and XXX depends on something else,
the do_package_qa teask for that something may not run. Users would
generally expect it to have though.
Add in the missing dependency to ensure that do_build does trigger
the right package_qa tasks.
(From OE-Core rev: e0beb64c6d3cf1d649f79a8704fb25cdf83b4a8b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake preserve network for non-local uid, refer [1],
so check_userns is not needed for non-local uid
[1] https://git.openembedded.org/bitbake/commit/?id=4eafae7904bae6e5c6bc50356e8a9077f2e207fa
(From OE-Core rev: 808d0cece22bcbee15236717e158da247cbedaf1)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race condition when iterating directories which are being
altered whilst iterating, which is something that can and does happen
when do_package_qa runs at the same time as eg do_package_write_ipkg
(the opkg metadata is written inside the build tree). The race is that
naive code will list a directory contents and then stat() each name to
determine if its a directory or file. The classic failure that we see
is that CONTROL/ is found on a listdir but deleted by the time the stat
happens, so is incorrectly listed as a file (because it is not a
directory).
Since Python 3.5, os.walk() uses scandir() instead of listdir() which
mitigates this race by returning the file type alongside the name, so
a stat is no longer needed to identify the type.
However, cachedpath.walk() was copied from Python before this, so it
uses listdir() and has this race condition. Since I changed insane to
use cachedpath.walk()[1] I inadvertently reintroduced this race.
I believe there's actually no need to use cachedpath.walk() and a
logical fix is to simply use os.walk():
With os.walk() each directory is listed and categorised in a single
os.scandir() as the underlying syscall, getdents64, returns the type.
However, cachedpath.walk() uses os.listdir() which ignores the type
field returned and has to do a stat() on every file to determine the
type.
Thus, we should switch users of cachedpath.walk() to os.walk(): there's
no real gain in what is effectively just a prefetch for the stat cache,
but depending on what the calling code does may result in more stat()
calls than needed.
In the future we may want to redesign cachedpath to reimplement walk so
that it can also cache the DirEntry instances as returned by scandir()
as that will avoid needing to call stat() at all in many cases. However
I believe we should instead use a caching pathlib.Path instance instead.
[1] cad3c8 insane: use oe.cachedpath.CachedPath instead of os.path
(From OE-Core rev: 22e4486d65e4874bf48d89160d69118f318278e8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The functions behave slightly differently to the functions they're
caching and the use in insane.bbclass isn't compatible. For now, to
avoid build failures, switch back to the stat calls. We may be able
to improve cachedpath or change the call sites.
(From OE-Core rev: fa771ae887ab5152f043748cf3419735831bcf7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exception handler tries to read() /etc/passwd file in sysroot
and if file doesn't exist for any reason then it raises FileNotFoundError
exception which mask the original source of the problem and makes
debugging of the issue more difficult.
Fixes:
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/codebuild/output/src1899304708/src/build/tmp-container/work/core2-64-oe-linux/emqx-bin/4.3.12/recipe-sysroot/etc/passwd'
(From OE-Core rev: 4ad9a0e0b11eb7bc5a3dd45fc8945e094ea949e9)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reorder and comment the architecture checks to make it clearer what they
are actually checking.
(From OE-Core rev: 78db9e79e1a307ffb8436e26656bfb98efb513bc)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The insane QAPATHTESTs make many os.stat() calls, the majority of which
are redundant with caching as the initial sweep does a stat() on every
entry to determine if it is a file or a directory, and from then on each
test that does further stat()s is redundant as the tree doesn't change.
Switch os.stat() and friends (os.path.isfile(), etc) to use a common
oe.cachedpath.CachedPath() instance that is shared between all of the
functions, meaning only one stat is done.
In my test case of ltp:do_package_qa, this reduces the time taken from
44s to 37s.
(From OE-Core rev: cad3c889439fd6a007debd6f2f6578f4a1e16c9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check whether the elf is not None first, before doing os.stat() calls
on disk. Also don't check anything that isn't a file, not just FIFOs.
(From OE-Core rev: 38454a2675f38c7db55efcb67bbb8b9fef7e0bf1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't actively do more work:
- Exit early if there are no packages being generated
- Don't iterate repeatedly when removing CONTROL and DEBIAN
- Extend a list with another list instead of appending item by item
- Remove unused variables
(From OE-Core rev: 79ffb8896d570dd935d3aea9d28ee20b52e1674a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the prepopulate function out of global scope, and access the
dictionary once instead of repeatedly.
This still results in each ELF being opened twice, but this avoids
opening all of the files at once and the ELFFile.open() call is fairly
fast.
(From OE-Core rev: cda3647b32703f43c4fe2af3bab977e5698633f6)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since oe-core 66f8a7 merged in 2023[1], ld sets DT_RUNPATH instead of
DT_RPATH when -rpath is specified, which we don't check for.
Update the insane tests to look at both RPATH and RUNPATH.
[1] oe-core 66f8a745668a067d8d763fa2af3e65f26c9c1ebe
(From OE-Core rev: d6c5076d179a3d5ebb74b719ec4d523c197c1918)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reduces the number of files that need to be swept by not scanning
eg the library symlinks, and means we can remove the explicit islink()
checks in many of the tests.
(From OE-Core rev: aa9ec4b5c719bf610ad953095d1111e4c257747e)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
