* [gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation
@ 2021-10-02 20:11 Sam James
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install Sam James
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg Sam James
0 siblings, 2 replies; 5+ messages in thread
From: Sam James @ 2021-10-02 20:11 UTC (permalink / raw
To: gentoo-portage-dev; +Cc: Sam James
Further fixes for when the VDB is corrupted by e.g. broken scanelf.
Already posted on GH a while ago at https://github.com/gentoo/portage/pull/744.
Sam James (2):
doebuild.py: check for inconsistent PROVIDES/image post-src_install
Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg
lib/_emerge/Binpkg.py | 28 +++++++++++++++++++++++++-
lib/portage/package/ebuild/doebuild.py | 24 +++++++++++++++++++++-
2 files changed, 50 insertions(+), 2 deletions(-)
--
2.33.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install
2021-10-02 20:11 [gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation Sam James
@ 2021-10-02 20:11 ` Sam James
2021-10-04 19:35 ` Alec Warner
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg Sam James
1 sibling, 1 reply; 5+ messages in thread
From: Sam James @ 2021-10-02 20:11 UTC (permalink / raw
To: gentoo-portage-dev; +Cc: Sam James
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).
At the point of installation (even if not merging), we need
to detect inconsistent metadata: PROVIDES should be populated
if we're installing any dynamic libraries. This suggests that
e.g. scanelf malfunctioned or some corruption occurred.
Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James <sam@gentoo.org>
---
lib/portage/package/ebuild/doebuild.py | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/lib/portage/package/ebuild/doebuild.py b/lib/portage/package/ebuild/doebuild.py
index 9650a8444..dc3fe3d97 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -3,6 +3,7 @@
__all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"]
+import glob
import grp
import gzip
import errno
@@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
) as f:
f.write(soname_deps.requires)
- if soname_deps.provides is not None:
+ if soname_deps.provides:
with io.open(
_unicode_encode(
os.path.join(build_info_dir, "PROVIDES"),
@@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out):
errors="strict",
) as f:
f.write(soname_deps.provides)
+ else:
+ # Let's check if we've got inconsistent results.
+ # If we're installing dynamic libraries (.so files), we should
+ # really have a PROVIDES.
+ # (This is a complementary check at the point of creation for the
+ # ingestion check in Binpkg.py)
+ # Note: we could check a non-empty PROVIDES against the list of .sos,
+ # but this doesn't gain us anything. We're interested in failure
+ # to properly parse the installed files at all, which should really
+ # be a global problem (e.g. bug #811462)
+ installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True)
+
+ if installed_dynlibs:
+ self._writemsg_level(
+ colorize(
+ "BAD",
+ "!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!",
+ ),
+ noiselevel=-1,
+ level=logging.ERROR,
+ )
if unrecognized_elf_files:
qa_msg = ["QA Notice: Unrecognized ELF file(s):"]
--
2.33.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg
2021-10-02 20:11 [gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation Sam James
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install Sam James
@ 2021-10-02 20:11 ` Sam James
2021-10-04 19:36 ` Alec Warner
1 sibling, 1 reply; 5+ messages in thread
From: Sam James @ 2021-10-02 20:11 UTC (permalink / raw
To: gentoo-portage-dev; +Cc: Sam James
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).
When unpacking a binpkg to be installed, we should check
for the existence of PROVIDES if we're installing any
dynamic libraries. If PROVIDES does not exist in that case,
this suggests that e.g. scanelf malfunctioned or some corruption occurred.
Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James <sam@gentoo.org>
---
lib/_emerge/Binpkg.py | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index c7dde69bd..9b876f354 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -2,7 +2,7 @@
# Distributed under the terms of the GNU General Public License v2
import functools
-
+import glob
import _emerge.emergelog
from _emerge.EbuildPhase import EbuildPhase
from _emerge.BinpkgFetcher import BinpkgFetcher
@@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge
from _emerge.EbuildBuildDir import EbuildBuildDir
from _emerge.SpawnProcess import SpawnProcess
from portage.eapi import eapi_exports_replace_vars
+from portage.output import colorize
from portage.util import ensure_dirs
from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
import portage
@@ -425,6 +426,31 @@ class Binpkg(CompositeTask):
self._async_unlock_builddir(returncode=self.returncode)
return
+ # Before anything else, let's do an integrity check.
+ (provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
+ if not provides:
+ # Let's check if we've got inconsistent results.
+ # If we're installing dynamic libraries (.so files), we should
+ # really have a PROVIDES.
+ # (This is a complementary check at the point of ingestion for the
+ # creation check in doebuild.py)
+ # Note: we could check a non-empty PROVIDES against the list of .sos,
+ # but this doesn't gain us anything. We're interested in failure
+ # to properly parse the installed files at all, which should really
+ # be a global problem (e.g. bug #811462)
+ installed_dynlibs = glob.glob(
+ self.settings["D"] + "/**/*.so", recursive=True
+ )
+ if installed_dynlibs:
+ self._writemsg_level(
+ colorize(
+ "BAD",
+ "!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!",
+ ),
+ noiselevel=-1,
+ level=logging.ERROR,
+ )
+
try:
with io.open(
_unicode_encode(
--
2.33.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install Sam James
@ 2021-10-04 19:35 ` Alec Warner
0 siblings, 0 replies; 5+ messages in thread
From: Alec Warner @ 2021-10-04 19:35 UTC (permalink / raw
To: gentoo-portage-dev; +Cc: Sam James
On Sat, Oct 2, 2021 at 1:11 PM Sam James <sam@gentoo.org> wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> At the point of installation (even if not merging), we need
> to detect inconsistent metadata: PROVIDES should be populated
> if we're installing any dynamic libraries. This suggests that
> e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> lib/portage/package/ebuild/doebuild.py | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/lib/portage/package/ebuild/doebuild.py b/lib/portage/package/ebuild/doebuild.py
> index 9650a8444..dc3fe3d97 100644
> --- a/lib/portage/package/ebuild/doebuild.py
> +++ b/lib/portage/package/ebuild/doebuild.py
> @@ -3,6 +3,7 @@
>
> __all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"]
>
> +import glob
> import grp
> import gzip
> import errno
> @@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
> ) as f:
> f.write(soname_deps.requires)
>
> - if soname_deps.provides is not None:
> + if soname_deps.provides:
The previous code checked if soname_deps.provides was None (or not.)
You have changed it to check if soname_deps.provides is true-ish (or not.)
Why did you change it?
> with io.open(
> _unicode_encode(
> os.path.join(build_info_dir, "PROVIDES"),
> @@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out):
> errors="strict",
> ) as f:
> f.write(soname_deps.provides)
> + else:
> + # Let's check if we've got inconsistent results.
> + # If we're installing dynamic libraries (.so files), we should
> + # really have a PROVIDES.
> + # (This is a complementary check at the point of creation for the
> + # ingestion check in Binpkg.py)
> + # Note: we could check a non-empty PROVIDES against the list of .sos,
> + # but this doesn't gain us anything. We're interested in failure
> + # to properly parse the installed files at all, which should really
> + # be a global problem (e.g. bug #811462)
> + installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True)
> +
> + if installed_dynlibs:
> + self._writemsg_level(
> + colorize(
> + "BAD",
> + "!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!",
> + ),
> + noiselevel=-1,
> + level=logging.ERROR,
> + )
>
> if unrecognized_elf_files:
> qa_msg = ["QA Notice: Unrecognized ELF file(s):"]
> --
> 2.33.0
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg Sam James
@ 2021-10-04 19:36 ` Alec Warner
0 siblings, 0 replies; 5+ messages in thread
From: Alec Warner @ 2021-10-04 19:36 UTC (permalink / raw
To: gentoo-portage-dev; +Cc: Sam James
On Sat, Oct 2, 2021 at 1:11 PM Sam James <sam@gentoo.org> wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> When unpacking a binpkg to be installed, we should check
> for the existence of PROVIDES if we're installing any
> dynamic libraries. If PROVIDES does not exist in that case,
> this suggests that e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> lib/_emerge/Binpkg.py | 28 +++++++++++++++++++++++++++-
> 1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
> index c7dde69bd..9b876f354 100644
> --- a/lib/_emerge/Binpkg.py
> +++ b/lib/_emerge/Binpkg.py
> @@ -2,7 +2,7 @@
> # Distributed under the terms of the GNU General Public License v2
>
> import functools
> -
> +import glob
> import _emerge.emergelog
> from _emerge.EbuildPhase import EbuildPhase
> from _emerge.BinpkgFetcher import BinpkgFetcher
> @@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge
> from _emerge.EbuildBuildDir import EbuildBuildDir
> from _emerge.SpawnProcess import SpawnProcess
> from portage.eapi import eapi_exports_replace_vars
> +from portage.output import colorize
> from portage.util import ensure_dirs
> from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
> import portage
> @@ -425,6 +426,31 @@ class Binpkg(CompositeTask):
> self._async_unlock_builddir(returncode=self.returncode)
> return
>
> + # Before anything else, let's do an integrity check.
> + (provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
> + if not provides:
> + # Let's check if we've got inconsistent results.
> + # If we're installing dynamic libraries (.so files), we should
> + # really have a PROVIDES.
> + # (This is a complementary check at the point of ingestion for the
> + # creation check in doebuild.py)
> + # Note: we could check a non-empty PROVIDES against the list of .sos,
> + # but this doesn't gain us anything. We're interested in failure
> + # to properly parse the installed files at all, which should really
> + # be a global problem (e.g. bug #811462)
> + installed_dynlibs = glob.glob(
> + self.settings["D"] + "/**/*.so", recursive=True
> + )
> + if installed_dynlibs:
> + self._writemsg_level(
> + colorize(
> + "BAD",
> + "!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!",
> + ),
> + noiselevel=-1,
> + level=logging.ERROR,
> + )
> +
Here you have written the same code (and long comment block) as the
last patch...I'd take that as an argument to extract it into a
function (check_provides or whatever) and not copy and paste the code
around.
-A
> try:
> with io.open(
> _unicode_encode(
> --
> 2.33.0
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-04 19:36 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-02 20:11 [gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation Sam James
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install Sam James
2021-10-04 19:35 ` Alec Warner
2021-10-02 20:11 ` [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg Sam James
2021-10-04 19:36 ` Alec Warner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox