View source for Project:Tinderbox

{{Project
|Name=Tinderbox
|Description=
|Email=tinderbox@gentoo.org
|IRC=gentoo-tinderbox
|ParentProject=Project:Gentoo
|PropagatesMembers=No
|Members={{Project Member
|Developer=User:Ago
|Role=tinderbox and CI
|IsLead=No
}}
}}

{{Note|This page is still a work in progress}}

== What is a tinderbox ==
tinderbox means a machine that compiles packages to find the most common build and qa issues.

== Origins ==
The 'tinderbox' term, was initially coined by [https://www.flameeyes.com/tinderbox flameeyes] that run tinderbox for a lot of years and we take this opportunity to thank him for the idea and the work done.

== Other tinderbox projects ==
At the time of writing, we have two other tinderbox projects that should not be confused with this.
See: 
* [[Project:Tinderbox-cluster|Tinderbox-cluster]] run by [[User:Zorry|zorry]]
* [https://zwiebeltoralf.de/tinderbox.html tinderbox] run by [[User:Toralf|toralf]]

== Why tinderbox born? ==
While tinderbox in general is a good stuff, there are two particular reason that made this project go alive:
# Because, during Arch Testing / Stabilization, there were a lot of bugs/regressions that nobody reported and they always blocked stabilizations so tinderbox was a way to anticipate those bugs.
# Surprisingly, there was a mail and a rant over irc from two different people who moved objection about the quality of tests. 

== Tinderbox's rules ==
Following there are the rules at least for tinderbox run by ago:
* The base system is ''~arch'' unless there is a tag/ref like ''tinderbox_stable''
* Bug reports come with a meaningful summary. If there isn't, there wasn't a pattern that match a common error and that happens frequently with src_test() failures. Maintainer is always encouraged to set a proper summary.
*  Common additional logs (like test-suite.log testlog.txt CMakeOutput.log CMakeError.log LastTest.log config.log testsuite.log autoconf.out) are automatically attached. If you need something else please ask for them.
* There may be an internal reference between round brackets on the “Discovered on” line. This is to understand where that failure was reproduced.
* If you see ‘ci’ as internal reference after you pushed a fix, it is very probably that the bug still exist, or there is another failure in the same ebuild phase.
* The build log contains by default:
# The commit of the tree at the time the ebuild was merged, for convenience there is a link
# emerge --info
# list of the emerge history (qlop -mv)
# the list of installed packages (qlist –ICvUSS)
* In case of open bugs regarding test failures, CI won't run tests because result in a waste of resources.
* When there is an open bug about a package, in case the bug is reproduced with a newer version, a comment on the bug is added and the summary is updated with new new version that reproduces the issue.


What else you need to know:
* If you can't reproduce the bug on your system, please try in an empty stage3 (or docker for convenience).
* If you see a compressed build log, is because the plain text version exceeds the limits on our bugzilla (1MB).
* When you close the bug with a resolution different from RESOLVED/FIXED please not be cryptic.
* If in the emerge history there are dev-lang/python-exec, sys-devel/gcc-config and sys-devel/binutils-config at the end, this is a test with USE=”-native-symlinks” against the package.
* If you are unsure if your commit fixes the reported issue, please always close the bug (as RESOLVED/TEST-REQUEST if you prefer) so the system will open a new bug in case the problem is still reproducible.
* Given the amount of handled bugs it's hard answer to all generic questions. QA warnings are produced by portage so if there is anything unclear you may want to ask publicly (on irc?) to reduce the response time.
* Overlays are supported.
* Bugs can be filed on github trackers in case Overlays have their own tracker there.
* There are kinds of tests that may produce an unstable system (let's says LTO as example). For this reason changes are applied per package instead of globally. That means that while comment #0 of the bug warns about LTO, if the failure is related to a DEPEND of a package that is about to be tested, it probably won't have anything to do with LTO

== After get a tinderbox bug ==
When you get a tinderbox bug, you must always click on links provided by tinderbox, because while you may know the content, in the meantime the wiki content may have been updated with new resources that can help you.
When you push a fix, in case of dubious fixes, feel free to close the bug as RESOLVED-> TEST-REQUEST. If the bug is still reproducible you will get another bug with an updated ''build.log''.
In case of bugs with tests failures, make sure to use ''`pkgdev commit '''-c'''`'' or close the bug before pushing the fix. Otherwise, as explained above, ''`CI`'' won't run tests.

== Tinderbox against a set of packages ==
You want to test something that involve a set of packages, let's say a new version of <code>dev-java/openjdk</code>, and you want to know how packages behave, you have multiple choices:
# Provide a list of packages in a syntax like ''"app-category/package1 app-category/package2 app-category/package3 ..."''. That means that bugs will be reported on https://bugs.gentoo.org
# Create a github repo that we will treat as an overlay. This repo will contain only the packages you want to test. This scenario is intended when you want to have a preview about how much bugs will be created, so tinderbox will file bugs as github issues and will open a 'fake' issue as <code>app-category/package1 has been build successfully</code> to give the opportunity to store successful build logs for further inspections. An example is here: https://github.com/mgorny/python-bump-testing

The following snippet may help you:
<code>
 PACKAGES="app-category/package1 app-category/package2 app-category/package3"
 OVERLAY="overlay"
 PORTDIR="$( portageq get_repo_path / gentoo )"
 for PACKAGE in ${PACKAGES}
 do
        if [ -d "${PORTDIR}"/"${PACKAGE}" ]
        then
                QATOM="$( qatom -C "${PACKAGE}" )"
                CATEGORY="$( echo "${QATOM}" | awk '{print $1}' )"
                PN="$( echo "${QATOM}" | awk '{print $2}' )"
                mkdir -p "${OVERLAY}"/"${CATEGORY}"
                cp -r "${PORTDIR}"/"${PACKAGE}" "${OVERLAY}"/"${CATEGORY}"/
        fi
 done
</code>

== Statistics ==
=== Per year ===
* 2020: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2020-04-01&chfieldto=2020-12-31&f1=reporter&limit=0&list_id=6765598&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org 5098 bugs]
* 2021: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2021-01-01&chfieldto=2021-12-31&f1=reporter&limit=0&list_id=6765600&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org 4818 bugs]
* 2022: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2022-01-01&chfieldto=2022-12-31&f1=reporter&limit=0&list_id=6765602&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org 5817 bugs]
* 2023: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2023-01-01&chfieldto=2023-12-31&f1=reporter&limit=0&list_id=6765604&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org 3811 bugs]
* 2024: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2024-01-01&chfieldto=2024-12-31&f1=reporter&limit=0&list_id=6765604&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org 5752 bugs]
* 2025: [https://bugs.gentoo.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=2025-01-01&chfieldto=2025-12-31&f1=reporter&limit=0&list_id=6765604&o1=equals&order=bug_id&product=Gentoo%20Linux&product=GURU&query_format=advanced&v1=ago%40gentoo.org NaN bugs]
=== Per system ===
{{Note|historically, bugs were not filed with the idea of have statistics based on targets, so results may be incomplete.}}
* [https://bugs.gentoo.org/buglist.cgi?f1=longdesc&f2=reporter&limit=0&o1=casesubstring&o2=equals&order=bug_id%20DESC&query_format=advanced&v1=MUSL-SYSTEM&v2=ago%40gentoo.org MUSL-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=OP&f3=longdesc&f4=longdesc&j2=OR&limit=0&o1=equals&o3=casesubstring&o4=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v3=LTO-SYSTEM&v4=%20%28lto%29 LTO-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&limit=0&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=CLANG-STRICTER-SYSTEM CLANG-STRICTER-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&limit=0&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=DASH-SYSTEM DASH-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=USR-MERGE-SYSTEM USR-MERGE-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=LIBARCHIVE-SYSTEM LIBARCHIVE-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=HARDENED-SYSTEM HARDENED-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=MUSL-CLANG-SYSTEM MUSL-CLANG-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=GCC-13-SYSTEM GCC-13-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=CYTHON3-SYSTEM CYTHON3-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=GCC-14-SYSTEM GCC-14-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=CLANG-17-SYSTEM CLANG-17-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=PYTHON-3.12-SYSTEM PYTHON-3.12-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=GLIBC-2.38-SYSTEM GLIBC-2.38-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=GLIBC-2.39-SYSTEM GLIBC-2.39-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=CLANG-LLD CLANG-LLD-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=BINUTILS-2.42-SYSTEM BINUTILS-2.42-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=SLIBTOOL-SYSTEM SLIBTOOL-SYSTEM]
* [https://bugs.gentoo.org/buglist.cgi?f1=reporter&f2=longdesc&o1=equals&o2=casesubstring&order=bug_id%20DESC&query_format=advanced&v1=ago%40gentoo.org&v2=GCC-15-SYSTEM GCC-15-SYSTEM]

== Goal of this page ==
It's a matter of fact that in the past tinderbox was done without any written rules. This led to constant discussions, so the goal of this page  is to make clear rules about:
* What to test
* How to test
* How to report
* What is fine to be reported

== Errors in the 'QA Notices' ==
Tinderbox basically copy-paste the 'QA Notice' into a bug report. If you think that tinderbox reported a wrong 'QA Notice', a false positive 'QA Notice' and so on, you should get in contact with the author of the 'QA Notice' unless it is a custom 'QA Notice' called 'Tinderbox QA Notice'.

== Useful bugzilla trackers ==
This is not a copy paste of all [https://bugs.gentoo.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&f1=keywords&o1=substring&order=bug_id%20DESC&product=Gentoo%20Linux&query_format=advanced&v1=tracker bugzilla trackers] but just the most common. The newest one goes at the top.
{| class="wikitable"
|-
! Summary !! Number || Note
|-
| sys-devel/gcc-14 porting || [https://bugs.gentoo.org/show_bug.cgi?id=914580 914580] ||
|-
| sys-devel/autoconf-2.72 || [https://bugs.gentoo.org/show_bug.cgi?id=893434 893434] ||
|-
| missing DEPEND || [https://bugs.gentoo.org/show_bug.cgi?id=893356 893356] ||
|-
|}

== Future ideas ==
Here are the things we can test in the future. Test them right now is not a good idea because they will generate a big amount of bugs that need an attention we can't give at this moment.

* compiler-rt ''(sys-devel/clang-common[default-compiler-rt] - sys-devel/clang-toolchain-symlinks[gcc-symlinks] - sys-devel/gcc-config[-native-symlinks])''
* llvm libc++ ''(sys-devel/clang-common[default-libcxx] - sys-devel/clang-toolchain-symlinks[gcc-symlinks] - sys-devel/gcc-config[-native-symlinks])''
* more from ''app-alternatives/'' packages
* sys-libs/fortify-headers
* Packages that overwrite CFLAGS/CXXFLAGS or add uncommon ones
* Packages that compile in ''src_install()''
* Packages that miss RDEPEND
* Build ''src_test()'' with Address Sanitizer

== See also ==
* {{See also|Project:Tinderbox/Common_Issues_Helper}}