Add --full-build-fallback-threshold option; bump to v1.0.4

README.md

@@ -1,4 +1,4 @@
-EMTEE 1 "Version 1.0.3: Nov 2019"
+EMTEE 1 "Version 1.0.4: Apr 2020"
 =================================
 
 [//]: # ( Convert to manpage using e.g. go-md2man -in=README.md -out=emtee.1 )
@@ -11,7 +11,7 @@ emtee - a faster-startup emerge -DuU --with-bdeps=y --keep-going @world (et al.)
 SYNOPSIS
 --------
 
-`emtee` [`-a`] [`-A`] [`-b`] [`-c`] [`-C`] [`-d`] [`-e` args] [`-E` args]
+`emtee` [`-a`] [`-A`] [`-b`] [`-c`] [`-C`] [`-d`] [`-e` args] [`-E` args] [`-f` NUM]
 [`-h`] [`-p`] [`-N`] [`-s` set] [`-S`] [`-v`] [`-V`] [`-z`]
 
 DESCRIPTION
@@ -88,6 +88,18 @@ OPTIONS
   options by setting them directly via `-E`, if you prefer. They are
   provided as syntactic sugar, for convenience.
 
+`-f`, `--full-build-fallback-threshold=`*NUM*
+  If the number of packages passed to the real emerge step
+  is >= *NUM*, then a dry-run will first be performed, to check that
+  the proposed set can be emerged consistently, and iff that
+  fails, then a full `emerge --emptytree @world`
+  run will be initiated, followed by (if successful)
+  `emerge --depclean`.  
+   
+  This functionality is provided as a fallback, to ensure that
+  fundamental changes that trigger many dependencies can be built
+  consistently. The default is **not** to do this dry-run test.
+
 `-h`, `--help`
   Prints a short help message, and exits.
 
@@ -183,6 +195,14 @@ The `emtee` process runs as follows:
    explicitly (via `-E`/`--emerge-args`) and implicitly, via one of
    the impacting options (`-v`/`--verbose`, `-a`/`--ask`,
    `-A`/`--alert`, `-p`/`--pretend` or `-z`/`--keep-going`).
+   
+   Note also that if `-f`/`--full-build-fallback-threshold` is used, and
+   the number of packages passed to this phase is >= *NUM*, then
+   a dry-run  will first be performed, to check that the proposed
+   set can be emerged consistently, and iff that  fails,
+   then  a  full `emerge --emptytree @world` run will be initiated,
+   followed by (if successful)  `emerge  --depclean`.  The
+   default is not to do any such dry run.
 
 BASIS
 -----
@@ -279,7 +299,7 @@ packages requiring upgrade increases.
 COPYRIGHT
 ---------
 
-Copyright © 2018-2019 sakaki
+Copyright © 2018-2020 sakaki
 
 License GPLv3+ (GNU GPL version 3 or later)  
 <http://gnu.org/licenses/gpl.html>

emtee

@@ -219,7 +219,7 @@ set -o pipefail
 # ********************** variables ********************* 
 
 PROGNAME="$(basename "${0}")"
-VERSION="1.0.3"
+VERSION="1.0.4"
 
 RED_TEXT="" GREEN_TEXT="" YELLOW_TEXT="" RESET_ATTS="" ALERT_TEXT=""
 if [[ -v TERM && -n "${TERM}" && "${TERM}" != "dumb" ]]; then
@@ -280,6 +280,10 @@ PREFIXSTRING="* "
 SHOWPREFIX="${GREEN_TEXT}${PREFIXSTRING}${RESET_ATTS}"
 SHOWSUFFIX=""
 
+# number of phase 4 builds at or above which an emerge failure will
+# trigger a full, ground-up rebuild; 0 means never do this
+declare -i FALLBACK_THRESHOLD=0
+
 # program arguments (booleans in this case)
 declare -i ARG_ASK=0 ARG_ALERT=0 ARG_HELP=0 ARG_VERSION=0
 declare -i ARG_VERBOSE=0 ARG_NEWUSE=0 ARG_PRETEND=0
@@ -443,6 +447,13 @@ Options:
   -e, --emptytree-emerge-args=ARGS
                         pass provided additional ARGS to the initial
                         (--emptytree) emerge stage
+  -f, --full-build-fallback-threshold=NUM
+                        if there are >=NUM packages to be rebuilt at the
+                        final phase, then dry-run the oneshot emerge of
+                        these first; if this fails, run a _full_
+                        --emptytree @world emerge, followed (if successful)
+                        by emerge --depclean (the default is not to
+                        do any of these steps)
   -E, --emerge-args=ARGS
                         pass provided additional ARGS to the 'real' emerge
                         e.g., use --emerge-args='--autounmask-write' to
@@ -503,7 +514,7 @@ process_command_line_options() {
     set +e
         # error trapping temporarily off while we parse
         # so we can handle issues ourselves
-        TEMP="$(getopt -o aAb:cCde:E:hpNs:SvVz: --long ask,alert,with-bdeps:,crosscheck,strict-crosscheck,debug,emptytree-emerge-args:,emerge-args:,help,pretend,newuse,set:,force-slot-rebuilds,verbose,version,keep-going: -n "${PROGNAME}" -- "${@}")"
+        TEMP="$(getopt -o aAb:cCde:E:f:hpNs:SvVz: --long ask,alert,with-bdeps:,crosscheck,strict-crosscheck,debug,emptytree-emerge-args:,emerge-args:,full-build-fallback-threshold:,pretend,newuse,set:,force-slot-rebuilds,verbose,version,keep-going: -n "${PROGNAME}" -- "${@}")"
         RC="${?}"
     set -e
     if ((RC!=0)); then
@@ -534,6 +545,11 @@ process_command_line_options() {
                     "") shift 2 ;;
                     *) EMERGEARGS="${2}" ; shift 2 ;;
                 esac ;;
+            -f|--full-build-fallback-threshold)
+                case "${2}" in
+                    "") shift 2 ;;
+                    *) FALLBACK_THRESHOLD="${2}" ; shift 2 ;;
+                esac ;;
             -h|--help) ARG_HELP=1 ; shift ;;
             -p|--pretend) ARG_PRETEND=1 ; shift ;;
             -N|--newuse) ARG_NEWUSE=1 ; USEOPT="--newuse" ; SHORTUSEOPT="N" ; shift ;;
@@ -972,14 +988,46 @@ perform_itemized_emerge() {
     #   1. deal with blocks / uninstalls correctly
     #   2. add in slot-change rebuilds automatically, and
     #   3. permit job parallelism (where enabled via EMERGE_DEFAULT_OPTS)
+    #
+    # if --full-build-fallback-threshold=N / -f N is set, and >=N FQAs are carried
+    # forward into this phase 4, then the emerge will first be
+    # dry-run, and if that _fails_ (indicating usually a stubborn
+    # blocker caused by e.g. a slot change in a low-level library with
+    # lots of deps) then a _full_ emptytree @world emerge will be
+    # triggered instead; this provides a failsafe for most
+    # cases
+    #
+    # also, if --extreme / -x is set, then emerge --depclean will be
+    # run at the (successful) conclusion of the emerge, to leave
+    # the system in a consistent state
+
+    declare -i TRY_STANDARD_EMERGE=1
 
     if ((NUM_TO_BUILD>0)); then
-        if ((ARG_DEBUG==1)); then
-            show "Issuing:"
-            echo "  emerge --oneshot ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} --keep-going=${KEEP_GOING_FLAG} ${EMERGEARGS} ${BUILD_FQAS}"
+        if ((FALLBACK_THRESHOLD>0 && NUM_TO_BUILD>=FALLBACK_THRESHOLD)); then
+            show "Fallback threshold met, validating phase 4 build list..."
+            show "(this may take some time)"
+            if emerge --oneshot --pretend ${EMERGEARGS} ${BUILD_FQAS} &>/dev/null; then
+                show "Build list validated!"
+            else
+                warning "NOT valid; initiating full --emptytree @world emerge"
+                # fallback
+                TRY_STANDARD_EMERGE=0
+                emerge ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} \
+                       --emptytree @world
+                show "Full emerge successful!"
+                show "Removing packages not required by @world set..."
+                emerge ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} --depclean
+            fi
+        fi
+        if ((TRY_STANDARD_EMERGE==1)); then
+            if ((ARG_DEBUG==1)); then
+                show "Issuing:"
+                echo "  emerge --oneshot ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} --keep-going=${KEEP_GOING_FLAG} ${EMERGEARGS} ${BUILD_FQAS}"
+            fi
+            emerge --oneshot ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} \
+                   --keep-going=${KEEP_GOING_FLAG} ${EMERGEARGS} ${BUILD_FQAS}
         fi
-        emerge --oneshot ${ASKFLAG} ${ALERTFLAG} ${VERBOSITYFLAG} ${PRETENDFLAG} \
-               --keep-going=${KEEP_GOING_FLAG} ${EMERGEARGS} ${BUILD_FQAS}
     else
         show "Nothing to emerge!"
     fi

emtee.1

@@ -1,11 +1,12 @@
-.TH EMTEE 1 "Version 1.0.3: Nov 2019"
+.nh
+.TH EMTEE 1 "Version 1.0.4: Apr 2020"
 .SH NAME
 .PP
-emtee \- a faster\-startup emerge \-DuU \-\-with\-bdeps=y @world (et al.)
+emtee \- a faster\-startup emerge \-DuU \-\-with\-bdeps=y \-\-keep\-going @world (et al.)
 
 .SH SYNOPSIS
 .PP
-\fB\fCemtee\fR [\fB\fC\-a\fR] [\fB\fC\-A\fR] [\fB\fC\-b\fR] [\fB\fC\-c\fR] [\fB\fC\-C\fR] [\fB\fC\-d\fR] [\fB\fC\-e\fR args] [\fB\fC\-E\fR args]
+\fB\fCemtee\fR [\fB\fC\-a\fR] [\fB\fC\-A\fR] [\fB\fC\-b\fR] [\fB\fC\-c\fR] [\fB\fC\-C\fR] [\fB\fC\-d\fR] [\fB\fC\-e\fR args] [\fB\fC\-E\fR args] [\fB\fC\-f\fR NUM]
 [\fB\fC\-h\fR] [\fB\fC\-p\fR] [\fB\fC\-N\fR] [\fB\fC\-s\fR set] [\fB\fC\-S\fR] [\fB\fC\-v\fR] [\fB\fC\-V\fR] [\fB\fC\-z\fR]
 
 .SH DESCRIPTION
@@ -21,9 +22,9 @@ It may be used (with appropriate options) in place of:
 
 .RS
 .IP \(bu 2
-\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-changed\-use \-\-keep\-going\fR \fI@world\fP,
+\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-changed\-use \-\-keep\-going @world\fR,
 .IP \(bu 2
-\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-newuse \-\-ask \-\-verbose \-\-keep\-going\fR \fI@world\fP;
+\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-newuse \-\-ask \-\-verbose \-\-keep\-going @world\fR;
 
 .RE
 
@@ -98,12 +99,27 @@ block\-driven unmerges.
   these arguments are \fInot\fP passed to the preliminary \fB\fCemerge\fR step; you
   need to use \fB\fC\-e\fR for that.
 .br
- 
+
 .br
   Note also that you can achieve the effect of the \fB\fC\-a\fR \fB\fC\-A\fR, \fB\fC\-p\fR and \fB\fC\-v\fR
   options by setting them directly via \fB\fC\-E\fR, if you prefer. They are
   provided as syntactic sugar, for convenience.
 
+.PP
+\fB\fC\-f\fR, \fB\fC\-\-full\-build\-fallback\-threshold=\fR\fINUM\fP
+  If the number of packages passed to the real emerge step
+  is >= \fINUM\fP, then a dry\-run will first be performed, to check that
+  the proposed set can be emerged consistently, and iff that
+  fails, then a full \fB\fCemerge \-\-emptytree @world\fR
+  run will be initiated, followed by (if successful)
+  \fB\fCemerge \-\-depclean\fR\&.
+.br
+
+.br
+  This functionality is provided as a fallback, to ensure that
+  fundamental changes that trigger many dependencies can be built
+  consistently. The default is \fBnot\fP to do this dry\-run test.
+
 .PP
 \fB\fC\-h\fR, \fB\fC\-\-help\fR
   Prints a short help message, and exits.
@@ -166,7 +182,7 @@ Filter this list, by marking each fully\-qualified atom
 (\fIFQA\fP=\fI$CATEGORY/$PF\fP) within it for building (or not). Begin
 with all \fIFQAs\fP unmarked.
 .br
- 
+\ 
 .br
 
 .RS
@@ -183,33 +199,34 @@ information to allow us to easily work this out).
 Then (pass 3), if \fB\fC\-S\fR/\fB\fC\-\-force\-slot\-rebuilds\fR is in use, for
 each marked package on the list whose slot or subslot is
 changing (also inferable from the phase 1 output), search
-\fI/var/db/pkg/<FQA>/RDEPENDS\fP (and \fIDEPENDS\fP, if
+\fI/var/db/pkg/FQA/RDEPENDS\fP (and \fIDEPENDS\fP, if
 \fB\fC\-\-with\-bdeps=y\fR, the default, is active) for any matching slot
 dependencies.  Mark each such located (reverse) dependency that
 is \fIalso\fP on the original \fB\fC\-\-emptytree\fR list (and not a block
 or uninstall) for build.
 .br
- 
+\&nbsp;
 .br
 Note that pass 3 is skipped by default, since the phase 4 emerge
 (aka the real \fB\fCemerge\fR) will automatically trigger any
 necessary slot rebuilds anyway, so it is redundant except for in a
 few esoteric situations.
 
 .RE
+
 .IP "  3." 5
 Iff \fB\fC\-c\fR/\fB\fC\-\-crosscheck\fR (or \fB\fC\-C\fR/\fB\fC\-\-strict\-crosscheck\fR) passed,
 compare the \fIFQA\fP build list produced by invoking \fB\fCemerge \-\-bdeps=y
 \-\-pretend \-\-deep \-\-update [\-\-changed\-use|\-\-newuse] [opts]\fR \fI@world\fP
 (adapted for specified options appropriately), with that produced
 by invoking \fB\fCemerge \-\-oneshot \-\-pretend [opts]\fR
-\fI<filtered-FQA-build-list-from-phase-2>\fP\&. If any differences are
+\fIfiltered\-FQA\-build\-list\-from\-phase\-2\fP\&. If any differences are
 found, report them (and, additionally, stop the build in such a
 case, if \fB\fC\-S\fR/\fB\fC\-\-strict\-crosscheck\fR specified). Also report
 a series of comparative (total elapsed wall\-clock) timings for both
 alternatives, for benchmarking purposes.
 .br
- 
+\&nbsp;
 .br
 Note: crosschecking should \fIonly\fP be used for reassurance or
 benchmarking, as it will, of necessity, be slower than the baseline
@@ -218,14 +235,23 @@ in total time cost (since the check involves running both that
 to improve emerge times, do \fInot\fP pass \fB\fC\-s\fR/\fB\fC\-S\fR\&.
 .IP "  4." 5
 Invoke the real \fB\fCemerge\fR, as: \fB\fCemerge \-\-oneshot [opts]\fR
-\fI<filtered-FQA-build-list-from-phase-2>\fP\&.
+\fIfiltered\-FQA\-build\-list\-from\-phase\-2\fP\&.
 .br
- 
+\&nbsp;
 .br
 Note that additional arguments may be passed to this invocation, both
 explicitly (via \fB\fC\-E\fR/\fB\fC\-\-emerge\-args\fR) and implicitly, via one of
 the impacting options (\fB\fC\-v\fR/\fB\fC\-\-verbose\fR, \fB\fC\-a\fR/\fB\fC\-\-ask\fR,
 \fB\fC\-A\fR/\fB\fC\-\-alert\fR, \fB\fC\-p\fR/\fB\fC\-\-pretend\fR or \fB\fC\-z\fR/\fB\fC\-\-keep\-going\fR).
+\&nbsp;
+.br
+Note also that if \fB\fC\-f\fR/\fB\fC\-\-full\-build\-fallback\-threshold\fR is used, and
+the number of packages passed to this phase is >= \fINUM\fP, then
+a dry\-run  will first be performed, to check that the proposed
+set can be emerged consistently, and iff that  fails,
+then  a  full \fB\fCemerge \-\-emptytree @world\fR run will be initiated,
+followed by (if successful)  \fB\fCemerge  \-\-depclean\fR\&.  The
+default is not to do any such dry run.
 
 .RE
 
@@ -238,7 +264,7 @@ Why is this approach faster? Well, the main claims behind \fB\fCemtee\fR are:
 An \fB\fC\-\-emptytree\fR \fB\fCemerge\fR of \fI@world\fP yields the same versioned package list
 that a \fB\fC\-\-deep \-\-update\fR \fB\fCemerge\fR would arrive at.
 .br
- 
+\&nbsp;
 .br
 That is, for \fB\fCemtee\fR to work, it must be true that for a consistent,
 depcleaned Gentoo system with a recently updated set of ebuild
@@ -247,7 +273,7 @@ invoked and runs successfully to conclusion, then an immediately
 following \fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-changed\-use \-\-update\fR
 \fI@world\fP will always be a no\-op.
 .br
- 
+\&nbsp;
 .br
 Or, to put it another way, we claim that the list of
 fully\-qualified atoms (\fIFQAs\fP, where an \fIFQA\fP is \fI$CATEGORY/$PF\fP)
@@ -260,7 +286,7 @@ as regards packages and versions, anyhow.
 It also contains sufficient information to simulate \fB\fC\-\-changed\-use\fR
 and \fB\fC\-\-newuse\fR\&.
 .br
- 
+\&nbsp;
 .br
 Of course, the issue is that in addition to new versions (\fI[N]\fP),
 package upgrades (\fI[U]\fP), downgrades (\fI[UD]\fP), new slots (\fI[NS]\fP)
@@ -269,7 +295,7 @@ huge number of reinstalls (\fI[R]\fP). Some of these will genuinely
 need doing (in light of changed USE flags etc.), but many,
 usually the vast majority, will be redundant.
 .br
- 
+\&nbsp;
 .br
 Fortunately, for common rebuild selections (such as \fB\fC\-\-changed\-use\fR
 and \fB\fC\-\-newuse\fR), we can easily identify which is which, using only
@@ -281,7 +307,7 @@ the remaining set with a \fI%\fP suffix, when \fB\fC\-\-verbose\fR is used.
 Producing such a list, and then shallow emerging it, reduces the net
 dependency calculation time.
 .br
- 
+\&nbsp;
 .br
 Finally, we also claim that for a Gentoo system with many installed
 packages, the time taken to 1) generate an \fB\fC\-\-emptytree\fR \fI@world\fP
@@ -340,12 +366,12 @@ packages requiring upgrade increases.
 
 .SH COPYRIGHT
 .PP
-Copyright © 2018\-2019 sakaki
+Copyright © 2018\-2020 sakaki
 
 .PP
 License GPLv3+ (GNU GPL version 3 or later)
 .br
-
+http://gnu.org/licenses/gpl.html
 \[la]http://gnu.org/licenses/gpl.html\[ra]
 
 .PP
@@ -355,8 +381,8 @@ There is NO WARRANTY, to the extent permitted by law.
 
 .SH AUTHOR
 .PP
-sakaki — send bug reports or comments to 
-\[la]sakaki@deciban.com\[ra]
+sakaki — send bug reports or comments to sakaki@deciban.com
+\[la]mailto:sakaki@deciban.com\[ra]
 
 .SH SEE ALSO
 .PP