From 31d669a34eb1b5f06d5629d0e2729e748b3cef73 Mon Sep 17 00:00:00 2001 From: sakaki Date: Thu, 17 Jan 2019 01:12:56 +0000 Subject: [PATCH] Add initial manpage, generated from README.md by go-md2man --- emtee.1 | 356 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 356 insertions(+) create mode 100644 emtee.1 diff --git a/emtee.1 b/emtee.1 new file mode 100644 index 0000000..8677249 --- /dev/null +++ b/emtee.1 @@ -0,0 +1,356 @@ +.TH EMTEE 1 "Version 1.0.0: Jan 2019" +.SH NAME +.PP +emtee \- a faster\-startup emerge \-DuU \-\-with\-bdeps=y @world (et al.) + +.SH SYNOPSIS +.PP +\fB\fCemtee\fR [\fB\fC\-a\fR] [\fB\fC\-A\fR] [\fB\fC\-b\fR] [\fB\fC\-c\fR] [\fB\fC\-C\fR] [\fB\fC\-d\fR] [\fB\fC\-e\fR args] [\fB\fC\-E\fR args] +[\fB\fC\-h\fR] [\fB\fC\-p\fR] [\fB\fC\-N\fR] [\fB\fC\-s\fR set] [\fB\fC\-S\fR] [\fB\fC\-v\fR] [\fB\fC\-V\fR] + +.SH DESCRIPTION +.PP +\fB\fCemtee\fR is a simple script to speed up \fI@world\fP updates on Gentoo +Linux systems, by significantly decreasing the time taken for the +\fB\fCemerge\fR(1)'s initial dependency resolution stage (often the dominant +component for SBCs and other resource\-challenged systems, if a binhost +is deployed). + +.PP +It may be used (with appropriate options) in place of: + +.RS +.IP \(bu 2 +\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-changed\-use\fR \fI@world\fP, +.IP \(bu 2 +\fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update \-\-newuse \-\-ask \-\-verbose\fR \fI@world\fP; + +.RE + +.PP +and similar invocations. + +.PP +For example, you could achieve the same effect as the above commands +with: + +.RS +.IP \(bu 2 +\fB\fCemtee\fR, and +.IP \(bu 2 +\fB\fCemtee \-\-newuse \-\-ask \-\-verbose\fR (or, equivalently, \fB\fCemtee \-\-Nav\fR) + +.RE + +.PP +respectively. + +.PP +Generally speaking, \fB\fCemtee\fR will take materially less time +to get the first actual build underway than its counterpart \fB\fCemerge +\-\-update\fR, while still targeting an identical build list, permitting +job parallelism, and triggering any required slot\-driven rebuilds, and +block\-driven unmerges. + +.SH OPTIONS +.PP +\fB\fC\-a\fR, \fB\fC\-\-ask\fR + Turns on interactive mode for the real \fB\fCemerge\fR step (i.e. + during phase 4, as described in \fIALGORITHM\fP, below). + +.PP +\fB\fC\-A\fR, \fB\fC\-\-alert\fR + Uses terminal bell notification for all interactive prompts in + the real \fB\fCemerge\fR step. Selecting this turns on \fB\fC\-a\fR by default. + +.PP +\fB\fC\-b\fR, \fB\fC\-\-with\-bdeps=y|n\fR + Specifies whether or not to pull\-in build time dependencies. NB, + defaults to \fB\fCy\fR if unspecified, in contrast to \fB\fCemerge\fR\&'s default + behaviour. + +.PP +\fB\fC\-c\fR, \fB\fC\-\-crosscheck\fR + Checks the final build list against the result obtained by running a + conventional \fB\fC\-\-deep\fR \fB\fCemerge\fR, and prints a \fB\fCdiff\fR(1) of the package + list; also provides a comparitive timing, for benchmarking purposes. + NB: does not check the \fIordering\fP of the packages in the two lists. + +.PP +\fB\fC\-C\fR, \fB\fC\-\-strict\-crosscheck\fR + As for \fB\fC\-c\fR, but exits with an error if the \fB\fCemtee\fR and conventional + \fB\fC\-\-deep \-\-update\fR \fB\fCemerge\fR lists do not match. + +.PP +\fB\fC\-d\fR, \fB\fC\-\-debug\fR + Prints some additional debugging information during the process. + +.PP +\fB\fC\-e\fR, \fB\fC\-\-emptytree\-emerge\-args=\fR\fIADDITIONAL\_ARGS\fP + Passes the specified arguments to the initial (\fB\fC\-\-pretend + \-\-emptytree\fR) \fB\fCemerge\fR step used to caclulate the (unfiltered) build + list. Note that these arguments are \fInot\fP passed to the real + \fB\fCemerge\fR step; you need to use \fB\fC\-E\fR for that. + +.PP +\fB\fC\-E\fR, \fB\fC==emerge\-args=\fR\fIADDITIONAL\_ARGS\fP + Passes the specified arguments to the real \fB\fCemerge\fR step. Note that + these arguments are \fInot\fP passed to the preliminary \fB\fCemerge\fR step; you + need to use \fB\fC\-e\fR for that. +.br +  +.br + Note also that you can achieve the effect of the \fB\fC\-a\fR \fB\fC\-A\fR, \fB\fC\-p\fR and \fB\fC\-v\fR + options by setting them directly via \fB\fC\-E\fR, if you prefer. They are + provided as syntactic sugar, for convenience. + +.PP +\fB\fC\-h\fR, \fB\fC\-\-help\fR + Prints a short help message, and exits. + +.PP +\fB\fC\-p\fR, \fB\fC\-\-pretend\fR + Passes the \fB\fC\-\-pretend\fR option to the real \fB\fCemerge\fR step, resulting in + a 'dry run' (nothing will actually be updated). + +.PP +\fB\fC\-N\fR, \fB\fC\-\-newuse\fR + Rebuild packages which have had any USE flag changes, even if those + changes don't affect flags the user has enabled (the more + conservative \fB\fC\-\-changed\-use\fR behaviour is the default). + +.PP +\fB\fC\-s\fR, \fB\fC\-\-set\fR=\fISET\fP + Uses the specified set (e.g. \fI@system\fP) in preference to the default + \fI@world\fP\&. You can pass regular package names here as well, but note + that using a regular \fB\fCemerge\fR is likely to be faster for such targets. + +.PP +\fB\fC\-S\fR, \fB\fC\-\-force\-slot\-rebuilds\fR + Checks for any slot\-change\-driven rebuilds, and adds these (reverse) + dependencies to the build list; this option should not generally be + required, as the real \fB\fCemerge\fR step will trigger such rebuilds + automatically anyway. + +.PP +\fB\fC\-v\fR, \fB\fC\-\-verbose\fR + Passes the \fB\fC\-\-verbose\fR option to the real \fB\fCemerge\fR step + +.PP +\fB\fC\-V\-\fR, \fB\fC\-\-version\fR + Prints \fB\fCemtee\fR\&'s version number, and exits. + +.SH ALGORITHM +.PP +The \fB\fCemtee\fR process runs as follows: + +.RS +.IP " 1." 5 +Derive a full, versioned build list for the \fI@world\fP set and its +entire deep dependency tree, via \fB\fCemerge \-\-with\-bdeps=y \-\-pretend +\-\-emptytree \-\-verbose [opts]\fR \fI@world\fP, which Portage can do +relatively quickly. The resulting list, as it is derived as if \fIno\fP +packages were installed to begin with, will automatically contain +all necessary packages at their 'best' versions (which may entail +upgrades, downgrades, new slots etc. wrt the currently installed +set). +.IP " 2." 5 +Filter this list, by marking each fully\-qualified atom +(\fIFQA\fP=\fI$CATEGORY/$PF\fP) within it for building (or not). Begin +with all \fIFQAs\fP unmarked. +.br +  +.br + +.RS +.IP \(bu 2 +Then (pass 1), mark anything which isn't a block, uninstall or reinstall for build; +.IP \(bu 2 +Then (pass 2), check each reinstall, to see if its \fIactive\fP +USE flag set is changing (default behaviour), or if \fIany\fP of +its USE flags are changing (\fB\fC\-N\fR/\fB\fC\-\-newuse\fR behaviour), and if +so, mark that package for build (fortunately, the \fB\fC\-\-verbose\fR +output from step 1 contains the necessary USE flag delta +information to allow us to easily work this out). +.IP \(bu 2 +Then (pass 3), if \fB\fC\-S\fR/\fB\fC\-\-force\-slot\-rebuilds\fR is in use, for +each marked package on the list whose slot or subslot is +changing (also inferable from the phase 1 output), search +\fI/var/db/pkg//RDEPENDS\fP (and \fIDEPENDS\fP, if +\fB\fC\-\-with\-bdeps=y\fR, the default, is active) for any matching slot +dependencies. Mark each such located (reverse) dependency that +is \fIalso\fP on the original \fB\fC\-\-emptytree\fR list (and not a block +or uninstall) for build. +.br +  +.br +Note that pass 3 is skipped by default, since the phase 4 emerge +(aka the real \fB\fCemerge\fR) will automatically trigger any +necessary slot rebuilds anyway, so it is redundant except for in a +few esoteric situations. + +.RE +.IP " 3." 5 +Iff \fB\fC\-c\fR/\fB\fC\-\-crosscheck\fR (or \fB\fC\-C\fR/\fB\fC\-\-strict\-crosscheck\fR) passed, +compare the \fIFQA\fP build list produced by invoking \fB\fCemerge \-\-bdeps=y +\-\-pretend \-\-deep \-\-update [\-\-changed\-use|\-\-newuse] [opts]\fR \fI@world\fP +(adapted for specified options appropriately), with that produced +by invoking \fB\fCemerge \-\-oneshot \-\-pretend [opts]\fR +\fI\fP\&. If any differences are +found, report them (and, additionally, stop the build in such a +case, if \fB\fC\-S\fR/\fB\fC\-\-strict\-crosscheck\fR specified). Also report +a series of comparative (total elapsed wall\-clock) timings for both +alternatives, for benchmarking purposes. +.br +  +.br +Note: crosschecking should \fIonly\fP be used for reassurance or +benchmarking, as it will, of necessity, be slower than the baseline +in total time cost (since the check involves running both that +\fIand\fP the newer, \fB\fC\-\-emptytree\fR\-based approach)! So, if your goal is +to improve emerge times, do \fInot\fP pass \fB\fC\-s\fR/\fB\fC\-S\fR\&. +.IP " 4." 5 +Invoke the real \fB\fCemerge\fR, as: \fB\fCemerge \-\-oneshot [opts]\fR +\fI\fP\&. +.br +  +.br +Note that additional arguments may be passed to this invocation, both +explicitly (via \fB\fC\-E\fR/\fB\fC\-\-emerge\-args\fR) and implicitly, via one of +the impacting options (\fB\fC\-v\fR/\fB\fC\-\-verbose\fR, \fB\fC\-a\fR/\fB\fC\-\-ask\fR, +\fB\fC\-A\fR/\fB\fC\-\-alert\fR, or \fB\fC\-p\fR/\fB\fC\-\-pretend\fR). + +.RE + +.SH BASIS +.PP +Why is this approach faster? Well, the main claims behind \fB\fCemtee\fR are: + +.RS +.IP " 1." 5 +An \fB\fC\-\-emptytree\fR \fB\fCemerge\fR of \fI@world\fP yields the same versioned package list +that a \fB\fC\-\-deep \-\-update\fR \fB\fCemerge\fR would arrive at. +.br +  +.br +That is, for \fB\fCemtee\fR to work, it must be true that for a consistent, +depcleaned Gentoo system with a recently updated set of ebuild +repositories, if \fB\fCemerge \-\-with\-bdeps=y \-\-emptytree\fR \fI@world\fP is +invoked and runs successfully to conclusion, then an immediately +following \fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-changed\-use \-\-update\fR +\fI@world\fP will always be a no\-op. +.br +  +.br +Or, to put it another way, we claim that the list of +fully\-qualified atoms (\fIFQAs\fP, where an \fIFQA\fP is \fI$CATEGORY/$PF\fP) +produced by running \fB\fCemerge \-\-with\-bdeps=y \-\-pretend \-\-emptytree +\-\-verbose\fR \fI@world\fP will always describe the same end state reached +by running \fB\fCemerge \-\-with\-bdeps=y \-\-deep \-\-update +[\-\-changed\-use|\-\-newuse]\fR \fI@world\fP from same starting conditions, +as regards packages and versions, anyhow. +.IP " 2." 5 +It also contains sufficient information to simulate \fB\fC\-\-changed\-use\fR +and \fB\fC\-\-newuse\fR\&. +.br +  +.br +Of course, the issue is that in addition to new versions (\fI[N]\fP), +package upgrades (\fI[U]\fP), downgrades (\fI[UD]\fP), new slots (\fI[NS]\fP) +blocks and uninstalls, such a list will generally also contain a +huge number of reinstalls (\fI[R]\fP). Some of these will genuinely +need doing (in light of changed USE flags etc.), but many, +usually the vast majority, will be redundant. +.br +  +.br +Fortunately, for common rebuild selections (such as \fB\fC\-\-changed\-use\fR +and \fB\fC\-\-newuse\fR), we can easily identify which is which, using only +the information provided by the \fB\fC\-\-pretend \-\-emptytree\fR \fB\fCemerge\fR +itself \- since in its output, changes to the USE flag active set +for a given package are shown with an \fI*\fP suffix, and changes to +the remaining set with a \fI%\fP suffix, when \fB\fC\-\-verbose\fR is used. +.IP " 3." 5 +Producing such a list, and then shallow emerging it, reduces the net +dependency calculation time. +.br +  +.br +Finally, we also claim that for a Gentoo system with many installed +packages, the time taken to 1) generate an \fB\fC\-\-emptytree\fR \fI@world\fP +\fIFQA\fP list for all packages, 2) filter this to leave only those +elements that actually \fIneed\fP an install or reinstall (given the +current package set and \fB\fC\-\-changed\-use\fR/\fB\fC\-\-newuse\fR +etc. preference); and 3) invoke a \fB\fC\-\-oneshot\fR \fB\fCemerge\fR on the +resulting list (of \fI=$CATEGORY/$PF\fP \fIFQAs\fP), to the point the first +build actually starts, can be up to an \fIorder of magnitude\fP less +than the equivalent time to first build commencement for a \fB\fC\-\-deep +\-\-update\fR based \fI@world\fP \fB\fCemerge\fR (for a system with many installed +packages and where the number of required updates is (relatively) +small). Yet, if the other claims above are correct, the resulting +merge lists for both approaches will be identical. Furthermore, +this real \fB\fC\-\-oneshot\fR \fB\fCemerge\fR will still deal with triggered slot +change rebuilds and soft block uninstalls for us, and (subject to +\fIEMERGE\_DEFAULT\_OPTS\fP) allow the scheduled builds to be fully +parallelized. + +.RE + +.SH ADVANTAGES +.PP +The speedup for the dependency phase just mentioned, can +translate to hours saved on slow SBCs with binhost backing (where the +build phase itself is relatively low cost). The efficiency gains fall +if a large number of packages require updating, however. + +.PP +Another advantage of this approach is that for some complex updates, +with many blockers, \fB\fCemerge \-\-with\-bdeps=y \-\-pretend \-\-emptytree +\-\-verbose\fR \fI@world\fP can sometimes derive a valid list of \fIFQAs\fP, in +cases where \fB\fCemerge \-\-with\-bdeps=y \-\-pretend \-\-deep \-\-update\fR \fI@world\fP +fails so to do, even with heavy backtracking (although this is a +comparatively rare situation). + +.PP +Note: in the context of this script, an \fIFQA\fP, or fully qualified +atom, is taken to be \fI$CATEGORY/$PF\fP, so for example: +\fIsys\-apps/package\-a\-1.0.4\_rc4\_p3\-r2\fP\&. + +.SH BUGS +.PP +A number of nice \fB\fCemerge\fR features don't work with \fB\fCemtee\fR, such as +\fB\fC\-\-changed\-deps\fR etc. The focus has been on \fB\fC\-\-changed\-use\fR and +\fB\fC\-\-newuse\fR, which are the most common. + +.PP +To operate correctly, \fB\fCemtee\fR needs to be able to parse the output +from \fB\fCemerge\fR\&. So, if the latter's format changes in the future, +expect breakage >< + +.PP +The script's efficiency gains degrade rapidly as the number of +packages requiring upgrade increases. + +.SH COPYRIGHT +.PP +Copyright © 2018\-2019 sakaki + +.PP +License GPLv3+ (GNU GPL version 3 or later) +.br + +\[la]http://gnu.org/licenses/gpl.html\[ra] + +.PP +This is free software, you are free to change and redistribute it. +.br +There is NO WARRANTY, to the extent permitted by law. + +.SH AUTHOR +.PP +sakaki — send bug reports or comments to +\[la]sakaki@deciban.com\[ra] + +.SH SEE ALSO +.PP +\fB\fCdiff\fR(1), \fB\fCemerge\fR(1), \fB\fCportage\fR(5)