New in.exe and with.exe tools

It's been a while since the last release, so here's the latest update...

New Tools and features

• in.exe: Execute a command in a different directory, then come back. Avoids having to remember to run popd afterwards.
  Ex: in .. cd displays the full name of the parent directory.
• with.exe: Execute a command with specific environment variables.
  Useful in the Windows cmd shell, which (contrary to Unix shells) does not have that feature built in.
  Ex: with DEBUG=1 build program.exe builds the debug version of your application.
• echoargs: Display echoargs arguments in Unix.
  Useful to diagnose command-line processing issues, like echoargs.bat and echoargs.ps1 in Windows.
• LODOS.mak: Restores the long-lost ability to build MS-DOS device drivers, TSRs, and MINICOM programs.
  More to come in this respect in future releases!

Changes

• update.exe: Now skips Thumbs.db files in Windows by default. Avoids annoying error messages when backing up big folder trees.
• zap.exe: Added the ability to get the list of pathnames to delete from stdin.
  Useful in combination with commands like findstr /M or ag -l, etc.
• cpuid.exe: Now also comes with an amd64 version, and with greatly improved features.
  It knows about many newer processors features. Option -q queries if a given feature is supported....
• junction.exe: Added option -t, modifying options -l and -r, to list all types of reparse points, and displaying their type.
• encoding.exe: Added option -b to report the presence of a Unicode BOM with a B suffix.
• cfdt.tcl: Added options to control the --m2n name prefix, and changed the default separator from "_" to " ".
• dirc.exe: Separate new option -I (ignore differences <= 2s) from -i (ignore time zone differences).
• 1clip.exe: Added options to query the size and locale of the clipboard data.
• 2clip.exe: Added option -N to remove the final CRLF.

And as usual, many small changes and bug fixes. See NEWS.md for details.

Oops

This bug fix release fixes two problems in version 1.21 released earlier this week:

• A regression, which caused several C executables to fail to correctly detect their input text encoding.
• The 2note.exe program failed to work with the new enhanced Notepad now being released for Windows 11 22H1.

For everything else, please refer to the version 1.21 release notes.

Junctions, loops, search, and more

Would you believe it... It's been almost a year since the last SysToolsLib release.
In between lots of things have changed. Here are the main ones:

New tools:

• junction.exe: Easily manage junctions. A faster and more powerful replacement for Microsoft SysInternals junction.exe.
  Includes unique features for easily managing junctions as if they were relative links, including on network drives.
  See the help screen for details and limitations.
• ShareInfo.exe: Get information about a shared folder on a remote server. Useful to improve the support for junctions on network drives.
• encoding.exe: Detect the text encoding for one or more files (Ex: Binary/ASCII/Windows system code page/UTF-8/UTF-16/UTF-32),
  with several optional heuristics for improving that detection.
• Search.bat: Query the Windows Search Service indexed files, using the Advanced Query Syntax. Very fast, as it's using the system index.
  The results are listed in an Explorer window, ordered by rank of relevance. Changing the view to Details allows reordering by date, etc.
• subsh: The Linux equivalent of subcmd.bat: Open a sub-shell, and change the prompt to show the depth in the shell stack.
• ShellApp.ps1: Enumerate shell applications, i.e. File Explorers and Control Panels.

Major changes under the hood:

• All tools using the MsvcLibX library (Ex: junction.exe, update.exe, etc) now manage junctions on network drives as if they were relative symbolic links.
  This is very useful to people using junctions as a poor man's symbolic link. See the update.exe help screen for details and limitations.
• Likewise, tools using the MsvcLibX library are now immune to link and junctions loops. They detect them, and avoid entering them.
  (Whereas, in case of a loop, Microsoft SysInternals junction.exe lists longer and longer paths, until reaching its redirection limit.)
  (I initially created the junction.exe tool as a test for that feature... After being bitten by actual loops on my hard disk! Loops happen!)
• Window.ps1: Major rewrite. Now much faster at enumerating windows, and outputs more information about them.

Other highlights

• Shell/Library.sh: A sourceable library of useful Bourne Shell routines. Much more limited than Library.bash,
  but useful when compatibility with all (and possibly old) Unix shells is desired.
• update.exe, zap.exe: Can now handle any NTFS file name, including usually impossible names like "end space " or "nul".
  (Which cannot be created by normal Windows apps, but can be created by Linux apps in the WSL.)
  To distinguish files like "nul" from their homonym "nul" system devices, always pass a path for the file. Ex: ".\nul"
• Batch\Library.bat: Added routines :GetEchoState, :ReplaceEquals (Two non trivial things to do in batch!), and :UrlEncode.
• sector.exe and gpt.exe can now manage partitions larger than 2TB.
• sector.exe has a new -lp option for listing partitions.
• conv.exe can now convert to and from UTF-16 and UTF-32.
• remplace.exe, conv.exe, trim.exe, detab.exe, deffeed.exe: If no change was made to the data, use the input file timestamp for the output. (Or preserve it when using the -= switch.)
• dirsize.exe: Added option -f to follow links to directories.
• smbios.exe: Decode several new table types.
• chars.exe: Fixed the output in the new Windows Terminal in code page 437.

And as usual, lots of bug fixes, etc.

Support for multiple Python versions simultaneously

Significantly improved pysetup.bat, and the other python management scripts for Windows

pysetup.bat now defaults to configuring the py.exe launcher as the default command for executing Python scripts.
When multiple instances of Python are configured correctly, it's possible to simultaneously run both Python 2.7 and Python 3.x scripts, by typing just their name (without the .py extension) at the command prompt. I'm preparing a blog post explaining how all that should be done, and how to fix common problems. (Spoiler: SysToolsLib's pysetup.bat will fix everything automatically for you!)
pysetup.bat, pip.bat and python.bat now have a common Python instance selection method, with instance #0 the current default. (Preferably py.exe as explained above.) Use option -? to get more information.

Other changes and new tools

• Renamed lessive.exe as trim.exe. (The French name meant nothing to anybody except for me!)
• Renamed the Bash directory as Shell, as most of the scripts there are actually generic Posix Shell scripts.
• Added an mcd function for Linux, alter-ego of the mcd.bat script for Windows. (Create a directory, and enter it, in a single command.)
• A new -C option in dirc.exe for Windows, displaying the file compression ratio.
• regx.bat: Added options -se and -ue to easily manage System and User Environment variables.
• which.exe does not report App Exec Link targets by default anymore, until we know how to reliably use that target with the other link parameters.

Important bug fixes

• Fixed the -z option in sector.exe, where a regression prevented erasing hard disks.
• Fixed dirsize which reported 0 on some Unix partitions.

And several other smaller improvements and bug fixes.

Christmas Treats

It's been a while since I last made a release, so here's a gift for Christmas!

New scripts

• distrib: Identify the Unix distribution and version. Works in any Posix shell, and in any Unix derivative.
• htmldec.tcl: Decode text strings that contain HTML entities. For use as a filter script with 1clip.exe/2clip.exe/12.bat.
• htmlenc.tcl: Encode text strings with the 3 basic HTML entities for & < >. For use as a filter script with 1clip.exe/2clip.exe/12.bat.

New features in existing scripts and programs:

• which.exe, dirc.exe, truename.exe now detect UWP applications execution links, and display their target.
• dirc.exe now reports WSL 2 symbolic links, and their targets. For WSL 1, it just reports it's an .
• which.exe now identifies pwsh.exe as PowerShell, and uses PowerShell rules when invoked inside pwsh.exe.
• TimeX.bat can now measure durations > 1 day, even over midnight, any day in the year.
• 2clip.exe, conv.exe now auto-detect UTF-8 and UTF-16 input, even without a BOM.
• 1clip.exe, 2clip.exe have new -8 and -16 options to force output to be UTF-8 or UTF-16.
• update.exe accepts target argument "D:=" as meaning "Same path as the source, but on drive D:"
• ipcfg.bat improved the network adapter name detection, and defined "VPN" as the alias for Pulse Secure VPN.
• conv.exe, detab.exe, lessive.exe, remplace.exe now accept -= as an alias for -same, meaning update a file in place.

Under the hood, in the MsvcLibX library

• The Windows build and release generation can now be done from the project root directory, like for Unix.
• All the Windows executables built go into the bin subdirectory in the project root directory.
• Functions readlink(), readdir(), lstat() now handle UWP App. Exec. links and WSL Linux symlinks as if they were standard symlinks.
• Added GNU C library routine asprintf(). Now used for debug output, where it avoid lots of fixed size buffers allocations.

Notes about UWP App. Exec. links

Universal Windows Platform (UWP) applications are new in Windows 10.
They are applications that can run on any types of Windows devices. (PCs, Phones, Xbox, IoT devices, etc)
Circa 2017, Windows 10 introduced a new kind of NTFS link, tagged as an IO_REPARSE_TAG_APPEXECLINK, or AppExecLink here for short.
Most UWP applications installations create an AppExecLink to the actual executable in "%LOCALAPPDATA%\Microsoft\WindowsApps".
This directory is in the Windows 10 PATH, and the links there avoid having to add each application's path to your global PATH.
These AppExecLinks differ from normal NTFS symbolic links in that they contain more than just a pointer to the executable.
I think the additional information helps Windows' CreateTask() speed up the UWP apps startup.
Anyway, 3 years later, Windows' cmd.exe and PowerShell 5.1 still report AppExecLinks as 0-bytes files.
Only the new PowerShell 7.0 correctly reports them as links.

Until now, the System Tools Library was not doing any better than cmd.exe.
But at last the new version released here handles them correctly.
For example if you have installed the Windows Terminal, which.exe now finds it correctly, and truename.exe and dirc.exe also:

C:\JFL\Temp>which wt
C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe

C:\JFL\Temp>which -v wt
# C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps\wt.exe # UWP App. Exec. Link
C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe

C:\JFL\Temp>which -l -v wt
# 2020-11-20 13:26:46        0 C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps\wt.exe -> C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe # UWP App. Exec. Link
2020-11-20 13:26:44    92672 C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe

C:\JFL\Temp>truename C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps\wt.exe
C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe

C:\JFL\Temp>dirc C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps\wt.exe

C:\Users\Larvoire\AppData\Local\Microsoft\WindowsApps

wt.exe -> C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe\wt.exe  <APPEXECL> 2020-11-20 13:26:46

1 files or directories listed.

C:\JFL\Temp>

Improved Linux and MacOS support

This release has much better support for Linux, and for the first time supports MacOS.
The build and installation procedures are the same for MacOS as for Linux, collectively referred to as Unix.

Other significant improvements

• All C tools now build in Unix without warnings.
• Unix make install is now more standard, using the Unix install command internally.
  It can also be dry-runned by running make -n install
• install: New script for installing select tools in Unix. (Contrary to make install which installs everything.)
• update.exe: Several new options, and a few old ones renamed for consistency:
  Renamed switches -e|--erase as -c|--clean. (By analogy with make clean.)
  Renamed option -T|-resettime as -R|-resettime.
  Added switches -T|--tree, independent of -E|--noempty. (As the 2018 change linking the two features did more harm than good.)
  Renamed options -S|--showdest as -D|--dest, and added -S|--source to explicit the default behavior.
  Added options -C|--command to display the equivalent shell commands. (And thus display both the source and dest. files.)
  Added option -B|--nobak to skip backup and temporary files. (*.bak, ~, ##)
• dirc.exe: Added option -B to skip backup and temporary files.
• halve.bat: New filtering function for use with 12.bat.
• sml2.exe: Now preserves head spaces, if any. This allows successfully converting indented blocks of XML.

Major bug fixes

• update.exe: Fixed a serious usability issue when the target was a link to a directory.
• backnum, dirc, dirsize, rd, redo, update, zap: Fixed an issue which sometimes caused failures in Unix.
• which: Fixed wildcards search in Unix.

Please as usual many small improvements and minor bug fixes

New scripts, and many improvements to old ones

New tools

• md2h.bat: Converts a markdown file to HTML, using GitHub APIs, then displays it in your Web browser.
  This is useful to verify that the markdown you typed in a README.md file will display as intended after being pushed into GitHub.
  Note: I’ve not found how to use GitHub’s actual CSS style sheets. So the generated HTML uses makeshift style sheets, that do not look as good as the real ones. This is not really an issue, since the objective is to verify the structure and contents of the HTML, not the aesthetics of the output.
• WhereAmI.bat|WhereAmI.py|WhereAmI.tcl: Three versions of the same script, displaying your location based on your IP address.
  I've copied these files over from another project I contributed to: https://github.com/JFLarvoire/today
  It works by querying a free Web service, that infers your location based on your IP address.
  Note if you're running this on an intranet with a web proxy, you'll get the location of the proxy, not your own location.
  Note on note: WhereAmI.bat uses the global proxy configured in Windows. The other two rely on the http-proxy environment variable.

Significant improvements

• cpuid.exe: Now decodes CPUID(7) “Structured Extended Feature Flags”. This lists all processor improvements released in the last 10 years.
  (Give it a try with cpuid -v to see what your processor is capable of!)
  It also now gets the processor Second Level Address Translation (SLAT) capability from WMI.
  Useful to know if the processor is compatible with HyperV 2016 and 2019, and with Docker.
• zap.exe: Can now delete multiple directories using wild cards. See the -? help screen for usage information.
• inicomp.exe: Now works with .ini files with sections split in several disjoint parts.
  Also added a new -f option to support files with non-standard settings without an =value.
• ShadowCopy.ps1: Major performance improvement when searching for the previous versions of a file.
• cascade.tcl: A new -l option to list cascadable applications, or list the cascadable windows for one application.
• regsub.tcl: Added several options for controlling Case [in]dependant matching; Line or global mode; First or all matches.
• AutoRun.cmd: AutoRun -i now installs the extension scripts from AutoRun.d\ listed in AutoRun.d\default.lst.
  This should make it much easier to configure AutoRun on new systems, from the contents of the SysTools.zip distribution.
  (Reminder: Among other good things, AutoRun allows having a history command in Windows, that can be piped to grep or findstr, like the history command in Linux.)

Plus as usual many other small improvements and bug fixes. See NEWS.md in the project root for the complete list.

Which now supports wildcards; Several new scripts

Main changes

• The which/which.exe program can now search for programs using wildcards.
  This is very useful when you don't quite remember the exact name of a program.
  Ex, try: which -a *zip* in Windows, or which -a '*zip*' in Unix

• New scripts paths.bat for Windows, and paths for Unix, showing the PATH with one path per line, and making it easy to reorder it.
  Optionally manages any other variable structured like PATH.
  More intuitive to use than the old AddPaths.bat scripts they replace.

• New scripts python.bat and pip.bat, for managing multiple instances of Python on Windows.
  (Useful now that Microsoft installs its own instances, without looking for non-Microsoft ones that may have been there before!)

• The new Get-Console.bat is a front-end to Get-Console.ps1, for capturing the console contents as coloured text into the clipboard.

• All Windows executables now include program properties, with a description, version, product infos, etc.

• The ARM64 versions of the C programs now build without any warning...
  But I've still not been able to test them for lack of a Windows 10/ARM64 test system.

Plus as usual many small improvements and bug fixes.

ARM64 builds; Powerful doskey aliases and AutoRun scripts management; Etc.

Major novelties:

• The build system now supports Visual Studio 2019, and ARM64 versions of all *.exe programs are provided.
  Important: I've not tested these ARM64 programs, as I don't have an ARM64 version of Windows.
  If you have one and can help, please tell me in issue #23.
• which.exe now implements a -i option for Windows, akin to which -i for Linux.
  It efficiently detects DOSKEY aliases, cmd.exe internal commands, and PowerShell aliases and functions.
  Run which -? for details on how to configure cmd.exe and PowerShell for that.
  You can also find a detailed presentation of which.exe there.
• A new AutoRun.cmd script, and an AutoRun.cmd.d subdirectory, to easily manage multiple AutoRun scripts.
  Several aliases definitions are provided in AutoRun.cmd.d: history, pid, which.
  See the Batch/AutoRun.cmd.d/README.md file for details.
  You can find a detailed presentation of AutoRun.cmd there.

New scripts:

• DumpLink.bat: Display the contents of Windows Explorer *.lnk shortcuts.
• urldecode.tcl: An old script for decoding URLs encoded with %XX hexadecimal codes.

Other changes:

• All *.exe programs -V/--version now display the minimum Windows version supported.
• which.exe: Added option -- to stop processing switches.
• ag.exe: The silver searcher now expands wildcards in command-line pathnames.
• dirc.exe: Added option -ct to report equal files with different times.
• update.exe: Added option -- to stop processing switches.
  Added option -P to show the file copy progress.
  Added option -T to reset the destination file time to that of the source file, if they have identical sizes.
• chars.exe: Added option -u to display Unicode characters or ranges.
  Added option -v to display verbose information.
  Improved error reporting when switching code pages.

Bug fixes:

• n.bat: Work around trailing spaces issue in Windows 10 v 2019-03.
• font.exe: Fixed the console font setting, which did not work well with TrueType fonts.
• cfdt.tcl: Fixed the --i2n option, which moved files to the current directory.
• regx.bat: Fixed the dir command when the pathname contains spaces.
• update.exe: Fixed a 2018-12-18 bug causing Error: Not enough arguments

A much improved console font management tool

This is a minor release with one important change:
The font.exe tool now has the ability to save and restore the console font.
This is very useful when another program unexpectedly changes the console font as a side effect.
This happens on one of my machines, where PowerShell always changes the font back to a Raster font. (I have no idea why, nor why this happens on this machine and not on the others.)
Anyway, to save and restore the cmd.exe shell font, use the following commands:

:# Save the current font settings in an environment variable
for /f "delims=" %f in ('font -s') do @set OLD_FONT=%f
:# Run the font-breaking command
powershell -c $PSVersionTable
:# Restore the initial font
font %OLD_FONT%

Another minor change in this release: The sml2.exe tool now defaults to ignoring XML parser warnings and errors. They can be re-enabled if desired.