NEWS

Snowball 2.2.0 (2021-11-10)
===========================

New Code Generators
-------------------

* Add Ada generator from Stephane Carrez (#135).

Javascript
----------

* Fix generated code to use integer division rather than floating point
  division.

  Noted by David Corbett.

Pascal
------

* Fix code generated for division.  Previously real division was used and the
  generated code would fail to compile with a "Incompatible types" error.

  Noted by David Corbett.

* Fix code generated for Snowball's `minint` and `maxint` constant.

Python
------

* Python 2 is no longer actively supported, as proposed on the mailing list:
  https://lists.tartarus.org/pipermail/snowball-discuss/2021-August/001721.html

* Fix code generated for division.  Previously the Python code we generated
  used integer division but rounded negative fractions towards negative
  infinity rather than zero under Python 2, and under Python 3 used floating
  point division.

  Noted by David Corbett.

Code Quality Improvements
-------------------------

* C#: An `among` without functions is now generated as `static` and groupings
  are now generated as constant.  Patches from James Turner in #146 and #147.

Code generation improvements
----------------------------

* General:

  + Constant numeric subexpressions and constant numeric tests are now
    evaluated at Snowball compile time.

Behavioural changes to existing algorithms
------------------------------------------

* german2: Fix handling of `qu` to match algorithm description.  Previously
  the implementation erroneously did `skip 2` after `qu`.  We suspect this was
  intended to skip the `qu` but that's already been done by the substring/among
  matching, so it actually skips an extra two characters.

  The implementation has always differed in this way, but there's no good
  reason to skip two extra characters here so overall it seems best to change
  the code to match the description.  This change only affects the stemming of
  a single word in the sample vocabulary - `quae` which seems to actually be
  Latin rather than German.

Optimisations to existing algorithms
------------------------------------

* arabic: Handle exception cases in the among they're exceptions to.

* greek: Remove unused slice setting, handle exception cases in the among
  they're exceptions to, and turn `substring ... among ...  or substring ...
  among ...` into a single `substring ... among ...` in cases where it is
  trivial to do so.

* hindi: Eliminate the need for variable `p`.

* irish: Minor optimisation in setting `pV` and `p1`.

* yiddish: Make use of `among` more.

Compiler
--------

* Fix handling of `len` and `lenof` being declared as names.

  For compatibility with programs written for older Snowball versions
  len and lenof stop being tokens if declared as names.  However this
  code didn't work correctly if the tokeniser's name buffer needed to
  be enlarged to hold the token name (i.e. 3 or 5 elements respectively).

* Report a clearer error if `=` is used instead of `==` in an integer test.

* Replace a single entry command list with its contents in the internal syntax
  tree.  This puts things in a more canonical form, which helps subsequent
  optimisations.

Build system
------------

* Support building on Microsoft Windows (using mingw+msys or a similar
  Unix-like environment).  Patch from Jannick in #129.

* Split out INCLUDES from CPPFLAGS so that CPPFLAGS can now be overridden by
  the user if required.  Fixes #148, reported by Dominique Leuenberger.

* Regenerate algorithms.mk only when needed rather than on every `make` run.

libstemmer
----------

* The libstemmer static library now has a `.a` extension, rather than `.o`.
  Patch from Michal Vasilek in #150.

Testsuite
---------

* stemtest: Test that numbers and numeric codes aren't damaged by any of the
  algorithms.  Regression test for #66.  Fixes #81.

* ada: Fix ada tests to fail if output differs.  There was an extra `| head
  -300` compared to other languages, which meant that the exit code of `diff`
  was ignored.  It seems more helpful (and is more consistent) not to limit how
  many differences are shown so just drop this addition.

* go: Stop thinning testdata.  It looks like we only are because the test
  harness code was based on that for rust, which was based on that for
  javascript, which was only thinning because it was reading everything into
  memory and the larger vocabulary lists were resulting in out of memory
  issues.

* javascript: Speed up stemwords.js.  Process input line-by-line rather than
  reading the whole file into memory, splitting, iterating, and creating an
  array with all the output, joining and writing out a single huge string.
  This also means we can stop thinning the test data for javascript, which we
  were only doing because the huge arabic test data file was causing out of
  memory errors.  Also drop the -p option, which isn't useful here and
  complicates the code.

* rust: Turn on optimisation in the makefile rather than the CI config.  This
  makes the tests run in about 1/5 of the time and there's really no reason to
  be thinning the testdata for rust.

Documentation
-------------

* CONTRIBUTING.rst: Improve documentation for adding a new stemming algorithm.

* Improve wording of Python docs.

Snowball 2.1.0 (2021-01-21)
===========================

C/C++
-----

* Fix decoding of 4-byte UTF-8 sequences in `grouping` checks.  This bug
  affected Unicode codepoints U+40000 to U+7FFFF and U+C0000 to U+FFFFF and
  doesn't affect any of the stemming algorithms we currently ship (#138,
  reported by Stephane Carrez).

Python
------

* Fix snowballstemmer.algorithms() method (#132, reported by kkaiser).

* Update code to generate trove language classifiers for PyPI.  All the
  natural languages we previously had stemmers for have now been added to
  PyPI's list, but Armenian and Yiddish aren't on it.  Patch from Dmitry
  Shachnev.

Code Quality Improvements
-------------------------

* Suppress GCC warning in compiler code.

* Use `const` pointers more in C runtime.

* Only use spaces for indentation in javascript code.  Change proposed by Emily
  Marigold Klassen in #123, and seems to be the modern Javascript norm.

New Snowball Language Features
------------------------------

* `lenof` and `sizeof` can now be applied to a literal string, which can be
  useful if you want to do calculations on cursor values.

  This change actually simplifies the language a little, since you can now use
  a literal string in any read-only context which accepts a string variable.

Code generation improvements
----------------------------

* General:

  + Fix bugs in the code generated to handle failure of `goto`, `gopast` or
    `try` inside `setlimit` or string-`$`.  This affected all languages (though
    the issue with `try` wasn't present for C).  These bugs don't affect any of
    the stemming algorithms we currently ship.  Reported by Stefan Petkovic on
    snowball-discuss.

  + Change `hop` with a negative argument to work as documented.  The manual
    says a negative argument to hop will raise signal f, but the implementation
    for all languages was actually to move the cursor in the opposite direction
    to `hop` with a positive argument.  The implemented behaviour is
    problematic as it allows invalidating implicitly saved cursor values by
    modifying the string outside the current region, so we've decided it's best
    to fix the implementation to match the documentation.

    The only Snowball code we're aware of which relies on this was the original
    version of the new Yiddish stemming algorithm, which has been updated not
    to rely on this.

    The compiler now issues a warning for `hop` with a constant negative
    argument (internally now converted to `false`), and for `hop` with a
    constant zero argument (internally now converted to `true`).

  + Canonicalise `among` actions equivalent to `()` such as `(true)` which
    previously resulted in an extra case in the among, and for Python
    we'd generate invalid Python code (`if` or `elif` with an empty body).
    Bug revealed by Assaf Urieli's Yiddish stemmer in #137.

  + Eliminate variables whose values are never used - they no longer have
    corresponding member variables, etc, and no code is generated for any
    assignments to them.

  + Don't generate anything for an unused `grouping`.

  + Stop warning "grouping X defined but not used" for a `grouping` which is
    only used to define other another `grouping`.

* C/C++:

  + Store booleans in same array as integers.  This means each boolean is
    stored as an int instead of an unsigned char which means 4 bytes instead of
    1, but we save a pointer (4 or 8 bytes) in struct SN_env which is a win for
    all the current stemmers.  For an algorithm which uses both integers and
    booleans, we also save the overhead of allocating a block on the heap, and
    potentially improve data locality.

  + Eliminate duplicate generated C comment for sliceto.

* Pascal:

  + Avoid generating unused variables.  The Pascal code generated for the
    stemmers we ship is now warning free (tested with fpc 3.2.0).

* Python:

  + End `if`-chain with `else` where possible, avoiding a redundant test
    of the variable being switched on.  This optimisation kicks in for an
    `among` where all cases have commands.  This change seems to speed up `make
    check_python_arabic` by a few percent.

New stemming algorithms
-----------------------

* Add Serbian stemmer from stef4np (#113).

* Add Yiddish stemmer from Assaf Urieli (#137).

* Add Armenian stemmer from Astghik Mkrtchyan.  It's been on the website for
  over a decade, and included in Xapian for over 9 years without any negative
  feedback.

Optimisations to existing algorithms
------------------------------------

* kraaij_pohlmann: Use `$v = limit` instead of `do (tolimit setmark v)` since
  this generates simpler code, and also matches the code other algorithm
  implementations use.

  Probably for languages like C with optimising compilers the compiler
  will generate equivalent code anyway, but e.g. for Python this should be
  an improvement.

Code clarity improvements to existing algorithms
------------------------------------------------

* hindi.sbl: Fix comment typo.

Compiler
--------

* Don't count `$x = x + 1` as initialising or using `x`, so it's now handled
  like `$x += 1` already is.

* Comments are now only included in the generated code if command like option
  -comments is specified.

  The comments in the generated code are useful if you're trying to debug the
  compiler, and perhaps also if you are trying to debug your Snowball code, but
  for everyone else they just bloat the code which as the number of languages
  we support grows becomes more of an issue.

* `-parentclassname` is not only for java and csharp so don't disable it if
  those backends are disabled.

* `-syntax` now reports the value for each numeric literal.

* Report location for excessive get nesting error.

* Internally the compiler now represents negated literal numbers as a simple
  `c_number` rather than `c_neg` applied to a `c_number` with a positive value.
  This simplifies optimisations that want to check for a constant numeric
  expression.

Build system
------------

* Link binaries with LDFLAGS if it's set, which is needed for some platform
  (e.g. OpenEmbedded).  Patch from Andreas Müller (#120).

* Add missing dependencies of algorithms.go rule.

Testsuite
---------

* C: Add stemtest for low-level regression tests.

Documentation
-------------

* Document a C99 compiler as a requirement for building the snowball compiler
  (but the C code it generates should still work with any ISO C compiler).

  A few declarations mixed with code crept in some time ago (which nobody's
  complained about), so this is really just formally documenting a requirement
  which already existed.

* README: Explain what Snowball is and what Stemming is (#131, reported by Sean
  Kelly).

* CONTRIBUTING.rst: Expand section on adding a new generator.

* For Python snowballstemmer module include global NEWS instead of
  Python-specific CHANGES.rst and use README.rst as the long description.
  Patch from Dmitry Shachnev (#119).

* COPYING: Update and incorporate Python backend licensing information which
  was previously in a separate file.

Snowball 2.0.0 (2019-10-02)
===========================

C/C++
-----

* Fully handle 4-byte UTF-8 sequences.  Previously `hop` and `next` handled
  sequences of any length, but commands which look at the character value only
  handled sequences up to length 3.  Fixes #89.

* Fix handling of a 3-byte UTF-8 sequence in a grouping in `backwardmode`.

Java
----

* TestApp.java:

  - Always use UTF-8 for I/O.  Patch from David Corbett (#80).

  - Allow reading input from stdin.

  - Remove rather pointless "stem n times" feature.

  - Only lower case ASCII to match stemwords.c.

  - Stem empty lines too to match stemwords.c.

Code Quality Improvements
-------------------------

* Fix various warnings from newer compilers.

* Improve use of `const`.

* Share common functions between compiler backends rather than having multiple
  copies of the same code.

* Assorted code clean-up.

* Initialise line_labelled member of struct generator to 0.  Previously we were
  invoking undefined behaviour, though in practice it'll be zero initialised on
  most platforms.

New Code Generators
-------------------

* Add Python generator (#24).  Originally written by Yoshiki Shibukawa, with
  additional updates by Dmitry Shachnev.

* Add Javascript generator.  Based on JSX generator (#26) written by Yoshiki
  Shibukawa.

* Add Rust generator from Jakob Demler (#51).

* Add Go generator from Marty Schoch (#57).

* Add C# generator.  Based on patch from Cesar Souza (#16, #17).

* Add Pascal generator.  Based on Delphi backend from stemming.zip file on old
  website (#75).

New Snowball Language Features
------------------------------

* Add `len` and `lenof` to measure Unicode length.  These are similar to `size`
  and `sizeof` (respectively), but `size` and `sizeof` return the length in
  bytes under `-utf8`, whereas these new commands give the same result whether
  using `-utf8`, `-widechars` or neither (but under `-utf8` they are O(n) in
  the length of the string).  For compatibility with existing code which might
  use these as variable or function names, they stop being treated as tokens if
  declared to be a variable or function.

* New `{U+1234}` stringdef notation for Unicode codepoints.

* More versatile integer tests.  Now you can compare any two arithmetic
  expressions with a relational operator in parentheses after the `$`, so for
  example `$(len > 3)` can now be used when previously a temporary variable was
  required: `$tmp = len $tmp > 3`

Code generation improvements
----------------------------

* General:

  + Avoid unnecessarily saving and restoring of the cursor for more commands -
    `atlimit`, `do`, `set` and `unset` all leave the cursor alone or always
    restore its value, and for C `booltest` (which other languages already
    handled).

  + Special case handling for `setlimit tomark AE`.  All uses of setlimit in
    the current stemmers we ship follow this pattern, and by special-casing we
    can avoid having to save and restore the cursor (#74).

  + Merge duplicate actions in the same `among`.  This reduces the size of the
    switch/if-chain in the generated code which dispatch the among for many of
    the stemmers.

  + Generate simpler code for `among`.  We always check for a zero return value
    when we call the among, so there's no point also checking for that in the
    switch/if-chain.  We can also avoid the switch/if-chain entirely when
    there's only one possible outcome (besides the zero return).

  + Optimise code generated for `do <function call>`.  This speeds up "make
    check_python" by about 2%, and should speed up other interpreted languages
    too (#110).

  + Generate more and better comments referencing snowball source.

  + Add homepage URL and compiler version as comments in generated files.

* C/C++:

  + Fix `size` and `sizeof` to not report one too high (reported by Assem
    Chelli in #32).

  + If signal `f` from a function call would lead to return from the current
    function then handle this and bailing out on an error together with a
    simple `if (ret <= 0) return ret;`

  + Inline testing for a single character literals.

  + Avoiding generating `|| 0` in corner case - this can result in a compiler
    warning when building the generated code.

  + Implement `insert_v()` in terms of `insert_s()`.

  + Add conditional `extern "C"` so `runtime/api.h` can be included from C++
    code.  Closes #90, reported by vvarma.

* Java:

  + Fix functions in `among` to work in Java.  We seem to need to make the
    methods called from among `public` instead of `private`, and to call them
    on `this` instead of the `methodObject` (which is cleaner anyway).  No
    revision in version control seems to generate working code for this case,
    but Richard says it definitely used to work - possibly older JVMs failed to
    correctly enforce the access controls when methods were invoked by
    reflection.

  + Code after handling `f` by returning from the current function is
    unreachable too.

  + Previously we incorrectly decided that code after an `or` was
    unreachable in certain cases.  None of the current stemmers in the
    distribution triggered this, but Martin Porter's snowball version
    of the Schinke Latin stemmer does.  Fixes #58, reported by Alexander
    Myltsev.

  + The reachability logic was failing to consider reachability from
    the final command in an `or`.  Fixes #82, reported by David Corbett.

  + Fix `maxint` and `minint`.  Patch from David Corbett in #31.

  + Fix `$` on strings.  The previous generated code was just wrong.  This
    doesn't affect any of the included algorithms, but for example breaks
    Martin Porter's snowball implementation of Schinke's Latin Stemmer.
    Issue noted by Jakob Demler while working on the Rust backend in #51,
    and reported in the Schinke's Latin Stemmer by Alexander Myltsev
    in #58.

  + Make SnowballProgram objects serializable.  Patch from Oleg Smirnov in #43.

  + Eliminate range-check implementation for groupings.  This was removed from
    the C generator 10 years earlier, isn't used for any of the existing
    algorithms, and it doesn't seem likely it would be - the grouping would
    have to consist entirely of a contiguous block of Unicode code-points.

  + Simplify code generated for `repeat` and `atleast`.

  + Eliminate unused return values and variables from runtime functions.

  + Only import the `among` and `SnowballProgram` classes if they're actually
    used.

  + Only generate `copy_from()` method if it's used.

  + Merge runtime functions `eq_s` and `eq_v` functions.

  + Java arrays know their own length so stop storing it separately.

  + Escape char 127 (DEL) in generated Java code.  It's unlikely that this
    character would actually be used in a real stemmer, so this was more of a
    theoretical bug.

  + Drop unused import of InvocationTargetException from SnowballStemmer.
    Reported by GerritDeMeulder in #72.

  + Fix lint check issues in generated Java code.  The stemmer classes are only
    referenced in the example app via reflection, so add
    @SuppressWarnings("unused") for them.  The stemmer classes override
    equals() and hashCode() methods from the standard java Object class, so
    mark these with @Override.  Both suggested by GerritDeMeulder in #72.

  + Declare Java variables at point of use in generated code.  Putting all
    declarations at the top of the function was adding unnecessary complexity
    to the Java generator code for no benefit.

  + Improve formatting of generated code.

New stemming algorithms
-----------------------

* Add Tamil stemmer from Damodharan Rajalingam (#2, #3).

* Add Arabic stemmer from Assem Chelli (#32, #50).

* Add Irish stemmer from Jim O'Regan (#48).

* Add Nepali stemmer from Arthur Zakirov (#70).

* Add Indonesian stemmer from Olly Betts (#71).

* Add Hindi stemmer from Olly Betts (#73). Thanks to David Corbett for review.

* Add Lithuanian stemmer from Dainius Jocas (#22, #76).

* Add Greek stemmer from Oleg Smirnov (#44).

* Add Catalan and Basque stemmers from Israel Olalla (#104).

Behavioural changes to existing algorithms
------------------------------------------

* Portuguese:

  + Replace incorrect Spanish suffixes by Portuguese suffixes (#1).

* French:

  + The MSDOS CP850 version of the French algorithm was missing changes present
    in the ISO8859-1 and Unicode versions.  There's now a single version of
    each algorithm which was based on the Unicode version.

  + Recognize French suffixes even when they begin with diaereses.  Patch from
    David Corbett in #78.

* Russian:

  + We now normalise 'ё' to 'е' before stemming.  The documentation has long
    said "we assume ['ё'] is mapped into ['е']" but it's more convenient for
    the stemmer to actually perform this normalisation.  This change has no
    effect if the caller is already normalising as we recommend.  It's a change
    in behaviour they aren't, but 'ё' occurs rarely (there are currently no
    instances in our test vocabulary) and this improves behaviour when it does
    occur.  Patch from Eugene Mirotin (#65, #68).

* Finish:

  + Adjust the Finnish algorithm not to mangle numbers.  This change also
    means it tends to leave foreign words alone.  Fixes #66.

* Danish:

  + Adjust Danish algorithm not to mangle alphanumeric codes. In particular
    alphanumeric codes ending in a double digit (e.g. 0x0e00, hal9000,
    space1999) are no longer mangled.  See #81.

Optimisations to existing algorithms
------------------------------------

* Turkish:

  + Simplify uses of `test` in stemmer code.

  + Check for 'ad' or 'soyad' more efficiently, and without needing the
    strlen variable.  This speeds up "make check_utf8_turkish" by 11%
    on x86 Linux.

* Kraaij-Pohlmann:

  + Eliminate variable x `$p1 <= cursor` is simpler and a little more efficient
    than `setmark x $x >= p1`.

Code clarity improvements to existing algorithms
------------------------------------------------

* Turkish:

  + Use , for cedilla to match the conventions used in other stemmers.

* Kraaij-Pohlmann:

  + Avoid cryptic `[among ( (])` ... `)` construct - instead use the same
    `[substring] among (` ... `)` construct we do in other stemmers.

Compiler
--------

* Support conventional --help and --version options.

* Warn if -r or -ep used with backend other than C/C++.

* Warn if encoding command line options are specified when generating code in a
  language with a fixed encoding.

* The default classname is now set based on the output filename, so `-n` is now
  often no longer needed.  Fixes #64.

* Avoid potential one byte buffer over-read when parsing snowball code.

* Avoid comparing with uninitialised array element during compilation.

* Improve `-syntax` output for `setlimit L for C`.

* Optimise away double negation so generators don't have to worry about
  generating `--` (decrement operator in many languages).  Fixes #52, reported
  by David Corbett.

* Improved compiler error and warning messages:

  - We now report FILE:LINE: before each diagnostic message.

  - Improve warnings for unused declarations/definitions.

  - Warn for variables which are used, but either never initialised
    or never read.

  - Flag non-ASCII literal strings.  This is an error for wide Unicode, but
    only a warning for single-byte and UTF-8 which work so long as the source
    encoding matches the encoding used in the generated stemmer code.

  - Improve error recovery after an undeclared `define`.  We now sniff the
    token after the identifier and if it is `as` we parse as a routine,
    otherwise we parse as a grouping.  Previously we always just assumed it was
    a routine, which gave a confusing second error if it was a grouping.

  - Improve error recovery after an unexpected token in `among`.  Previously
    we acted as if the unexpected token closed the `among` (this probably
    wasn't intended but just a missing `break;` in a switch statement).  Now we
    issue an error and try the next token.

* Report error instead of silently truncating character values (e.g. `hex 123`
  previously silently became byte 0x23 which is `#` rather than a
  g-with-cedilla).

* Enlarge the initial input buffer size to 8192 bytes and double each time we
  hit the end.  Snowball programs are typically a few KB in size (with the
  current largest we ship being the Greek stemmer at 27KB) so the previous
  approach of starting with a 10 byte input buffer and increasing its size by
  50% plus 40 bytes each time it filled was inefficient, needing up to 15
  reallocations to load greek.sbl.

* Identify variables only used by one `routine`/`external`.  This information
  isn't yet used, but such variables which are also always written to before
  being read can be emitted as local variables in most target languages.

* We now allow multiple source files on command line, and allow them to be
  after (or even interspersed) with options to better match modern Unix
  conventions.  Support for multiple source files allows specifying a single
  byte character set mapping via a source file of `stringdef`.

* Avoid infinite recursion in compiler when optimising a recursive snowball
  function.  Recursive functions aren't typical in snowball programs, but
  the compiler shouldn't crash for any input, especially not a valid one.
  We now simply limit on how deep the compiler will recurse and make the
  pessimistic assumption in the unlikely event we hit this limit.

Build system
------------

* `make clean` in C libstemmer_c distribution now removes `examples/*.o`.
  (#59)

* Fix all the places which previously had to have a list of stemmers to work
  dynamically or be generated, so now only modules.txt needs updating to add
  a new stemmer.

* Add check_java make target which runs tests for java.

* Support gzipped test data (the uncompressed arabic test data is too big for
  github).

* GNUmakefile: Drop useless `-eprefix` and `-r` options from snowball
  invocations for Java - these are only meaningful when generating C code.

* Pass CFLAGS when linking which matches convention (e.g. automake does it) and
  facilitates use of tools such as ASan.  Fixes #84, reported by Thomas
  Pointhuber.

* Add CI builds with -std=c90 to check compiler and generated code are C90
  (#54)

libstemmer
----------

* Split out CPPFLAGS from CFLAGS and use CFLAGS when linking stemwords.

* Add -O2 to CFLAGS.

* Make generated tables of encodings and modules const.

* Fix clang static analyzer memory leak warning (in practice this code path
  can never actually be taken).  Patch from Patrick O. Perry (#56)

Documentation
-------------

* Added copyright and licensing details (#10).

* Document that libstemmer supports ISO_8859_2 encoding.  Currently hungarian
  and romanian are available in ISO_8859_2.

* Remove documentation falsely claiming that libstemmer supports CP850
  encoding.

* CONTRIBUTING.rst: Add guidance for contributing new stemming algorithms and
  new language backends.

* Overhaul libstemmer_python_README.  Most notably, replace the benchmark data
  which was very out of date.