Skip to content

Latest commit

 

History

History
27 lines (22 loc) · 1.46 KB

FORMS.md

File metadata and controls

27 lines (22 loc) · 1.46 KB
Raw

Specification of form manipulation

Specification of the value-to-form processing in Lexibank datasets:

The value-to-form processing is divided into two steps, implemented as methods:

  • FormSpec.split: Splits a string into individual form chunks.
  • FormSpec.clean: Normalizes a form chunk.

These methods use the attributes of a FormSpec instance to configure their behaviour.

  • brackets: {'[': ']', '{': '}', '(': ')'} Pairs of strings that should be recognized as brackets, specified as dict mapping opening string to closing string
  • separators: ;/,~ Iterable of single character tokens that should be recognized as word separator
  • missing_data: ('-',) Iterable of strings that are used to mark missing data
  • strip_inside_brackets: True Flag signaling whether to strip content in brackets (and strip leading and trailing whitespace)
  • replacements: [] List of pairs (source, target) used to replace occurrences of source in formswith target (before stripping content in brackets)
  • first_form_only: False Flag signaling whether at most one form should be returned from split - effectively ignoring any spelling variants, etc.
  • normalize_whitespace: True Flag signaling whether to normalize whitespace - stripping leading and trailing whitespace and collapsing multi-character whitespace to single spaces
  • normalize_unicode: None UNICODE normalization form to use for input of split (None, 'NFD' or 'NFC')