You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Oct 2, 2023. It is now read-only.
Some parsers log progress (e.g. every X parsed rows) and information about parsed files, some others do nothing. We should define a pattern and apply to all parsers.
Suggestion (as seen in the corpwatch parser):
log file name on start of parsing a specific file
then log every 100.000 rows/records
log finished count of records for this file
repeat
This allows to keep track how source file sizes develop over time and to spot data source errors.
The text was updated successfully, but these errors were encountered:
One aspect to consider here: GitHub Actions has a tendency to kill jobs that don't print any output for an extended period. So we should output something, every now and then.
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Some parsers log progress (e.g. every X parsed rows) and information about parsed files, some others do nothing. We should define a pattern and apply to all parsers.
Suggestion (as seen in the corpwatch parser):
This allows to keep track how source file sizes develop over time and to spot data source errors.
The text was updated successfully, but these errors were encountered: