New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: add check for NaN validation loss in EarlyStopping #28

Merged

Bjarten merged 2 commits into main from fix/early-stopping-nan-check

Oct 15, 2024

Owner

Bjarten commented Oct 15, 2024

This PR addresses an issue where EarlyStopping incorrectly treats nan validation losses as an improvement, often caused by exploding gradients.

Key changes:

Added np.isnan(val_loss) check to ensure that nan validation losses are ignored.
Updated the logic to ensure that the patience counter and model checkpointing are unaffected by nan values.
Introduced a new unit test, test_validation_loss_nan, to verify that EarlyStopping behaves correctly when nan values are encountered during training.

Closes #16

Bjarten added 2 commits

October 15, 2024 12:53


          fix: add check for nan in early stopping

efe20ab


          ci: update tests for new early stopping check

f4f1b80

Bjarten merged commit 676686b into main

5 checks passed

Bjarten deleted the fix/early-stopping-nan-check branch

October 15, 2024 04:04

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet