How to save checkpoint and later load it?

[syaifulnizamyahya]

I have this to load existing checkpoint, if checkpoint doesnt exist, fit and save the checkpoint.

if force_train==True or os.path.exists(checkpoint_fullpath)==False:
    print("Initializing a new model.")
    engine.fit(datamodule=datamodule, model=model)
else:
    print(f"Loading model from checkpoint: {checkpoint_fullpath}")
    model = Fastflow.load_from_checkpoint(checkpoint_fullpath)

Later when I load the model and run test, the result is extremely low. Unlike when first run fit and test.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │    0.2809523642063141     │
│       image_F1Score       │            0.0            │
│        pixel_AUROC        │    0.49691861867904663    │
└───────────────────────────┴───────────────────────────┘

Here are my model, callbacks and engine setup.

model = Fastflow(backbone="resnet18", flow_steps=8)

callbacks = [
    ModelCheckpoint(
        dirpath=checkpoint_dir, 
        filename=checkpoint_name, 
        mode="max",
        monitor="pixel_AUROC",
        # save_top_k=1, #default is =1. Shoule we set this to 1?
    ),
    EarlyStopping(
        monitor="pixel_AUROC",
        mode="max",
        patience=3,
    ),
]

engine = Engine(
    callbacks=callbacks,
    pixel_metrics="AUROC",
    accelerator="auto",
    devices=1,
    logger=False,
)

What is the best way to save checkpoint and load checkpoint?