How to test a finetuned model on custom dataset?

[songgaojundeng]

My findtuned model based on moirai_1.0_R_small was saved in outputs/finetune/moirai_1.0_R_small folder. I wonder how can I use/load this fine-tuned model for evaluation.

Could you provide more examples/explanations on it? Thank you.

[gorold]

https://github.com/SalesforceAIResearch/uni2ts/blob/main/cli/conf/finetune/model/moirai_1.0_R_small.yaml

https://github.com/SalesforceAIResearch/uni2ts/blob/main/cli/conf/finetune/model/moirai_small.yaml

Please see these 2 files, depending if you want to load the huggingface_hub file or the lightning checkpoint.

[dany4142]

Hi @gorold,

I'm having a couple of issues with the model fine-tuning and loading process:

1. After fine-tuning the model and saving it in the directory specified above, I'm not seeing a .ckpt file that I can load. Could you please clarify the expected output files after the fine-tuning step?

2. Assuming I have the correct saved files from the fine-tuning process, how do I load the fine-tuned model checkpoint so that I can use it to make predictions on a pandas DataFrame? I want to ensure I'm loading the model properly so I can test its performance on new data.

Any guidance you can provide on these two points would be greatly appreciated. Let me know if you need any additional information from me.

Thanks!

[songgaojundeng]

@dany4142 Hey check your default.yaml under finetune folder.

callbacks:
    - _target_: lightning.pytorch.callbacks.LearningRateMonitor
      logging_interval: epoch
    - _target_: lightning.pytorch.callbacks.ModelCheckpoint
      dirpath: ${hydra:runtime.output_dir}/checkpoints
      monitor: val/PackedNLLLoss
      save_weights_only: true
      mode: min
      save_top_k: 1
      every_n_epochs: 1

Maybe you didn't set the validation set properly or maybe the val/PackedNLLLoss so you did not save any checkpoint. Also check the warning.

[songgaojundeng]

@linyuan13 Hey I tested it in notebook, which looks like:

if finetune:
            model = MoiraiForecast.load_from_checkpoint(f"..some path/example_run/checkpoints/epoch=4-step=250.ckpt",local_files_only=True,
                prediction_length=dim_outputseqlen,
                context_length=dim_inputseqlen,
                patch_size=PSZ,
                num_samples=1000,
                target_dim=1,
                feat_dynamic_real_dim=ds.num_feat_dynamic_real,
                past_feat_dynamic_real_dim=ds.num_past_feat_dynamic_real,
            )