update train_sts_seed_optimization with SentenceTransformerTrainer

[JINO-ROHIT]

This PR updates the example script train_sts_seed_optimization.py with SentenceTransformerTrainer.

I also noticed the documentation was quite outdated when I was referring for some args, should we try and look to update them too?

@tomaarsen

[tomaarsen]

The SentenceTransformerTrainingArguments has a warmup_ratio=0.1 that we can use instead.

[tomaarsen]

I don't think this is used right now

[tomaarsen]

I think the stop_after isn't actually making it stop after this many steps.
Normally you can use max_steps, but then I think it messes with the scheduler, ideally we want the scheduler to be "normal" but then still stop after stop_after steps, but I'm not sure if that's the old behaviour either.

[tomaarsen]

I'm also curious what you mean with the outdated docs - I'd also like that to be fixed if possible.

[JINO-ROHIT]

ahh yeah ill change those.

sorry about the docs, i was accidentally referring to the old fit method here - https://www.sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#:~:text=steps_per_epoch%20%E2%80%93%20Number%20of%20training%20steps,the%20DataLoader%20size%20from%20train_objectives.&text=warmup_steps%20%E2%80%93%20Behavior%20depends%20on%20the,to%20the%20maximal%20learning%20rate. and saw args like steps_per_epoch and warmup_steps that werent there in the Trainer

[JINO-ROHIT]

also, i dont quite understand the stop_after bit as well, is a custom callback expected?

[tomaarsen]

also, i dont quite understand the stop_after bit as well, is a custom callback expected?

Makes sense, this is a little confusing. I think the idea is that we create 1 epoch of e.g. 100k steps. The seed (e.g. for data sampling) has been shown to be fairly important for training embedding models, so we want to train e.g. the first 30k steps out of the 100k and then see where we're at. Then we can pick the seed that performed the best after just a bit of training.

But if we use max_steps=0.3 * total_steps, then our scheduler will also recognize that we're only doing 30k steps, and update accordingly.

Instead, we want the scheduler to think that we're doing 100k steps, but indeed we want the training to stop after 30k (or stop_after * total_steps). I think a custom callback is indeed a good solution, I'll whip something up and add it to this PR.

• Tom Aarsen

[tomaarsen]

This was my final log with this script at the default parameters:

Current sts-dev_spearman_cosine Scores per Seed: {7: 0.8510304677377857,
 9: 0.8487004831766769,
 5: 0.8486879350254634,
 1: 0.8446502257325709,
 2: 0.8422240179194527,
 4: 0.8402418940176778,
 6: 0.8381279059862979,
 0: 0.8367594493128546,
 3: 0.831016545559169,
 8: 0.8271094833035064}

So there's indeed a pretty big difference, 0.827 VS 0.851.

• Tom Aarsen

[JINO-ROHIT]

ah thanks for explaining, this makes sense.
whup, there still seems to be a diff with your custom callback? is it possible because of the randomness of the seed itself?

[tomaarsen]

A difference? Between the evaluation scores you mean?
Indeed, those are because of the different seed (which impacts a lot in the training process, from data sampling to dropout, and also weight initialization if you're training fully from scratch).

• Tom Aarsen

[JINO-ROHIT]

ahh okay okay

[tomaarsen]

Thanks for tackling this!

• Tom Aarsen