[meta] Kick off a 2024-H2 training run

[gregtatum]

We'd like to kick off another training run to fix quality issues in spring-2024 where the models weren't shippable and pick up a few languages we skipped. This meta issue will collect issues I'd like to get done before kicking it off. Also I'd like to incorporate more cost saving work.

This does not cover CJK.

Tasks

Beta Give feedback

1. Investigate CTranslate2 for translating sentences with teacher model firefox-translations-training#165
   cost & perf +1
   
2. BLOCKING MozIntGemm is not working for larger translation models
3. Experiment with student model parameters #894
   experiment quality +2
   
4. Investigate removing teacher ensemble training firefox-translations-training#778
   cost & perf experiment +2
   
5. Improve GPU utilization for "translate" tasks firefox-translations-training#785
   cost & perf +1
6. Improve GPU utilization in student training firefox-translations-training#783
   cost & perf +1
7. Limit the amount of data used for distillation #905
   cost & perf +1

Name	To "en"	Release	From "en"	Release
Russian	-2.67	Released	-5.23	Nightly
Hungarian	-2.55	Released	-4.50	Nightly
Slovak	-2.77	Released	-5.71	Nightly
Arabic
Hebrew
Serbian	-3.56	Released
Vietnamese	-2.91	Released
Latvian	-3.38	Released	-5.10	Nightly
Persian	-6.05	Nightly	-9.17	Nightly
Lithuanian	-3.22	Released	-5.82	Nightly
Ukrainian	-3.18	Released	-4.68	Nightly
Bosnian	-7.81	Nightly