Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distributed training setup #4

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from
Draft

Distributed training setup #4

wants to merge 27 commits into from

Conversation

MicPie
Copy link

@MicPie MicPie commented Jan 12, 2022

PR for the distributed training setup.

@MicPie
Adaptations CLIP model class for ddp training 1.
7833706
@MicPie
Draft ddp training 1.
88bb5d9
@MicPie
Draft ddp training 2 incl. dumming dataset and bug fixes.
f037153
@MicPie
Draft ddp training 3 incl. webdataset and fixes.
cf83155
@MicPie
Fix data timing measurement point.
211b549
@MicPie
Add checkdataloading argument to debug dataloading.
db3a65d
@MicPie
Added seeding, use of all latents and minor cleanups.
0ac974e
@MicPie
Added code for testing with all latents for debugging.
9ad4fa6
@MicPie
Added code for testing with all latents for debugging 2.
286f850
@MicPie
Added distributed_backends from https://github.com/lucidrains/DALLE-p…
4880436 
…ytorch (state commit: 5a255eab032bcd821c2038c808b9682e485b3f1a)
@MicPie
Backup all_gather backprop setups for transfer.
31f58ac
@MicPie
fix for merge
cf7327a
@MicPie
Merge branch 'lucidrains-main' into hvd
142d79d
@MicPie
Added pytorch ddp backend.
874ca70
@MicPie
Adapted pytorch ddp backend class name and added class to __init__.
c880877
@MicPie
update pytorch ddp setup for multi-gpu tests
dfb5c99
@MicPie
update pytorch ddp setup for multi-gpu tests 2
20c7797
@MicPie
update pytorch ddp setup for multi-gpu tests 2
c988f5f
@MicPie
update pytorch ddp setup for multi-gpu tests 3
50a994a
@MicPie
update pytorch ddp setup for multi-gpu tests 4
34558ea
@MicPie
update pytorch ddp setup for multi-gpu tests 3
117ba82
@MicPie
cleanup
7feff0f
@MicPie
setup for distributed abstraction training
e39cba5
@MicPie
Update x-clip to only use local samples for the loss and general clea…
6b086b8 
…nup.
@MicPie
big cleanup and grad cache test setup
095bd0e
@MicPie
distributed training setup 1
ec28d1c
@MicPie
Merge branch 'distributed_training_setup' of github.com:MicPie/x-clip…
5338ecb 
… into distributed_training_setup
@MicPie
Copy link
Author

MicPie commented Jan 12, 2022 •
edited
Loading

Packages I currently work on:

  • grad cache
  • pytorch AMP FP16 training
  • lr schedule

Other packages that will be needed:

  • review and check (web)dataset setup incl. text mask output and validation dataset
  • add accuracy logging
  • add ImageNet eval
  • 8bit adam/zero optimizer
  • test horovod training if needed
  • test deep speed training if needed
  • see small TO DOs in the code base

Other stuff:

  • add Hopfield network for CLOOB (InfoLOOB is there)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@MicPie