[FEA] distributed autograd primitives (for tensors of variable size o…

…n each rank) (#105) * initial commit * add tests and fix bugs * format code * proper pytest markups * add a few docstrings * add test remark * update changelog * remove usage of all_to_all_single, address feedback * update docstrings to be more precise about bwd ops * format code * fix typo --------- Co-authored-by: Maximilian Stadler <mstadler.nvidia.com>
NVIDIA · Aug 22, 2023 · bdb10aa · bdb10aa
1 parent 012abfc
commit bdb10aa
Show file tree

Hide file tree

Showing 4 changed files with 1,111 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Added distributed utilities to create process groups and orthogonal process groups.
 - Added distributed AFNO model implementation.
+- Added distributed utilities for communication of buffers of varying size per rank.
 
 ### Changed