-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgraded folding and CP layers #129
Conversation
Benchmark results for this PR.
|
This is not merged yet because I plan to implement more fine-grained tests for each layer, rather than just testing a PC. |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some notes for myself. some for discussion.
inputs = F.pad(inputs, [0, 0, 0, 0, 0, 1], value=-float("inf")) | ||
# It should be the neutral element of a group. | ||
# For now computations are in log-space, thus 0 is our pad value. | ||
inputs = F.pad(inputs, [0, 0, 0, 0, 0, 1], value=0) # pylint: disable=not-callable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pad can be fused into cat?
New features
Notes
prod_exp
implementation to simplify the current code for now. Before merging I will run some benchmarks to better understand how much faster theprod_exp
implementation is. Do we already have some benchmarks on this? It seems to me we did not find a case where that tiny portion of parameters space is useful, especially if we allow for unnormalized parameters. Also, if we have CP layers with products with more than two inputs, it is always better to compute them in log-space for numerical stability.Additional features we are now able to implement easily
fold
andunfold
incirkit.models.functional
, thus making folding optional. Theunfold
method would enable us to cache intermediate computations, e.g., for compression or to do fast inverse transform sampling.Closes #113.
Closes #116.