Skip to content

Commit

Permalink
Updated description of gumbel-softmax
Browse files Browse the repository at this point in the history
  • Loading branch information
Alexander März committed Aug 23, 2023
1 parent 3908ccc commit 496f646
Showing 1 changed file with 11 additions and 11 deletions.
22 changes: 11 additions & 11 deletions xgboostlss/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,20 +185,20 @@ def gumbel_softmax_fn(predt: torch.tensor,
Gumbel-softmax function used to ensure predt is adding to one.
The Gumbel-softmax distribution is a continuous distribution over the simplex, which can be thought of as a "soft"
version of a categorical distribution. It’s a way to draw samples from a categorical distribution in a
differentiable way. The motivation behind using the Gumbel-Softmax is to make the discrete sampling process of
categorical variables differentiable, which is useful in gradient-based optimization problems. To sample from a
Gumbel-Softmax distribution, one would use the Gumbel-max trick: add a Gumbel noise to logits and apply the softmax.
Formally, given a vector z, the Gumbel-softmax function s(z,tau)_i for a component i at temperature tau is
defined as:
version of a categorical distribution. It’s a way to draw samples from a categorical distribution in a
differentiable way. The motivation behind using the Gumbel-Softmax is to make the discrete sampling process of
categorical variables differentiable, which is useful in gradient-based optimization problems. To sample from a
Gumbel-Softmax distribution, one would use the Gumbel-max trick: add a Gumbel noise to logits and apply the softmax.
Formally, given a vector z, the Gumbel-softmax function s(z,tau)_i for a component i at temperature tau is
defined as:
s(z,tau)_i = frac{e^{(z_i + g_i) / tau}}{sum_{j=1}^M e^{(z_j + g_j) / tau}}
s(z,tau)_i = frac{e^{(z_i + g_i) / tau}}{sum_{j=1}^M e^{(z_j + g_j) / tau}}
where g_i is a sample from the Gumbel(0, 1) distribution. The parameter tau (temperature) controls the sharpness
of the output distribution. As tau approaches 0, the mixing probabilities become more discrete, and as tau
approaches infty, the mixing probabilities become more uniform. For more information we refer to
where g_i is a sample from the Gumbel(0, 1) distribution. The parameter tau (temperature) controls the sharpness
of the output distribution. As tau approaches 0, the mixing probabilities become more discrete, and as tau
approaches infty, the mixing probabilities become more uniform. For more information we refer to
Jang, E., Gu, Shixiang and Poole, B. "Categorical Reparameterization with Gumbel-Softmax", ICLR, 2017.
Jang, E., Gu, Shixiang and Poole, B. "Categorical Reparameterization with Gumbel-Softmax", ICLR, 2017.
Arguments
---------
Expand Down

0 comments on commit 496f646

Please sign in to comment.