From 60a7b5bb2456346d773bc45d88c5fcba68119ac7 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Fri, 11 Oct 2024 05:49:39 +0000 Subject: [PATCH] build based on 8939ff8 --- dev/.documenter-siteinfo.json | 2 +- dev/api/index.html | 8 ++++---- dev/constructors/bernoulli/index.html | 2 +- dev/constructors/binomial/index.html | 2 +- dev/constructors/continuous_bernoulli/index.html | 2 +- dev/constructors/gamma/index.html | 2 +- dev/constructors/gaussian/index.html | 2 +- dev/constructors/negative_binomial/index.html | 2 +- dev/constructors/pareto/index.html | 2 +- dev/constructors/poisson/index.html | 2 +- dev/constructors/weibull/index.html | 2 +- dev/index.html | 2 +- dev/math/appendix/expectation/index.html | 2 +- dev/math/appendix/gamma/index.html | 2 +- dev/math/appendix/gaussian/index.html | 2 +- dev/math/appendix/inverses/index.html | 2 +- dev/math/appendix/poisson/index.html | 2 +- dev/math/bregman/index.html | 2 +- dev/math/intro/index.html | 2 +- dev/math/objectives/index.html | 2 +- dev/math/references/index.html | 2 +- 21 files changed, 24 insertions(+), 24 deletions(-) diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 85041e9..c83ea28 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.11.0","generation_timestamp":"2024-10-11T05:33:36","documenter_version":"1.7.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.11.0","generation_timestamp":"2024-10-11T05:49:33","documenter_version":"1.7.0"}} \ No newline at end of file diff --git a/dev/api/index.html b/dev/api/index.html index 4116c9b..c5ed660 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,5 +1,5 @@ -API Documentation · ExpFamilyPCA
GitHub

API Documentation

Contents

Index

Functions

The core of the ExpFamilyPCA.jl API is the EPCA abstract type. All supported and custom EPCA specifications are subtypes of EPCA and include three methods in their EPCA interface: fit!, compress and decompress.

ExpFamilyPCA.fit! — Function
fit!(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real

Fits the EPCA model on the dataset X.

Arguments

  • epca::EPCA: The EPCA model.
  • X::AbstractMatrix{T}: (n, indim) - The input training data matrix. Rows are observations. Columns are features or variables.

Keyword Arguments

  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.

Returns

  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.

Usage

Input:

using ExpFamilyPCA
+API Documentation · ExpFamilyPCA
GitHub
API Documentation
Contents
Index
Functions
The core of the ExpFamilyPCA.jl API is the EPCA abstract type. All supported and custom EPCA specifications are subtypes of EPCA and include three methods in their EPCA interface: fit!, compress and decompress.
ExpFamilyPCA.fit! — Function
fit!(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Fits the EPCA model on the dataset X.
Arguments
  • epca::EPCA: The EPCA model. 
  • X::AbstractMatrix{T}: (n, indim) - The input training data matrix. Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
Input:
using ExpFamilyPCA
 using Random; Random.seed!(1)
 
 # Create the model
@@ -16,11 +16,11 @@
 Iteration: 50/200 | Loss: 11.07389383509631
 Iteration: 100/200 | Loss: 10.971490262772905
 Iteration: 150/200 | Loss: 10.886018474442618
-Iteration: 200/200 | Loss: 10.718703556787007
source
ExpFamilyPCA.compress — Function
compress(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Compresses the input data X with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • X::AbstractMatrix{T}: (n, indim) - The input data matrix (can differ from the training data). Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
# Generate some random test data
+Iteration: 200/200 | Loss: 10.718703556787007
source
ExpFamilyPCA.compress — Function
compress(epca::EPCA, X::AbstractMatrix{T}; maxiter::Integer = 100, verbose::Bool = false, steps_per_print::Integer = 10) where T <: Real
Compresses the input data X with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • X::AbstractMatrix{T}: (n, indim) - The input data matrix (can differ from the training data). Rows are observations. Columns are features or variables. 
Keyword Arguments
  • maxiter::Integer = 100: The maximum number of iterations performed during loss minimization. Defaults to 100. May converge early.
  • verbose::Bool = false: A flag indicating whether to print optimization progress. If set to true, prints the loss value and iteration number at specified intervals (steps_per_print). Defaults to false.
  • steps_per_print::Integer = 10: The number of iterations between printed progress updates when verbose is set to true. For example, if steps_per_print is 10, progress will be printed every 10 iterations. Defaults to 10.
Returns
  • A::AbstractMatrix{T}: (n, outdim) - The compressed data.
Usage
# Generate some random test data
 m = 10
 Y = rand(0:1, m, indim)
 
 # Compress the test data using the fitted model from the previous example
-Y_compressed = compress(epca, Y)
source
ExpFamilyPCA.decompress — Function
decompress(epca::EPCA, A::AbstractMatrix{T}) where T <: Real
Decompress the compressed matrix A with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • A::AbstractMatrix{T}: (n, outdim) - A compressed data matrix.
Returns
  • X̂::AbstractMatrix{T}: (n, indim) - The reconstructed data matrix approximated using EPCA model parameters.
Usage
Y_reconstructed = decompress(epca, Y)
source
Options
ExpFamilyPCA.Options — Type
Options(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), A_init_value::Real = 1.0, A_lower::Union{Real, Nothing} = nothing, A_upper::Union{Real, Nothing} = nothing, A_use_sobol::Bool = false, V_init_value::Real = 1.0, V_lower::Union{Real, Nothing} = nothing, V_upper::Union{Real, Nothing} = nothing, V_use_sobol::Bool = false, low = -1e10, high = 1e10, tol = 1e-10, maxiter = 1e6)
Defines a struct Options for configuring various parameters used in optimization and calculus. It provides flexible defaults for metaprogramming, initialization values, optimization boundaries, and binary search controls.
Fields
  • metaprogramming::Bool: Enables metaprogramming for symbolic calculus conversions. Default is true.
  • μ::Real: A regularization hyperparameter. Default is 1.
  • ϵ::Real: A regularization hyperparameter. Default is eps().
  • A_init_value::Real: Initial value for parameter A. Default is 1.0.
  • A_lower::Union{Real, Nothing}: Lower bound for A, or nothing. Default is nothing.
  • A_upper::Union{Real, Nothing}: Upper bound for A, or nothing. Default is nothing.
  • A_use_sobol::Bool: Use Sobol sequences for initializing A. Default is false.
  • V_init_value::Real: Initial value for parameter V. Default is 1.0.
  • V_lower::Union{Real, Nothing}: Lower bound for V, or nothing. Default is nothing.
  • V_upper::Union{Real, Nothing}: Upper bound for V, or nothing. Default is nothing.
  • V_use_sobol::Bool: Use Sobol sequences for initializing V. Default is false.
  • low::Real: Lower bound for binary search. Default is -1e10.
  • high::Real: Upper bound for binary search. Default is 1e10.
  • tol::Real: Tolerance for stopping binary search. Default is 1e-10.
  • maxiter::Real: Maximum iterations for binary search. Default is 1e6.
source
ExpFamilyPCA.NegativeDomain — Function
NegativeDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the negative domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = -1: Initializes A with a negative value.
  • A_upper = -1e-4: Upper bound for A is constrained to a small negative value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
ExpFamilyPCA.PositiveDomain — Function
PositiveDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the positive domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = 1: Initializes A with a positive value.
  • A_upper = 1e-4: Upper bound for A is constrained to a small positive value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
Miscellaneous
ExpFamilyPCA.decompress — Function
decompress(epca::EPCA, A::AbstractMatrix{T}) where T <: Real
Decompress the compressed matrix A with the EPCA model.
Arguments
  • epca::EPCA: The fitted EPCA model.[1] fit! should be called before compress.
  • A::AbstractMatrix{T}: (n, outdim) - A compressed data matrix.
Returns
  • X̂::AbstractMatrix{T}: (n, indim) - The reconstructed data matrix approximated using EPCA model parameters.
Usage
Y_reconstructed = decompress(epca, Y)
source
Options
ExpFamilyPCA.Options — Type
Options(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), A_init_value::Real = 1.0, A_lower::Union{Real, Nothing} = nothing, A_upper::Union{Real, Nothing} = nothing, A_use_sobol::Bool = false, V_init_value::Real = 1.0, V_lower::Union{Real, Nothing} = nothing, V_upper::Union{Real, Nothing} = nothing, V_use_sobol::Bool = false, low = -1e10, high = 1e10, tol = 1e-10, maxiter = 1e6)
Defines a struct Options for configuring various parameters used in optimization and calculus. It provides flexible defaults for metaprogramming, initialization values, optimization boundaries, and binary search controls.
Fields
  • metaprogramming::Bool: Enables metaprogramming for symbolic calculus conversions. Default is true.
  • μ::Real: A regularization hyperparameter. Default is 1.
  • ϵ::Real: A regularization hyperparameter. Default is eps().
  • A_init_value::Real: Initial value for parameter A. Default is 1.0.
  • A_lower::Union{Real, Nothing}: Lower bound for A, or nothing. Default is nothing.
  • A_upper::Union{Real, Nothing}: Upper bound for A, or nothing. Default is nothing.
  • A_use_sobol::Bool: Use Sobol sequences for initializing A. Default is false.
  • V_init_value::Real: Initial value for parameter V. Default is 1.0.
  • V_lower::Union{Real, Nothing}: Lower bound for V, or nothing. Default is nothing.
  • V_upper::Union{Real, Nothing}: Upper bound for V, or nothing. Default is nothing.
  • V_use_sobol::Bool: Use Sobol sequences for initializing V. Default is false.
  • low::Real: Lower bound for binary search. Default is -1e10.
  • high::Real: Upper bound for binary search. Default is 1e10.
  • tol::Real: Tolerance for stopping binary search. Default is 1e-10.
  • maxiter::Real: Maximum iterations for binary search. Default is 1e6.
source
ExpFamilyPCA.NegativeDomain — Function
NegativeDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the negative domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = -1: Initializes A with a negative value.
  • A_upper = -1e-4: Upper bound for A is constrained to a small negative value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
ExpFamilyPCA.PositiveDomain — Function
PositiveDomain(; metaprogramming::Bool = true, μ::Real = 1, ϵ::Real = eps(), low::Real = -1e10, high::Real = 1e10, tol::Real = 1e-10, maxiter::Real = 1e6)
Returns an instance of Options configured for optimization over the positive domain. Sets defaults for A and V parameters while keeping the remaining settings from Options.
Specific Settings
  • A_init_value = 1: Initializes A with a positive value.
  • A_upper = 1e-4: Upper bound for A is constrained to a small positive value.
  • V_init_value = 1: Initializes V with a positive value.
  • V_lower = 1e-4: Lower bound for V is constrained to a small positive value.
Other fields inherit from the Options struct.
source
Miscellaneous
  • 1If compress is called before fit!, X will compressed using unfitted starting weights.

+Compressor for `CompressedBeliefMDPs.jl`.
source
  • 1If compress is called before fit!, X will compressed using unfitted starting weights.
diff --git a/dev/constructors/bernoulli/index.html b/dev/constructors/bernoulli/index.html index f0af0c6..21d7080 100644 --- a/dev/constructors/bernoulli/index.html +++ b/dev/constructors/bernoulli/index.html @@ -1,2 +1,2 @@ -Bernoulli · ExpFamilyPCA
GitHub

Bernoulli EPCA

Math

NameBernoulliEPCA
$G(\theta)$$\log(1 + e^\theta)$
$g(\theta)$$\frac{e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, 1)$
$\Theta$ Spacereal
Appropriate Databinary

$G$ is the softplus function and $g$ is the logistic function.

Documentation

ExpFamilyPCA.BernoulliEPCA — Function
BernoulliEPCA(indim::Integer, outdim::Integer; options = Options(μ = 0.5))

Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Bernoulli · ExpFamilyPCA
GitHub

Bernoulli EPCA

Math

NameBernoulliEPCA
$G(\theta)$$\log(1 + e^\theta)$
$g(\theta)$$\frac{e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, 1)$
$\Theta$ Spacereal
Appropriate Databinary

$G$ is the softplus function and $g$ is the logistic function.

Documentation

ExpFamilyPCA.BernoulliEPCA — Function
BernoulliEPCA(indim::Integer, outdim::Integer; options = Options(μ = 0.5))

Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/binomial/index.html b/dev/constructors/binomial/index.html index 3394019..3198345 100644 --- a/dev/constructors/binomial/index.html +++ b/dev/constructors/binomial/index.html @@ -1,2 +1,2 @@ -Binomial · ExpFamilyPCA
GitHub

Binomial EPCA

Math

NameBinomialEPCA
$G(\theta)$$n \log(1 + e^\theta)$
$g(\theta)$$\frac{n e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, n)$
$\Theta$ Spacereal
Appropriate Datacount
$n$$n \geq 0$ (number of trials)

$G$ is the scaled softplus function and $g$ is the scaled logistic function.

Documentation

ExpFamilyPCA.BinomialEPCA — Function
BinomialEPCA(indim::Integer, outdim::Integer, n::Integer; options::Options = Options(μ = 0.5))

Binomial EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • n::Integer: A known parameter representing the number of trials (nonnegative).
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the binomial distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Binomial · ExpFamilyPCA
GitHub

Binomial EPCA

Math

NameBinomialEPCA
$G(\theta)$$n \log(1 + e^\theta)$
$g(\theta)$$\frac{n e^\theta}{1+e^\theta}$
$\mu$ Space[1]$(0, n)$
$\Theta$ Spacereal
Appropriate Datacount
$n$$n \geq 0$ (number of trials)

$G$ is the scaled softplus function and $g$ is the scaled logistic function.

Documentation

ExpFamilyPCA.BinomialEPCA — Function
BinomialEPCA(indim::Integer, outdim::Integer, n::Integer; options::Options = Options(μ = 0.5))

Binomial EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • n::Integer: A known parameter representing the number of trials (nonnegative).
  • options::Options: Optional parameters (default: μ = 0.5).

Returns

  • epca: An EPCA subtype for the binomial distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/continuous_bernoulli/index.html b/dev/constructors/continuous_bernoulli/index.html index 450d7cc..a03707f 100644 --- a/dev/constructors/continuous_bernoulli/index.html +++ b/dev/constructors/continuous_bernoulli/index.html @@ -1,2 +1,2 @@ -Continuous Bernoulli · ExpFamilyPCA
GitHub

Continuous Bernoulli EPCA

Math

NameContinuousBernoulliEPCA
$G(\theta)$$\log \frac{e^\theta - 1}{\theta}$
$g(\theta)$$\frac{\theta-1}{\theta} + \frac{1}{e^\theta - 1}$
$\mu$ Space[1]$(0, 1) \setminus \{\frac{1}{2}\}$
$\Theta$ Space$\mathbb{R} \setminus \{ 0 \}$
Appropriate Dataunit interval

Documentation

ExpFamilyPCA.ContinuousBernoulliEPCA — Function
ContinuousBernoulliEPCA(indim::Integer, outdim::Integer; options::Options = Options(μ = 0.5))

Continuous Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.25).

Returns

  • epca: An EPCA subtype for the continuous Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Continuous Bernoulli · ExpFamilyPCA
GitHub

Continuous Bernoulli EPCA

Math

NameContinuousBernoulliEPCA
$G(\theta)$$\log \frac{e^\theta - 1}{\theta}$
$g(\theta)$$\frac{\theta-1}{\theta} + \frac{1}{e^\theta - 1}$
$\mu$ Space[1]$(0, 1) \setminus \{\frac{1}{2}\}$
$\Theta$ Space$\mathbb{R} \setminus \{ 0 \}$
Appropriate Dataunit interval

Documentation

ExpFamilyPCA.ContinuousBernoulliEPCA — Function
ContinuousBernoulliEPCA(indim::Integer, outdim::Integer; options::Options = Options(μ = 0.5))

Continuous Bernoulli EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters (default: μ = 0.25).

Returns

  • epca: An EPCA subtype for the continuous Bernoulli distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/gamma/index.html b/dev/constructors/gamma/index.html index 68c7061..1a86b5e 100644 --- a/dev/constructors/gamma/index.html +++ b/dev/constructors/gamma/index.html @@ -1,2 +1,2 @@ -Gamma · ExpFamilyPCA
GitHub

Gamma EPCA

Math

NameGammaEPCA or ItakuraSaitoEPCA
$G(\theta)$$-\log(-\theta)$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datapositive

Documentation

ExpFamilyPCA.GammaEPCA — Function
GammaEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -2, A_upper = -eps(), V_lower = eps()))

Gamma EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -2).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Gamma · ExpFamilyPCA
GitHub

Gamma EPCA

Math

NameGammaEPCA or ItakuraSaitoEPCA
$G(\theta)$$-\log(-\theta)$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datapositive

Documentation

ExpFamilyPCA.GammaEPCA — Function
GammaEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -2, A_upper = -eps(), V_lower = eps()))

Gamma EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -2).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/gaussian/index.html b/dev/constructors/gaussian/index.html index ae8ec36..e37ec3d 100644 --- a/dev/constructors/gaussian/index.html +++ b/dev/constructors/gaussian/index.html @@ -1,2 +1,2 @@ -Gaussian · ExpFamilyPCA
GitHub

Gaussian EPCA

NameGaussianEPCA or NormalEPCA
$G(\theta)$$\theta^2 / 2$
$g(\theta)$$\theta$
$\mu$ Space[1]real
$\Theta$ Spacereal
Appropriate Datacontinuous

The Gaussian EPCA objective is equivalent to regular PCA.

Documentation

ExpFamilyPCA.NormalEPCA — Function
GaussianEPCA(indim::Integer, outdim::Integer; options::Options = Options())

An EPCA model with Gaussian loss.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Gaussian distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Gaussian · ExpFamilyPCA
GitHub

Gaussian EPCA

NameGaussianEPCA or NormalEPCA
$G(\theta)$$\theta^2 / 2$
$g(\theta)$$\theta$
$\mu$ Space[1]real
$\Theta$ Spacereal
Appropriate Datacontinuous

The Gaussian EPCA objective is equivalent to regular PCA.

Documentation

ExpFamilyPCA.NormalEPCA — Function
GaussianEPCA(indim::Integer, outdim::Integer; options::Options = Options())

An EPCA model with Gaussian loss.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Gaussian distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/negative_binomial/index.html b/dev/constructors/negative_binomial/index.html index 39edd79..6b60770 100644 --- a/dev/constructors/negative_binomial/index.html +++ b/dev/constructors/negative_binomial/index.html @@ -1,2 +1,2 @@ -Negative Binomial · ExpFamilyPCA
GitHub

Negative Binomial EPCA

NameNegativeBinomialEPCA
$G(\theta)$$-r \log(1 - e^\theta)$
$g(\theta)$$- \frac{r e^\theta}{1 - e^\theta}$
$\mu$ Space[1]positive
$\Theta$ Spacenegative
Appropriate Datacount
$r$$r > 0$ (number of successes)

Documentation

ExpFamilyPCA.NegativeBinomialEPCA — Function
NegativeBinomialEPCA(indim::Integer, outdim::Integer, r::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Negative binomial EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • r::Integer: A known parameter of the negative binomial distribution representing the number of successes (positive).
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -1).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Negative Binomial · ExpFamilyPCA
GitHub

Negative Binomial EPCA

NameNegativeBinomialEPCA
$G(\theta)$$-r \log(1 - e^\theta)$
$g(\theta)$$- \frac{r e^\theta}{1 - e^\theta}$
$\mu$ Space[1]positive
$\Theta$ Spacenegative
Appropriate Datacount
$r$$r > 0$ (number of successes)

Documentation

ExpFamilyPCA.NegativeBinomialEPCA — Function
NegativeBinomialEPCA(indim::Integer, outdim::Integer, r::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Negative binomial EPCA.

Arguments

  • indim::Integer: The dimension of the input space.
  • outdim::Integer: The dimension of the latent (output) space.
  • r::Integer: A known parameter of the negative binomial distribution representing the number of successes (positive).
  • options::Options: Optional configuration parameters for the EPCA model.
    • A_init_value: Initial fill value for matrix A (default: -1).
    • A_upper: The upper bound for the matrix A, default is -eps().
    • V_lower: The lower bound for the matrix V, default is eps().
Tip

Try using options = NegativeDomain() if you encounter domain errors when calling fit! or compress.

Returns

  • epca: An instance of an EPCA subtype.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/pareto/index.html b/dev/constructors/pareto/index.html index a687999..3932113 100644 --- a/dev/constructors/pareto/index.html +++ b/dev/constructors/pareto/index.html @@ -1,2 +1,2 @@ -Pareto · ExpFamilyPCA
GitHub

Pareto EPCA

Math

NameParetoEPCA
$G(\theta)$$-\log(-1 - \theta) + \theta \log m$
$g(\theta)$$\log m - \frac{1}{\theta + 1}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ \log{m} \}$
$\Theta$ Spacenegative
Appropriate Dataheavy-tail
$m$$m > 0$ (minimum value)

Documentation

ExpFamilyPCA.ParetoEPCA — Function
ParetoEPCA(indim::Integer, outdim::Integer, m::Real; options::Options = Options(μ = 2, A_init_value = 2, A_lower = 1 / indim, V_init_value = -2, V_upper = -1))

Pareto EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • m::Real: A known parameter of the Pareto distribution representing the minimum value in the support.
  • options::Options: Optional parameters for model initialization:
    • μ: Default value 2.
    • A_init_value: Initial value for matrix A (default: 2).
    • A_lower: Lower bound for matrix A (default: 1 / indim).
    • V_init_value: Initial value for matrix V (default: -2).
    • V_upper: Upper bound for matrix V (default: -1).

Returns

  • epca: An EPCA subtype for the Pareto distribution.
source
Tip

If your compression converges to a constant matrix, try processing your data to reduce the maximum (e.g., divide your data by a large constant, take the logarithm).

  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Pareto · ExpFamilyPCA
GitHub

Pareto EPCA

Math

NameParetoEPCA
$G(\theta)$$-\log(-1 - \theta) + \theta \log m$
$g(\theta)$$\log m - \frac{1}{\theta + 1}$
$\mu$ Space[1]$\mathbb{R} \setminus \{ \log{m} \}$
$\Theta$ Spacenegative
Appropriate Dataheavy-tail
$m$$m > 0$ (minimum value)

Documentation

ExpFamilyPCA.ParetoEPCA — Function
ParetoEPCA(indim::Integer, outdim::Integer, m::Real; options::Options = Options(μ = 2, A_init_value = 2, A_lower = 1 / indim, V_init_value = -2, V_upper = -1))

Pareto EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • m::Real: A known parameter of the Pareto distribution representing the minimum value in the support.
  • options::Options: Optional parameters for model initialization:
    • μ: Default value 2.
    • A_init_value: Initial value for matrix A (default: 2).
    • A_lower: Lower bound for matrix A (default: 1 / indim).
    • V_init_value: Initial value for matrix V (default: -2).
    • V_upper: Upper bound for matrix V (default: -1).

Returns

  • epca: An EPCA subtype for the Pareto distribution.
source
Tip

If your compression converges to a constant matrix, try processing your data to reduce the maximum (e.g., divide your data by a large constant, take the logarithm).

  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/poisson/index.html b/dev/constructors/poisson/index.html index 1417900..4ac56b9 100644 --- a/dev/constructors/poisson/index.html +++ b/dev/constructors/poisson/index.html @@ -1,2 +1,2 @@ -Poisson · ExpFamilyPCA
GitHub

Poisson EPCA

NamePoissonEPCA
$G(\theta)$$e^\theta$
$g(\theta)$$e^\theta$
$\mu$ Space[1]positive
$\Theta$ Spacereal
Appropriate Datacount, probability

Poisson EPCA minimizes the generalized KL divergence making it well-suited for compressing probability profiles. Poisson EPCA has also been used in reinforcement learning to solve partially observed Markov decision processes (POMDPs) with belief compression (Roy et al., 2005).

Documentation

ExpFamilyPCA.PoissonEPCA — Function
PoissonEPCA(indim::Integer, outdim::Integer; options::Options = Options())

Poisson EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Poisson distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Poisson · ExpFamilyPCA
GitHub

Poisson EPCA

NamePoissonEPCA
$G(\theta)$$e^\theta$
$g(\theta)$$e^\theta$
$\mu$ Space[1]positive
$\Theta$ Spacereal
Appropriate Datacount, probability

Poisson EPCA minimizes the generalized KL divergence making it well-suited for compressing probability profiles. Poisson EPCA has also been used in reinforcement learning to solve partially observed Markov decision processes (POMDPs) with belief compression (Roy et al., 2005).

Documentation

ExpFamilyPCA.PoissonEPCA — Function
PoissonEPCA(indim::Integer, outdim::Integer; options::Options = Options())

Poisson EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters.

Returns

  • epca: An EPCA subtype for the Poisson distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/constructors/weibull/index.html b/dev/constructors/weibull/index.html index 6dea28e..2edff8d 100644 --- a/dev/constructors/weibull/index.html +++ b/dev/constructors/weibull/index.html @@ -1,2 +1,2 @@ -Weibull · ExpFamilyPCA
GitHub

Weibull EPCA

NameWeibullEPCA
$G(\theta)$$-\log(-\theta) - \log k$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} / \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datanonnegative continuous

WeibullEPCA omits it the known shape parameter $k$ since it does not affect the Weibull EPCA objective.

Documentation

ExpFamilyPCA.WeibullEPCA — Function
WeibullEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Weibull EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters for model initialization. Default NegativeDomain().

Returns

  • epca: An EPCA subtype for the Weibull distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
+Weibull · ExpFamilyPCA
GitHub

Weibull EPCA

NameWeibullEPCA
$G(\theta)$$-\log(-\theta) - \log k$
$g(\theta)$$-\frac{1}{\theta}$
$\mu$ Space[1]$\mathbb{R} / \{ 0 \}$
$\Theta$ Spacenegative
Appropriate Datanonnegative continuous

WeibullEPCA omits it the known shape parameter $k$ since it does not affect the Weibull EPCA objective.

Documentation

ExpFamilyPCA.WeibullEPCA — Function
WeibullEPCA(indim::Integer, outdim::Integer; options::Options = Options(A_init_value = -1, A_upper = -eps(), V_lower = eps()))

Weibull EPCA.

Arguments

  • indim::Integer: Dimension of the input space.
  • outdim::Integer: Dimension of the latent (output) space.
  • options::Options: Optional parameters for model initialization. Default NegativeDomain().

Returns

  • epca: An EPCA subtype for the Weibull distribution.
source
  • 1$\mu$ space refers to the space of valid regularization parameters, not to the expectation parameter space.
diff --git a/dev/index.html b/dev/index.html index 3a084bc..91098a3 100644 --- a/dev/index.html +++ b/dev/index.html @@ -13,4 +13,4 @@ Y_compressed = compress(poisson_epca, Y; maxiter=200, verbose=true) X_reconstructed = decompress(poisson_epca, X_compressed) -Y_reconstructed = decompress(poisson_epca, Y_compressed)

Supported Distributions

DistributionExpFamilyPCA.jlObjectiveLink Function $g(\theta)$
BernoulliBernoulliEPCA$\log(1 + e^{\theta - 2x\theta})$$\frac{e^\theta}{1 + e^\theta}$
BinomialBinomialEPCA$n \log(1 + e^\theta) - x\theta$$\frac{ne^\theta}{1 + e^\theta}$
Continuous BernoulliContinuousBernoulliEPCA$\log\left(\frac{e^\theta - 1}{\theta}\right) - x\theta$$\frac{\theta - 1}{\theta} + \frac{1}{e^\theta - 1}$
Gamma¹GammaEPCA or ItakuraSaitoEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$
Gaussian²GaussianEPCA or NormalEPCA$\frac{1}{2}(x - \theta)^2$$\theta$
Negative BinomialNegativeBinomialEPCA$-r \log(1 - e^\theta) - x\theta$$\frac{-re^\theta}{e^\theta - 1}$
ParetoParetoEPCA$-\log(-1 - \theta) + \theta \log m - x\theta$$\log m - \frac{1}{\theta + 1}$
Poisson³PoissonEPCA$e^\theta - x\theta$$e^\theta$
WeibullWeibullEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$

¹: For the Gamma distribution, the link function is typically based on the inverse relationship.

²: For Gaussian, also known as Normal distribution, the link function is the identity.

³: The Poisson distribution link function is exponential.

+Y_reconstructed = decompress(poisson_epca, Y_compressed)

Supported Distributions

DistributionExpFamilyPCA.jlObjectiveLink Function $g(\theta)$
BernoulliBernoulliEPCA$\log(1 + e^{\theta - 2x\theta})$$\frac{e^\theta}{1 + e^\theta}$
BinomialBinomialEPCA$n \log(1 + e^\theta) - x\theta$$\frac{ne^\theta}{1 + e^\theta}$
Continuous BernoulliContinuousBernoulliEPCA$\log\left(\frac{e^\theta - 1}{\theta}\right) - x\theta$$\frac{\theta - 1}{\theta} + \frac{1}{e^\theta - 1}$
Gamma¹GammaEPCA or ItakuraSaitoEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$
Gaussian²GaussianEPCA or NormalEPCA$\frac{1}{2}(x - \theta)^2$$\theta$
Negative BinomialNegativeBinomialEPCA$-r \log(1 - e^\theta) - x\theta$$\frac{-re^\theta}{e^\theta - 1}$
ParetoParetoEPCA$-\log(-1 - \theta) + \theta \log m - x\theta$$\log m - \frac{1}{\theta + 1}$
Poisson³PoissonEPCA$e^\theta - x\theta$$e^\theta$
WeibullWeibullEPCA$-\log(-\theta) - x\theta$$-\frac{1}{\theta}$

¹: For the Gamma distribution, the link function is typically based on the inverse relationship.

²: For Gaussian, also known as Normal distribution, the link function is the identity.

³: The Poisson distribution link function is exponential.

diff --git a/dev/math/appendix/expectation/index.html b/dev/math/appendix/expectation/index.html index 8cc06c0..76fef55 100644 --- a/dev/math/appendix/expectation/index.html +++ b/dev/math/appendix/expectation/index.html @@ -7,4 +7,4 @@ &= \int x \exp(x \theta - G(\theta)) h(x) \, dx \\ &= \int x p_\theta(x) \, dx \\ &= \mathbb{E}_\theta[X]. -\end{aligned}\]

+\end{aligned}\]

diff --git a/dev/math/appendix/gamma/index.html b/dev/math/appendix/gamma/index.html index 188a1fc..2eb26af 100644 --- a/dev/math/appendix/gamma/index.html +++ b/dev/math/appendix/gamma/index.html @@ -11,4 +11,4 @@ &= \frac{p}{q} - \log \frac{p}{q} - 1, \end{aligned}\]

so $B_F$ is the Itakura-Saito (Itakura and Saito, 1968) distance as desired. Further, the EPCA objective is

\[\begin{aligned} B_F(x \| g(\theta)) = \frac{p}{g(\theta)} - \log \frac{p}{g(\theta)} - 1 = -p\theta - \log(-p\theta) - 1. -\end{aligned}\]

+\end{aligned}\]

diff --git a/dev/math/appendix/gaussian/index.html b/dev/math/appendix/gaussian/index.html index bc5f0e7..ed96fdd 100644 --- a/dev/math/appendix/gaussian/index.html +++ b/dev/math/appendix/gaussian/index.html @@ -6,4 +6,4 @@ &= \frac{1}{2} \big[ \langle A, A \rangle - 2\langle B, A \rangle + \langle B, B \rangle \big] \\ &= \frac{1}{2} \langle A - B, A - B \rangle \\ &= \frac{1}{2} \| A - B \|_F^2. -\end{aligned}\]

Similarly, the Bregman divergence induced from the log-partition of the Gaussian $G(\theta) = \theta^2/2$ is the squared Euclidean distance.

+\end{aligned}\]

Similarly, the Bregman divergence induced from the log-partition of the Gaussian $G(\theta) = \theta^2/2$ is the squared Euclidean distance.

diff --git a/dev/math/appendix/inverses/index.html b/dev/math/appendix/inverses/index.html index 0dce38b..eceb2c7 100644 --- a/dev/math/appendix/inverses/index.html +++ b/dev/math/appendix/inverses/index.html @@ -6,4 +6,4 @@ &= \nabla_\mu \Big[ \mu \theta - G(\theta)\Big] \\ &= \theta + \mu \nabla_\mu \theta - g(\theta) \nabla_\mu \theta \\ &= \theta. \\ -\end{aligned}\]

The converse is similar, so $f = g^{-1}$.

+\end{aligned}\]

The converse is similar, so $f = g^{-1}$.

diff --git a/dev/math/appendix/poisson/index.html b/dev/math/appendix/poisson/index.html index 4f6f4f1..141b4c0 100644 --- a/dev/math/appendix/poisson/index.html +++ b/dev/math/appendix/poisson/index.html @@ -9,4 +9,4 @@ &= F(p) - F(q) - \langle f(q), p - q \rangle \\ &= p \log p - p - q \log q + q - \langle \log q, p - q \rangle \\ &= p \log \frac{p}{q} - p + q. -\end{aligned}\]

so $B_F$ is generalized KL-divergence.

+\end{aligned}\]

so $B_F$ is generalized KL-divergence.

diff --git a/dev/math/bregman/index.html b/dev/math/bregman/index.html index eea05e6..1c9d7f9 100644 --- a/dev/math/bregman/index.html +++ b/dev/math/bregman/index.html @@ -2,4 +2,4 @@ Bregman Divergences · ExpFamilyPCA
GitHub

Bregman Divergences

Bregman divergences (Bregman, 1967) play a central role in the probabilistic framework of EPCA. They generalize the concept of distance between two points but do not necessarily satisfy the properties of a traditional metric (such as symmetry or the triangle inequality). Instead, Bregman divergences provide a flexible way to measure differences between data points, making them useful in applications like clustering, optimization, and information theory.

Understanding Bregman divergences is essential for EPCA because they link the exponential family of probability distributions to loss functions used in optimization, allowing us to generalize PCA to non-Gaussian data.

Definition

Formally, the Bregman divergence (Bregman, 1967) $B_F$ associated with a function $F(\theta)$ is defined as

\[B_F(p \| q) = F(p) - F(q) - \langle f(p), p - q \rangle\]

where

  • $F(\mu)$ is a strictly convex and continuously differentiable function,
  • $f(\mu) = \nabla_\mu F(\mu)$ is the convex conjugate (defined later) of $F$,
  • and $\langle \cdot, \cdot \rangle$ denotes an inner product.

Intuitively, the Bregman divergence expresses the difference at $p$ between $F$ and its first-order Taylor expansion about $q$.

Properties

Unlike traditional metrics, Bregman divergences are not generally symmetric (i.e., $B_F(p \| q) \neq B_F(q \| p)$) and do not usually satsify the triangle inequality. However, they are always non-negative ($B_F(p \| q) \geq 0$) and equal $0$ if and only if $p = q$.

Info

While the full EPCA objective is always non-negative, the EPCA loss may be negative because ExpFamilyPCA.jl uses transformed objectives that are equivalent but not equal to minimizing a sum of Bregman divergences.

The Exponential Family

The natural exponential family is a broad class of probability distributions that includes many common distributions such as the Gaussian, binomial, Poisson, and gamma distributions. A probability distribution belongs to the exponential family if its probability density function (or mass function for discrete variables) can be expressed in the following canonical form:

\[p_\theta(x) = h(x) \exp(\langle \theta, x \rangle - G(\theta) )\]

where

  • $\theta$ is the natural parameter (also called the canonical parameter) of the distribution,
  • $h(x)$ is the base measure, independent of $\theta$,
  • and $G(\theta)$ is the log-partition function (also called the cumulant function) defined as:

\[G(\theta) = \log \int h(x) \exp(\langle \theta, x \rangle) \, dx.\]

The log-partition function $G(\theta)$ ensures that the probability distribution integrates (or sums) to $1$.

Key Parameters

  1. Natural Parameter ($\theta$): This parameter controls the distribution’s shape in its canonical form. For example, the natural parameter for the Poisson distribution is $\log \lambda$.
  2. Expectation Parameter ($\mu$): This is the expected value of the sufficient statistic,[1] computed as the mean of the data under the distribution. In exponential family distributions, it is related to the natural parameter through the gradient of the log-partition function:

\[\mu = \mathbb{E}_{\theta}[X] = \nabla_\theta G(\theta) = g(\theta)\]

where $E_\theta$ is the expectation with respect to the distribution $p_\theta$. A derivation is provided in the appendix. Similarly, we also have $\theta = f(\mu)$.

The Legendre Transform

To understand the relationship between the natural parameters $\theta$ and the expectation parameters $\mu$, we introduce the concept of convex conjugates and the Legendre transform. For a convex function $F$, its convex conjugate (or dual) $F^*$ is defined as:[2]

\[F^*(\theta) = \sup_{\mu} [\langle \theta, \mu \rangle - F(\mu)].\]

The convex conjugate is an involution ($F^{**} = F$) meaning the Legendre transform allows us to convert back and forth between the natural and expectation parameter spaces. In the next section, we see how ExpFamilyPCA.jl exploits the rich mathematical structure of the Legendre transform to discover multiple specifications of the Bregman divergence.

Bregman Loss Functions

An important relationship connects exponential family distributions and Bregman divergences: minimizing the negative log-likelihood of an exponential family distribution is equivalent (up to a constant) to minimizing a Bregman divergence between the observed data and the distribution's expectation parameter (Azoury and Warmuth, 2001; Forster and Warmuth, 2002). This connection is fundamental in extending PCA to EPCA.

Consider the negative log-likelihood of an exponential family distribution:

\[-\ell(x; \theta) = G(\theta) - \langle x, \theta \rangle.\]

Our goal is to show that this expression is equivalent (up to a constant) to the Bregman divergence $B_F(x \| \mu)$. First, recall that $G$ is the convex conjugate (dual) of $F$, so:

\[G(\theta) = \langle \theta, \mu \rangle - F(\mu).\]

Next, substitute $G$ back into the negative log-likelihood:

\[\begin{aligned} -\ell(x; \theta) &= (\langle \theta, \mu \rangle - F(\mu)) - \langle x, \theta \rangle \\ &= -F(\mu) - \langle \theta, x - \mu \rangle. -\end{aligned}\]

The last line is equivalent to $B_F(x \| \mu)$ up to a constant, meaning we can interpret the Bregman divergence as a loss function that generalizes the maximum likelihood of any exponential family distribution. In the next section, we expand this idea to derive specific constructors for the EPCA objective.

  • 1The sufficient statistic for the natural exponential family is simply the identity.
  • 2Duality also refers to the concept in convex analysis (Boyd and Vandenberghe, 2004).
+\end{aligned}\]

The last line is equivalent to $B_F(x \| \mu)$ up to a constant, meaning we can interpret the Bregman divergence as a loss function that generalizes the maximum likelihood of any exponential family distribution. In the next section, we expand this idea to derive specific constructors for the EPCA objective.

diff --git a/dev/math/intro/index.html b/dev/math/intro/index.html index 5f32c63..3c5f88d 100644 --- a/dev/math/intro/index.html +++ b/dev/math/intro/index.html @@ -9,4 +9,4 @@ & & B_F(X \| g(\Theta)) + \epsilon B_F(\mu_0 \| g(\Theta)) \\ & \text{subject to} & & \mathrm{rank}\left(\Theta\right) = k. -\end{aligned}\]

In this formulation:

On the next page, we dive deeper into Bregman divergences. We’ll explore their properties and how they connect to the exponential family, providing a solid foundation for understanding the probabilistic framework of EPCA.

+\end{aligned}\]

In this formulation:

On the next page, we dive deeper into Bregman divergences. We’ll explore their properties and how they connect to the exponential family, providing a solid foundation for understanding the probabilistic framework of EPCA.

diff --git a/dev/math/objectives/index.html b/dev/math/objectives/index.html index 530a6f7..e1c0196 100644 --- a/dev/math/objectives/index.html +++ b/dev/math/objectives/index.html @@ -19,4 +19,4 @@ g(θ) = exp(θ) poisson_epca = EPCA(indim, outdim, B, g, Val((:B, :g)))

7. Using $\tilde{B}$ and $g$

Similarly, using the transformed Bregman divergence $\tilde{B}(p, q) = B_F(p \| g(q))$ along with $g$ is straightforward. Given that $\tilde{B}$ is just $B_F$ evaluated at $g(q)$, and we already have the link function $g$, defining the EPCA objective is almost too obvious to mention. While mathematically plain, this specification is useful in practice to avoid domain errors that make optimization difficult with certain exponential family members (e.g., gamma, negative binomial, Pareto).

Example:

Bg(x, θ) = exp(θ) - x * θ + x * log(x) - x
 g(θ) = exp(θ)
-poisson_epca = EPCA(indim, outdim, Bg, g, Val((:Bg, :g)))

Conclusion

In summary, the EPCA objective function and the decompression function $g$ can be derived from various components. The flexibility afforded by Julia's multiple dispatch and symbolic differentiation capabilities allows for efficient computation of EPCA in different scenarios. Depending on the form of the data and the problem at hand, one can choose the most convenient set of components to define and compute the EPCA objective.

+poisson_epca = EPCA(indim, outdim, Bg, g, Val((:Bg, :g)))

Conclusion

In summary, the EPCA objective function and the decompression function $g$ can be derived from various components. The flexibility afforded by Julia's multiple dispatch and symbolic differentiation capabilities allows for efficient computation of EPCA in different scenarios. Depending on the form of the data and the problem at hand, one can choose the most convenient set of components to define and compute the EPCA objective.

diff --git a/dev/math/references/index.html b/dev/math/references/index.html index bce22db..a48b443 100644 --- a/dev/math/references/index.html +++ b/dev/math/references/index.html @@ -1,2 +1,2 @@ -References · ExpFamilyPCA
GitHub

References

+References · ExpFamilyPCA
GitHub

References