XGBoostLSS Feature Requests

[StatMixedML]

Dear Community,

In case you find the XGBoostLSS library interesting for your research / use-case, please use this forum for posting new feature requests you would like to see in the future.

I am currently working on a more efficient package structure. So stay tuned.

[kmedved]

Interesting! Can you provide more details?

Also, is there any chance of getting an sklearn compatible API?

[StatMixedML]

@kmedved Thanks for your comment!

In its current version, there are potential problems when it comes to serialization of the models. So I am planning to have a better class structure of the package. Also, I fully base the package on PyTorch functions for the derivatives that make it easier to generalize.

Currently, I am not planning to have a sklearn integration. What advantages do you see in that?

[kmedved]

The main advantage of sklearn is just integration with the rest of the python DS ecosystem. It's very convenient to have a model that you can use as a drop-in replacement for testing purposes, and to interact with the other parts of the scikit-learn ecosystem.

Either way, I appreciate all your work on this package, but just a suggestion to help with adoption. Thanks!

[StatMixedML]

I appreciate your suggestion, don't get me wrong. It is just that embedding it into the sklearn framework creates a lot of outside dependencies, which I want to avoid.

[HiramCillo]

I want to try the multi-target regression, because I want to try it on a task called " match_modality" for single-cell multiomics El vie, 25 nov 2022 a las 4:18, Alexander März ***@***.***>) escribió:

…

Dear Community, In case you find the XGBoostLSS library interesting for your research / use-case, please use this forum for posting new feature requests you would like to see in the future. I am currently working on a more efficient package structure. So stay tuned. — Reply to this email directly, view it on GitHub <#24>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH5MCOOX7N7PTSPAUV2WK73WKB77TANCNFSM6AAAAAASLDVEJA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[StatMixedML]

@HiramCillo Thanks for your comment!

Right now, the multivariate version can be slow to estimate for high-dimensional outputs. I am also currently working on adding a new boosting model to the repo that is faster.

Of what dimension are we talking in terms of number of targets for the multi-output regression?

[StatMixedML]

What extensions (additional distributions, runtime, etc.) of XGBoostLSS do you consider important to make it more competitive in comparison to other approaches?

[joseortiz3]

I'm wondering if the natural gradient approach (e.g. in NGBoost) can be incorporated into XGBoostLSS? It might improve convergence and stability, provided the hessians corresponding to the natural gradients can be analytically found. Seems like putting natural gradients into XGBoostLSS would certainly be faster/more optimized than NGBoost since XGBoost would be the underlying framework.

[StatMixedML]

Thanks for your interest in the project!

I am not exactly sure how NGBoost is working. As far as XGBoostLSS is concerned, it trains all parameters of distribution, using a Newton-Raphson type of algorithm, i.e., a second order Taylor approximation of the loss function

where for each distributional parameter $\theta_{m}$, gradients and hessians are calculated

Since gradients and hessians do not have to be analytically derived, we can leverage automatic differentiation to derive them numerically. This allows to also use complex loss functions, such as Normalizing Flows as shown in Distributional Gradient Boosting Machines or multivariate distributions as show in Multi-Target XGBoostLSS Regression.

You also mention that

[...] It might improve convergence and stability [...]

How would natural gradients (including hessians) be different from the XGBoostLSS gradients and hessians?

[joseortiz3]

How would natural gradients (including hessians) be different from the XGBoostLSS gradients and hessians?

Natural gradients differ from normal gradients in that the "best direction" for loss improvement is invariant to a user's choice of how they parameterize a given distribution. It uses a different notion of distance, one that is better suited for probability distributions (e.g. KL divergence instead of euclidian). From the NGBoost paper:

The paper's algorithm is a standard psuedo-residuals-fitting (gradient-fitting) booster, but it doesn't yet leverage hessians like XGBoost. They've worked out the natural gradients analytically for many different distributions explained here. I believe adopting the NGBoost natural gradients in XGBoostLSS parameter updates could be the best of both worlds.

[StatMixedML]

After skimming through the paper briefly, it says

From (8) we see that using MLE and the log-likelihood as the optimization metric, the natural gradient is given by the product of the inverse of the Fisher information matrix (which is a matrix that describes the curvature of the objective function with respect to the model parameters) and the usual gradient. Hence, scaling the usual gradient by the inverse fisher information matrix can provide a more accurate and efficient update direction for the model parameters in certain scenarios where the curvature of the objective function is significant, such as when the parameters are highly correlated, or the parameter space is highly curved.

Since XGBoostLSS currently optimizes on the log-likelihood only (while NGBoost also allows to optimize on scoring rules such as CRPS), one could try scaling the gradients in XGBoostLSS with the inverse Hessian (given the that the Hessian is invertible), since the negative Hessian evaluated at the MLE is the same as the observed Fisher information matrix evaluated at the MLE. However, due to efficiency reasons, XGBoostLSS currently only uses the diagonal of the Hessian matrix, and hence, disregards the cross-derivatives between distributional parameters. I am not sure if NGBoost makes use of the full Hessian matrix, also using the off-diagonal elements. Anyone knows this?

Also, it is important to note that inverting a Hessian might easily lead to numerical instabilities, especially for high-dimensional parameter settings, or when parameters are co-dependent. In extreme cases, this can cause the inverse even to be undefined, resulting in inaccurate or unreliable results. One way to mitigate this would be to add a small positive value to the diagonal elements of the Hessian to improve numerical stability during its inversion. Anyone knows how NGBoost handles such cases?