diff --git a/DESCRIPTION b/DESCRIPTION index 7c78c57..5c64b37 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,10 +1,10 @@ Package: fairness Title: Algorithmic Fairness Metrics -Version: 1.1.1 +Version: 1.2.0 Authors@R: c(person('Nikita', 'Kozodoi', email = 'n.kozodoi@icloud.com', role = c('aut', 'cre')), person('Tibor', 'V. Varga', email = 'tirgit@hotmail.com', role = c('aut'), comment = c(ORCID = '0000-0002-2383-699X'))) Maintainer: Nikita Kozodoi -Description: Offers various metrics of algorithmic fairness. Fairness in machine learning is an emerging topic with the overarching aim to critically assess algorithms (predictive and classification models) whether their results reinforce existing social biases. While unfair algorithms can propagate such biases and offer prediction or classification results with a disparate impact on various sensitive subgroups of populations (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities), fair algorithms possess the underlying foundation that these groups should be treated similarly / should have similar outcomes. The fairness R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. These methods are described by Calders and Verwer (2010) , Chouldechova (2017) , Feldman et al. (2015) , Friedler et al. (2018) and Zafar et al. (2017) . The package also offers convenient visualizations to help understand fairness metrics. +Description: Offers calculation, visualization and comparison of algorithmic fairness metrics. Fair machine learning is an emerging topic with the overarching aim to critically assess whether ML algorithms reinforce existing social biases. Unfair algorithms can propagate such biases and produce predictions with a disparate impact on various sensitive groups of individuals (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities). Fair algorithms possess the underlying foundation that these groups should be treated similarly or have similar prediction outcomes. The fairness R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. These methods are described by Calders and Verwer (2010) , Chouldechova (2017) , Feldman et al. (2015) , Friedler et al. (2018) and Zafar et al. (2017) . The package also offers convenient visualizations to help understand fairness metrics. License: MIT + file LICENSE Language: en-US Encoding: UTF-8 diff --git a/NEWS.md b/NEWS.md index 5685d3c..b657527 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,8 @@ +# fairness 1.2.0 +- added support for continuous group feature with binning options +- added group size to metric outputs +- small updates in documentation + # fairness 1.1.1 - added check and conversion of data to `data.frame` in metric functions - quiet AUC computation in `roc_parity()` diff --git a/R/fairness-package.R b/R/fairness-package.R index 94fbff9..c83dd3b 100644 --- a/R/fairness-package.R +++ b/R/fairness-package.R @@ -1,6 +1,6 @@ #' fairness: Algorithmic Fairness Metrics #' -#' The \strong{fairness} package offers various metrics of algorithmic fairness. Fairness in machine learning is an emerging topic with the overarching aim to critically assess algorithms (predictive and classification models) whether their results reinforce existing social biases. While unfair algorithms can propagate such biases and offer prediction or classification results with a disparate impact on various sensitive subgroups of populations (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities), fair algorithms possess the underlying foundation that these groups should be treated similarly / should have similar outcomes. The fairness R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. The package also offers convenient visualizations to help understand fairness metrics. +#' The \strong{fairness} package offers calculation, visualization and comparison of algorithmic fairness metrics. Fair machine learning is an emerging topic with the overarching aim to critically assess whether ML algorithms reinforce existing social biases. Unfair algorithms can propagate such biases and produce predictions with a disparate impact on various sensitive groups of individuals (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities). Fair algorithms possess the underlying foundation that these groups should be treated similarly or have similar prediction outcomes. The \strong{fairness} R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. The package also offers convenient visualizations to help understand fairness metrics. #' #' @details #' \tabular{ll}{ @@ -8,7 +8,7 @@ #' Depends: \tab R (>= 3.5.0)\cr #' Type: \tab Package\cr #' Version: \tab 1.1.1\cr -#' Date: \tab 2020-07-24\cr +#' Date: \tab 2020-11-18\cr #' License: \tab MIT\cr #' LazyLoad: \tab Yes #' } diff --git a/README.md b/README.md index 6bee5a9..5f56564 100644 --- a/README.md +++ b/README.md @@ -18,16 +18,16 @@ ## Package overview -The fairness R package provides tools to easily calculate algorithmic fairness metrics across different sensitive groups based on model predictions in a binary classification task. It also provides opportunities to visualize and compare other prediction metrics between the groups. +The fairness R package provides tools to calculate algorithmic fairness metrics across different sensitive groups based on model predictions in a binary classification task. It also provides opportunities to visualize and compare other prediction metrics between the sensitive groups. -The package contains functions to compute the most commonly used metrics of algorithmic fairness such as: +The package contains functions to compute the commonly used fair machine learning metrics such as: - Demographic parity - Proportional parity - Equalized odds - Predictive rate parity -In addition, the following comparisons are also implemented: +In addition, the following metrics are implemented: - False positive rate parity - False negative rate parity @@ -37,7 +37,7 @@ In addition, the following comparisons are also implemented: - ROC AUC comparison - MCC comparison -The comprehensive tutorial on using the package is provided in [this blogpost](https://kozodoi.me/r/fairness/packages/2020/05/01/fairness-tutorial.html). We recommend that you go through the blogpost, as it contains a more in-depth description of the fairness package compared to this README. You will also find a brief tutorial in the fairness [vignette](https://github.com/kozodoi/fairness/blob/master/vignettes/fairness.Rmd): +The comprehensive tutorial on using the package is provided in [this blogpost](https://kozodoi.me/r/fairness/packages/2020/05/01/fairness-tutorial.html). We recommend that you go through the post, as it contains a more in-depth description of the fairness package compared to this README. You will also find a brief tutorial in the fairness [vignette](https://github.com/kozodoi/fairness/blob/master/vignettes/fairness.Rmd): ```r vignette('fairness') diff --git a/man/acc_parity.Rd b/man/acc_parity.Rd index fda6489..bf1e73b 100644 --- a/man/acc_parity.Rd +++ b/man/acc_parity.Rd @@ -13,7 +13,8 @@ acc_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ acc_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw accuracy metrics for all groups and metrics standardized for the base group (accuracy parity metric). Lower values compared to the reference group mean lower accuracies in the selected subgroups} diff --git a/man/dem_parity.Rd b/man/dem_parity.Rd index cc22dc3..20fe454 100644 --- a/man/dem_parity.Rd +++ b/man/dem_parity.Rd @@ -13,7 +13,8 @@ dem_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ dem_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Absolute number of positive classifications for all groups and metrics standardized for the base group (demographic parity metric). Lower values compared to the reference group mean lower number of positively predicted observations in the selected subgroups} diff --git a/man/equal_odds.Rd b/man/equal_odds.Rd index 0ab464e..80e796b 100644 --- a/man/equal_odds.Rd +++ b/man/equal_odds.Rd @@ -13,7 +13,8 @@ equal_odds( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ equal_odds( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw sensitivities for all groups and metrics standardized for the base group (equalized odds parity metric). Lower values compared to the reference group mean lower sensitivities in the selected subgroups} diff --git a/man/fairness.Rd b/man/fairness.Rd index 8ef2e5a..c73dad8 100644 --- a/man/fairness.Rd +++ b/man/fairness.Rd @@ -5,7 +5,7 @@ \alias{fairness} \title{fairness: Algorithmic Fairness Metrics} \description{ -The \strong{fairness} package offers various metrics of algorithmic fairness. Fairness in machine learning is an emerging topic with the overarching aim to critically assess algorithms (predictive and classification models) whether their results reinforce existing social biases. While unfair algorithms can propagate such biases and offer prediction or classification results with a disparate impact on various sensitive subgroups of populations (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities), fair algorithms possess the underlying foundation that these groups should be treated similarly / should have similar outcomes. The fairness R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. The package also offers convenient visualizations to help understand fairness metrics. +The \strong{fairness} package offers calculation, visualization and comparison of algorithmic fairness metrics. Fair machine learning is an emerging topic with the overarching aim to critically assess whether ML algorithms reinforce existing social biases. Unfair algorithms can propagate such biases and produce predictions with a disparate impact on various sensitive groups of individuals (defined by sex, gender, ethnicity, religion, income, socioeconomic status, physical or mental disabilities). Fair algorithms possess the underlying foundation that these groups should be treated similarly or have similar prediction outcomes. The \strong{fairness} R package offers the calculation and comparisons of commonly and less commonly used fairness metrics in population subgroups. The package also offers convenient visualizations to help understand fairness metrics. } \details{ \tabular{ll}{ @@ -13,7 +13,7 @@ Package: \tab fairness\cr Depends: \tab R (>= 3.5.0)\cr Type: \tab Package\cr Version: \tab 1.1.1\cr -Date: \tab 2020-07-24\cr +Date: \tab 2020-11-18\cr License: \tab MIT\cr LazyLoad: \tab Yes } diff --git a/man/fnr_parity.Rd b/man/fnr_parity.Rd index 2bbc072..2fe03d0 100644 --- a/man/fnr_parity.Rd +++ b/man/fnr_parity.Rd @@ -13,7 +13,8 @@ fnr_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ fnr_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw false negative rates for all groups and metrics standardized for the base group (false negative rate parity metric). Lower values compared to the reference group mean lower false negative error rates in the selected subgroups} diff --git a/man/fpr_parity.Rd b/man/fpr_parity.Rd index 5f5cc1d..f5d3c5e 100644 --- a/man/fpr_parity.Rd +++ b/man/fpr_parity.Rd @@ -13,7 +13,8 @@ fpr_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ fpr_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw false positive rates for all groups and metrics standardized for the base group (false positive rate parity metric). Lower values compared to the reference group mean lower false positive error rates in the selected subgroups} diff --git a/man/mcc_parity.Rd b/man/mcc_parity.Rd index e895c58..25b2510 100644 --- a/man/mcc_parity.Rd +++ b/man/mcc_parity.Rd @@ -13,7 +13,8 @@ mcc_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ mcc_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw Matthews Correlation Coefficient metrics for all groups and metrics standardized for the base group (parity metric). Lower values compared to the reference group mean Matthews Correlation Coefficients in the selected subgroups} diff --git a/man/npv_parity.Rd b/man/npv_parity.Rd index 027a142..55f4fda 100644 --- a/man/npv_parity.Rd +++ b/man/npv_parity.Rd @@ -13,7 +13,8 @@ npv_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ npv_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw negative predictive values for all groups and metrics standardized for the base group (negative predictive value parity metric). Lower values compared to the reference group mean lower negative predictive values in the selected subgroups} diff --git a/man/pred_rate_parity.Rd b/man/pred_rate_parity.Rd index 5196a57..7d5a7a4 100644 --- a/man/pred_rate_parity.Rd +++ b/man/pred_rate_parity.Rd @@ -13,7 +13,8 @@ pred_rate_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ pred_rate_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw precision metrics for all groups and metrics standardized for the base group (predictive rate parity metric). Lower values compared to the reference group mean lower precisions in the selected subgroups} diff --git a/man/prop_parity.Rd b/man/prop_parity.Rd index 95a9ce2..3f603a3 100644 --- a/man/prop_parity.Rd +++ b/man/prop_parity.Rd @@ -13,7 +13,8 @@ prop_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ prop_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw proportions for all groups and metrics standardized for the base group (proportional parity metric). Lower values compared to the reference group mean lower proportion of positively predicted observations in the selected subgroups} diff --git a/man/roc_parity.Rd b/man/roc_parity.Rd index 1b7ed1b..0b91dbd 100644 --- a/man/roc_parity.Rd +++ b/man/roc_parity.Rd @@ -4,7 +4,15 @@ \alias{roc_parity} \title{ROC AUC parity} \usage{ -roc_parity(data, outcome, group, probs, preds_levels = NULL, base = NULL) +roc_parity( + data, + outcome, + group, + probs, + preds_levels = NULL, + base = NULL, + group_breaks = NULL +) } \arguments{ \item{data}{The dataframe that contains the necessary columns.} @@ -17,7 +25,9 @@ roc_parity(data, outcome, group, probs, preds_levels = NULL, base = NULL) \item{preds_levels}{The desired levels of the predicted binary outcome. If not defined, levels of the outcome variable are used.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw ROC AUC metrics for all groups and metrics standardized for the base group (parity metric). Lower values compared to the reference group mean lower ROC AUC values in the selected subgroups} diff --git a/man/spec_parity.Rd b/man/spec_parity.Rd index ec08401..743209a 100644 --- a/man/spec_parity.Rd +++ b/man/spec_parity.Rd @@ -13,7 +13,8 @@ spec_parity( preds_levels = NULL, outcome_base = NULL, cutoff = 0.5, - base = NULL + base = NULL, + group_breaks = NULL ) } \arguments{ @@ -33,7 +34,9 @@ spec_parity( \item{cutoff}{Cutoff to generate predicted outcomes from predicted probabilities. Default set to 0.5.} -\item{base}{Base level for sensitive group comparison} +\item{base}{Base level for sensitive group comparison.} + +\item{group_breaks}{If group is continuous (e.g., age): either a numeric vector of two or more unique cut points or a single number >= 2 giving the number of intervals into which group feature is to be cut.} } \value{ \item{Metric}{Raw specificity metrics for all groups and metrics standardized for the base group (specificity parity metric). Lower values compared to the reference group mean lower specificities in the selected subgroups}