Add DenseClus Implementation notebook for jumpstart

[srushtii-aws]

Summary

Add DenseClusImplentation notebook for jumpstart, it includes:

• Kmeans implementation on numerical features
• umap embeddings + hdbscan on numerical and categorical features separately
• DenseClus implementation on all features together using 5 umap combine methods
• Analysis on the results
• Conclusion

[bharven]

few notes on the analysis, mostly revolving around pulling in some stats to understand what the points in the clusters actually look like. code quality looks good!

[bharven]

can we try and get a sense of whether the clusters are valuable using descriptive stats? I think it would be cool to understand number/characteristics of numerical vs categorical clusters, then see characteristics of of denseclus clusters that use information from both.

simplified dummy example:
have numerical clusters with age 25-30 working 40 hours/week, ages 16-20 working 20 hours/week. also have categorical clusters with phd degree and academia, HS education and Entrepreneurship.

I think it would be very interesting to identify and compare these groupings to groupings that we find across both feature sets (ie people 20-30 with BS working 60 hrs/week in private sector). could illustrate that we miss information when we look at the two different sets of features in a silo

[srushtii-aws]

Good idea. Added descriptive stats for numerical vs categorical clusters and also for denseclus clusters that use information from both. Also added some observations there.

[bharven]

adding descriptive stats here may help defend why intersection_union_mapper yielded best results - can talk through the clusters and their intuitive meanings

[bharven]

is this the same visual as above? if so, this part could alternatively be the deep dive into the types of datapoints in each cluster

[srushtii-aws]

Added a section for 'Profiling the Clusters' to deep dive into clusters by DenseClus

[srushtii-aws]

Resolved most of the comments. Added descriptive stats for analysis on clusters formed and observations.

[bharven]

few more analysis comments!

[bharven]

FYI can't leave comments on specific lines because diff is too big (and github vs code extension doesn't support commenting on notebooks microsoft/vscode-pull-request-github#3462). Will try and be descriptive in comments.

[bharven]

add a comment explaining why we only take native_country = " United-States"

[bharven]

Add a markdown description about what/why we are doing in "Create UMAP embeddings & Fit HdbScan for Numerical and Categorical features separately" section (ie "a seemingly straightforward approach may be to try clustering numerical and categorical features separately. lets use this as a baseline to compare against.. ")

also include brief overview of what UMAP and HDBSCAN are - can probably pull this from other notebooks

[bharven]

thoughts about baseline separate numerical/categorical cluster analysis:

• cluster results don't look super meaningful, could this be improved with hyperparameter optimization? this may be too much for this notebook though, especially considering this is just supposed to be a baseline and we get reasonable denseclus results. I'm open either way here, any thoughts?
• in the select_dtypes line why are we dropping segment then adding it back in the next line?
• can we expand the analysis to look at more than just mean? I think other descriptive stats might help with the story telling (but understand cluster quality is not good so there isnt much of a story to tell)
• can we see the columns used for categorical clustering?
• categorical analysis points 2 and 3 seem to be conflicting: we are saying both that there is a small finite space where we can have points + we have a large sparse space

[bharven]

can we update all of the plots to have appropriate x/y axis labels (or remove the labels) instead of None

[bharven]

@momonga-ml ready for your review