Release 0.5.0 · vanvalenlab/deepcell-label

This release primarily focuses on the new addition of cell type annotation to DeepCell Label. In addition to implementing the data structures and state machines required to track cell type data, we have introduced a large number of UI and QOL features to maximize the efficiency of cell type labeling, including channel expression calculations / plotting and in-browser training with uncertainty quantification for active learning labeling workflows. Another major addition is an overhaul of canvas rendering by changing the data structure used to store cell-value mapping, allowing for huge increases in performance in actions that involve canvas renders.

🚀 Features

Canvas rendering overhaul, greatly improved speed and overlap support… @ykevu (#502)

… for cell types

What

Canvas renderings have previously relied on a cellMatrix of size (numValues, numCells), which is both memory inefficient and led to very slow responsiveness due to the reliance on looping through the matrix even in the gpu.js kernel
We replace cellMatrix with a cell-value mapping { mappings, lengths }. Here is the format of the data structure:
- mappings: An array of arrays where the i th array contains all cells that map to value i. The array is then padded with zeroes such that each array is the same length
- lengths: An array where the i th element denotes how many cells value i maps to
Now, we have a very simple and fast algorithm to color cell label interiors for a given pixel:

value = labelArray[y][x]
numCells = numCellsArray[value]
for (let i = 0; i < numCells; i++) {
    cell = cellMapping[numCells[i]]
    color = colorMap[cell]
    appendColorToOverlap(color)
}

Anecdotally, on a Keren dataset this led to changing label opacity taking almost 3 seconds to less than 0.3 seconds (more than an order of magnitude speedup)
The algorithm for anything involving outlines is a bit more complicated, and looks quite ugly, but it should still be significantly faster even with the nested for loops, due to the fact that overlaps should basically always be very sparse.
Memory-wise, this is also more efficient, even with the padding (which is required for these algorithms to work with gpu.js, since jagged arrays result in incorrect array lookups since the GPU is probably just looking forward by a set amount of memory to a pointer or something)--in the worst, case, even with a certain pixel where every cell in the entire frame overlaps at that point, the mappings array is actually the exact same size as cellMatrix :D

Areas of improvement

It is likely that the ugly outline algorithms could be cleaned up a bit or made more efficient
cellSelectionCanvas.js may need some work; it still loops through the total number of selected cells, which can be quite a bit (as much as the number of cells in a canvas), and the max loop iteration is still 5000, so any selection with more than 5000 cells will have invisible selections

Increase visibility of multiselected cells for general use @ykevu (#481)

What

Addressing #480 and #464
We change the multiselect canvas rendering to increase its visibility--the width of the white outlines is increased to 2, and the interior is lit up with white 50% opacity pixels. Additionally, an animation can be enabled that alternates the interior on and off to create a light-up flashing effect (there is a warning on the toggle for people sensitive to flashing lights)
We leverage this new multiselect to add a button + keybinding to multiselect all cells with multiple labels, for reasoning explained in #464
Some additional UI and keybinding tweaks were made for QOL, such as moving tooltip positions and changing keybindings, most notably hover toggle from Shift to X
Importantly, this canvas can be extended naturally to account for, say, high uncertainty cells from the in-browser trainer or imported from a model prediction for future features/use

Track opened cell types for UI logic and new features @ykevu (#478)

What

The cell type panel is open is now tracked in the editCellTypes machine state context
The cell type that is open determines what celltype you are adding to when you are in add or remove mode now, rather than which button you pressed.
You can now cycle between the cells within that cell type with buttons / arrow keys if you have the cell info panel open
Adds keybindings for various cell type functions

Cell info @ykevu (#476)

What

Adds a "CELL" panel to the right-side UI of cell types that shows, for the current selected cell, the cell ID, the current cell types it is assigned to, the ability to add it to a new cell type and remove labels, and a bar chart showing the normalized channel expressions if the calculation has been made

Feature: drag celltype cards for reordering @ykevu (#469)

What

Enables the user to drag cell type cards to reorder them--this is purely a UI feature and does not affect the underlying cell types data at all
Introduces this with the react-beautiful-dnd library

Notes

The opacity slider is now wrapped in a bit of a gross div as a workaround for the drag behavior not recognizing MUI sliders and thus not allowing you to drag the slider

Export channels on submit/download @ykevu (#468)

What

Channel names were not exported when a project is exported, meaning that most importantly submitting a project makes the channel names disappear upon loading from the output bucket
This is now implemented

Add log y scale button for histogram and save layout changes @ykevu (#443)

The UI currently looks pretty ugly and the interfacing between scatter and histogram is still pretty gross (a TODO to potentially also wrap this up in custom plotly buttons could maybe solve this, alternatively hooking into state machine but that might be overkill) but it works. Plotly does indeed have log axes for histograms, I was just looking at the wrong thing.

Also, Plotly related UI stuff is pretty hard to test unfortunately.

Closes #442

Refactor idbWebWorker to separate indexedDB storage into multiple obj… @ykevu (#471)

…ect stores

What

We noticed that any time some sort of edit is made (including adding a single cell to a cell type), there would be a large amount of disk usage and waiting before the IndexedDB was done updating.
This was because we were using one object to store all persisted context in the IndexedDB, so even adding one element to the cellTypes json, for example, entailed rewriting the entire raw image, labeled mask, etc. to IndexedDB, which was very undesireable especially with larger images
By separating into separate object stores instead, we can update cellTypes.json, for example, while leaving the raw arrays untouched

Celltype UI toolbar with new toggles @ykevu (#461)

What

The cell type controls panel on the left now has a toolbar instead of the giant ADD CELLS button. It includes:

Add Cells button (same as before, but as an icon now)
Marker Panel open button (same as before, but as part of the toolbar now)
Cell type writing mode toggle (Defaults to "overwrite," so that adding Cell 1 which is labeled Type A to Type B removes it from Type A). This addresses #453.
Cell type hovering toggle (Defaults to off, but if on, you can hover over cells to view what cell type(s) it is in). This addresses #452.

There are also tooltips for each button, but a TODO is to add keybindings for them in some form

Add toggle to NavBar to switch between light and dark mode @ykevu (#450)

Adds an icon button to the navbar in the top right to toggle between light and dark mode themes. It turned out that two lines of default dark mode from MUI and adding the cssbaseline component made this essentially good enough out of the box.

Lower priority TODOs:

The plotly components are still blinding white in dark mode, but they would definitely be a pain to sync up
Cypress test

Marker panel visualization and editing modal @ykevu (#449)

Adds a popup component that allows the user to visualize the marker panel and make edits. There are still a number of features that would be useful to implement, but I wanted to get this in soon and move on to more important things, since, though it looks pretty, in practicality this feature shouldn't really be used much (hopefully, if our project creation scripts work as intended, not at all) by an annotator such as Martin, other than as a non-interactive reference.

Specifically, the following is implemented by these changes:

The user can click an icon that pops out the modal with a table visualizing the marker panel, with colors/lit up chips that denote whether the cell type or channel is matched with the current image
The user can double click cells to make edits to the cell types or channels in the marker panel
Channels now must be case-sensitive, exact matches to be detected (this can lead to inconveniences for sure, so am still thinking about how best to approach this, but the projects for Martin should have properly mapped channel names and such anyway)

The following is not implemented that would theoretically be useful in general cases:

Persistence of edits, whether through import/export or IndexedDB
Tracking of edits with UNDO/REDO (I'm not even sure this would be desirable)
Ability to add new entries to the marker panel (ie. cell type that we have not considered)
A visual list of the image's channels to reference when constructing a marker panel

Update marker panel with Elora's version @ykevu (#446)

Update the hard-coded marker panel with Elora's more accurate version that uses the master lists. We discussed potential features to improve the matching/suggestion with user editing, but that can be done in a separate PR.

Add button for deleting channels, remove original display controls @ykevu (#441)

It can be cumbersome to remove channels due to the remove button being accessible only through a popup menu, so I added a trash icon button so users can quickly delete channels they don't want to see.

Additionally, the original display controls on the left are now removed in favor of purely using the top display control panel.

Model training overhaul: SNGP model, uncertainty visualization, and custom embedding support @ykevu (#436)

What

Custom embeddings can be imported for visualization and training by including embeddings.json in an uploaded zip file
Various UI tweaks and fixes
Implement tabs for different visualizations in the training modal, including new training set histogram
Implement SNGP model using custom-written layers and functions adapted from Python Tensorflow
Uncertainty for predictions can be visualized and calculated for model inference

Reintroduce instructions including cell type documentation @ykevu (#440)

What

Moves the instructions into a modal that can be opened by clicking a help icon on the navbar
Added instructions for all the new features and notable limitations of cell type annotation and training
Reintroduced instructions testing into Cypress

Assume no overlaps when using cell types mode for drastically improve… @ykevu (#437)

…d speed

Assume that we are using data with no overlaps when in the cell types tab so that canvas responsiveness is much faster. This does mean that overlaps will cause potentially breaking visual issues if you want to do cell type labeling
Additionally, the labeledCanvas will be set transparent when doing cell type edits, since it doesn't make too much sense to use it when doing cell type labeling
Finally, note that the original cellTypeCanvas that supports overlaps is still in the code base, just not used until a better optimization solution can be reached
The cell type hover UI is also disabled for now, since it was more annoying than useful while testing due to a bug where it would persist

Implement buttons for matching cell types with marker panel @ykevu (#425)

What

Addresses #418

Implements buttons that fuzzy match cell type names to a marker panel and can open the relevant matched channels
UI changes and revamping of the cell type accordion
- Icon buttons to save space with tooltips
- Chips to show user the matched cell type names and channels that dynamically change color
- Various minor changes (eg. layout, some icons, etc.)

Allow calculations and training across all frames @ykevu (#424)

What

Addresses #413

Implemented functions to calculate and visualize mean/total embeddings across all frames
Implemented UI controls that allow users to specify whether they want to calculate/visualize/train/predict on the current frame or across all frames
Added documentation to trainingMachine functions since those functions are a bit of a mess
Added unit tests for new cross-frame calculation functions

Implement centroid calculations and channel expression unit testing @ykevu (#402)

What

Adds ability to calculate centroids of each cell in order to spatially visualize them on the plotly chart and graph them
Adds unit testing for both the new centroid calculations and for the original mean and total calculations in channelExpressionMachine.js

Implement cell types editing, plotting, and prototypes for training @ykevu (#400)

What

This merges in all of the work that has been done on implementing the new cell types features, including importing, editing, adding, removing cell types as well as channel marker calculations, plotting, and in-browser training and prediction.

Enable basic functionality for channel name editing and import + refine cell types controls (#393)

Cell type colors now mix properly (for cells with multiple cell types as well as overlapping cells)
Hovering over cells now show which cell type(s) that cell is labeled with
Implement remove cells mode
Added instructions for cell type labeling
Deleting cells deletes them from cell types now
Cell types are separately tracked by feature (segmentation)
Color map bug fixes
Channel names can now be edited
Display controls visually tweaked
Cell type opacity can be toggled and adjusted
Channel name can be imported from OME.tiff metadata

Implement channel expression calculations and plotting features + added linear opacity scaling (#394)

Calculate mean and total channel expressions for an image and plot the values
Clusters can be selected using plotly and added to specified cell types
Other UI changes: linear label opacity scaling, zooming in with the mouse scroll up instead of out
Cell grid removed.

Add UMAP visualization, remove and cancel buttons, and other UI tweaks/fixes (#395)

UMAP calculated for mean and total (some bugs associated)
Scatter points turn white when selected
Add/remove/cancel set of buttons added
Minor UI tweaks and bug fixes
Number of cells is now indicated in each cell type
Toggle all button can toggle all cell types
Scatter/histogram can be toggled with radio group
Editing prompt position fixed
Changed thickness of selected points on plot
Fixed resizing to perform resizing by channel
Fixed max iterations for cell type rendering

Optimize multi-frame canvas rendering, implement prototype for in-browser training (#396)

Added icons and made various small QOL and visual changes to the UI like plot titles
Training/prediction tab added to prototype transfer learning feature
Web worker memory bug "fixed" as the original raw array is no longer passed to the web worker, so larger files can be loaded now. This causes the issue of making it impossible to export the raw original though, so will be reverted
Relevant canvases now use a smaller cell matrix and use the actual list of cells on a given frame
Training loss plot
Training does not auto-predict
Additional UI changes: hover tooltip persisting on edges, plotly settings, etc.
Training tab moved to modal
New state machine specifically for training created
User can now specify percentage of training data to be used for training vs. validation (100% training not implemented)
Validation curve added
Confusion matrix calculated and displayed based on the validation data (not fully implemented for larger data)

Misc. testing and code clean-up

Fixed a toggle all bug
Fixed a number of linting and formatting issues
Added cypress test to test cell type controls
CI changes/fixes

Implement cell types machine, basic controls, and canvas rendering @ykevu (#392)

What

Adds editCellTypesMachine.js and cellTypesMachine.js and event bus to track and control state/context of cell types
Allow adding new cell types, editing their color and names, deleting them, and adding cells to the cell types
Export cell types in zip
Canvas rendering of cell types with colors

🐛 Bug Fixes

Remove cell type selection component when no cell types @ykevu (#514)

What

When there are no cell types the cell type selection component for adding or removing selected cells from a cell type on the right panel throws warnings; by checking if there are cell types, this warning will be suppressed

Cell type and channel name state management fixes @ykevu (#498)

What

Fixes a bug where undoing a cell type name change does not update the UI immediately (#448)
Fixes bug where changing channel names does not update UI immediately (#496) (see description of #473 )
Channel names are now updated in IDB (aka edited channel names will persist on reload)

Update instructions panel with bug fix and new cell type features @ykevu (#491)

What

Fixes and revamps the layout display section of the instructions, addressing #447
Updates the cell types relevant sections of the instructions with the new buttons and features (cell info, new tool bar and modes, etc.)

Change sizing for modal fit @ykevu (#487)

For some reason, (perhaps from Chrome updates?) some sizing issues cropped up on some of the components in the modals, so some size changes are made to make things fit properly in their containers.

Adjust height and margin of cell expression plot @ykevu (#485)

What

Adjusts the height and margin of cell expression plot to account for larger number of channels and variety of channel name lengths as referenced in #484 . This is difficult to test since we don't have a easy way of changing the channel names list so this may have issues again in the future, but the plot looked good for a variety of lists that I looked at (Hickey, Liu, 2d example)

Move toggle logic back to state machine, fix various bugs @ykevu (#473)

What

The array that controls what cell types are toggled on or off is moved back to the state machine, as it should be--I found that the fix to #470 was because we have to reassign arrays to new arrays for React to tell that a change was made (the Javascript equivalent of having to assign a variable to list.copy() instead of assigning to list)
I also fixed bugs relating to how the array is initialized and changed when cell types are turned on, turned off, deleted, added, etc. in various combinations
TODO: The array will still bug out if you, say delete a cell type and then undo that deletion (see #427). This can be fixed by making cellTypesMachine register the UI and have the toggle logic (and opacities should fall under this too) tracked with UNDO/REDO; however, this is a bit of a pain and there are more prioritized things that need to be done, so that will be for a future PR and I'll reopen a issue

Fix memory leak caused by continuously returning OutlineCellCanvas @ykevu (#465)

What

In AddCellTypeCanvas.js and AddDaughterCanvas.js, we have:

 if (!cell) {
    return null;
  }

I don't have a deep enough understanding of React to know exactly why is the case under the hood, but because of this code, the OutlineCellCanvas component is regenerated without disposing of its memory every time a cell goes from selected to not selected in the context of add mode (which equates to adding a cell to a cell type -> going from returning the component to returning null, alternating). This is especially bad because OutlineCellCanvas calls useCellMatrix(), which is a hook that, if needed, reallocates an approximately N x N matrix where N is the highest ID cell in your current frame.

Ultimately, this means that for the Keren dataset approximately 120 mb are reallocated everytime you add a cell in the final frame. I'm fairly confident that the issue is gone when this piece of code is removed though (whew).

Closes #460

🧰 Maintenance

Update documentation for set up and testing @ykevu (#520)

Updates the documentation with more detail and troubleshooting tips for first time set up, local use, and local testing.

More reqs cleanup @rossbar (#519)

What

Cleanup backend requirements

Why

Improve dev experience

MAINT: Followup to matplotlib removal. @rossbar (#518)

Followup to gh-517

MAINT: Rm dependency on matplotlib. @rossbar (#517)

Matplotlib was only used in one place for normalization. Replace the usage to reduce dependency footprint.

MAINT: Fix deprecated code and unpin numpy. @rossbar (#516)

What

Fix a few instances in the code where np.stack was called on non-iterables

Why

This pattern was deprecated - fixing it allows us to upgrade to latest numpy

Update documentation with cellTypes features @ykevu (#490)

What

Edits the documentation in the file format and the architecture to explain the new state machines and format specifications related to cell types
Adds a link to the documentation on the landing page to direct users to file format documentation

Bump webpack from 5.69.1 to 5.76.1 in /frontend @dependabot (#435)

Bumps webpack from 5.69.1 to 5.76.1.

Release notes

Sourced from webpack's releases.

v5.76.1

Fixed

Added assert/strict built-in to NodeTargetPlugin

Revert

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

v5.76.0

Bugfixes

Avoid cross-realm object access by @Jack-Works in webpack/webpack#16500

Improve hash performance via conditional initialization by @lvivski in webpack/webpack#16491

Serialize generatedCode info to fix bug in asset module cache restoration by @ryanwilsonperkin in webpack/webpack#16703

Improve performance of hashRegExp lookup by @ryanwilsonperkin in webpack/webpack#16759

Features

add target to LoaderContext type by @askoufis in webpack/webpack#16781

Security

CVE-2022-37603 fixed by @akhilgkrishnan in webpack/webpack#16446

Repo Changes

Fix HTML5 logo in README by @jakebailey in webpack/webpack#16614

Replace TypeScript logo in README by @jakebailey in webpack/webpack#16613

Update actions/cache dependencies by @piwysocki in webpack/webpack#16493

New Contributors

@Jack-Works made their first contribution in webpack/webpack#16500

@lvivski made their first contribution in webpack/webpack#16491

@jakebailey made their first contribution in webpack/webpack#16614

@akhilgkrishnan made their first contribution in webpack/webpack#16446

@ryanwilsonperkin made their first contribution in webpack/webpack#16703

@piwysocki made their first contribution in webpack/webpack#16493

@askoufis made their first contribution in webpack/webpack#16781

Full Changelog: webpack/webpack@v5.75.0...v5.76.0

v5.75.0

Bugfixes

experiments.* normalize to false when opt-out

avoid NaN%

show the correct error when using a conflicting chunk name in code

HMR code tests existance of window before trying to access it

fix eval-nosources-* actually exclude sources

fix race condition where no module is returned from processing module

fix position of standalong semicolon in runtime code

Features

add support for @import to extenal CSS when using experimental CSS in node

... (truncated)

Commits

21be52b Merge pull request #16804 from webpack/chore-patch-release
1cce945 chore(release): 5.76.1
e76ad9e Merge pull request #16803 from ryanwilsonperkin/revert-16759-real-content-has...
52b1b0e Revert "Improve performance of hashRegExp lookup"
c989143 Merge pull request #16766 from piranna/patch-1
710eaf4 Merge pull request #16789 from dmichon-msft/contenthash-hashsalt
5d64468 Merge pull request #16792 from webpack/update-version
67af5ec chore(release): 5.76.0
97b1718 Merge pull request #16781 from askoufis/loader-context-target-type
b84efe6 Merge pull request #16759 from ryanwilsonperkin/real-content-hash-regex-perf
Additional commits viewable in compare view

Maintainer changes

This version was pushed to npm by evilebottnawi, a new releaser for webpack since your current version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
@dependabot use these labels will set the current labels as the default for future PRs for this repo and language
@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Force cypress action to lower version @ykevu (#432)

What

Cypress e2e tests have started failing 100% of the time because on Feb 13, 2023, Github began defaulting to Node.js 18. Using the setup-node action to force node version to 16.x should bring us back to where we were before.

Bump default actions to latest version. @rossbar (#426)

Bump action versions to keep CI current, c.f. vanvalenlab/deepcell-tf#653

Fix cypress CI typo and add non-prepended AWS keys @ykevu (#420)

What

Fix typo that was causing cypress test not to run
Add non-prepended AWS keys for the backend to initialize properly

Apply precommit @rossbar (#401)

What

Applies linting from pre-commit hooks so the linting job stops failing. Also updates the pre-commit hooks for the linters and adds the linting commit to .git-blame-ignore-revs so that git blame is not cluttered by uninteresting formatting changes.

Prepend env vars with CYPRESS\_ @rossbar (#407)

According to the cypress-io action docs, env vars prepended this way are automatically forwarded to the cypress environment.

Follow-up to #406 in response to @ykevu 's observation that the test was not being run on main post-merge either.

What we expect:

Test to be skipped on PR opening/push (i.e. label. tests will only have 5 tests)
Test to run post-merge (i.e. label. tests will have 6 tests, all green)