Screenshot of the example solo dataset displayed in Voxel51
pysolotools-fiftyone is a python package for viewing and interacting with solo datasets using the Voxel51 viewer.
This package allows the user to create a new Voxel51 viewer in either a web browser or jupyter notebook.
SOLO datasets are generated by Unity's Perception package. Please see here for more information about the SOLO schema.
It supports the displaying of the following annotations:
- Bounding Boxes
- 3D Bounding Boxes
- Keypoints
- Instance Segmentation
- Semantic Segmentation
- Depth
- Occlusion
- Normals
- Pixel Position
- Install Anaconda or Miniconda (recommended).
- Install pre-commit.
- Install OpenEXR.
- Make sure
pipis installed
pip install pysolotools-fiftyoneThe Solo Fiftyone viewer is executed on the command line with the following command line:
pysolotools-fiftyone examples/groceriesThis will launch a new fiftyone viewer in your web browser.
SOLO FiftyOne can be executed in Jupyter Notebooks.
To use in a notebook:
from pysolotools_fiftyone.solo_fiftyone import run_in_notebook
path="./groceries"
run_in_notebook(path)An example notebook has been included in the repository for your convenience.
The fiftyone will create an entry for each frame. Each entry will consist of a group with at least an RGB capture of the Unity game screen. If the dataset was created with Normals or Pixel Position labelers, the image group will also contain a PNG image for each one of them. These 3 types create the background of the frame, and the labels are drawn on to of them. The background image can be changed with the group dropdown located at the upper right of the screen.
Group dropdown
Labels are drawn on top of the background group image. These can be toggled on and off via the Labels selector left of the image.
Label selector
2D bounding boxes and label type for each visible labeled object in the image. Detailed information includes the amount of visible pixels of the object.
Bounding boxes in Voxel51
Semantic segmentation labels each pixel of an image with a label type.
Semantic segmentation in Voxel51
Each pixel of the image is labeled with the ID of the object that it represents.
Instance segmentation in Voxel51
Display keypoint (generally, but not limited to, a human skeleton) information.
Keypoints in Voxel51
3D volume of object in the scene.
3D Bounding Boxes in Voxel51
Pixel distance from the camera in the scene displayed in Plotly's virdis scale. The color scale goes from purple to yellow, with purple representing objects closer to the camera.
Depth image in Voxel51, purple pixels are closer to the camera, yellow pixels are farthest away
Object occlusion, or the amount of the object you can see, displayed in Plotly's virdis scale. The scale goes from yellow (fully visible) to purple (fully occluded). Hovering over a pixel will give you the exact occlusion value.
Occlusion in Voxel51, yellow is fully visible, purple is fully occluded
The normals image displays an object's surface normal in a 3 channel image. The red channel represents the normal's x value, the green channel represents the y channel, and the blue channel represents the normal's z value.
Normals in Voxel51
The pixel position image displays a pixels cartesian coordinate in the camera's space. The red channel represents the pixel position's x value, the green channel represents the y channel, and the blue channel represents the z value. The image displayed is a 8bit per channel representation of a 32bit per channel file format, so some accuracy is lost in the display.
Pixel Position in Voxel51
Inside the examples directory there are two sample datasets
- groceries: A dataset of grocery items that have all supported labels except keypoints
- keypoints: A simple dataset that has keypoint annotations
The Unity Computer Vision tools are open-source, and we encourage and welcome contributions. If you wish to contribute, be sure to review our contribution guidelines and code of conduct.
For feature requests, bugs, or other issues, please file a GitHub issue using the provided templates we will investigate as soon as possible.
- When running with a web browser, it will open with a subset of your frames visible. The data is still being imported in the background, and refreshing the browser will update the viewer will the current set of loaded scenes.
- To properly see some heatmap overlays (for instance depth and occlusion) make sure that you have Voxel51 settings (gear icon at the top of the screen) fields Color by set to label and Label opacity to highest (all of the way to the right)
