Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pytests for decoding pipeline #1155

Merged
merged 10 commits into from
Nov 26, 2024
Merged

Add pytests for decoding pipeline #1155

merged 10 commits into from
Nov 26, 2024

Conversation

CBroz1
Copy link
Member

@CBroz1 CBroz1 commented Oct 4, 2024

Description

This PR adds coverage of the decoding pipeline to our test suite. The tests themselves could use greater specificity - these are generic checks of the right object type or column names in a dataframe. Maybe a future PR could use this infrastructure to ensure the data are processed correctly?

Coverage
---------- coverage: platform linux, python 3.9.19-final-0 -----------
Name                                                       Stmts   Miss  Cover   Missing
----------------------------------------------------------------------------------------
src/spyglass/common/common_behav.py                          276     42    85%   48-52, 85, 200, 228, 314-318, 322-351, 354, 398-399, 440, 453, 465-471, 501, 531, 544-546, 564, 642-645, 648-649, 655, 678-682
src/spyglass/common/common_dandi.py                          122    122     0%   1-332
src/spyglass/common/common_device.py                         224     42    81%   91, 124, 292-304, 308, 402-403, 422-452, 464, 573-611, 656, 710-713, 737, 770-774, 816-834
src/spyglass/common/common_dio.py                             52      3    94%   41-45, 129
src/spyglass/common/common_ephys.py                          313    178    43%   74, 114, 184, 194-203, 228-270, 305-310, 313, 327, 400-404, 439, 474-565, 569-578, 582-583, 648-734, 759-958, 962-963
src/spyglass/common/common_filter.py                         188     19    90%   23-24, 92-96, 99-102, 138-139, 341-344, 381, 392-400, 474
src/spyglass/common/common_interval.py                       195     16    92%   52-53, 120-125, 140-145, 163-166, 536, 546
src/spyglass/common/common_lab.py                            111     11    90%   62, 69-70, 154-155, 269, 277-278, 310, 315-316
src/spyglass/common/common_nwbfile.py                        246    107    57%   214, 228, 245-246, 289-309, 353-356, 363, 370, 449-525, 552-619, 636-655, 676-677, 691, 696-701, 709-714, 723, 727-731, 764, 784-798
src/spyglass/common/common_position.py                       256     25    90%   28-30, 184, 354-355, 362-363, 674-740
src/spyglass/common/common_region.py                          14      0   100%
src/spyglass/common/common_sensors.py                         24      2    92%   38-39
src/spyglass/common/common_session.py                         70      8    89%   155-159, 171, 177, 189-192
src/spyglass/common/common_subject.py                         21      3    86%   40-41, 63
src/spyglass/common/common_task.py                           108     33    69%   34-40, 122-123, 139-142, 166, 196-225, 241, 251-258
src/spyglass/common/common_usage.py                          148     32    78%   108, 129-130, 197, 206, 214, 251, 263-267, 275-279, 313, 338-342, 346-357
src/spyglass/common/errors.py                                  2      0   100%
src/spyglass/common/populate_all_common.py                    49     12    76%   40-46, 89-92, 161-163, 166-172
src/spyglass/common/prepopulate/prepopulate.py                44     31    30%   19, 24-74, 86-95
src/spyglass/common/signal_processing.py                      16     16     0%   1-57
src/spyglass/decoding/decoding_merge.py                       77     11    86%   44, 63-64, 82-83, 119-121, 139-147
src/spyglass/decoding/utils.py                                15      8    47%   30-39
src/spyglass/decoding/v1/clusterless.py                      169     20    88%   239-242, 271, 343, 553-601
src/spyglass/decoding/v1/core.py                              94     12    87%   59, 72-89, 254, 260, 262, 277
src/spyglass/decoding/v1/dj_decoder_conversion.py             73      5    93%   32, 168-169, 188-189
src/spyglass/decoding/v1/sorted_spikes.py                    156     29    81%   199-202, 230, 302, 356-372, 411, 421, 480-526
src/spyglass/decoding/v1/utils.py                              6      0   100%
src/spyglass/decoding/v1/waveform_features.py                108     10    91%   119, 132-134, 189, 272-273, 293-296, 358
src/spyglass/lfp/analysis/v1/lfp_band.py                     136     10    93%   107, 242, 244, 264, 278-282, 361-365, 435, 463
src/spyglass/lfp/lfp_electrode.py                             24      0   100%
src/spyglass/lfp/lfp_imported.py                              12      0   100%
src/spyglass/lfp/lfp_merge.py                                 22      0   100%
src/spyglass/lfp/v1/lfp.py                                    66      4    94%   128-132, 192-193
src/spyglass/lfp/v1/lfp_artifact.py                           56     25    55%   139-210
src/spyglass/lfp/v1/lfp_artifact_MAD_detection.py             37     29    22%   42-63, 81, 107, 129-137, 157-176
src/spyglass/lfp/v1/lfp_artifact_difference_detection.py      79     69    13%   72-224, 253-280
src/spyglass/linearization/merge.py                           14      0   100%
src/spyglass/linearization/utils.py                            0      0   100%
src/spyglass/linearization/v0/main.py                         49     24    51%   62-64, 71-74, 88-92, 127-192, 196
src/spyglass/linearization/v1/main.py                         55      6    89%   66-69, 83-87, 203
src/spyglass/position/position_merge.py                       22      0   100%
src/spyglass/position/v1/dlc_reader.py                       112     24    79%   26, 40, 46-47, 53, 59-60, 63, 72, 76, 82-83, 141-143, 154, 158-171, 223, 227
src/spyglass/position/v1/dlc_utils.py                        371    152    59%   58, 61, 69, 72, 97-100, 104, 149-163, 186, 212-225, 230-237, 249-263, 278-296, 315, 334, 415, 424, 430, 465, 478-509, 513-540, 559-574, 588-634, 639-664, 734, 807, 818
src/spyglass/position/v1/dlc_utils_makevid.py                246    118    52%   70, 80, 118-119, 168-169, 195-199, 221, 236-244, 259, 318-340, 346-480, 483-490, 498-505, 508-557
src/spyglass/position/v1/position_dlc_centroid.py            113      8    93%   185, 208-216, 222, 239, 262
src/spyglass/position/v1/position_dlc_cohort.py               47      1    98%   122
src/spyglass/position/v1/position_dlc_model.py               122     27    78%   36-40, 104, 207, 222, 236, 296-340
src/spyglass/position/v1/position_dlc_orient.py               75     10    87%   76, 140-160, 172
src/spyglass/position/v1/position_dlc_pose_estimation.py     139      4    97%   75, 82, 128, 266
src/spyglass/position/v1/position_dlc_position.py            180     32    82%   57, 107, 217-218, 224-230, 367-369, 375, 398, 401, 423, 462-482
src/spyglass/position/v1/position_dlc_project.py             214     90    58%   92, 94, 130-203, 270, 299, 311-316, 328, 344-362, 406-407, 438-445, 452-455, 479-504
src/spyglass/position/v1/position_dlc_selection.py           125      2    98%   236, 393
src/spyglass/position/v1/position_dlc_training.py             89      5    94%   49, 130-131, 190-191
src/spyglass/position/v1/position_trodes_position.py         105      3    97%   70, 307-308
src/spyglass/spikesorting/analysis/v1/group.py                92     34    63%   78-80, 109-128, 178, 189-197, 201, 216, 234-249, 279, 294
src/spyglass/spikesorting/analysis/v1/unit_annotation.py      41      2    95%   75, 138
src/spyglass/spikesorting/imported.py                         64     35    45%   49-58, 65, 72, 94-113, 121-134, 139-150
src/spyglass/spikesorting/spikesorting_merge.py               91     19    79%   106, 134-142, 214-229, 260
src/spyglass/spikesorting/utils.py                            97     48    51%   76-78, 89, 102-103, 111-115, 120-124, 146-152, 161-204, 237, 243-248, 250-255, 260-278, 282
src/spyglass/spikesorting/v1/artifact.py                      86     25    71%   69, 101-102, 256, 297-331, 350-372
src/spyglass/spikesorting/v1/curation.py                     157     32    80%   144-162, 241-266, 357-364, 388, 414, 439-443, 478-480
src/spyglass/spikesorting/v1/figurl_curation.py               86     55    36%   47-58, 69-103, 125-169, 175-176, 188, 206-273
src/spyglass/spikesorting/v1/metric_curation.py              191     29    85%   88, 134, 139-141, 162, 193-194, 260, 313, 403, 448, 457-458, 462, 507-526, 585
src/spyglass/spikesorting/v1/metric_utils.py                  19      5    74%   43-50, 58
src/spyglass/spikesorting/v1/recording.py                    202     29    86%   82, 85, 122, 157-158, 245, 384-395, 417-432, 453-460, 481, 486, 509-510
src/spyglass/spikesorting/v1/sorting.py                      118     20    83%   100, 131-132, 205, 209, 256, 272-278, 322-351, 387
src/spyglass/spikesorting/v1/utils.py                         27     15    44%   62-109
src/spyglass/utils/database_settings.py                       99     15    85%   147, 162, 173-175, 179-183, 196, 212-215, 223, 226
src/spyglass/utils/dj_graph.py                               488     34    93%   38-39, 55-56, 125, 263, 306, 375, 407, 473, 482, 522-531, 629, 646-648, 663, 716, 929, 949, 955-956, 992, 1020, 1111-1113, 1126, 1136, 1138, 1158, 1182
src/spyglass/utils/dj_helper_fn.py                           195     94    52%   78, 101, 167-177, 182-191, 196-199, 292, 301-302, 313, 341-342, 372-436, 452-459, 479-495, 511-523, 538-539, 555-569
src/spyglass/utils/dj_merge_tables.py                        301     76    75%   35, 37, 43-47, 74-75, 91, 153, 179, 235, 241, 323-325, 345, 351-352, 361, 413, 419, 457-463, 484-503, 533, 614, 620, 631, 678, 686, 708-714, 718, 725, 748, 767, 803-804, 814, 824-830, 834-844, 851-854, 879, 884, 914-925
src/spyglass/utils/dj_mixin.py                               368     66    82%   238-249, 391-431, 452-453, 461, 513, 549, 589, 605, 617-618, 626-636, 650-656, 675-678, 761, 768, 911, 937-938, 958-1026, 1037-1038
src/spyglass/utils/logging.py                                 17      2    88%   27-28
src/spyglass/utils/nwb_helper_fn.py                          175     36    79%   53-90, 109, 154-159, 202, 205, 214, 238, 262-263, 307, 325, 435, 548, 565, 574-592
src/spyglass/utils/position.py                                13      2    85%   27-28
src/spyglass/utils/spikesorting.py                             9      2    78%   13, 18
src/spyglass/utils/sql_helper_fn.py                           84     17    80%   45, 58-62, 91, 150, 184-185, 195-205
----------------------------------------------------------------------------------------
TOTAL                                                       8707   2132    76%
===================== 279 passed, 12 skipped in 202.96s (0:03:22) =====================

This motivated a couple edits to the pipeline itself

  • decoding_merge: previously had a couple places that ran fetch1 on unrestricted/uninstanced class objects. Swapping out for merge_restrict_class fixed these issues
  • decoding_merge.create_decoding_view: previously expected a tuple from fetch_linear_position_info, which caused errors because it returns just the dataframe
  • clusterless.py and core.py: swapped out raising error for logging error to align with broader package insert patterns
  • core.py: Partially addressing DecodingParameters.insert_default errors on new declaration #1158 with a check of whether or not vars will work on the passed params.
  • `unit_annotation

Checklist:

  • No. This PR should be accompanied by a release: (yes/no/unsure)
  • N/a. If release, I have updated the CITATION.cff
  • No. This PR makes edits to table definitions: (yes/no)
  • N/a. If table edits, I have included an alter snippet for release notes.
  • N/a. If this PR makes changes to position, I ran the relevant tests locally.
  • I have updated the CHANGELOG.md with PR number and description.
  • N/a. I have added/edited docs/notebooks to reflect the changes

@CBroz1 CBroz1 marked this pull request as ready for review October 9, 2024 15:54
@edeno
Copy link
Collaborator

edeno commented Nov 26, 2024

Looks good to me pending conflict resolution

@edeno edeno merged commit 37ddfc1 into LorenFrankLab:master Nov 26, 2024
17 checks passed
@CBroz1 CBroz1 deleted the dpt branch November 27, 2024 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants