in colab: GPUs: None detected

[delaroob]

Hi everyone,

I'm trying to continue training a SLEAP network in Colab. I've done the process (importing the same stuff, running the same code blocks etc.) several times in the past few days without any problems, however, it seems like I can't connect to any GPUs.

As the matter of fact, I can't run anything in colab right now except for like saving variables, importing packages and stuff that doesn't really require much comp power. Deeplabcut doesn't work either, the runtime colapses and restarts without further information.

In runtime python3 with a v100 GPU is selected and I still have 122 comp units available.

Thanks in advance for any help and let me know if additional information is required to solve the issue!

Here is the stuff I run (it's basically the demo notebook):

!pip uninstall -qqq -y opencv-python opencv-contrib-python
!pip install -qqq "sleap[pypi]>=1.3.3"

from google.colab import drive
drive.mount('/content/drive/')

(i've already done the next "iteration" of training yesterday, so I skipped the unzip and training part, since I just wanted to run inference and predict instances)

!sleap-track "/content/drive/MyDrive/sleap/colab2/male.mp4" -m "/content/drive/MyDrive/sleap/colab2/models/231213_081111.single_instance"

output:

INFO:numexpr.utils:NumExpr defaulting to 8 threads.
2023-12-17 16:30:34.863435: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:34.863471: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Started inference at: 2023-12-17 16:30:37.969681
Args:
{
│   'data_path': '/content/drive/MyDrive/sleap/colab2/male.mp4',
│   'models': ['/content/drive/MyDrive/sleap/colab2/models/231213_081111.single_instance'],
│   'frames': '',
│   'only_labeled_frames': False,
│   'only_suggested_frames': False,
│   'output': None,
│   'no_empty_frames': False,
│   'verbosity': 'rich',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': None,
│   'tracking.tracker': None,
│   'tracking.max_tracking': None,
│   'tracking.max_tracks': None,
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

2023-12-17 16:30:37.999611: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-12-17 16:30:37.999983: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:38.000129: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:38.000255: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:38.000375: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:38.045719: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-12-17 16:30:38.046198: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Versions:
SLEAP: 1.3.3
TensorFlow: 2.8.4
Numpy: 1.22.4
Python: 3.10.12
OS: Linux-6.1.58+-x86_64-with-glibc2.35

System:
GPUs: None detected.

Video: /content/drive/MyDrive/sleap/colab2/male.mp4
2023-12-17 16:30:38.122476: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2023-12-17 16:30:41.717931: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -36 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -18 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -18 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "101" frequency: 2000 num_cores: 8 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 32768 l2_cache_size: 1048576 l3_cache_size: 40370176 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -18 } dim { size: -40 } dim { size: -41 } dim { size: 1 } } }
Predicting... ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   5% ETA: 0:49:26 4.0 FPS

[delaroob]

update, in the DLC notebook, installing torch kindof solved the issue (it is painfully slow but at least the code runs...), so i've tried installing some packages in my SLEAP notebook too:

!pip install torch==1.12
!pip install uvicorn
!pip install python-multipart
!pip install kaleido
!pip install fastapi

...and ran some stuff, in case it helps

when:

import torch
torch.cuda.is_available()

output:

True

when:

!nvidia-smi

output:

Mon Dec 18 14:42:03 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-SXM2-16GB           Off | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P0              25W / 300W |      0MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |

and i've tried running the sleap-track again several times, and it still says GPUs: none detected

[talmo]

Hi @delaroob,

Thanks for pointing this out to us! From my tests, it looks like any version of TensorFlow older than 2.15 is no longer detecting GPUs on Google Colab.

We figured this day would come at some point... We don't control Google Colab :(

We're working on our own alternative to Google Colab, but in the meantime you could try Paperspace which offers another alternative.

We'll be trying to expand the version compatibility in SLEAP with newer versions of TensorFlow, but it's a complex process due to other conflicting dependencies.

I'm converting this to an issue so that we can track the progress.

Thanks and sorry that the Colab approach is not currently working!

Talmo

[delaroob]

Hi @talmo,

thanks for the reply. I will try out the paperspace approach and see how it goes.

So there isn't really a way around it in Colab, is there? And if I understand correctly, I should expect the same behaviour from any other program that uses TF<2.15 (I guess like DLC)?
Also, is there anything I can do? Is there any point to contact someone "from Colab"? It just happened so fast, I've been getting dependency warnings for like a month now but until a few days ago stuff was still working:(( .

[talmo]

Hi @talmo,

thanks for the reply. I will try out the paperspace approach and see how it goes.

So there isn't really a way around it in Colab, is there? And if I understand correctly, I should expect the same behaviour from any other program that uses TF<2.15 (I guess like DLC)?

That's my understanding, but let me know if you discover otherwise! It may be that their software is compatible with TF>=2.15 though -- we're just using some functionality that changed in newer versions.

Also, is there anything I can do? Is there any point to contact someone "from Colab"? It just happened so fast, I've been getting dependency warnings for like a month now but until a few days ago stuff was still working:(( .

Not that I'm aware of. You can try the colabtools GitHub repo, but they recommend actually asking questions under the google-colaboratory tag on StackOverflow. You'll actually find on there that other applications broke as well with the recent update of Colab to CUDA 12 [1].

In general though, Google Colab is managed by Google, so you have to contact them directly. The FAQ suggests going to Help --> Send feedback...

[delaroob]

found a solution here (apologies, I have no clue how to quote properly):
DeepLabCut/DeepLabCut#2465

I think the reason that Colab now uses 1% of the GPU is because Google made an update to CUDA 12.2 recently. Tensorflow-2.10.0 automatically installs on Colab, so it needs CUDA 11.2 and cuDNN 8.1 (according to the Linux GPU section here). A temporary solution is to connect to a GPU runtime -> click tools -> command palette -> type in and select 'use fallback runtime'. But this will only work until early Jan unfortunately.

Originally posted by @H-Dempsey in DeepLabCut/DeepLabCut#2465 (comment)

tried it out, with the SLEAP notebook, and it works!

[talmo]

Awesome, thanks @delaroob!! I'll update #1644 with some of these workarounds.