Are these recoverable errors?

[rrazavipour]

Using dvc==3.56.0, I see the following errors, high number of similar ones when I run a dvc pull --verbose

Traceback (most recent call last):
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -   File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 611, in _load_from_storage
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -     _load_from_object_storage(trie, entry, storage)
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -   File "/opt/conda/lib/python3.10/site-packages/dvc_data/index/index.py", line 547, in _load_from_object_storage
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -     obj = Tree.load(storage.odb, root_entry.hash_info, hash_name=storage.odb.hash_name)
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -   File "/opt/conda/lib/python3.10/site-packages/dvc_data/hashfile/tree.py", line 193, in load
2024-11-13 10:30:45 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset - with obj.fs.open(obj.path, "r") as fobj:
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -   File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 324, in open
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -     return self.fs.open(path, mode=mode, **kwargs)
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -   File "/opt/conda/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 131, in open
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset -     return open(path, mode=mode, encoding=encoding)  # noqa: SIM115
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset - FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/13/47ee4f701aaca444787c32b0b39bde.dir'
2024-11-13 10:30:46 -0800 - dagster - INFO - __ASSET_JOB - 631d350a-5ef1-46d9-8206-57c5e833585b - mlmodel_device_make_model_dataset - 2024-11-13 18:29:54,805 DEBUG: failed to load ('18c2837b-3da6-4e69-bbc7-e7adbfb4ee49',) from storage local (/opt/ml/input/data/train/test_mm_code/.dvc/cache) - [Errno 2] No such file or directory: '/opt/ml/input/data/train/test_mm_code/.dvc/cache/c8/da2382c563ff9139dd577dadf5473e.dir'

What causes these errors?
Ways to avoid them?
Can dvc pull be restarted and recover from the errors?

[shcheklein]

@rrazavipour can you give a bit more details please? I see that you run it within some environment with dagster? could you describe it (is it local machine, or some automation, etc)

Usually this error means that someone (or some process) didn't do dvc push. What remote storage do you use with DVC?

[rrazavipour]

the problem happens in many different contexts, locally, on a Mac, EC2 with an FSx Lustre filesystem, in Dagster...
so the sequence of events are usually git clone or git pull, that gives me the "*.dvc" files, followed by a "dvc pull --verbose"

My remote storage is S3
[core]
analytics = false
remote = xxx
['remote "xxx"']
url = s3:/xxx/mlmodel-deviceidentification-dataset/dvc
region = us-west-2

[rrazavipour]

sorry I was doing dvc --version, you want dvc version

dvc version
DVC version: 3.56.0 (pip)

Platform: Python 3.10.14 on macOS-14.5-x86_64-i386-64bit
Subprojects:
dvc_data = 3.16.7
dvc_objects = 5.1.0
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.3.8
Supports:
http (aiohttp = 3.10.10, aiohttp-retry = 2.9.1),
https (aiohttp = 3.10.10, aiohttp-retry = 2.9.1),
s3 (s3fs = 2024.10.0, boto3 = 1.35.36)
Config:
Global: /Users/rezarazavipour/Library/Application Support/dvc
System: /Library/Application Support/dvc

[rrazavipour]

/Library/Caches/dvc/repo$ ls
42076f9e50496d199f1e6a63757cec6b b4829e627b60a3364e4341fae024716f ea6f569afa8007f064edcc98188dfcaa
4ef8232d6edcab6c03568bc07243e4d8 c62549c3608df13c3a1d5636548dfc37 f087f181142741a09ec00265570ed888
766b7b5c200556a43c8982067651433e cb522001f88226d50af087736620d85e
875d683f0717fb81695ca880348d938d e55c1098df4aad3211978864dec176b8

[shcheklein]

please run dvc version inside the repo that is causing troubles or do a fresh clone and run it there. You will see the exact path for that repo to the site cache dir.

[rrazavipour]

DVC version: 3.56.0 (pip)

Platform: Python 3.9.13 on macOS-10.16-x86_64-i386-64bit
Supports:
gs (gcsfs = 2023.12.2.post1),
hdfs (fsspec = 2022.1.0, pyarrow = 6.0.1),
webhdfs (fsspec = 2022.1.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
s3 (s3fs = 2022.1.0, boto3 = 1.20.24)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s2s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s2s1
Repo: dvc, git
(base) ~/data/ml-data$ Traceback (most recent call last):
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/dvc/main.py", line 4, in
from dvc.main import main
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/dvc/init.py", line 6, in
import dvc.logger
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/dvc/logger.py", line 8, in
from dvc.progress import Tqdm
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/dvc/progress.py", line 6, in
import fsspec
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/fsspec/init.py", line 13, in
from .exceptions import FSTimeoutError
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/fsspec/exceptions.py", line 4, in
import asyncio
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/asyncio/init.py", line 21, in
from .base_events import *
File "/Users/rezarazavipour/opt/anaconda3/lib/python3.9/site-packages/asyncio/base_events.py", line 296
future = tasks.async(future, loop=self)
^
SyntaxError: invalid syntax

[rrazavipour]

pip freeze | grep dvc
dvc==3.56.0
dvc-data==3.16.7
dvc-http==2.32.0
dvc-objects==5.1.0
dvc-render==1.0.2
dvc-s3==3.2.0
dvc-studio-client==0.21.0
dvc-task==0.4.0
dvclive==0.9.0