Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--c Flag for Continuing Training Not Implemented in MONAI nnUNet Runner #8237

Open
Kirscher opened this issue Nov 25, 2024 · 0 comments
Open

Comments

@Kirscher
Copy link

Kirscher commented Nov 25, 2024

Describe the bug

I encountered an issue while using the MONAI nnUNetV2Runner. The --c flag, which is mentioned in the documentation and associated with continuing training from the latest checkpoint, does not seem to be implemented in the nnUNetV2Runner code. Including this flag in the command leads to an error indicating that it is unrecognized.

To Reproduce

Run the following command:
python -m monai.apps.nnunet nnUNetV2Runner train_single_model --input_config "input.yaml" --config "3d_fullres" --fold 1 --c
Observe the error:
error: unrecognized arguments: True

Expected behavior
The --c flag should:

Be recognized as a valid argument.
Enable continuation of training from the latest checkpoint without manually specifying a checkpoint file.

Environment

================================
Printing MONAI config...
================================
MONAI version: 1.4.0+18.g13b96aed
Numpy version: 1.26.4
Pytorch version: 2.5.1+cu124
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 13b96aedc48ad2da16149490b06a1a6bd8361335
MONAI __file__: /home/<username>/tutorials/venv/lib/python3.12/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.0
Nibabel version: 5.3.2
scikit-image version: 0.24.0
scipy version: 1.14.1
Pillow version: 11.0.0
Tensorboard version: 2.18.0
gdown version: 5.2.0
TorchVision version: 0.20.1+cu124
tqdm version: 4.67.0
lmdb version: 1.5.1
psutil version: 6.1.0
pandas version: 2.2.3
einops version: 0.8.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 2.18.0
pynrrd version: 1.1.1
clearml version: 1.16.5

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 24.04.1 LTS
Platform: Linux-6.8.0-49-generic-x86_64-with-glibc2.39
Processor: x86_64
Machine: x86_64
Python version: 3.12.3
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/kirscher/.config/Code/logs/20241125T093256/ptyhost.log', fd=43, position=434, mode='a', flags=33793), popenfile(path='/snap/code/176/usr/share/code/v8_context_snapshot.bin', fd=103, position=0, mode='r', flags=32768)]
Num physical CPUs: 12
Num logical CPUs: 24
Num usable CPUs: 24
CPU usage (%): [46.3, 42.1, 61.7, 35.9, 46.0, 64.4, 31.0, 68.1, 41.3, 56.4, 62.3, 26.9, 35.1, 17.1, 48.6, 49.4, 34.1, 34.5, 38.8, 50.7, 21.4, 84.8, 32.6, 17.7]
CPU freq. (MHz): 3514
Load avg. in last 1, 5, 15 mins (%): [22.6, 23.9, 24.1]
Disk usage (%): 94.2
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 62.6
Available memory (GB): 41.7
Used memory (GB): 18.1

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: NVIDIA RTX A4500
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 56
GPU 0 Total memory (GB): 19.7
GPU 0 CUDA capability (maj.min): 8.6

Additional context
Using the standard nnUNetv2_train command outside of the MONAI runner framework bypasses the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant