GeoAI_2021_unit_04_EX_Prepare_your_data

[Baldl]

No description provided.

[FloFranz]

Applying the function prepare_ds , I get the following error message:

2021-12-08 15:10:34.928189: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2021-12-08 15:10:34.929902: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2021-12-08 15:10:34.932636: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: LAPTOP-5NT7CML9
2021-12-08 15:10:34.933100: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: LAPTOP-5NT7CML9
2021-12-08 15:10:34.972827: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

I think there is a problem with TensorFlow. At the beginning of this course we had to install it. I installed it in Miniconda and the example script that we have at the beginning worked. Any ideas to solve this problem?

[gisma]

@FloFranz I have no Idea I made tests with the scripts that were available at the beginning of the course and it did work. so I have to hand over now to @Baldl

[jp-hecht]

@FloFranz As far as I know this message occurs when you have installed tensorflow on a CPU (as described in unit 1) and not a GPU. This message should not be a problem for further proceeding. Could you reshape your tensorflow dataset to an image?

[FloFranz]

@jp-hecht I can take a subset of the created array but the training_dataset and validation_dataset look like this:

<MapDataset shapes: ((None, None, None, None), (None, None, None, None)), types: (tf.float32, tf.float32)>

[jp-hecht]

@FloFranz I think you have the desired output. The None is just a placeholder for a (currently) unknown values.

[Puliester]

When I try to install tensorflow it doesn't work, I get the following Error message:

ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.* (from versions: none)
ERROR: No matching distribution found for tensorflow==2.7.*
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Error: Error installing package(s): ""tensorflow==2.7.*""

Any idea how to solve this?

[gisma]

@Puliester You may cross check the answer from session 1 #2 (comment) I think your libraries are not up to date, so please check the suggested possible solutions.

[IvoJJ]

Hello, i get also get Warningmessges, when i apply the prepare_ds function:

2021-12-12 12:49:09.885335: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/R/lib:/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/lib/server
2021-12-12 12:49:09.885371: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2021-12-12 12:49:09.885402: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ivo-ThinkPad-X240): /proc/driver/nvidia/version does not exist
2021-12-12 12:49:09.885806: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

And it seems that i get the right dataoutput afterwards:

<MapDataset shapes: ((None, None, None, None), (None, None, None, None)), types: (tf.float32, tf.float32)>

However, if i proceed and try to reshape i get the following error:

it <- as_iterator(training_dataset)
it <- tensorflow::iterate(as_iterator(training_dataset))

Error in py_iterate(it, f) :
InvalidArgumentError: input must have 3 channels but instead has 1 channels.
[[{{node adjust_saturation/AdjustSaturation}}]] [Op:IteratorGetNext]

I think this might be due to a wrong tensorflow installation but i dont know how to fix it.

[jp-hecht]

@IvoJJ

I think this might be due to a wrong tensorflow installation but i dont know how to fix it.

You could simply check if tensorflow is working appropriate with
tf$constant("Hello Tensorflow")
Desired output:
tf.Tensor(b'Hello Tensorflow', shape=(), dtype=string)
or run the MNIST example from the first unit

Error in py_iterate(it, f) :InvalidArgumentError: input must have 3 channels but instead has 1 channels.[[{{node adjust_saturation/AdjustSaturation}}]] [Op:IteratorGetNext]

Maybe your input file path or the input image itself has the wrong number of channels e.g. you switched the mask with the image or vice versa. Please check this in your code.

[gisma]

W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/R/lib:/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/lib/server

@IvoJJ it seems that your cuda is not available or not in the path. Pls check the following things in the shell:

# find libcuda.so.1
# first look if it is in the path
echo $LD_LIBRARY_PATH
# now search for it 
sudo find /usr/ -name 'libcuda.so.*' 

Depending on your system it should be something like:

/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so.460.91.03
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.460.91.03
/usr/local/cuda-11.0/compat
...

If you find it add it to your path

export LD_LIBRARY_PATH=/path/of/libcuda.so.1:$LD_LIBRARY_PATH

Note cuda 11 is requested.

[IvoJJ]

@gisma there were no returns when using

echo $LD_LIBRARY_PATH   and
sudo find /usr/ -name 'libcuda.so.*' 

@jp-hecht
the installation in the warm-up R spatial was seamless and the test script did also work. I also get the right output when using

tf$constant("Hello Tensorflow")

I dont think i got a switch in my data. I use

files <- data.frame(
   img = list.files(
      file.path(envrmt$path_model_training_data_dop),
      full.names = TRUE,
      pattern = "*.png"
   ),
   mask = list.files(
      file.path(envrmt$path_model_training_data_bui),
      full.names = TRUE,
      pattern = "*.png"
   )
)

in the bui folder are the mask files (only 2 values, or black and white). Then i split:

data <- initial_split(files, prop = 0.8)

Afterwards i build training and validation dataset:

# one more parameter
batch_size = 8

training_dataset <-
   prepare_ds(
      training(data),
      train = TRUE,
      predict = FALSE,
      model_input_shape = model_input_shape,
      batch_size = batch_size
   )


validation_dataset <-
   prepare_ds(
      testing(data),
      train = FALSE,
      predict = FALSE,
      model_input_shape = model_input_shape,
      batch_size = batch_size
   )

Then my output is:

training_dataset

<MapDataset shapes: ((None, None, None, None), (None, None, None, None)), types: (tf.float32, tf.float32)>

but maybe im not seeing something.

Thanks for the help!

[gisma]

@IvoJJ I am pretty sure that you do NOT have a GPU onboard of your laptop?

If so try to reininstall

library(keras)
install_keras(method = c("conda"),
              conda = "auto", version = "default", tensorflow = "cpu")

If not give tensorflow = "gpu" a try

[IvoJJ]

@gisma no i only have onboard graphics, is this a problem?

[gisma]

@IvoJJ nope no problem but as far as I see it from your error messages is your system looking for a valid cuda library. So please try to re-install it with tensorflow = cpu