UserWarning: Valid PROJ data directory not found #1350

DOptimusPrime · 2023-10-16T04:52:21Z

I am running this code but for, I realized that the FileNotFound Error is happening because it is looking for a file on another old computer? how do I solve that issue? Is it related to the Pyproj warning:

C:\Users\hdela\.conda\envs\chur2\python.exe D:\Chur\Manager\main.py batch_compute_precipitation 
running return period: 500
running return period: 500
executing 2 processes
['python', 'D:/Chur/Engine/main.py', 'precipitation', 'D:/Chur/Manager/precipitation_definitions/54ac8997-efe7-4877-922c-f10e68f4dc84']
['python', 'D:/Chur/Engine/main.py', 'precipitation', 'D:/Chur/Manager/precipitation_definitions/f563a296-bc30-4474-ac18-7f272fd58e14']
C:\Users\hdela\.conda\envs\chur2\lib\site-packages\pyproj\__init__.py:91: UserWarning: Valid PROJ data directory not found. Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.
  warnings.warn(str(err))
C:\Users\hdela\.conda\envs\chur2\lib\site-packages\pyproj\__init__.py:91: UserWarning: Valid PROJ data directory not found. Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.
  warnings.warn(str(err))
Traceback (most recent call last):
  File "D:\Chur\Engine\main.py", line 1676, in <module>
    main(sys.argv[1:])
  File "D:\Chur\Engine\main.py", line 606, in main
    engine.precipitationSimulator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 46, in initialize
    self.generator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 154, in initialize
    self.fh = Dataset(self.ncfile, mode='r')
  File "src\netCDF4\_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src\netCDF4\_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200703010100_200704010000.nc'
Traceback (most recent call last):
  File "D:\Chur\Engine\main.py", line 1676, in <module>
    main(sys.argv[1:])
  File "D:\Chur\Engine\main.py", line 606, in main
    engine.precipitationSimulator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 46, in initialize
    self.generator.initialize()
  File "D:\Chur\Engine\Models\Precipitation\precipitation.py", line 154, in initialize
    self.fh = Dataset(self.ncfile, mode='r')
  File "src\netCDF4\_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src\netCDF4\_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200710010100_200711010000.nc'
D:/Chur/Manager/precipitation_definitions/f563a296-bc30-4474-ac18-7f272fd58e14
2023-10-15 23:44:33-- engine: creating output directories at D:/Chur/Engine/Data/SimulationData//Precipitation/f563a296-bc30-4474-ac18-7f272fd58e14/ --
D:/Chur/Manager/precipitation_definitions/54ac8997-efe7-4877-922c-f10e68f4dc84
2023-10-15 23:44:33-- engine: creating output directories at D:/Chur/Engine/Data/SimulationData//Precipitation/54ac8997-efe7-4877-922c-f10e68f4dc84/ --
executed 2 processes

Process finished with exit code 0

My environment variables:
C:\Users\hdela.conda\envs\chur2\Lib\site-packages\pyproj
C:\Users\hdela.conda\envs\chur2\Library\share\proj

The text was updated successfully, but these errors were encountered:

djhoese · 2023-10-16T13:49:08Z

Why did you delete all of the questions on the issue template? They are there for a reason. That reason is that we need that information to give you any useful kind of answer. Please answer these questions and then we'll see if we can help:

#### Installation method/steps
 - Installation method (conda, pip wheel, from source, etc...)
 - How did you install PROJ? Where is it installed?
 - Please provide all commands/steps you used to install pyproj and PROJ.

#### Environment Information
 - pyproj version you are attempting to install
 - PROJ version (Execute `proj` command and give version here.)
 - Python version (`python -c "import sys; print(sys.version.replace('\n', ' '))"`)
 - Operation System Information (`python -c "import platform; print(platform.platform())"`)

DOptimusPrime · 2023-10-17T05:54:47Z

Rel. 8.1.1, September 1st, 2021
usage: PROJ [-bdeEfiIlmorsStTvVwW [args]] [+opt[=arg] ...] [file ...]

This is my environment information:

3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:39:05) [MSC v.1929 64 bit (AMD64)]
Windows-10-10.0.22621-SP0

I installed it through conda-forge:

These are all the package.:

 Name                    Version                   Build  Channel
attrs                     23.1.0             pyh71513ae_1    conda-forge
boost-cpp                 1.74.0               h9f4b32c_8    conda-forge
brotli                    1.1.0                hcfcfb64_1    conda-forge
brotli-bin                1.1.0                hcfcfb64_1    conda-forge
brotli-python             1.1.0            py39h99910a6_1    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
cairo                     1.16.0            hb19e0ff_1008    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cfitsio                   3.470                h0af3d06_7    conda-forge
cftime                    1.6.2            py39hc266a54_1    conda-forge
charset-normalizer        3.3.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           win_pyh7428d3b_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.1            py39h1f6ef14_1    conda-forge
curl                      7.86.0               heaf79c2_1    conda-forge
cycler                    0.12.0             pyhd8ed1ab_0    conda-forge
decorator                 4.4.2                      py_0    conda-forge
expat                     2.5.0                h63175ca_1    conda-forge
fiona                     1.8.20           py39hea8b339_2    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               hbde0cde_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.43.0           py39ha55989b_0    conda-forge
freetype                  2.12.1               hdaf720e_2    conda-forge
freexl                    1.0.6                h67ca5e6_1    conda-forge
gdal                      3.3.3            py39h3f5efd6_0    conda-forge
geopandas                 0.9.0              pyhd8ed1ab_1    conda-forge
geopandas-base            0.9.0              pyhd8ed1ab_1    conda-forge
geos                      3.9.1                h39d44d4_2    conda-forge
geotiff                   1.7.0                h350af67_3    conda-forge
gettext                   0.21.1               h5728263_0    conda-forge
hdf4                      4.2.15               h1b1b6ef_5    conda-forge
hdf5                      1.12.1          nompi_h2a0e4a3_104    conda-forge
icu                       68.2                 h0e60522_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-resources       6.1.0              pyhd8ed1ab_0    conda-forge
importlib_resources       6.1.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2023.2.0         h57928b3_50496    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hcfcfb64_3    conda-forge
kealib                    1.4.15               h8995ca9_0    conda-forge
kiwisolver                1.4.5            py39h1f6ef14_1    conda-forge
krb5                      1.19.3               h1176d77_0    conda-forge
lcms2                     2.14                 h90d422f_0    conda-forge
lerc                      4.0.0                h63175ca_0    conda-forge
libblas                   3.9.0              18_win64_mkl    conda-forge
libbrotlicommon           1.1.0                hcfcfb64_1    conda-forge
libbrotlidec              1.1.0                hcfcfb64_1    conda-forge
libbrotlienc              1.1.0                hcfcfb64_1    conda-forge
libcblas                  3.9.0              18_win64_mkl    conda-forge
libcurl                   7.86.0               heaf79c2_1    conda-forge
libdeflate                1.14                 hcfcfb64_0    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgdal                   3.3.3                h08edb8f_0    conda-forge
libglib                   2.78.0               he8f3873_0    conda-forge
libhwloc                  2.9.1                h51c2c0f_0    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
libkml                    1.3.0             hd45a9bc_1016    conda-forge
liblapack                 3.9.0              18_win64_mkl    conda-forge
libnetcdf                 4.8.1           nompi_h1cc8e9d_102    conda-forge
libpng                    1.6.39               h19919ed_0    conda-forge
libpq                     13.8                 h04fd8be_0    conda-forge
librttopo                 1.1.0                hb340de5_6    conda-forge
libspatialindex           1.9.3                h39d44d4_4    conda-forge
libspatialite             5.0.1                h6a90f17_9    conda-forge
libsqlite                 3.43.0               hcfcfb64_0    conda-forge
libssh2                   1.10.0               h680486a_3    conda-forge
libtiff                   4.4.0                hc4f729c_5    conda-forge
libwebp-base              1.3.2                hcfcfb64_0    conda-forge
libxcb                    1.13              hcd874cb_1004    conda-forge
libxml2                   2.10.4               hc3477c8_0    conda-forge
libzip                    1.9.2                hfed4ece_1    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
lz4-c                     1.9.3                h8ffe710_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mapclassify               2.5.0              pyhd8ed1ab_1    conda-forge
matplotlib-base           3.7.1            py39haf65ace_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munch                     4.0.0              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
netcdf4                   1.6.0           nompi_py39hf113b1f_100    conda-forge
networkx                  2.6.2              pyhd8ed1ab_0    conda-forge
numpy                     1.20.3           py39h6635163_2    conda-forge
openjpeg                  2.5.0                hc9384bd_1    conda-forge
openssl                   1.1.1w               hcfcfb64_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    1.5.3            py39h2ba5b7c_1    conda-forge
pcre                      8.45                 h0e60522_0    conda-forge
pcre2                     10.40                h17e33f8_0    conda-forge
pillow                    9.2.0            py39h595c93f_3    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h63175ca_0    conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    conda-forge
pooch                     1.7.0              pyhd8ed1ab_4    conda-forge
poppler                   21.09.0              h24fffdf_3    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                13.8                 h392df6f_0    conda-forge
proj                      8.1.1                h1cfcee9_2    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyproj                    3.2.1            py39h39b2389_2    conda-forge
pysocks                   1.7.1              pyh0701188_6    conda-forge
python                    3.9.15          h0269646_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rtree                     1.0.1            py39h09fdee3_3    conda-forge
scikit-learn              1.2.2            py39hb6325b5_1    conda-forge
scipy                     1.10.1           py39hfbf2dce_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   1.7.1            py39haadaec5_5    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.43.0               hcfcfb64_0    conda-forge
tbb                       2021.9.0             h91493d7_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tiledb                    2.3.4                h78dabda_0    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodedata2              15.1.0           py39ha55989b_0    conda-forge
uriparser                 0.9.7                h1537add_1    conda-forge
urllib3                   2.0.6              pyhd8ed1ab_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
xarray                    2023.1.0           pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.4                h63175ca_3    conda-forge
xorg-libxau               1.0.11               hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hcfcfb64_5    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge

djhoese · 2023-10-17T14:29:52Z

Thank you. Are you activating your conda environment before using it?

djhoese · 2023-10-17T14:31:15Z

Side note: I'm a little surprised you're getting such an old version of pyproj, although that's not the problem here most likely. When you created your environment did you specify any version limits for any of your dependencies?

DOptimusPrime · 2023-10-17T14:35:12Z

Yes. This is what I needed to run my code so pyproj came as a dependency:
networkx== 2.6.2
shapely== 1.7.1
geopandas== 0.9.0
numpy==1.20.3
That is why my version of pyproj is so old.

djhoese · 2023-10-17T14:42:59Z

Is that "Yes" to the activation question? The various PROJ directory environment variables are meant to be set on environment activation.

DOptimusPrime · 2023-10-17T14:52:52Z

Yes. I think because I installed an older version version of geopandas it came with an older version of proj. I don't know if I answered your question correctly

djhoese · 2023-10-17T14:57:28Z

Ok, so at some point you are running conda activate chur2? Ok. There is a small-ish chance that something has been fixed since this older version of pyproj in the conda-forge builds, but it is hard to say. I didn't see anything obvious in the commit history on conda-forge. Let me see if I can reproduce the warnings on my linux box. It could also be that I'm just completely missing something so @snowman2 could probably answer this too.

DOptimusPrime · 2023-10-17T15:03:00Z

I run the following:
conda activate chur2
conda install -c conda-forge networkx== 2.6.2 shapely== 1.7.1 geopandas== 0.9.0 numpy== 1.20.3 fiona netcdf4
That was how I isntalled my packages.
Thank you for your help

djhoese · 2023-10-17T15:13:22Z

If you run python -c "import pyproj" do you see the warning?

Edit: I'm trying to figure out the smallest amount of code to reproduce the warning. I'm not seeing it on linux with these package versions, but it wouldn't surprise me that Windows builds have an issue that linux doesn't.

DOptimusPrime · 2023-10-17T15:23:23Z

It works fine:
(chur2) C:\Users\hdela>python -c "import pyproj"

(chur2) C:\Users\hdela>python
Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:39:05) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import pyproj

djhoese · 2023-10-17T15:32:17Z

Well the warning you were getting is supposed to show up in pyproj/__init__.py when you import it so I'm not sure why you would stop getting it now. You could do:

python -c "import warnings; warnings.simplefilter('error'); import pyproj"

but I wouldn't guess it would be a problem. So perhaps one of the other packages is interfering with pyproj? What order do you import things in your script? Also, it looks like you might be using some multiprocessing, is that right?

DOptimusPrime · 2023-10-17T15:40:38Z

Yes I am using some multiprocessing. These are the other import statement:
import sys

import numpy as np
import pandas as pd
import csv
import pickle
import math
import os
import json
import uuid
import shutil
import threading

from scipy.stats import norm
from time import strftime, time
import time
import cProfile, pstats, io
from pstats import SortKey

from osgeo.gdalconst import *
from osgeo import osr, ogr, gdal
import geopandas as gpd
import subprocess
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from mpl_toolkits.axes_grid1.inset_locator import inset_axes

import matplotlib as mpl

from fiona import collection, open as fiona_open
from shapely.geometry import mapping, shape, Point, LineString

DOptimusPrime · 2023-10-17T16:13:08Z

Do you think is it that warning that is making python to look for that folder on my old machine.
The last line of the error message where FileNotFoundError: [Errno 2] No such file or directory: b'C:/Users/dhoussou/chur/Chur/Engine/Data/SourceData/precipitation/MeteoSwiss/RdisaggH_ch01r.swisscors_200703010100_200704010000.nc'
That's the path on my old machine. I do not think but I am using a new machine so I am confused why that is? Do you think it has something to do with PROJ. I have looked through my code and I don't think I provided any path to my old machine so I am a bit confused.

djhoese · 2023-10-17T17:27:30Z

Did you copy your old machine's environment to your new machine?

No I don't think that warning has anything to do with your incorrect netcdf path being used besides that both issues could be pointing to the same problem: something got copied or configured incorrectly when switching to the new machine.

djhoese · 2023-10-17T17:30:35Z

Note: Even with that set of imports in a python file and with:

import warnings
warnings.simplefilter("always")

I don't see a pyproj warning on my linux environment.

snowman2 · 2024-08-27T03:09:34Z

Providing a Minimal Reproducible Example may help to debug the issue: https://stackoverflow.com/help/minimal-reproducible-example

DOptimusPrime added the installation-issues Issue related to installation problems. label Oct 16, 2023

snowman2 added the windows Windows related issue label Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UserWarning: Valid PROJ data directory not found #1350

UserWarning: Valid PROJ data directory not found #1350

DOptimusPrime commented Oct 16, 2023 •

edited by djhoese

Loading

djhoese commented Oct 16, 2023

DOptimusPrime commented Oct 17, 2023 •

edited by djhoese

Loading

djhoese commented Oct 17, 2023

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

djhoese commented Oct 17, 2023 •

edited

Loading

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

DOptimusPrime commented Oct 17, 2023

djhoese commented Oct 17, 2023

djhoese commented Oct 17, 2023

snowman2 commented Aug 27, 2024

UserWarning: Valid PROJ data directory not found #1350

UserWarning: Valid PROJ data directory not found #1350

Comments

DOptimusPrime commented Oct 16, 2023 • edited by djhoese Loading

djhoese commented Oct 16, 2023

DOptimusPrime commented Oct 17, 2023 • edited by djhoese Loading

djhoese commented Oct 17, 2023

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 • edited Loading

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 • edited Loading

djhoese commented Oct 17, 2023 • edited Loading

DOptimusPrime commented Oct 17, 2023 • edited Loading

djhoese commented Oct 17, 2023

DOptimusPrime commented Oct 17, 2023 • edited Loading

DOptimusPrime commented Oct 17, 2023

djhoese commented Oct 17, 2023

djhoese commented Oct 17, 2023

snowman2 commented Aug 27, 2024

DOptimusPrime commented Oct 16, 2023 •

edited by djhoese

Loading

DOptimusPrime commented Oct 17, 2023 •

edited by djhoese

Loading

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

djhoese commented Oct 17, 2023 •

edited

Loading

DOptimusPrime commented Oct 17, 2023 •

edited

Loading

DOptimusPrime commented Oct 17, 2023 •

edited

Loading