Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble Exporting to netcdf #98

Open
JonathonMSmith opened this issue Sep 9, 2022 · 4 comments
Open

Trouble Exporting to netcdf #98

JonathonMSmith opened this issue Sep 9, 2022 · 4 comments

Comments

@JonathonMSmith
Copy link
Collaborator

Describe the bug
After creating an sgp4 instrument with missions I have tried to save to netcdf.
When I use the to_netcdf function I get the error

AttributeError: NetCDF: Not a valid data type or _FillValue type mismatch

I tried saving a cnofs ivm instrument with to_netcdf and got a different error, but I was still unable to save it.

I'm wondering if this is because I'm not setting the kwargs correctly.

To Reproduce

Python 3.9.13 (main, Aug 25 2022, 18:29:29) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pysat

In [2]: sat1 = pysat.Instrument('missions', 'sgp4')

In [3]: sat1.load(2018,1)
<ipython-input-3-46da04f1f5b8>:1: DeprecationWarning: Meta now contains a class for global metadata (MetaHeader). Default attachment of global attributes to Instrument will be Deprecated in pysat 3.2.0+. Set `use_header=True` in this load call or on Instrument instantiation to remove this warning.
  sat1.load(2018,1)

In [4]: sat1.to_netcdf4('/Users/jmsmit37/sat1_30_degrees.nc')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [4], line 1
----> 1 sat1.to_netcdf4('/Users/jmsmit37/sat1_30_degrees.nc')

File ~/anaconda3/envs/missions/lib/python3.9/site-packages/pysat/_instrument.py:3618, in Instrument.to_netcdf4(self, fname, base_instrument, epoch_name, zlib, complevel, shuffle, preserve_meta_case, export_nan, unlimited_time, modify)
   3615 inst = self if modify else self.copy()
   3617 # Write the output file
-> 3618 pysat.utils.io.inst_to_netcdf(inst, fname=fname,
   3619                               base_instrument=base_instrument,
   3620                               epoch_name=epoch_name, zlib=zlib,
   3621                               complevel=complevel, shuffle=shuffle,
   3622                               preserve_meta_case=preserve_meta_case,
   3623                               export_nan=export_nan,
   3624                               unlimited_time=unlimited_time)
   3626 return

File ~/anaconda3/envs/missions/lib/python3.9/site-packages/pysat/utils/io.py:1816, in inst_to_netcdf(inst, fname, base_instrument, epoch_name, mode, zlib, complevel, shuffle, preserve_meta_case, check_type, export_nan, unlimited_time, meta_translation, meta_processor)
   1809 cdfkey = out_data.createVariable(epoch_name, 'i8',
   1810                                  dimensions=(epoch_name),
   1811                                  zlib=zlib,
   1812                                  complevel=complevel,
   1813                                  shuffle=shuffle)
   1815 # Attach epoch metadata
-> 1816 cdfkey.setncatts(export_meta[epoch_name])
   1818 # Attach the time index to the data
   1819 cdfkey[:] = (inst.index.values.astype(np.int64)
   1820              * 1.0E-6).astype(np.int64)

File src/netCDF4/_netCDF4.pyx:4408, in netCDF4._netCDF4.Variable.setncatts()

File src/netCDF4/_netCDF4.pyx:1694, in netCDF4._netCDF4._set_att()

File src/netCDF4/_netCDF4.pyx:1965, in netCDF4._netCDF4._ensure_nc_success()

AttributeError: NetCDF: Not a valid data type or _FillValue type mismatch

Expected behavior
netcdf file is produced from the instrument

Desktop (please complete the following information):

  • OS: Mac OS Monterey
  • Version 12.5
  • python version 3.9.13
  • pysat version 3.0.4
  • pysatMissions version 0.3.3
  • netCDF4 version 1.6.0

Additional context
All the dependencies for pysatMissions were installed from the git repo using
pip install -r requirements.txt

apexpy and aacgmv2 were installed with pip

@jklenzing
Copy link
Member

Linking to pysat/pysat#585

Adding output file tests in the core code will help maintain support.

@jklenzing
Copy link
Member

If you add the epoch_name='time', it works. I think this is a bug in pysat, since it assumes the epoch_name in the instrument is 'Epoch', but will not let you write with that name.

@jklenzing
Copy link
Member

lol, just ran into this bug again.

@jklenzing
Copy link
Member

TODO: test other modules, then transfer issue if root cause in core package

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants