-
Notifications
You must be signed in to change notification settings - Fork 21
ESGF Publication Test
Verify you have a test project in /esg/config/esgcet/esg.ini, eg.
project_options =
cmip5 | CMIP5 | 1
ipcc4 | IPCC Fourth Assessment Report | 2
test | Test Project | 3
the default dataroot is set up:
thredds_dataset_roots =
esg_dataroot | /esg/data
and you have /esg/config/esgcet/esg.test.ini (if not you can fetch from github ESGF/config repo).
Make sure that in esg.ini you either comment/remove the Globus service line or replace the #DEFAULTENDPOINT# with your actual endpoint uuid. If you don't know yet, then best to comment/remove the line.
(run as root for basic publishing test):
source /usr/local/conda/bin/activate esgf-pub
esgtest_publish # additonal instructions may follow
Download test dataset
mkdir /esg/data/test; cd /esg/data/test
wget -O sftlf.nc http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/externals/sftlf.nc
Execution user
Must be different from root. Must have recursive write permission to /esg/content/thredds/esgcet
groupadd publication
useradd -G publication esg-user
chown -R tomcat.publication /esg/content/thredds/esgcet/
chmod -R g+rw /esg/content/thredds/esgcet/
Mapfile generation
su esg-user
source /usr/local/conda/bin/activate esgf-pub # if you are running esgf v2.4.x or earlier, please upgrade. Otherwise you'll be missing Conda
If you have a newer (>=3) of the publisher, use esgprep. Automatically names your mapfile to the dataset name + .map The esgprep command to generate the mapfile is additionally version dependent. In esgprep >=2.8 the command line is esgmapfile. Earlier versions use "esgprep mapfile". Note that the dataset name for esgprep must begin with "test".
esgmapfile --dataset test.ipsl.fr.mytest --project test /esg/data/test
--or--
esgprep mapfile --dataset test.ipsl.fr.mytest --project test /esg/data/test
followed by:
mv mapfiles/test.ipsl.fr.mytest.map test_mapfile.txt
Older versions (esg-publisher 2.x.x):
esgscan_directory --dataset ipsl.fr.test.mytest --project test /esg/data/test > test_mapfile.txt
Publication step 1 - Data Node Local Thredds Server
esgpublish --service fileservice --map test_mapfile.txt --project test --thredds
Then check http://my-data-node/thredds/catalog/esgcet/catalog.html
Publication step 2 - Peer Index Node
Authenticate with an ESGF user which has publisher permissions (or rootAdmin). See the CoG site administration guide for managing user permissions: https://www.earthsystemcog.org/projects/cog/esgf_cog_data_access_control_setup
myproxy-logon -T -s [idp_node] -p 7512 -l [login] -o ~/.globus/certificate-file -b
Publish
esgpublish --service fileservice --map test_mapfile.txt --project test --noscan --publish
Then check the cog site - http://my-index-node/ If you are testing a site with both index and data for the first time, please refer to the CoG setup guide:
https://www.earthsystemcog.org/projects/cog/esgf_cog_data_search_configuration
Unpublish
esgunpublish --map test_mapfile.txt --database-delete