Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non-reproducible SSP with NorESM2.0.9 #588

Open
mvdebolskiy opened this issue Nov 8, 2024 · 26 comments
Open

Non-reproducible SSP with NorESM2.0.9 #588

mvdebolskiy opened this issue Nov 8, 2024 · 26 comments
Assignees
@DirkOlivie @TomasTorsvik @mvdebolskiy
Labels
bug Something isn't working

Comments

@mvdebolskiy
Copy link

Describe the bug
Please provide a clear and concise description of what the bug is.

  • NorESM version: release-noresm2.0.9
  • HPC platform: betzy, fram
  • Compiler (if applicable): intel
  • Compset (if applicable): NSSP585frc2
  • Resolution (if applicable): f19_tn14
  • Error message (if applicable): None

To Reproduce
Steps to reproduce the behavior:

  1. Follow instructions here
  2. set STOP_N=1,STOP_OPTION=nyears,REST_N=1,REST_OPTION=nyears,RESUBMIT=3
  3. set ./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=02:30:00
  4. submit case
  5. Finish run
  6. ncdiff history

Expected behavior
The results should be B4B or at least roundoff with /nird/projects/NS9560K/noresm/cases/NSSP585frc2_f19_tn14_20191014/.

Screenshots
image
image

Additional context

@adagj copied me to /cluster/projects/nn2345k/olivie/cases-cmip6/NSSP585frc2_f19_tn14_20191014 for me and I've checked PElayouts and env_*.xml to match (apart from batch and mach_specific).
@mvdebolskiy mvdebolskiy added the bug Something isn't working label Nov 8, 2024
@TomasTorsvik
Copy link
Contributor

@mvdebolskiy - thanks for reporting, I didn't look at SSP runs when testing.
Could be useful to know if this is specific to SSP585, all SSP simulations or something more general.
I don't have time for testing now, will try to look at this over the weekend.

@oyvindseland
Copy link

Did you run it on Fram or Betzy?

@mvdebolskiy
Copy link
Author

@oyvindseland I ran on both. All are different with the history files on nird.

@DirkOlivie
Copy link
Contributor

The original simulation was on Fram. Has the new one been run with the same number of PEs for the atmosphere (768)?

@mvdebolskiy
Copy link
Author

@DirkOlivie Yes. I ran on fram too.
I used your case as a reference. diff between your env_mach_pes.xml and the one in my case says 0 differences.

@mvdebolskiy
Copy link
Author

Also, ran N1850 and it is also has differences with history on nird. I wonder if I am doing something wrong.

@mvertens
Copy link

mvertens commented Nov 8, 2024

Were there SoureMods or xml changes that were used in the runs on NIRD? Are the case directories still around?

@mvdebolskiy
Copy link
Author

NSSP585frc2_f19_tn14_20191014 uses MICOM2 instead of blom.
Main differences in namelists are in ocn_in.
atm_in for release-noresm2.0.9 has aerosol tables while NSSP585frc2_f19_tn14_20191014 does not have them.
This is Dirk's old case's readme.

2019-10-13 21:26:53: ./create_newcase --case /cluster/projects/nn2345k/olivie/cases-cmip6/NSSP585frc2_f19_tn14_20191014 --res f19_tn14 --compset NSSP585frc2 --mach fram --project nn2345k --user-mods-dir cmip6_noresm_hifreq_xaer --run-unsupported
 ---------------------------------------------------
2019-10-13 21:26:53: Compset longname is SSP585_CAM60%NORESM%FRC2_CLM50%BGC-CROP_CICE%NORESM-CMIP6_MICOM%ECO_MOSART_SGLC_SWAV_BGC%BDRDDMS
 ---------------------------------------------------
2019-10-13 21:26:53: Compset specification file is /cluster/projects/nn2345k/olivie/noresm-cmip6/noresm2-20191012-scenarios/cime_config/config_compsets.xml
 ---------------------------------------------------
2019-10-13 21:26:53: Pes     specification file is /cluster/projects/nn2345k/olivie/noresm-cmip6/noresm2-20191012-scenarios/cime_config/config_pes.xml
 ---------------------------------------------------
2019-10-13 21:26:53: Forcing is 
 ---------------------------------------------------
2019-10-13 21:26:53: Using None coupler instances
 ---------------------------------------------------
2019-10-13 21:26:53: Component ATM is CAM cam6 physics:
 ---------------------------------------------------
2019-10-13 21:26:53: ATM_GRID is 1.9x2.5
 ---------------------------------------------------
2019-10-13 21:26:53: Component LND is clm5.0:BGC (vert. resol. CN and methane) with prognostic crop:
 ---------------------------------------------------
2019-10-13 21:26:53: LND_GRID is 1.9x2.5
 ---------------------------------------------------
2019-10-13 21:26:53: Component ICE is Sea ICE (cice) model version 5 :with NORESM modifications appropriate for CMIP6 experiments
 ---------------------------------------------------
2019-10-13 21:26:53: ICE_GRID is tnx1v4
 ---------------------------------------------------
2019-10-13 21:26:53: This component includes user_mods /cluster/projects/nn2345k/olivie/noresm-cmip6/noresm2-20191012-scenarios/components/cice/cime_config/usermods_dirs/noresm-cmip6
 ---------------------------------------------------
2019-10-13 21:26:53: Component OCN is MICOM default:MICOM/Ecosystem:
 ---------------------------------------------------
2019-10-13 21:26:53: OCN_GRID is tnx1v4
 ---------------------------------------------------
2019-10-13 21:26:53: Component ROF is MOSART: MOdel for Scale Adaptive River Transport
 ---------------------------------------------------
2019-10-13 21:26:53: ROF_GRID is r05
 ---------------------------------------------------
2019-10-13 21:26:53: Component GLC is Stub glacier (land ice) component
 ---------------------------------------------------
2019-10-13 21:26:53: GLC_GRID is null
 ---------------------------------------------------
2019-10-13 21:26:53: Component WAV is Stub wave component
 ---------------------------------------------------
2019-10-13 21:26:53: WAV_GRID is null
 ---------------------------------------------------
2019-10-13 21:26:53: ESP_GRID is None
 ---------------------------------------------------
2019-10-13 21:26:53: INFORMATION ABOUT YOUR GIT VERSION CONTROL SYSTEM :
 ---------------------------------------------------
2019-10-13 21:26:53: remote branch:origin	https://DirkOlivie@github.com/metno/noresm-dev.git (fetch)
origin	https://DirkOlivie@github.com/metno/noresm-dev.git (push)
 ---------------------------------------------------
2019-10-13 21:26:53: git branch:* featureCESM2.1.0-OsloDevelopment 11073fc [origin/featureCESM2.1.0-OsloDevelopment] Merge branch 'featureCESM2.1.0-OsloDevelopment' of https://github.com/metno/noresm-dev into featureCESM2.1.0-OsloDevelopment
  master                           0e21727 [origin/master] settings for parallel IO and processors count for quarter degree ocean with HAMOCC
 ---------------------------------------------------
2019-10-13 21:26:53: git log:commit 11073fc88428f330dc17fb6762a78ce4992c560b
Merge: ede8130 1888282
Author: Alf Kirkevåg <alf.kirkevag@met.no>
Date:   Fri Oct 11 11:33:26 2019 +0200

    Merge branch 'featureCESM2.1.0-OsloDevelopment' of https://github.com/metno/noresm-dev into featureCESM2.1.0-OsloDevelopment
 ---------------------------------------------------

@JorgSchwinger
Copy link
Contributor

@mvdebolskiy @oyvindseland @DirkOlivie @mvertens

I have set-up and tested a scenario that was originally run on betzy in 2021 (NSSP534frc2_f19_tn14_20210427). This is branched from NSSP585frc2_f19_tn14_20191014 (hybrid restart) at 2040-01-01. The original simulation was done with release2.0.5

I get bit-for-bit identical results with release2.0.9, also after a restart.

Did we ever test bit-for-bit reproducibility on fram after there was an upgrade some time ago?

@mvdebolskiy
Copy link
Author

@JorgSchwinger I am not comparing to 2.0.5, but rather to cmip6 simulations made in 2019.

@JorgSchwinger
Copy link
Contributor

Yes, I know but 2.0.5 IS (on of the versions of) the CMIP6 code of the model. It is (should be) bit-for-bit compatible with releases 2.0.0-2.0.4 (the SSP5-3.4 I tested is a CMIP6 simulation)

@mvdebolskiy
Copy link
Author

I am comparing my simulations against what is listed here
And located in here: /projects/NS9560K/noresm/cases.

@JorgSchwinger
Copy link
Contributor

My point is: I'm not sure if we expect to be able to reproduce old fram simulations bit-for-bit? We tested bfb regularly on betzy, and my test shows that that 2.0.9 still gives bfb for CMIP6 simulations run on betzy.

@mvdebolskiy
Copy link
Author

But where I can find new simulations that were done on betzy? Also, are they submitted to ESGF?

@DirkOlivie
Copy link
Contributor

Hi Matvey, a CMIP6 experiment run on Betzy long ago (December 2020) is
/cluster/projects/nn9560k/olivie/cases-cmip6/NF1850norbc_f19_20201226 (the data is stored on nird). Are you looking for something like that?
I can try to rerun it with NorESM2.0.9 on Betzy (not today), but you can also try to if you like. Best regards, Dirk

@oyvindseland
Copy link

Since I had a 2.0.9 set-up up and running I tested the case
/cluster/projects/nn2345k/oyvinds/NorESM2-CMIP6/cases/NF1850norbc_test209_20241112
reproduces Dirk's case BFB.

I do not know why there is a SourceMods subroutine in the case so I copied that file to my case folder as well.

@JorgSchwinger
Copy link
Contributor

The NSSP534frc2_f19_tn14_20210427 is on nird in /projects/NS9560K/noresm/cases/

It has also been published on ESGF (ssp534-over)

(I already tested that it is bfb with release 2-0.9)

@mvdebolskiy
Copy link
Author

The NSSP534frc2_f19_tn14_20210427 is on nird in /projects/NS9560K/noresm/cases/

It has also been published on ESGF (ssp534-over)

(I already tested that it is bfb with release 2-0.9)

Can you point to your case?
Because I suspect that the instructions in the docs which I have followed are not correct.

@JorgSchwinger
Copy link
Contributor

On bezty:

/cluster/projects/nn2345k/schwinger/cases/NSSP534frc2_f19_tn14_20210427

@mvdebolskiy
Copy link
Author

mvdebolskiy commented Nov 19, 2024 •
edited
Loading

I do not have access to nn2345k. Whoever is the PI there has to add me, or you can copy the case to nn9560k.

Also, are you cloning your old cases?

@JorgSchwinger
Copy link
Contributor

Ok, copied:

/cluster/projects/nn9560k/schwinger/cases/NSSP534frc2_f19_tn14_20210427

I didn't use the clone command. I executed the create_newcase command as found in the README.case file.

@oyvindseland
Copy link

Not a scenario, but when I tested Dirk's simulation above I also used create_newcase not clone

@mvdebolskiy
Copy link
Author

mvdebolskiy commented Nov 20, 2024 •
edited
Loading

@JorgSchwinger
I am looking at your case:

grep REFCASE env*

env_run.xml:    <entry id="RUN_REFCASE" value="NSSP585frc2_f19_tn14_20191014">
env_run.xml:    <entry id="GET_REFCASE" value="FALSE">
env_run.xml~:    <entry id="RUN_REFCASE" value="case.std">
env_run.xml~:    <entry id="GET_REFCASE" value="FALSE">

grep REFDATE env*
env_run.xml:    <entry id="RUN_REFDATE" value="2040-01-01">
env_run.xml~:    <entry id="RUN_REFDATE" value="0001-01-01">

I will try that. Can you try to make a case that starts in 2015 from a history run?

@oyvindseland can you put your casedir into /cluster/projects/nn9560/ on fram?

@oyvindseland
Copy link

Copied the case folder to fram /cluster/projects/nn9560k/oyvinds/cases/NF1850norbc_test209_20241112
but I do not know if that specific case have been tested on Fram with earlier versions

@DirkOlivie
Copy link
Contributor

@mvdebolskiy I have now tested SSP585 with NorESM2.0.9 on Fram. I obtain results which are different from the CMIP6 simulation NSSP585frc2_f19_tn14_20191014.

@adagj
Copy link
Contributor

adagj commented Nov 21, 2024

@DirkOlivie
These two were run and FRAM and are BFB:

NSSP585frc2_f09_tn14_20191105
NSSP585frc2_f09_tn14_20200919

But that is a rather long time ago. So what happen between then and now that caused the BFB to fail? Could upgrades on FRAM alone cause that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DirkOlivie DirkOlivie

@TomasTorsvik TomasTorsvik

@mvdebolskiy mvdebolskiy

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mvertens @oyvindseland @adagj @DirkOlivie @JorgSchwinger @TomasTorsvik @mvdebolskiy