Bringing our environment to modern standards

[valeriupredoi]

This is an umbrella issue that is motivated by our dependencies environment being fairly patchy and out of date; there are some outstaning issues at play:

• the need for NCL to be moved to an "optional dependency": ncl is lagging behind with every Python and co upgrade, specifically for us, it makes it impossible to:
  • move to support Python 3.13 via Support Python 3.13 #3805
  • move to support recent versions of esmpy>=8.6
  • start supporting numpy>=2
  • altogether get rid of importlib_metadata <8 (predicated by older esmpy, predicated, in turn, by NCL)
• unpin curl, hopefully with a correct curl=8.11, caused by the issues it has with Julia installing packages, see Unpin curl #3758
• unpin cartopy, see Unpin cartopy #3769 -> this is done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811 - it is safe to do that since we pin iris there
• unpin pandas - that will be done when we release ESMValCore v2.12 since the fix for pandas is there now
• unpin numpy<2 ie upper bound - iris 3.11 is safe for numpy>=2 - we should free it too, and consider issues with diagnostics not being numpy2 compatible as they come, probably during the release process for v2.12; unpinning upper bound for `numpy done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811
• unpin r-base <4.3 - I can't remember why we are doing that, I need to look into it