We should set up a repository that automatically cross-compares the outputs of multiple (all?) CO2 calculators whenever they are updated?
GitHub actions probably a good platform? For example, this is how we do it for KGen.
3 Likes
Nice idea! I think that these tests should probably run only on a new version release rather than testing every commit(?), so this doesn’t replace the CI tests for the separate tools. Which many of them don’t have, so a first step could be writing a standard set of tests and implementing them for the different tools separately.
1 Like
Definitely doesn’t replace diagnostic tests within individual packages.
I was thinking of this as a set of higher-level checks (e.g. running the over-constrained GLODAP data? Replicating the tests in Orr et al., 2015?) that could be run every time a new version is released, to allow both the inter-comparison of packages, as well as a ‘performance history’ of each individual package?
@mvdh shall I set up a scaffold for this in the SystemCO2 GitHub?
Please do!
There are a bunch of test files which might be useful for this in the existing PyCO2SYS CI suite, though they are a bit scattered around and not brilliantly documented… here they are anyway:
This overlaps with an idea I’d like to get funded which has the same idea but a larger scope - a site for validating calculation codes of any kind. It provides a standard dataset which you download, process it, and upload the result - then it will tell you whether your calculation got the right answer.
For this smaller community it should be possible to build an automatic tool as you suggest, but I see potential synergies…
1 Like
Sounds great! Perhaps this could be the seed that grows into something larger.
Hoping to start getting the infrastructure together later this week… but I’m always a time optimist!
hi, incase its useful, we maintain a scientific Python package that is also on github for calculating air-sea fluxes (its configurable for multiple gases and momentum and called FluxEngine). We have a traceable dataset that is included within the python install package and users are then guided to run a verification script when they install the main python package. the verification run tests a standard output against this reference dataset to confirm everything has installed as expected (its also one of the tests we run to check a new versions before they are released).
only mentioning here incase its useful as our python package is not relevant for your carbonate system intercomparison effort (but we do use PyCO2sys for aspects of studying the gas fluxes and surface waters and its one of the dependencies for installing FluxEngine).
1 Like
Thanks for pointing this out Jamie, it’s a nice idea, an extra level of testing beyond what we had currently thought of. Will take a look at how this implemented in FluxEngine and see if we can do something similar.
I have no time estimate on my idea, so no pressure from me! But happy to be in the conversation.