Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update perfect scores statistics groups with the new statistics #222

Open
21 tasks
TatianaBurek opened this issue Oct 25, 2022 · 0 comments
Open
21 tasks
Assignees
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle requestor: METplus Team METplus Development Team type: enhancement Improve something that it is currently doing

Comments

@TatianaBurek
Copy link
Collaborator

Describe the Enhancement

MET added a few statistics like CRPS but these statistics were never included into perfect scores statistics groups for adjusting the perfect score. Without this , the calculation of the scorecard can be invalid.
Need to:

  • identify missing statistics in perfect_score_adjustment method from utils.py
  • for the each new statistic identify the group using John's suggestion:

As for CRPS, this section in Appendix C contains a description of it. Near the bottom, it states:
The score can be interpreted as a continuous version of the mean absolute error (MAE). Thus, the score is negatively oriented, so smaller is better. Further, similar to MAE, bias will inflate the CRPS. Thus, bias should also be calculated and considered when judging forecast quality using CRPS.
So, like MAE, 0 is a perfect score for CRPS. Appendix C is definitely the best place to look for the interpretation of stats, including the definition of what's "perfect".

  • include the new statistics into the corresponding group

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Linked issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@TatianaBurek TatianaBurek added type: enhancement Improve something that it is currently doing alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle requestor: METplus Team METplus Development Team labels Oct 25, 2022
@TatianaBurek TatianaBurek added this to the METcalcpy-2.0 milestone Oct 25, 2022
@TatianaBurek TatianaBurek self-assigned this Oct 25, 2022
@bikegeek bikegeek modified the milestones: METcalcpy-2.0, METcalcpy 2.1.0-beta2 Mar 1, 2023
@JohnHalleyGotway JohnHalleyGotway modified the milestones: METcalcpy 2.1.0-beta2, METcalcpy 2.1.0 Mar 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle requestor: METplus Team METplus Development Team type: enhancement Improve something that it is currently doing
Projects
No open projects
Development

No branches or pull requests

3 participants