Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiment API #53

Open
michelole opened this issue Jun 14, 2017 · 2 comments
Open

Experiment API #53

michelole opened this issue Jun 14, 2017 · 2 comments

Comments

@michelole
Copy link
Member

It would be nice to have an easy way to run lots of experiments (maybe in parallel?) and get the relevant metrics in a spreadsheet.

@michelole
Copy link
Member Author

@plopezgarcia suggested the following in a email:

As I suggested a few weeks ago, at some point the code should freeze, and every experiment should have input parameters that should come from a file stored in the repository for reference and tracking, which represents a specific experiment, like:

experiment001.json:

"{geneExpansion:true,
addTreatmentKeywords:true,
addSexDemographics:true,
translateDemographicsToAgeGroup:true,

....}"

or something of that sort.

This will enable us to run all sorts of experiments and variations in one go, analyze, and plot the results as desired and which whatever tools we want, and finally pick the experiments that provide the best results.?

Ideally, we should have two input experiment folders, one for the automatic runs, and one for the manual runs, which can even be directly the elastic search query if Stefan/Zdenko are willing to curate them really accurately.

@michelole michelole added the P0 label Jul 7, 2017
michelole pushed a commit to michelole/trec2017 that referenced this issue Jul 14, 2017
michelole pushed a commit to michelole/trec2017 that referenced this issue Jul 14, 2017
ExperimentsBuilder eases the creation of a set of experiments with a
fluent interface and syntactic sugar for decorators.

This relates to bst-mug#53.
@michelole michelole added P1 and removed P0 P1 labels Jul 24, 2017
@michelole
Copy link
Member Author

TODO/Wishlist:

  • addResults() deprecated
  • read from JSON
  • parallel processing / thread-safe
  • hash Experiment and store all results with unique ID
  • CSVWriter for experiments
  • expose TrecEval @ Experiment class
  • ExperimentBuilder should provide a single experiment
  • ExperimentSetBuilder provides related experiments
  • validation rules

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant