Document arff-trace for parameter sweeps

[mfeurer]

Currently, there is no documentation of how the optimization trace of a parameter sweep is stored. To the best of my knowledge, this file describes it:

http://www.openml.org/data/download/2035727/optimization_trace6946920946676621819.arff

And some documentation, pasted from an email from @janvanrijn

The following fields are mandatory: repeat, fold, iteration, evaluation and selected. Furthermore, all fields that involve an optimized parameter are indicated with the prefix "parameter_" (in line with my email from May 24). All other fields are facultative.

[mfeurer]

Also, it would be great to have documented on how to save the runtime of individual models, as would be interesting in other contexts (such as openml/openml-python#1038 (comment))

[janvanrijn]

Hi Matthias,

Thanks for reminding me of this. The current documentation is far from perfect, but I am happy to put in an effort to improve it.

• I found this old document, that consists of a description of the run uploads: https://github.com/openml/OpenML/wiki/Run-Uploading (note that this document is no longer visible to users)
• Additionally, this is what is now documented (it misses some information, and got some information wrong from the top of my head): https://docs.openml.org/OpenML_definition/ -> Run -> Trace

My proposal would be the following:

• I can integrate the information from the currently hidden document into the currently public document
• I can add a link to an example trace document

Any other suggestions?

[mfeurer]

That sounds like a good proposal.

[joaquinvanschoren]

Hi all,

Did you see the API docs? https://new.openml.org/api
Under run: /run/trace/{id}
I do notice now that the example has a broken link.

More documentation is always better, but I would at least cross-link them?

[janvanrijn]

Hi Joaquin, Thanks for this new documentation, but there seems to be a broken link: https://new.openml.org/api/v1/xml_example/run.trace I think this documentation also doesn't describe the arff file that needs to be uploaded. I will add a short description at the docs.openml.org site, presuming that this documentation will also be available on the new website? Best, Jan Best, Jan Op za 10 apr. 2021 om 00:16 schreef Joaquin Vanschoren < ***@***.***>:

…

Hi all, Did you see the API docs? https://new.openml.org/api Under run: /run/trace/{id} I do notice now that the example has a broken link. More documentation is always better, but I would at least cross-link them? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#413 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARPX6QALDOJIFUGY7MFRRLTH54EZANCNFSM4DKDHNZA> .

[janvanrijn]

I created two PR's, the one above (which should be evaluated first), and then one on the documentation repo (which could be reviewed after)

Happy to hear your suggestions

[mfeurer]

Hey, I just checked https://github.com/openml/docs/pull/15/files and the links aren't working yet. Does there need to be a release of the server for them to appear?

[mfeurer]

The documentation PR openml/docs#15 just got merged. @joaquinvanschoren who can update the new API docs to add a link to the new description?