Exception with KeyError: 'oml:tag' when fetching benchmarks

[bailuding]

Description

I followed the instruction on https://docs.openml.org/benchmark/#openml100 to fetch the details of benchmarks. but I encountered KeyError: 'oml:tag' for both open ml 100 and openml-cc18.

I use the latest openml package and Python 3.6

Steps/Code to Reproduce

Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import openml
benchmark_suite = openml.study.get_study('OpenML-CC18','tasks')
Traceback (most recent call last):
File "", line 1, in
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 33, in get_study
for tag in _multitag_to_list(result_dict, 'oml:tag'):
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 8, in _multitag_to_list
if isinstance(result_dict[tag], list):
KeyError: 'oml:tag'

Expected Results

No error is thrown.

Actual Results

Traceback (most recent call last):
File "", line 1, in
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 33, in get_study
for tag in _multitag_to_list(result_dict, 'oml:tag'):
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 8, in _multitag_to_list
if isinstance(result_dict[tag], list):
KeyError: 'oml:tag'

Versions

Python 3.6.6, openml 0.8.0

[joaquinvanschoren]

@janvanrijn @mfeurer
For the 'new' studies the tag that identifies the study is no longer returned (and the benchmarking studies are 'new' studies): https://www.openml.org/api/v1/study/99

@janvanrijn: is this by design? Then the openml-python package needs to be updated (and probably other bindings as well). If not then this is likely a bug in the API?

[bailuding]

@janvanrijn @mfeurer
For the 'new' studies the tag that identifies the study is no longer returned (and the benchmarking studies are 'new' studies): https://www.openml.org/api/v1/study/99

@janvanrijn: is this by design? Then the openml-python package needs to be updated (and probably other bindings as well). If not then this is likely a bug in the API?

How can I avoid the exception and get the benchmark in Python then?

[janvanrijn]

Currently, studies do no longer return tags by default (only legacy studies do, as announced in email thread on benchmark suites).

This is indeed a problem in openmlpython not covered by the unit tests. We should update this.

[joaquinvanschoren]

OK, so what is the most practical way to proceed?
Do we make a new minor release of the python API with a fix?
Or do we make the server return the tag until the APIs are updated and released?

[joaquinvanschoren]

I checked and it this change doesn't affect the R API.
@janvanrijn: I assume that you checked the Java API?
I'll make a PR for the python API now. We still need to decide whether we do a minor release of update the REST API.

[mfeurer]

Thanks for checking this @joaquinvanschoren

We can also make a full release of the python API once this is integrated as there have been plenty of changes already. I guess we only need to wait for a fix to #646 prior to doing so to maintain backwards compatibility.

[joaquinvanschoren]

Well, it looks like @janvanrijn already fixed this in commit a2a4ade

So that means that the development version should work

[joaquinvanschoren]

@bailuding To work around this, you can install the development version of the python API until we made a new release: https://openml.github.io/openml-python/master/contributing.html#installation

[janvanrijn]

That PR should be in the latest release already (0.8.0) if I am not mistaken.

[joaquinvanschoren]

Hmm, I don't see it here: #617
So I guess it's not in 0.8.0

I added a PR with the new unit test: #649

[mfeurer]

Fixed via #649. Please wait for the next release @bailuding for a fix on pypi.

[bailuding]

@bailuding To work around this, you can install the development version of the python API until we made a new release: https://openml.github.io/openml-python/master/contributing.html#installation

Hi @joaquinvanschoren

I installed the develop branch, and the 'oml:tag' exception is gone. However, now I got another exception when trying to get the X and y:

'numpy.ndarray' object has no attribute 'columns'
Traceback (most recent call last):
File ".\test.py", line 532, in download_openml_benchmark
X, y = task.get_X_and_y() # get the data (not used in this example)
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\tasks\task.py", line 125, in get_X_and_y
dataset_format='array', target=self.target_name
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\datasets\dataset.py", line 538, in get_data
x = self._convert_array_format(x, dataset_format, attribute_names)
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\datasets\dataset.py", line 378, in _convert_array_format
for column_name in data.columns
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

Do you know how I can work around this?

Package version:
numpy==1.15.0+mkl
openml==0.8.0

I am following this instruction to get the task (the Python example):

https://docs.openml.org/benchmark/#openml100

[joaquinvanschoren]

@bailuding
Looks like this is related to #646

In short, you can't use cached datasets from the 0.8.0 with the development version (at least not now). Workaround is to delete your cache and try again. Most likely you just need to run:

rm -rf ~/.openml/cache