Problem rendering run.txt in webbrowser and using docker push


while working on step 2 of milestone 2 ( we encountered a problem trying to render our local run.txt in the webbrowser as a html file.

Upon running the following tira-run command in the command line we were experiencing the error below.

tira-run --output-directory ${PWD}/bm25-output --image python-never-heard-of-qrels --allow-network true --command “diffir --dataset iranthology-python-never-heard-of --web $outputDir/run.txt > $outputDir/run.html”

Traceback (most recent call last):
File “/usr/bin/diffir”, line 8, in
File “/usr/lib/python3.8/site-packages/diffir/”, line 71, in main
diff(args.runfiles, config, cli=args.cli, web=args.web)
File “/usr/lib/python3.8/site-packages/diffir/”, line 86, in diff
html = task.web(runs)
File “/usr/lib/python3.8/site-packages/diffir/”, line 564, in web
json_data = self.json(*runs)
File “/usr/lib/python3.8/site-packages/diffir/”, line 412, in json
diff_query_objects = self.create_query_objects(
File “/usr/lib/python3.8/site-packages/diffir/”, line 138, in
qrels = dataset.qrels_dict()
File “/usr/lib/python3.8/site-packages/ir_datasets/formats/”, line 193,
in qrels_dict
for qrel in qrels_handler.qrels_iter():
File “/usr/lib/python3.8/site-packages/ir_datasets/formats/”, line 433,
in qrels_iter
yield from self._qrels_internal_iter(self._qrels_dlc)
File “/usr/lib/python3.8/site-packages/ir_datasets/formats/”, line 449,
in _qrels_internal_iter
raise RuntimeError(f’expected 4 columns, got {len(cols)}')
RuntimeError: expected 4 columns, got 6

It was expecting four columns of input but apparently was given 6. Is this due to our run.txt having 6 different values in each line? What are we supposed to do now?

Our run.txt looks like this (I only copied the first three of the many, many lines)

1 0 2016.sigirconf_conference-2016.224 1 20.548162321552805 BM25 1 0 2013.sigirconf_conference-2013.184 2 20.44424889331691 BM25 1 0 2009.cikm_conference-2009.1 3 20.42082291927 BM25

Can you also please confirm that our next steps would be to evaluate the effectiveness of our relevance judgments and sending you the qrels as well as the effectiveness score.

Best regards

Python never heard of him

Dear @Risi,

I think the problem is not the run file, it instead looks like the problem is in the qrels file.
The Stacktrace has lines like for qrel in qrels_handler.qrels_iter() in it, and than it says that it expected 4 columns, but got 6, so I guess the formatting error is not in the run file, but in the qrels.

Can you share an example of your qrels here?
(Or send me the link to your git repository, possibly in a private message.)

And yes, when the qrels are finished, you use your relevance judgments that are persisted in the qrels file to evaluate the effectiveness of your baseline retrieval system, and you then send the qrels and the effectiveness score.

Thanks and best regards,


Dear Maik,

thank you for your answer. You were completely right and to be totally honest I do not know how it was even possible for me to miss that. There was an error in our qrels file where a line was seperated into six elements.

Best regards

Dear Maik,

unfortunately one last error arose at the very last step.
Even after successfully logging in to docker my docker push request

docker push

gets denied with

5c661f4f9c2a: Preparing da407e07afe8: Preparing 78e273bf7706: Preparing 8bf90768ad67: Preparing 357c7a353b0b: Preparing 78500dd1c866: Waiting bf76138dca99: Waiting 714358772bec: Waiting 25dacf94f94f: Waiting c76370b76bd8: Waiting 95129a5fe07e: Waiting denied: requested access to the resource is denied

and thus not allowing us to select our docker image while trying to create the docker submission. I tried completely restarting docker, the console and my system to no avail. What could have gone wrong?

Best regards.

Dear Maurice,

Have you used the docker login command that is shown a your personal submission page under “Docker Submission” => “Upload Images”?

I dobule-checked that the login command there works.
Maybe the authentication was expired for some reason and you have to login again?

If you have further problems with the login, we best solve this via private messages.

Best Regards,


Dear Maik,

the command does in fact work. There seemed to be a problem with some rights of the Linux Subsystem. We could solve this by forcing a reload while docker was open. There are no further questions. :slight_smile:

Best regards

Dear Maurice,

That’s perfect, thanks for your update!

Just out of curiosity in case others face the same problem: What do you mean with forcing a reload while docker is open? (re-executing the docker login command?)

Best regards,


Dear Maik,

I had to fully reload the Linux Subsystem and therefore forceclosing docker. After reopening docker everything worked fine.

Best regards

1 Like