Problem rendering run.txt in webbrowser and using docker push

Risi · June 3, 2023, 1:07pm

Hey,

while working on step 2 of milestone 2 (https://github.com/mam10eks/ir-lab-sose-2023/tree/main/milestone-02) we encountered a problem trying to render our local run.txt in the webbrowser as a html file.

Upon running the following tira-run command in the command line we were experiencing the error below.

tira-run --output-directory ${PWD}/bm25-output --image python-never-heard-of-qrels --allow-network true --command “diffir --dataset iranthology-python-never-heard-of --web $outputDir/run.txt > $outputDir/run.html”

1 Traceback (most recent call last): File “/usr/bin/diffir”, line 8, in sys.exit(main()) File “/usr/lib/python3.8/site-packages/diffir/run.py”, line 71, in main diff(args.runfiles, config, cli=args.cli, web=args.web) File “/usr/lib/python3.8/site-packages/diffir/run.py”, line 86, in diff html = task.web(runs) File “/usr/lib/python3.8/site-packages/diffir/run.py”, line 564, in web json_data = self.json(*runs) File “/usr/lib/python3.8/site-packages/diffir/run.py”, line 412, in json diff_query_objects = self.create_query_objects( File “/usr/lib/python3.8/site-packages/diffir/run.py”, line 138, in create_query_objects qrels = dataset.qrels_dict() File “/usr/lib/python3.8/site-packages/ir_datasets/formats/base.py”, line 193, in qrels_dict for qrel in qrels_handler.qrels_iter(): File “/usr/lib/python3.8/site-packages/ir_datasets/formats/trec.py”, line 433, in qrels_iter yield from self._qrels_internal_iter(self._qrels_dlc) File “/usr/lib/python3.8/site-packages/ir_datasets/formats/trec.py”, line 449, in _qrels_internal_iter raise RuntimeError(f’expected 4 columns, got {len(cols)}') RuntimeError: expected 4 columns, got 6

It was expecting four columns of input but apparently was given 6. Is this due to our run.txt having 6 different values in each line? What are we supposed to do now?

Our run.txt looks like this (I only copied the first three of the many, many lines)


1 0 2016.sigirconf_conference-2016.224 1 20.548162321552805 BM25
1 0 2013.sigirconf_conference-2013.184 2 20.44424889331691 BM25
1 0 2009.cikm_conference-2009.1 3 20.42082291927 BM25

Can you also please confirm that our next steps would be to evaluate the effectiveness of our relevance judgments and sending you the qrels as well as the effectiveness score.

Best regards

Python never heard of him

maik_froebe · June 3, 2023, 4:11pm

Dear @Risi,

I think the problem is not the run file, it instead looks like the problem is in the qrels file.
The Stacktrace has lines like for qrel in qrels_handler.qrels_iter() in it, and than it says that it expected 4 columns, but got 6, so I guess the formatting error is not in the run file, but in the qrels.

Can you share an example of your qrels here?
(Or send me the link to your git repository, possibly in a private message.)

And yes, when the qrels are finished, you use your relevance judgments that are persisted in the qrels file to evaluate the effectiveness of your baseline retrieval system, and you then send the qrels and the effectiveness score.

Thanks and best regards,

Maik

Risi · June 3, 2023, 5:41pm

Dear Maik,

thank you for your answer. You were completely right and to be totally honest I do not know how it was even possible for me to miss that. There was an error in our qrels file where a line was seperated into six elements.

Best regards

Risi · June 3, 2023, 6:06pm

Dear Maik,

unfortunately one last error arose at the very last step.
Even after successfully logging in to docker my docker push request


docker push registry.webis.de/code-research/tira/tira-user-ir-lab-sose-2023-/milestone-02:0.0.1

gets denied with


5c661f4f9c2a: Preparing
da407e07afe8: Preparing
78e273bf7706: Preparing
8bf90768ad67: Preparing
357c7a353b0b: Preparing
78500dd1c866: Waiting
bf76138dca99: Waiting
714358772bec: Waiting
25dacf94f94f: Waiting
c76370b76bd8: Waiting
95129a5fe07e: Waiting
denied: requested access to the resource is denied

and thus not allowing us to select our docker image while trying to create the docker submission. I tried completely restarting docker, the console and my system to no avail. What could have gone wrong?

Best regards.

maik_froebe · June 4, 2023, 9:57am

Dear Maurice,

Have you used the docker login command that is shown a your personal submission page under “Docker Submission” => “Upload Images”?

I dobule-checked that the login command there works.
Maybe the authentication was expired for some reason and you have to login again?

If you have further problems with the login, we best solve this via private messages.

Best Regards,

Maik

Risi · June 4, 2023, 11:44am

Dear Maik,

the command does in fact work. There seemed to be a problem with some rights of the Linux Subsystem. We could solve this by forcing a reload while docker was open. There are no further questions.

Best regards

maik_froebe · June 4, 2023, 6:59pm

Dear Maurice,

That’s perfect, thanks for your update!

Just out of curiosity in case others face the same problem: What do you mean with forcing a reload while docker is open? (re-executing the docker login command?)

Best regards,

Maik

Risi · June 5, 2023, 10:37pm

Dear Maik,

I had to fully reload the Linux Subsystem and therefore forceclosing docker. After reopening docker everything worked fine.

Best regards