Image Retrieval for Arguments Dataset 23

Hi, im trying to run my docker container on the 2023 Dataset for the Task “Image Retrieval for Arguments”. Sadly, its missing. And none of the validation datasets contain image-text.txt files. How should I submit our software?

Kind Regards,

Hi, I am sorry for the delay. I am now copying all files. The dataset should be ready in about half a day.

I copied the dataset, but it seems the current version of TIRA can not handle the size. Can you tell me which of the input files you are using, so that I can, for now, reduce the dataset size so that you can at least submit while we are looking for a permanent solution?

Thank you. We only access the topics file and for each image: image.webp, text.txt and image-text.txt

Hi, the TIRA team now fixed the issue regarding large datasets (or rather, as it turned out, datasets with many files).

I now updated the datasets in TIRA:

  • main-2023 (full rank): this is the dataset you can use for submission
  • toy-2023 (full rank): this is the same dataset, but with only 3 topics and only 23 images, so you can use that to test your approach.

For both datasets a validator script is run at the end to check the output format. You should then be able to see in TIRA something like this when you click on “EVALUATION”: