Image Retrieval for Arguments Dataset 23

Touché
1

Hi, im trying to run my docker container on the 2023 Dataset for the Task “Image Retrieval for Arguments”. Sadly, its missing. And none of the validation datasets contain image-text.txt files. How should I submit our software?

Kind Regards,
Maximilian

2

Hi, I am sorry for the delay. I am now copying all files. The dataset should be ready in about half a day.

3

I copied the dataset, but it seems the current version of TIRA can not handle the size. Can you tell me which of the input files you are using, so that I can, for now, reduce the dataset size so that you can at least submit while we are looking for a permanent solution?

4

Thank you. We only access the topics file and for each image: image.webp, text.txt and image-text.txt

5

Hi, the TIRA team now fixed the issue regarding large datasets (or rather, as it turned out, datasets with many files).

I now updated the datasets in TIRA:

  • main-2023 (full rank): this is the dataset you can use for submission
  • toy-2023 (full rank): this is the same dataset, but with only 3 topics and only 23 images, so you can use that to test your approach.

For both datasets a validator script is run at the end to check the output format. You should then be able to see in TIRA something like this when you click on “EVALUATION”:

image
image918×269 17.9 KB