Question about JSON run submission

SCAI
1

Hi guys,

we have submitted a couple of runs today on the rewritten questions test dataset and submitted the output json runs to the TIRA system.

We are not sure if the submissions are accepted by the system and if they are correct.

Could you please let us know if our submissions are okay?

Thanks,
Team Ultron

2

Hi there,

Thank you for the submission! We now checked it and you are missing 258 answers (~2% of 15024), but it seems fine otherwise. Can you check your submission again?

Regards,
Johannes

3

Sorry, you said “today” but I just noticed the runs we reviewed were from Saturday. The ones from today you have not evaluated yet. There was an issue with the evaluation machine, but it should work again now. Can you try to evaluate them like you did for your earlier runs?

Thanks!
Johannes

4

Thank you very much for your prompt reply.

I have started the evaluation (clicked on “run”) on the submissions for today.

Can you please check the submissions, if possible?

Regarding the “missing answers”. I checked, we have added “none” for answers where the model couldn’t generate any meaningful text. Is that fine?

Also, for the submission files, I couldn’t see the name of the files we submitted, it
s just “submission N”. It would be hard to map these to the corresponding model later. Is there any way to check the original file name? Or should I re-submit again with a mapping from my side?

Please let me know.

Thanks,
Shashank

5

Hi, I now checked your evaluation runs and they seem fine (2 even had all answers). But yes, just writing “none” seems fine.

I currently don’t see that we save the original file name somewhere, but maybe I can pass that question to the implementor of this functionality… @maik_froebe?

6

Thanks for the hint with the filename.
Unfortunately, the file name is not available on the server. I have added a new “Run Name” field to the upload form.

@ultron can you please upload again and use this additional field?

Best Regards,
Maik

7
8