Submission testing file

Touché
1

Good morning,

What test.tsv file should we upload?
In Zenodo it is said “Future versions of this dataset will contain more arguments and be split into “-training”, “-validation”, and “-testing” files to represent the corresponding sets for the evaluation.” but I cant find the testing file to predict.

Thanks in advanced.

2

Hi, at the moment you can only upload predictions/labels for the ACL dataset (the one that is there on Zenodo). Basically, you could upload the labels-training.tsv to get max score. Expect the testing file to be there on Monday.

3

And there is only one early bird?

Thanks,
Nordin

4

Yes, we want you to make a good guess of what is your best model so far, without allowing too much optimization towards the test set (that would kind of ruin the purpose of it being the test set). So you can only submit one run for early bird. If you have more runs in TIRA, you can just tell us which one you want to have scores for.

5

Hi again,

I just uploaded my early bird, is there any way of knowing if the file I uploaded has an error?

Thanks in advanced,
Nordin

6

Hi Nordin,

I just checked and your submission has indeed been successful! If it were not, we would contact you and give you some time to resubmit. If you want to update your submission you can also still do so. Just tell me.

7

Hi Johannes,

I uploaded submissions for validation and training files and made sure the evaluation was successful and there were results. I was not able to push them on to the leaderboard because they were not reviewed. I also uploaded a test submission for the early bird. While I was able to run the evaluate, I cannot see any results. Should I be able see any results? I don’t want to miss my chance to the early bird submission.

8

Hi, thanks for letting me know. I just checked your test submission and it seems fine. It is intended that you do not see the results yet. You can still submit another run if you like, just tell me for which one you then want to get the results

9

Hi Johannes,

Thanks for the info. I uploaded another run because my earlier upload was a wrong version. Please always use my latest run to get me the results. I say this because I may upload another version if I find another error before the end of today.

10

:+1: will do as you say :wink: