Hello!
I’ve tried to perform an evaluation on some new runs, but I believe the automatic evaluation system is down.
Will it be up or is it possible to get the ground truth of test dataset?
Thank you in advance!
Gonçalo Raposo
Dear Gonçalo,
the submission system is still online.
Do you have problems with uploading your files?
The labeling process for the ground truth of new runs is still ongoing, but the ground truth data that we have used for the preliminary evaluation is available on Zenodo [1].
Best Regards,
Maik
[1] - QReCC - Question Rewriting in Conversational Context | Zenodo
Hello Maik!
Thank you for your response.
I wondered if it was still ongoing, because the results have been hidden for a while.
I thought I read somewhere that the ground truth was first made available and later removed. But this are great news!
The ground truth used for the preliminary evaluation is the file qrecc-test.json, right?
Thank you once again!
Gonçalo Raposo
Oh, you are right, the ground truth was removed.
So for the moment you still can evaluate your runs with TIRA.
I think the ground truth data will be available online soon (I will double-check this, and will keep you posted).
Best Regards,
Maik
Oh…
Regarding TIRA, is it unusual to take this long to unveil the results?
Best regards,
Gonçalo
Ah, yes, sorry for this delay.
The review process is manual, and we did not check for new submissions in the last days.
I have unblinded the results for you.
Please do not hesitate to submit more runs, just drop me a line so that I unblind them fast (because otherwise we probably only look every few days).
Best Regards,
Maik
Dear Gonçalo,
we will make the ground truth data used for the preliminary evaluation results available again on Zenodo, probably we will be able to do this tomorrow.
Best Regards,
Maik
Thank you very much Maik! Looking forward to it.
Best regards,
Gonçalo
Dear all,
The new version of the dataset contains the ground truth we used for automatic evaluation:
Regards,
Johannes