About validation indicators and data sets

PAN
1

Dear TRIA Platform Team,

I hope this message finds you well. I have a few questions regarding the PAN24 competition that I hope you can help clarify. Thank you in advance for your assistance!

  1. Which of the following datasets should be used as the standard for evaluating models?
  • pan24-generative-authorship-smoke
  • pan24-generative-authorship-tiny-smoke
  • pan24-generative-authorship-train
  1. After submitting the system and having the models evaluated using the aforementioned datasets, are there any further steps required?
  2. Could you please provide the deadlines for system submission, announcement of rankings, paper submission, and the dates for the conference?

I look forward to your response and thank you for your support.

2

Dear Zepeng,

Thanks for reaching out!

The final datasets on which the models will be evaluated are not ready yet. The datasets pan24-generative-authorship-smoke and pan24-generative-authorship-tiny-smoke are there to ensure that the submissions work (pan24-generative-authorship-train is only a copy of the training dataset for completeness). The final test datasets will include LLM generated texts from the Voight-Kampff Task at the ELOQUENT Lab where participants try to obfuscate that text was generated by an LLM. The submissions for the ELOQUENT lab are due by next monday (i.e., 06.05.2024), so the final test datasets for the PAN authorshop task will be constructed slightly after that.

From our perspective, running the models on the aformentioned datasets is enough at the moment, as soon as the test dataset is ready, we will run all submissions on the final test data.

I will ask for the corresponding deadlines and report back.

Best regards,

Maik

3

I can now report back with the esimations of the deadlines, they are aligned with the official CLEF deadlines.

I.e., it is not 100% clear yet, because of the alignment with the ELOQUENT Lab, but the deadlines will be roughly:

  • (soft) system submission deadline on next monday, 06.05.2024 AOE
  • notebook papers will have a deadline of 31.05.2024 (that is official by CLEF, so we can not extend this one)
  • We try to make the evaluation results available around the 20.05.2024

Sorry that it is not 100% exact, the only really “hard” deadline is the notebook paper submission deadline, all others are rather flexible (especially the system submission deadline).

Best regards,
Maik

4

Got it, thank you for your reply!

1 Like