HowTo: Metric Submission

SCAI SCAI Eval Metric Submission
1

You should have prepared a Docker image of your approach. To get you started, look at our simplicity metric.

  1. Open the task’s submission page
  2. Click on SUBMIT, registering your team if necessary.
    image
    image821×270 18.3 KB
  3. Click on DOCKER SUBMISSION and +.
    image
    image901×253 9.1 KB
  4. Follow the instructions, installing tira and executing the Docker submission with tira-run. The tira-run command will download the data automatically. YOUR-COMMAND has to ignore the entrypoint. Use $inputDataset/* for the dataset file and $outputDir/run.json for the output file. Check the output in tira-output/run.json.
  5. Click on PUSH NEW DOCKER IMAGE to get your personalized instructions to push the image to TIRA. Click REFRESH IMAGES after it was pushed to then be able to select if from the Docker Image dropdown.
  6. Put the same command you used in 4. into the command text field.
  7. Click NEXT and copy the two commands to check your approach another time.
  8. Click EDIT to brifly describe your software.
  9. Select resources and the scai-eval24-2023-09-26 dataset, then click “RUN”.
  10. Wait depending on your approach. Then reload the page and check your submission.
1 Like