How to organize a IR task with ir_datasets

Getting Started
1

This guide shows you how to organize an information retrieval shared task or experiment with ir_datasets in TIRA. Please contact the TIRA admins or comment below this thread if you want to organize a Shared Task using TIRA and have any questions or problems, we are happy to help.

We also have a dedicated tutorial for the ir-lab in 2023.

Screencast

Here we provide a short screencast on how organizers of shared tasks can deploy their shared task in TIRA:

If you prefer written instructions, the content of the screencast above is documented here:

Step-by-Step Guide: How to organize your IR task with ir_datasets in TIRA

For this tutorial, we assume that you have already prepared a docker image with the ir_datasets integration for your task (or that the dataset you want to use is already in the main branch of ir_datasets so that our default ir_datasets image already covers your dataset). If you do not yet have prepared this docker image, please follow the ir_datasets tutorial for TIRA that will guide you to create your docker image for a new dataset in 15 minutes.

Step 0: Requirements

To finish the tutorial, you will need a Docker installation on your machine. We will use the docker image webis/tira-ir-datasets-starter:0.0.47-pangram as example on how to add a new custom dataset via ir_datasets.

Step 1: Register To TIRA

Please create create an account and login to TIRA.

This tutorial is available in written instructions but also as a screencast that guides through all steps in a video/screencast .

Step 2: Add a Orga-Team for Your Organization

You can skip this step if you are already part of an organizer team in TIRA. If one of your colleagues already has created the organizer team, please let them add your account to the existing team (to see existing groups, please look at https://www.tira.io/g).

If you want to create a new organizer team, please click on “Organize Task” on the start page of TIRA and fill out the form.

Step 3: Add a Task

If the task already exists, you can skip this step.

To add a new task you must be part of an organizer team (get invited to a team if the team already exists, or go back to step 2). Please click on “Add Task” on the start page of TIRA and fill out the form to add a new task.

Step 4: Add a Dataset to your Task

Click on your new task on the start page of TIRA to go to the task page.
Click on “Import Existing Dataset” to import the new dataset via your ir_dataset docker image.
To import the ir_datasets data from the tutorial, please add pangrams as dataset name and webis/tira-ir-datasets-starter:0.0.47-pangram as docker image (this image is the result from the tutorial uploaded to docker hub).

Click on “Import IRDS Dataset” after you have filled out the dataset name and the docker image. Importing might take a while (please do not click the button multiple times).

Step 5: Final End-to-End Tests

We now have imported the task to TIRA.
To run a baseline, please click on “Submit” on your task page.

Following the general ir_datasets TIRA tutorial, we use existing baselines:

  • webis/tira-ir-starter-pyterrier:0.0.1-base as image, and
  • /workspace/run-pyterrier-notebook.py --input $inputDataset --output $outputDir --notebook /workspace/full-rank-pipeline.ipynb as command, which employs BM25 retrieval.

On your submission page (after clicking on “Submit” on the task page), please do the following:

  1. Click on “Docker Submission” → “Upload Images” to upload your image to your dedicated registry. You can tag the image webis/tira-ir-starter-pyterrier:0.0.1-base accordingly, e.g., if your image should have the name registry.webis.de/code-research/tira/tira-user-<YOUR-USER-ACCOUNT>/pyterrier:0.0.1, please run docker pull webis/tira-ir-starter-pyterrier:0.0.1-base to pull the image to your system, then docker tag webis/tira-ir-starter-pyterrier:0.0.1-base registry.webis.de/code-research/tira/tira-user-<YOUR-USER-ACCOUNT>/pyterrier:0.0.1 to tag the image accordingly, and subsequently push the image as documented in your personalized documentation for uploading images.
  2. Click on “Docker Submission” → “Add Container” to add your new software. Select the just uploaded docker image (you might have to click on “Refresh Images” and/or wait a bit so that the image is added to the drop down) and speficy as command /workspace/run-pyterrier-notebook.py --input $inputDataset --output $outputDir --notebook /workspace/full-rank-pipeline.ipynb. Then click on “Add Container”.
  3. Refresh the page so that the new software is shown.
  4. Click on “Docker Submission” → your newly added software, select the resources for execution, the dataset, and then click on “Run Container” to run your software.

Congratulations, you now have added your dataset to TIRA and have executed a first baseline run.
After your run was successfully executed, you can see the results, evaluation, and leaderboards.