TIRA is a platform for reproducible participation in shared tasks from IR, NLP, and machine learning.
As a participant, you can register for tasks and submit your software using TIRA. TIRA will execute your software on a dataset, evaluate the output, and report the scores back to you.
As a task organizer, you can provide (private) datasets to participants, manage submissions, and review the results.
Submissions:
- How to make a software submission with docker.
- How to use very big models (e.g., LLMs or other Hugging Face transformers) on A100 GPUs in TIRA
This written guide shows how to submit a (software) solution to TIRA via docker. - How to make an upload submission
This written guide shows how to submit a solution to TIRA by uploading a file with predictions. - Full participant workflow using Docker (written version)
This screencast shows how to register for a task and submit a (software) solution to TIRA via docker. - Examples
A curated list of TIRA-compatible baselines from various tasks. These baselines include individual build and submission instructions.
User Management
- How to register for a task
This written guide shows how to register for a task if the task requires registration. - How to add someone to your group
This written guide shows how you can add a partner to your registration group, so you can submit solutions under the same name.
Task Management:
- How to write an Docker Evaluator
This written guide shows you how to build an evaluator that will evaluate submissions, we also have a range of existing evaluators. - Dedicated tutorial for the ir-lab in 2023.
A dedicated tutorial on how to organize an IR task specific for the IR lab in 2023. - How to organize an IR task (using the ir_datasets integration)
This screencast shows how to set up a new information retrieval task on TIRA using the ir_datasets integration. Please contact the TIRA admins or comment below this thread if you want to organize a Shared Task using TIRA and have any questions or problems, we are happy to help. - How to organize a task (screencast)
This screencast shows how to set up a new task on TIRA. Please contact the TIRA admins or comment below this thread if you want to organize a Shared Task using TIRA and have any questions or problems, we are happy to help.
Experimentation:
- How to do post-hoc-experimentation (Screencast)
This Screencast describes how to run large-scale experiments using collections of datasets and software submissions.
Legacy Guides
- How to make a software submission with virtual machines (deprecated)
This written guide shows how to submit a (software) solution to TIRA using a TIRA virtual machine. This method is reserved for legacy use cases and is not officially supported anymore.