Abdominal Circumference Operator-agnostic UltraSound measurement Banner
Join

Phases and rules

Challenge phases

The ACOUSLIC-AI challenge takes place in two phases:

  • Preliminary Development Phase (Duration: 3 months. Submissions allowed: 10 total)

Teams interested in participating must register for the ACOUSLIC-AI challenge at acouslic-ai.grand-challenge.org. During this phase, they can use the Public Training and Development Set to develop and train AI models using their own computing resources or public platforms such as Google Colaboratory and Kaggle. Each team is allowed to submit one trained algorithm in a Docker container for evaluation for a maximum of 10 times. These algorithms are tested on the grand-challenge.org platform using a Hidden Validation and Tuning Set, with results reflected on a real-time, public leaderboard. This setup ensures that the evaluation images remain confidential and prevents any tampering with AI predictions. Before implementing your own algorithm using this template, we recommend that you first upload a GC algorithm based on this unaltered template. This initial step is important because it may take multiple submissions to resolve any issues with it. 

  • Final Test Phase (Duration: 2 weeks. Submissions allowed: 1 total)

At the conclusion of the Preliminary Development Phase, teams may submit their best AI algorithm for final evaluation on the Hidden Test Set. In the Final Test Phase, teams are allowed to submit their final AI algorithm one time only. This evaluation will be conducted on the Hidden Test Set. The algorithms' performance on this cohort will be used to establish the final rankings, which will determine the top three best-performing models.

During both phases (Development and Testing), the algorithms will be run on the grand-challenge.org platform.

Allowed number of submissions per phase

For each submission, you can only submit one AI model at a time. During the Preliminary Development Phase, algorithm performance will be assessed using the Hidden Validation and Tuning set, and team rankings will be updated in real-time on its corresponding live, public leaderboard.  A maximum of 10 submissions is permitted during the Preliminary Development Phase.

In the Final Test Phase, teams are allowed to submit a one-time single AI algorithm for assessment. This evaluation will be conducted on the Hidden Test Set. The algorithms' performance on this cohort will be used to establish the final rankings, which will determine the top best-performing models. During both phases (Development and Testing), the algorithms will be run on the grand-challenge.org platform.

Team Participation and Submission Protocol

Each participating team is permitted to have multiple members; however, only one designated team member is allowed to submit the algorithm for evaluation. To comply with submission requirements:

  • Identify and agree upon one team member who will act as the official submitter for your team.
  • This individual will be responsible for uploading the AI model to both the Preliminary Development Phase and Final Test Phase.
  • When submitting, ensure that a PDF document is included with your submission. This document must list all the names of the team members.
  • If multiple submissions from the same team are detected, the team will be disqualified from the challenge.

Rules

  • Participants may choose to join the challenge either individually or as part of a team. Each participant in a team - even if composed of a single participant - can only be a member of a single team.
  • Any individual participating with multiple or duplicate Grand Challenge profiles will be disqualified.
  • ranking on the validation/testing leaderboards, true names and affiliations [university, institute or company (if any), country] must be displayed accurately on verified Grand Challenge profiles, for all participants.
  • Members of sponsoring or organizing centers (i.e. Radboud University Medical Center, Delft Imaging Systems) may participate in the challenge, but are not eligible for prizes or the final ranking in the Testing Phase.
  • This challenge only supports the submission of fully automated methods in Docker containers. It is not possible to submit semi-automated or interactive methods.
  • All Docker containers submitted to the challenge will be run in an offline setting (i.e. they will not have access to the internet, and cannot download/upload any resources). All necessary resources (e.g. pre-trained weights) must be encapsulated in the submitted containers a priori.
  • Participants are allowed to use pre-trained AI models based on computer vision and/or medical imaging datasets (e.g. ImageNet, Medical Segmentation Decathlon). They can also use external datasets to train their AI algorithms. However, such data and/or models must be published under a permissive license (within one month of the Preliminary Development Phase deadline) to give all other participants a fair chance at competing. They must also clearly state the use of external data in their submission, using the algorithm name [e.g. "Abdominal Circumference AI Model (trained w/ private data)"], algorithm page and/or a supporting publication/URL. 
  • To participate in the Final Test Phase, participants must submit a short paper in pdf format on their methodology (2–3 pages) and a public/private URL to their source code on GitHub (hosted with a permissive license). We take these measures to ensure credibility and reproducibility of all proposed solutions, and to promote open-source AI development.
  • Although teams are permitted to publish their results independently, there is an embargo period of 6 months to ensure all analyses and computations outlined in this proposal can be conducted. This means that for the first six months following the end of the challenge (August 27th, 2024), the challenge organizers retain exclusive rights to publish the collective results, after which teams may proceed with their individual publications. However, we acknowledge the value of early research dissemination and, therefore, permit the submission of arXiv preprints during the embargo period.
  • Organizers of the ACOUSLIC-AI challenge reserve the right to disqualify any participant or participating team, at any point in time, on grounds of unfair or dishonest practices.
  • All participants reserve the right to drop out of the ACOUSLIC-AI challenge and forego any further participation. However, they will not be able to retract their prior submissions or any published results till that point in time.