23rd INTERNATIONAL CONFERENCE ON MEDICAL IMAGE
COMPUTING & COMPUTER ASSISTED INTERVENTION
4-8 OCTOBER 2020
LIMA CONVENTION CENTER / LIMA-PERU

THE MICCAI REVIEW PROCESS

CONTEXT

The purpose of this document is to define the Review Process for the MICCAI conference. The document has been initiated and is endorsed by the MICCAI 2017, 2018, 2019 and 2020 organizing committees, to be passed on each year, from successive MICCAI organizing committees, with the goal to refine and include feedback from the full set of future conference organising committees to create a "sliding window” and consistency between years.

The document is heavily inspired by the previous Review Process developed by S. Ourselin, D. Hawkes, N. Navab, P. Golland, N. Ayache, and G. Fichtinger. The current review process version was prepared jointly by the MICCAI 2017, 2018, 2019, 2020 committees with the following people involved: L. Joskowicz, M. Descoteaux, D.L. Collins, L. Meier-Hain, P. Jannin, S. Duchesne, C. Davatzikos, A.F. Frangi, J.A. Schnabel, D. Shen, T. Liu, A. Martel, D. Mateus, D. Stoyanov, M. Reyes, D. Racoceanu, P. Abolmasumi. We are grateful for all their contributions.

The policies set within are born of a desire to enhance the previous process, by specifically:

  • simplifying the process to increase compliance and enforceability;
  • lowering the burden on participants;
  • increasing review quality and retention;
  • clarifying roles and responsibilities to increase understanding from all participants;
  • decreasing the appearance of arbitrariness in the decision-making process, by being more transparent; and
  • reducing costs and carbon footprint.

As in previous MICCAI conferences, the goal of the MICCAI conference review process is to select the best papers in each discipline. This selection should be fair, taking into consideration the specialized nature of our discipline and the size of our community; efficient, in not wasting valuable time, effort and funding from our peers; and just, in exploiting the consensus of peer comments.

GENERAL

This summary is made public so that all participants understand the review process and can plan accordingly.

The MICCAI 2020 review process will be overseen by the following individuals:

Program Chairs:

  • Purang Abolmaesumi
  • Anne Martel
  • Diana Matteus
  • Danail Stoyanov
  • S. Kevin Zhou
  • Maria A. Zuluaga

Submission Platform Manager:

  • Kitty Wong

General Chairs:

  • Daniel Racoceanu
  • Leo Joskowicz

The program committee of MICCAI 2020 consists of two bodies of participants that are essential to the review process:

  • Area Chairs (ACs); and
  • The College of Reviewers (Reviewers)

The following documents supplement the following guide, and will be made available on the MICCAI 2020 conference website:

Stage 0: Call for ACs

The MICCAI Society should maintain a database of past Area Chairs and reviewers. Excellent ACs and reviewers should be identified after each event. The MICCAI 2020 organizing committee will build on the list provided by MICCAI 2019. The information about the number of times a member has been an AC will be kept indefinitely. The information on excellence of reviewers or ACs will be maintained in the system.

MICCAI 2020 also will issue a call for participation in the program committee via the newsletter and mailshots. Individuals will submit relevant information regarding their past participations in MICCAI and other similar conferences, as well as other biographical details.

Individuals will be chosen from the database, the received invitations and other sources, and invited to join the MICCAI 2020 program committee. When they apply, ACs will need to update their Toronto Paper Matching System entries, create/update their CMT profiles and associate themselves as being either MIC, CAI or MICCAI. They will need to use their institutional email address. The MICCAI Program Chairs will then select the program committee using the following guidelines:

  • The program committee composition should be representative of all MICCAI topics, and with a range of seniority;
  • There will be ~70 ACs initially recruited in this phase, balanced between MIC, CAI and MICCAI experts, with a spread of expertise based on past MICCAI history;
  • For MICCAI 2020, we will aim to have the best program committee possible while aiming for gender parity (50% male, 50% female) and balanced demographics.

A webinar may be held to explain the review process to ACs and secure their commitment early in the process.

Details and schedule for AC duties include the following:

You will be asked to update the keywords and CMT and upload your relevant papers to TPMS by the end of February. An initial teleconference will be held in March to explain the full review process and to discuss reviewing criteria. After the full paper submission deadline on 3rd March 2020, the first phase of your work will run from 10-15 March 2020 for the initial reviewer assignment period and it is essential that you are available during this week. The paper review period will run from 1-14 April 2020 during which time you are expected to monitor the progress of the reviewers. Between the 14-20 April you will need to assess the quality of the reviews, provide feedback for authors and recommend papers for accept, reject or rebuttal. The second phase will then start once the rebuttals have been submitted and between 13-26 May you will be asked to provide final rankings on your original paper allocations. In addition, you will be asked to act as a meta-reviewer and rank a further 16 papers. These are time intensive tasks and it is essential that you will be available during these crucial periods. Your responsibilities will end with the program committee teleconferences in June 2019. A detailed timeline is posted on the Conference webpage.

By accepting to serve on the program committee, you agree to this timeline.

Stage 1: Reviewer Database

The list of potential reviewers is primarily based on the list of reviewers who will respond to the Call for Reviewers, which will be disseminated to active reviewers of the last MICCAI conference, to which are added all authors at that conference. Further, student reviewers must be at least enrolled at Ph.D. and have two or more published articles in a related field to qualify as reviewers. ACs will contribute to updating and expanding the list of reviewers.

Each potential reviewer will be invited to review for MICCAI. By accepting to review:

  • the reviewers commit to review a maximum of 6 papers,
  • the reviewers will be asked to create or update their CMT and TPMS profiles,
  • the reviewers will be required to use their institutional/work address as primary contact.

Stage 2: Intention to Submit

Authors will provide an intention to submit two weeks prior to the paper submission deadline. Authors must provide: i) a list of co-authors and their affiliations, ii) Google scholar and DBPL profiles for each co-author where possible, iii) the title of the paper, and iv) the abstract of the paper. This information will be used to start building the assignment of papers to ACs and potential reviewers. Authors and co-authors must use their institutional/work email address.

Stage 3: Paper Submission

Authors will submit papers in CMT. The MICCAI Conference review process will be double-blind, i.e., the names of the authors will be hidden from the Area Chairs and Reviewers, and the names of Reviewers and Area Chairs will not be revealed to the authors. To achieve this, papers must be properly anonymized before submission. At the discretion of the Program Chairs, a paper may undergo an outright rejection when it is in blatant breach of these anonymization rules. Area Chairs and Reviewers may bring concerns about such a breach to the attention of the Program Chairs.

Each paper must be submitted with Primary and Secondary areas selected from the CMT system, and should also indicate up top three relevant keywords in the "keyword” section of the paper template. Authors also must identify to which stream the paper belongs, i.e., either MIC, CAI, or MICCAI. These areas, the stream, and the paper itself, will be used to generate suggested reviewers using the automated TPMS paper matching system embedded in the CMT system (see Stages 5 and 6).

Stage 4: Additional AC's Enrolment

The goal of this step is to ascertain the breadth and depth of expertise required within the program committee in response to the initial submissions from authors. Given that it is almost impossible to accurately predict which domains will be most represented in any conference edition, this step will allow for the adjustment of the program committee composition.

The Program Chairs may then invite approximately 10 additional AC members having an identified expertise from the database of potential ACs and reviewers. Those who agree to join, must update their CMT profiles and associate themselves as being MIC, CAI or MICCAI, but at this point the balance between these three groups will be proportional to their representation in the submissions.

If required, a webinar will be held for all AC members to provide early statistics and explain the process once again.

Stage 5: Paper Allocation to ACs

Each paper will be assigned to an AC automatically by the CMT system, based on the ranked lists provided by the authors and the TPMS. (Note that the AC remains blind to the paper authorship.) The Program Chairs will check the assignments to make sure all papers received a good assignment.

The essential role of the AC is to move the paper through the review process, up until the decision by the program committee. ACs will use their knowledge of the topic and of the appropriate reviewers to ensure the best (most informative) reviews.

For MICCAI 2020, a maximum of 25 papers will be allocated to each AC. This is necessary to ensure a proper statistical distribution in the ranking to follow (cf. Stage 7). The number of ACs will be adjusted accordingly. Thus, for approximately 70 ACs, MICCAI 2020 is expecting to handle around 1,750 submitted papers.

Stage 6: Paper Allocation to Reviewers

The goal of the paper allocation is to find the most appropriate reviewers in terms of expertise for a given paper. This step is achieved in two phases using the CMT system:

In phase 1, the CMT system will provide a list of potential reviewers for each paper to the AC. This ordered list will be generated based on keywords, TPMS, and other tools use by the computer vision community. Using their expertise and judgement, the AC will re-order this list as needed and can add additional reviewers to the list (including those suggested by the authors, if deemed reasonable). The TPMS system will re-optimize matching of reviewers to papers based on the ACs suggested ranked list of reviewers (at least 6 reviewers on the list). The optimization will consider the ordered list from the AC, while load balancing across all papers, reviewers, and ACs. Each reviewer will be assigned an initial list of 25 papers.

In phase 2, reviewers will bid for papers by categorising them into "I am interested in reviewing this paper”, "I can review this paper if need be”, "I am not interested in reviewing this paper”, "I have a conflict with this paper”. The TPMS system will then re-optimize matching of reviewers to papers based on the ranked list provided by the reviewers, while load balancing across all papers, reviewers and ACs. Note that the reviewers and secondary ACs remain blind to the paper authorship at all times. CMT permitting, primary ACs will be blinded to the final set of reviewers.

This procedure has the following advantages over previous methods:

  • Better matching of paper and reviewer; should yield better reviews
  • Better conflict identification and resolution
  • Reduced workload for AC
  • The streamlined process saves several steps (reviewer acceptance, reassignment of papers), and thus saves time in the process
  • In line with other conferences, thus leveraging existing reviewer experience with CMT
  • Automated procedure with fixed dates ensures that procedure is more likely to stay on track
  • Potential race condition for reviewers is avoided

The Program Chairs will ensure ACs have re-ordered reviewers on time and can provide additional opinions if a borderline situation cannot be resolved by the AC members.

Stage 7: Review

The goal of the review step is to provide constructive feedback of each submitted paper.

The reviewers will:

  • Provide a comprehensive, fair review
  • Provide a composite score
  • Recommend papers for orals and awards
  • Self-declare their expertise for each paper (passing knowledge, knowledgeable, expert)
  • Rank papers to provide additional assessment

The AC will log in frequently and react to the reviewers' actions: remind reviewers to login and download the papers, and especially ask for a more detailed or fair review. This is critical. Should the AC be unsatisfied with the quality of a review, and failing to get further feedback from the reviewer, then the AC can ask for additional reviewer(s) input on the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified in the reviewer database.

Stage 8: Early Paper Decisions and Rebuttal Process

The Area Chairs will recommend clear acceptance, identify borderline papers for rebuttal, and clear rejection of papers based on consistent reviews and scores. The Program Chairs recommend that:

  • For papers that have three clear acceptance recommendations, the paper is recommended for outright acceptance; the AC can recommend the paper is sent for rebuttal if the AC opinion differs from those of reviewers, however, the AC cannot outright reject the paper.
  • For papers that have three clear reject recommendations, the paper is recommended for outright rejection; the AC can recommend the paper is sent for rebuttal if the AC opinion differs from those of reviewers, however, cannot outright accept the paper.
  • If any paper has at least one recommendation for acceptance, the paper is sent for rebuttal.

This year, the conference will aim for an acceptance rate of ~30%.

We anticipate that after the initial review in this stage, ~25% will be accepted outright; ~ 55% will be rejected, and the remaining borderline papers will be sent for rebuttal. Approximately 40% of these borderline papers will ultimately be accepted.

Borderline papers will enter Stage 9 (below). The goal of the rebuttal process is to provide authors of borderline papers the opportunity to highlight possible misinterpretations or inaccuracies in the reviewer's findings, and inform the AC's final recommendation for scoring. The primary AC will provide a meta review for all cases where they disagree with the reviewers comments.

  • Reviews of the papers are sent to the authors
  • Authors have one week to submit their rebuttal
  • Only the ACs will read the rebuttal during this process

Stage 9: Paper Scoring of Borderline Papers

The goal of this step is to provide a numerical score and rank for each paper assigned to the AC.

After rebuttals are entered, each AC will provide for the borderline papers a recommendation to either "accept” or "reject”. ACs will further provide a meta-review for each assigned paper, consistent with their recommendation and based on the assessment of the reviews and rebuttal.

In addition, each AC will:

  • rank all allocated papers
  • select between 0-2 nominations for oral papers
  • recommend papers for Student Awards
  • recommend 2-3 Reviewer Awards

Stage 10: Acceptance Process

The goal of the decision process is to establish the final list of accepted papers. Prior to this stage, the Program Chairs will update the ACs regarding the total number of orals and selected papers to be awarded at the conference after consultation with the Organizing Committee and the MICCAI Board.

Following rebuttal, in addition to the remaining ~5 papers in each AC's pool emerging from the first round (i.e. initial 25 minus the top 25% and bottom 55%), each AC will be assigned the remaining set of ~5 papers each from two other ACs (so each AC will look at ~15 papers). This assignment of borderline papers will be made by the Program Chairs. ACs will be asked to read the reviews, the rebuttal, and the papers if necessary, and recommend "Accept” or "Reject”. The final "Accept” will be by majority voting (i.e. if at least two ACs recommend "Accept”). By this process, and a wind-up conference call, we hope to add 40% of borderline papers to the "Accepted” pool.

After having switched to the CMT, and TPMS systems for MICCAI 2017, 2018, and 2019, and after having collected significant data before, during, and after the program committee meetings, we will again replace the program committee meeting by a set of dedicated teleconferences between the MICCAI 2020 Program Chairs and the ACs to inform about the overall results and statistics of the review process to discuss and resolve any arising issues, and to collect their feedback.

Stage 11: Oral Decision Process (1 week)

The goal of this step is to select the best submitted papers that justify an oral presentation.

The number and format of the oral sessions will be determined by the Organizing Committee, in consultation with the MICCAI Board and will reflect the overarching objectives of the conference. The recommendations of the ACs will be taken into consideration when deciding which papers to select for full oral presentation. There will be an emphasis on providing Young Investigators with the opportunity to present and there will minimally be 1 CAI oral session.

Three teleconferences will be held, in three time zones (GMT ±8h) to discuss the outcome of the MICCAI review process, statistics and any arising issues, and to collect feedback and recommendations for any further improvements of the process.

Stage 12: Official Notification to Authors

The goal of this step is to inform authors about the outcome of the review process.

The Program Chairs will issue the following, via email, to all authors:

  • Complete statistics for the process (ACs, number of submissions, number of reviewers, number of reviews, number of accepted papers, number of orals); and
  • The author's acceptance (oral, poster) or rejection status for their paper.

A complete list of acceptance and orals will then be drafted for dissemination as a program. The Program Chairs will record each complaint and follow up.