AREA CHAIR GUIDELINES

Area Chairs Guidelines and Responsibilities

This document provides detailed guidelines to Area Chairs (AC) for MICCAI 2021. Please read the instructions carefully if this is your first time as AC for MICCAI, and just as carefully if you have been on the program committee before, as there are some changes in the review process compared to previous MICCAIs. Specifically, we adopted the AC model, in which each AC has specific expertise and more responsibilities for the papers assigned to him/her. Being an AC is a demanding task that involves managing the review process for a set of papers, selecting papers for orals and awards, assessing reviewer quality, and interacting with the Program Chairs (PCs) and other ACs. You have a strong influence on the final decision, please take it seriously and be fair.

Having successfully moved to a double-blinded peer-review process at MICCAI 2017, there will be no face-to-face Program Committee meeting for MICCAI 2021. Instead there will be a teleconference before the review process begins on March 8 and March 9 to explain the process and to answer any questions related to the review process, and a second teleconference near the end of the process, where the MICCAI 2021 PCs will communicate the overall results and statistics of the review process and any arising issues to the ACs, while at the same time collecting feedback from the ACs.

The role of AC therefore continues to entail a significant commitment.

Important Dates for Area Chairs is available from this page.

1- Responsibilities of the Area Chairs

ACs are expected to play an active role in ensuring high quality, timely, fair and constructive reviews. ACs also help the PCs select the best papers to be presented at MICCAI 2021. The ACs are involved in the following steps of the review process

Paper Submission and Assignment to Area Chairs
(Deadline for paper submissions: March 3rd, midnight, Pacific Time)

Authors will submit their papers using Microsoft's Conference Managing Toolkit (CMT). The MICCAI Conference review process will be double-blind, i.e. the ACs and reviewers won't know the authors, and the authors won't know the reviewers or ACs. Paper assignment to ACs will be completed automatically using CMT and the Toronto Paper Matching System (TPMS), which has been successfully used for past MICCAIs. The system is complex, considering paper matching scores, subject areas, conflicts and global load balancing - but remember, it is automatic and not necessarily optimal for each individual AC. If you feel a paper has been incorrectly assigned to you, please let program-chairs@miccai2021.org know; we will re-assign the paper to another AC and you will get a replacement paper that has been manually selected. We anticipate that such reassignments will be rare. The quality of the conference depends to a large degree on the reviewers, therefore we encourage ACs to urge their network, especially postdocs and students from their group, to sign-up for reviewing.

Reviewer Assignment - Phase 1
(Deadline for reviewer assignment: March 17th)

Note that MICCAI 2021 will be running double-blind - neither reviewers nor ACs will see the author list, nor will the authors see the reviewers or ACs. Reviewers will not see the ACs names. The CMT system will provide a list of reviewers for each paper to the AC. This ordered list is generated based on subject areas (relevance) and TPMS. Based on their expertise and judgment, the AC will select a ranked list of 10-15 possible reviewers; this is needed to ensure that the bidding and matching system works smoothly. You can filter the list by subject areas and sort by relevance according to keyword matching [score 0-1.0, 1.0=best match] or TPMS rank [1 - 1400,1=best match] to help you choose reviewers. Select your reviewers and save for each paper.

While possible, it is difficult to add a reviewer who is currently not in the reviewer database because we cannot automatically identify conflicts and maintain load balancing through CMT and TPMS. If you wish to add a reviewer not in the database, you must (1) ensure that the person is willing to review, and then (2) contact submission platform manager Kitty Wong (submission_support@miccai2021.org) and we will include the reviewer manually within CMT for that paper during Phase 2. Given the overhead involved in this process, it is expected that it will be a relatively infrequent exception.

Reviewer Assignment - Phase 2
(Deadline for Phase 2: March 29th)

Once all ACs have selected the initial set of reviewers, a reviewer paper bidding process will start, after which the CMT system will re-optimize matching of reviewers to papers based on a large weighting placed on the AC's suggested ranked list of reviewers and the reviewers ranking of paper preferences. This final optimization will consider the TPMS score, relevance, and conflict domains from CMT while performing load balancing across all papers, reviewers and ACs. The highest scoring three reviewers on the revised list will be sent the paper for review automatically.

*New*

Reviewers will be given 3 days to accept or decline the review assignment. This is to remove unresponsive reviewers and to give reviewers the chance to decline in case they feel unable to perform a good and unbiased review. We will ask reviewers to use the "decline” option only if absolutely needed.

Reviewing
(Deadline for reviews: April 20th)

Each AC will be responsible for shepherding the review process for a maximum of 25 papers. The main duty of the AC during this phase is to monitor the quality of the reviews. If reviews are too brief, lack quality, or are inappropriate in some way then it is the ACs responsibility to contact the reviewer and ask them for improvements. Should the AC be unsatisfied with the quality of a review, and fail to get further feedback from the reviewer, then the AC will be able to ask (program-chairs@miccai2021.org) for additional reviewer(s)' input on the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified in the reviewer database and may not be re-invited next year.

The submission platform manager will be responsible for reminding reviewers about deadlines and for monitoring whether reviewers have logged into the system. If 3 reviews have not been provided by the required date then the PCs will work in collaboration with the ACs to secure additional reviews.

ACs Meta-reviewing and preliminary ranking+scoring
(Deadline for meta-reviews and ACs preliminary scoring May 7th)

After the review deadline April 20, ACs will need to provide meta reviews which summarize the key strengths and weaknesses of the paper identified by reviewers, express the opinion of the AC about the paper, and justify the ACs preliminary recommendation. If all reviewers agree on a paper and the AC follows their recommendation, the report can be simple. If there is disagreement among the reviewers, it is the ACs job to reconcile the reviews, weigh the pros and cons of the paper, and make a well-founded justification for their preliminary recommendation. If you as AC disagree with the reviewers, you can go against their recommendation, but you will need to explain why. We expect ACs to be able to make decisions based on the reviews and their own reading of the papers in most cases, but if you would like to ask for clarification from the reviewers, you can communicate with your reviewers within CMT. Detailed instructions will be posted on: https://miccai2021.org/en/INSTRUCTIONS-TO-AREA-CHAIRS.html

For papers that will be sent for rebuttal, ACs should identify any questions/points that need to be specifically addressed. Please remind authors that the purpose of the rebuttal is to provide clarification or to point out misunderstandings, it is not to promise additional experiments! Please also see section 2. Best Practices of Being an Area Chair below.

ACs should also select appropriate keywords and categories for each paper; this information will be used to help in planning poster and oral sessions.

ACs will then rank the quality of the papers in their batch and score each paper as an early accept/ rebuttal / early reject. The PCs will send out guidance on the proportion of papers to place in each category once the total number of papers submitted is known.

We will also provide some aggregate information on the distribution of reviewer scores as each AC may not have a representative sample. Although we do not want to increase the number of rebuttal papers too much, ACs are able to exercise discretion when making their decisions.

New in MICCAI 2021 is a reproducibility checklist that authors have to fill out upon submission which should be taken into account. Please be aware that authors are not required to meet all criteria on the checklist, but rather mark criteria relevant to their paper.

During this period, the PCs may contact the ACs for suggestions of suitable reviewers for papers that are missing 3 reviews. The assignment of additional reviewers will be made by the submission platform manager as it will be necessary to check for conflicts first.

New this year: Please be aware that we intend to make reviews including meta-reviews and rebuttals of accepted papers publicly available (without disclosing the reviewers/ACs names).

Rebuttal and author response
(Deadline for submission of rebuttal: May 21th)

The rebuttal process provides a way for authors to correct possible misinterpretations in the reviewers' findings and inform the AC's final recommendation. The rebuttals will be published alongside the reviews and meta-review if the paper is accepted. Also authors of early accept pacers will be invited to write a response that will be published alongside the reviews. Once the reviews of the papers have been sent to the authors, authors have one week to submit their rebuttal/response. During this period the PCs will assign an additional 2 ACs to each paper sent for rebuttal.

Secondary meta reviewing and ranking+scoring of rebuttal papers
(Deadline: June 4th)

Rebuttals and all reviews will be visible to the primary AC and 2 secondary ACs. The primary AC will update their meta review in light of the rebuttal and the 2 new ACs have to provide a meta review explaining their recommendation as well. All 3 ACs will then rank the papers in their batch in this stage and make a binary accept/reject recommendation, which will be used by the PCs to make the final decisions. We expect ACs to be able to make their independent recommendations based on the reviews and their own reading of the papers in most cases, but if you would like to ask for clarification from the reviewers or the other ACs assigned to a paper, you can communicate the other assigned ACs and reviewers within CMT. Detailed instructions will be posted on: https://miccai2021.org/en/INSTRUCTIONS-TO-AREA-CHAIRS.html

The PCs will provide guidance about the proportion of papers to be accepted or rejected, but it is anticipated that there will be some variation between individual ACs.

Program Committee Teleconferences
(Week 23-24)

We invite and strongly encourage all ACs to participate in a teleconference, where the MICCAI 2021 PCs will communicate the overall results and statistics of the review process and present any arising issues to the ACs. In these teleconferences, ACs also will be asked to provide feedback on the overall review process and formulate recommendations on any need for adjustments or improvements. Specific details about the program and responsibilities will be announced prior to these teleconferences.

2- Best Practices of Being an Area Chair

This paragraph should remind all ACs of best practices[1] of being an AC.

  • Written report: One of the most crucial duties of ACs is the preparation of a written report ("meta-review”) on each paper. This is where the AC justifies his/her recommendation to accept/reject a paper. The reports represent a good quality review process and should highlight clearly why the decision was reached, it is not simply a summary of the reviews. If all reviewers agree on a paper, the report can be simple, while trying to encourage authors and provide constructive feedback. If there is even a slight disagreement on the reviews, it is your job to reconcile the reviews and make a well-founded justification for your final decision. Clarify if there are reviewer comments you did not agree with. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal, focusing on points that could change your mind whether to accept or reject the paper. For the second report (after rebuttal phase), if there is a rebuttal clarify if it removed the concerns. Please be aware that we intend to make the report (of both primary and secondary ACs) of accepted papers publicly available (without disclosing the ACs names).
  • Conflicts of interest: If you identify any paper with which you might have a conflict of interest please notify the PCs immediately so the paper can be re-assigned. DO NOT talk to any other AC about papers assigned to you without prior approval from the PCs, as there may be several other ACs conflicted with the paper. Also, DO NOT talk to any other AC about your own paper(s) (the paper(s) on which you are an author) or a paper with which you have a conflict, during this whole process. Please remember that it is unacceptable to include anyone as a co-author who has been directly or indirectly involved with the manuscript at any stage during the decision process either as a Reviewer, Area Chair or as part of the PCs; this includes any direct follow- on publications (e.g. MICCAI special journal issues).
  • Attitude: Be aware that you have a strong influence on the decision for a paper. Take your job very seriously and be fair. Be professional and willing to listen to other reviewers and ACs. Do not give in to undue influence from anyone.
  • At the conference: Please keep track of your accepted papers; for talks, have a question or two prepared for the speakers to stimulate discussion. For posters, go see the poster and ask questions.

[1] Acknowledgements: These guidelines were inspired by CVPR 2014 guidelines: http://www.pamitc.org/cvpr14/ac_guidelines.php