This page provides detailed guidelines to Area Chairs of CVPR 2013 as to how the process will unfold and what are their responsibilities.
The ACs help identify suitable reviewers, assign reviewers to papers, and evaluate the reviews as well as the authors' responses. ACs are expected to play an active role in ensuring higher quality and more timely reviews, and in initiating discussions amongst reviewers (anonymously), and in looking at authors' responses to the reviews. At the AC meeting in February 2013, ACs will work with other ACs (in pairs and then in panels) to decide which papers to accept and will write consolidation reports to explain the decisions.
We want to remind all ACs of some of the basic (and obvious) practices of being an AC.
- You have a strong influence on a decision on a paper. Take it seriously and be fair.
- DO NOT talk to any other AC about papers assigned to you until you are told about who is on your panel and who your buddy is. There may be several other ACs conflicted with the paper.
- The Program Co-Chairs did not submit any papers, so they are not in direct conflict with any papers and they know all authors of all papers and also all reviewers/ACs. They will NOT be involved with any decisions of papers from institutions with which they are in conflict. You can send them a question about any paper and if one of them is conflicted, another one will respond to you without including the conflicted PC Chair.
- DO NOT talk to any other AC about your own paper (the paper you are an author on) or a paper you have some conflict with, during this whole process, unless that AC and you are already collaborators and both of you have an already defined conflict.
- Be professional and willing to listen to other reviewers and ACs. Do not give in to undue influence from anyone.
- Remember, it is not fair to dismiss any review without looking at other reviews and reading the paper yourself to make an evaluation. Outright rejecting a review is not acceptable without a clearly articulated supporting argument. Area chairs who wish to make a decision that is not clearly supported by the reviews will have to present the paper to their panel at the meeting.
- Senior members of the committee should guide and advise the younger and first-time members. First-time members, please feel free to ask for help as needed.
Area Chair Consolidation Reports are the MOST CRUCIAL aspect of the review process of CVPR. This is where the Area Chair justifies his/her recommendation to accept/reject a paper. The Program Chairs will read all Consolidation Reports and we have committed ourselves to ensure that these reports are representative of a good quality review process. These reports should highlight why the decision was reached. If all reviewers agree on a paper, this consolidation report can be simple, but feel free to encourage authors and provide constructive feedback. If there is even a slight disagreement on the reviews, it is your job to clarify why and how the disagreement was resolved. Again, just saying "reviewer 1 was wrong" without any reasons is not acceptable. Explain why. For the most divergent reviews, explain the process you went through to reach a decision and if you and others read the paper, try to add some details from your and others reviews with the consolidation reports. Overruling all three reviewers to make a decision on a paper is not acceptable. If you strongly feel that the paper deserves that, make sure to discuss it in detail with your AC Buddy and also in your AC Panel. In such cases, (a) you should read the paper completely, (b) provide a detailed consolidation report, and (c) solicit confirmation of the decision from your panel. The program chairs will review and confirm or reverse any such decisions that do not have the consensus support of the panel.
During the reviewing phase, please periodically log in to the CMT site and monitor the review progress.
The Program Chairs are going to assign papers to you based on (a) papers from authors not in conflict with you (see conflict of interest in the Reviewer Instructions), (b) subject areas you chose when you registered on the review site, (c) subject areas, the authors chose when they registered the paper, and (d) results from the “Toronto" document matching system. As can be imagined, this will not be perfect as sometimes the Program Chairs have to do some load-balancing. Area Chairs should look through all the papers assigned to them and ensure that they are (1) not conflicted and (2) are knowledgeable in the subject area.
For each paper you are assigned you must initially select 7 or more candidate reviewers during the reviewer assignment step. After all ACs have completed this step, the CMT system will then select 3 from each candidate list, subject to necessary constraints on reviewer load. If it is not able to assign 3 from your list, it will assign fewer than 3, and you will be required to iterate again suggesting additional reviewers. CMT will not assign a reviewer to a paper that you have not suggested. Several iterations may be necessary if you only suggest the minimal number of candidate reviewers, and those will occur in a relatively short cycle (24 or 48 hours). If you plan to only assign the minimal number of reviewers, please expect to be available on short notice to add a few more to the papers which are not satisfied on the first round. If you will be unavailable for several days, it is better to assign 10 or more candidate reviewers. Please assign as many candidate reviewers as possible to each paper, as long as you are comfortable getting any subset of 3 actually assigned. You must assign at least 7, and the more you assign, the greater the chances you will not be required to suggest additional reviewer names for that paper. While CMT will not automatically do so, the program chairs may personally assign reviewers based on preference keyword match or any other scheme necessary to papers of ACs who do not respond in a timely manner.
Here are some suggestions on how to proceed. It can take anywhere from 4-6 minutes a paper do this, if you look at the paper and other info to help make a decision. Considering a max load of 35 papers, you can see that this may take some time. See Reviewer Instructions for details of how to use the system to assign reviewers.
- You have been chosen as an Area Chair because of your specific expertise in certain areas of vision and also your general expertise in computer vision. When you look at a paper, you may just come up with a name or three of ideal reviewers. Consider these people as the reviewers for this, but do also consider things like, (a) are they current in the field and (b) will they do a good job in a timely manner.
- Look at the paper, especially the Introduction, Related Work, and Citations at the end. See who they refer to and whose work they are building on. Often the cited people would be ideal reviewers.
- Remember that the system will give you some recommendations, based on matching subject areas and document statistics. This may provide a good starting point, but will not be perfect.
- Usually, we suggest that you choose a reviewer you know and trust. But the vision community is large. If the system recommends someone you do not know, look at their webpage or their papers on ACM or IEEE digital libraries or Google Scholar and the like.
- Be sure to spend a decent amount of time doing this. The quality of the reviewer assignments has a tremendous impact on the quality of the reviews and the final decisions.
- You may find it helpful to maintain your own spreadsheet with your list of preferred reviewers for each paper and the final paper assignments.
After you have ranked reviewers, the Program Chairs, with the review system's help will distribute papers to the reviewers. During this phase, we recommend that you log into the system to check if any reviews are being entered. Remember that the reviewers do not know your identity, so it is best if you do not contact them and/or identify yourself. This is yet another form of "anonymity" in the review process, as we don't want the reviewers to be unduly influenced by anyone. If reviews are showing up, look at them and see if (a) the reviewer has added sufficient detail and (b) has used appropriate tone in the review, and most importantly (c) provided a knowledgeable review. Contact the reviewers via the review system to get them to clarify issues from the review. Doing this during the process will reduce how much you need to do at the deadline when reviews are due.
After the review deadline has passed, check if all reviews are in. If not, send them a note and nicely demand that they finish the reviews ASAP. The system will also send reminders to them.
-
In "Area Chair" console, select "Consolidation Reports".
-
You should be able to see all the reviewers' names and the review status right next to each. For this incarnation of CMT, we just highlight cases where reviews have not been submitted (in red).
-
For each paper, you can:
- View all the reviews ("View All").
- Email the reviewers anonymously via CMT ("Email Reviewers"). You can select all the assigned reviewers for that paper (default), or just those not done with reviewing, or specific reviewer(s). For the last item, you will be able to edit the recipient list. Remember: Don't identify yourself or the other reviewers in the email.
-
You can also choose to bulk email all reviewers or all reviewers who have not completed their reviews (for all the papers in your stack). See below the title "Consolidation Reports" ("Email Reviewer"). Mouse over the "Email Reviewer" box and you will see the two options (mentioned earlier) from which to choose.
-
You can view all reviews for all papers in a single webpage. Just select "View All Reviews" in the line right below the title "Consolidation Reports".
-
Important note on emailing via CMT: Since any email sent on your behalf is anonymously sent through CMT, the communication is one way only. Please tell your email recipients not to respond to the email, because there is no automatic routing of email to you. So, be very specific about what you would like the reviewer(s) to do, e.g., improve their review in a certain way.
-
It is the area chair's responsibility to contact and urge the reviewers to finish in a timely manner.
Right after the review deadline, please look through ALL the reviews.. If any review appears unreasonable (e.g., one-liner comments or overly harsh), please contact the reviewer and ask for clarification. Remember that communication is only one-way, so be specific about what you need from the reviewer. Check again in one or two days to see if there is any update to that review. Since we will not be keeping track of your emails, so please do the book-keeping yourself (best to make a note of what you've done). Again, you may find it helpful to create and maintain your own spreadsheet to keep track of the reviews, scores, and any high-level discussion items or to-do lists.
Please start a discussion any time you feel that the overall scores and/or the comments are insufficient to allow you to make a decision (e.g., all "Borderlines"). At the very least, initiate a discussion if the difference between the maximum and minimum scores is greater than 2 levels.
You play the role of a moderator in the discussion. While it is ideal to reach a consensus, don't feel obligated to force one - the discussions are mainly to allow you to make more informed decisions. It is ok if the reviewers end up not changing their overall scores as long as you've gained a better understanding of how they arrived at their scores. Stick to the facts, and do not influence the outcome of the discussion by imposing your views on the paper. However, if you disagree with the majority view, say so, provide compelling reasons, and solicit feedback from the reviewers. If the discussion looks like it is getting out of hand (e.g., when a reviewer makes inappropriate remarks or a battle of words erupts), please contact us (cvpr...@gmail.com) immediately. Don't forget to let us know the paper number. For cases of "Borderlines", if the discussion appears to not to be heading towards any decision, we suggest you ask the question "If you had to choose now, would you accept or reject this paper?"
This feature can be accessed from the "Detailed Meta-Reviews and Discussions" page:
- You must do the following steps to enable the reviewers to participate in the discussion. Emailing the reviewers via CMT is NOT the right way to initiate the discussion.
- To initiate a discussion for a specific paper, first enable the discussion and then select "View/Post Message" under the "Discussion" column for that paper.
- In the "Paper Discussion" page, click on "Start A New Topic" to initiate the discussion (unless a reviewer took the initiative of starting one, in which case you reply; see the next bullet item). We suggest that your first post starts with a very brief summary of the reviews, a request to look at the other reviews for details, followed by specific things you want the reviewers to address.
- Once a discussion has been initiated, click on "Reply" on the far right to continue posting on the same discussion thread.
- Please sign your posts as "Area Chair", and identify the reviewers by the review number. Never identify yourself or other reviewers by name.
- Anytime a post is made (either by the area chair or a reviewer), the area chair and reviewers will receive an email notification from CMT with the subject that looks like "CVPR2013: New reviewer discussion posted for Paper ID XXX". There is a link in the email you can use to join the discussion (after logging in, you will be routed directly to the discussion page). Alternatively, you can just log in to CMT as usual.
- After you've posted, do not refresh the page(e.g., by hitting F5)! This will generate another post with the exact same message!
- In the discussion, you (as AC) can see the identities of the posters. However, each reviewer will NOT know the identities of the other posters.
- Once the author rebuttal period is over, you and the reviewers will be able to see the author rebuttal.
- Because of the frank nature of the discussions, the authors will not see them at any time. The consolidation reports will be visible to authors only after the AC meeting, when decisions are made known.
- Reviewer instructions for the discussion feature are given here . Note that reviewers can revise their reviews until the review deadline.
During the author rebuttal period reviewers are no longer allowed to modify their reviews.
You should be able to see the Author Rebuttal/Feedback to the reviews. .You can start a discussion using the discussion board (see Phase 3: Discussion Phase) with the reviewers of the papers, especially if the rebuttal is responding to a specific review. While, the reviews cannot be changed, you can take the input from the reviewers at this phase to help in your proposed decision and the consolidation report that you have drafted (or will!). Make sure to be clear about how the rebuttal was taken into account in the consolidation report. Remember, all consolidation reports are considered a draft and all decisions are just proposed and will be finalized at the AC Meeting.
To allow you to see the reviewer scores on the consolidation page, in the Area Chair "Consolidation Report" page, you can customize the page by selecting "Customize The View"->"Edit Review Columns". Select Q2 ("Overall Rating") and click on "OK" to see the updated table.
The Area Chair Meeting will occur February 23-24, 2013 in Los Angeles.
- I recognize a piece of work (i.e., I know the authors) should I recuse myself from the paper? They are not collaborators and I have no financial ties or other connections with them. My feeling is that I can be impartial and handle the review but it is your call.
If you think you can do the review impartially and fairly, and there is no direct conflict, we think you should review this paper, as we feel you are best qualified for it. Thanks for letting us know.
- I have been assigned a paper that might be pretty close to things I am working on at present (I don't know for sure since I didn't want to look at the full paper). I would like to decline handling that paper, if possible, on grounds of potential conflict of interest.
Thanks for letting us know. Yes, this does qualify as a conflict of interest so we will assign to some other Area Chair.
- I see a paper that I know is authored by a recent collaborator of mine (collaborated in last 3 years). I am reasonably confident I can still render a fair and an impartial review. But wanted to let you know.
This appears to be a conflict. While we do trust you to be fair and impartial, it is best to have someone else take over this paper. Thanks for letting us know. We will swap it.
- I have a paper assigned to me that appears to be very similar to one that I have just co-authored for CVPR 2013. I can be still objective in review of this paper, irrespective of my submission.
Thanks for telling us. We feel you are the expert in the area and trust you will do a good job. If you feel you cannot, then we will re-assign to another area chair.
- What should I do with a paper with the names of the authors explicitly mentioned?
Papers with authors’ names explicitly violate the anonymity requirement of CVPR and will be rejected. Let the Program Chairs know the paper id.
- What should I do with a paper that did not exactly follow the required format?
If the paper is still 2 column style CVPR format (missing the line numbers, paper id on every page, etc.) and not longer than 8 pages (font not too small, margins ok, etc.), then it is fine to assign reviewers. (We don't want to be too rigid.)
- What about papers over 8 pages long?
Overall, our hardline policy is (as the authors were warned!) that the paper will be rejected. If you see a paper with just a citation or 1-2 lines on page 9, then we are letting it go. But if the paper is grossly over onto page 9, then it is an "administrative" reject.
- Is there a race to assign reviewers? If I do not assign reviewers right away, will I lose all the good reviewers for papers assigned to me?
No, there is NO race. You are recommending 7+ reviewers for each paper and on Dec 7 we will optimize the assignments based on (a) reviewer load, (b) your choice/rank of a reviewer, and (c) subject area match to reviewer. We hope that this way, good reviewers per your choice will be assigned the paper for which you are an AC. However, we do not want to see all of you waiting until the last week to assign papers.
- I have scanned a paper and it seems very weak and will be rejected, what should I do?
Unfortunately, if the paper has been submitted and does not meet the criteria for an administrative reject (non-anonymous, too long, dual-submission, etc.), the paper has to be reviewed. Give the paper a fair chance and have it reviewed. We agree that sometimes this is a waste of reviewing resources, but we feel the authors deserve the best from us.
- When should I decide to request a new reviewer that is not in the database of reviewers?
After you scan a paper to see its content, some obvious names of reviewers will come to your mind. Look for them in the reviewer lists. All reviewers should be visible to you (there are over 700 of them, so it is a bit slow). They are sorted by "subject area relevance". If you do not see the person you wish to review the paper, consider adding them as an additional reviewer. Consider however that (a) this person may already have been asked to review for CVPR13 and may have declined and (b) this person may not have the time to review this time. So do not assume that the new reviewer will agree to do this review. Add an extra reviewer as a back-up for this one.
- When and where is the Area Chairs' Meeting?
AC Meeting will be held on Feb 23-24. All AC are expected to arrive on Feb 22 (Fri), 2013, and be there until the session ends on the evening of Feb 26 (Sun), which will then be followed by a dinner. All ACs are invited to stay an extra day, Feb 25 (Mon), 2013 to participate in an informal workshop on computer vision and related topics.
- When should we make reservations for the AC meeting?
You will be getting instructions from us about how to plan for your travel and how you will be reimbursed in a separate e-mail.
|
