Instructions for the Review Panel Chairman¶
Before the proposal deadline
- Verify the members of the proposal review panel: List of Reviewers
- Answer questions from review panel members. CC the APS User Office.
After the proposal deadline
- List of Proposals
- Run the
- Create a new project file
- Import the proposals (from the XML file)
- Create the list of reviewers or Import from previous .agup file
- Edit reviewers to assign topic weights
- Assign topic weights to each proposal
- Assign Reviewers to Proposals
- Update the Assignments on the Web site
- Prepare the boilerplate letter
- Send emails
List of Reviewers¶
- Identify panel members whose terms expire soon.
- Is a temporary reviewer necessary?
- Identify new candidates to replace expiring panel members. Evaluate panel members strengths in the various topics for review and compare with proposal weights in each topic. This will point out whether the panel has sufficient strength to review proposals on each topic.
- Confirm the list of reviewers for the upcoming cycle.
- Verify that each member can attend.
- Assign values for reviewer strength on each topic. Use a scale from 0 (no strength) to 1.0 (high strength), with 0.1 precision.
In various example screen images shown in this documentation, the details of specific individuals or proposals have been removed or obscured to preserve anonymity.
List of Proposals¶
Login to your panel’s proposal review web site, and click the “Download the list in XML” link. Each panel has a slightly different URL. Consult the following table or contact the APS User Office:
You might also consider using the “Print All Proposals” link to generate a single PDF file containing all the proposals and attachments. Any problem attachments will be noted on separate pages at the end of the document. Be sure to save that on-screen image to a file or it will go away when you close your browser.
When that page appears, it will look unformatted and not like XML. Don’t worry, underneath the page is XML but is rendered poorly.
In your browser, choose “view source” (Control-U on Firefox) and save to a file such as
raw-proposals-2015-2.xml(use the name of the proper review cycle).
This raw file may have characters that break the later XML processing. That’s why I save it as raw and then make a working copy next.
Copy this file to
proposals-2015-2.xmland make one change first. Add a
periodattribute (if not present) to the root element
Review_listwith a value of the current review cycle, such as:
Be on the lookout for:
- proposals that belong on another panel
- PUP or project proposals (they require additional reviewer work)
- identical or related proposals (may not be the same PI or beam line)
- ineligible reviewer(s) (ineligible since reviewer name is on the proposal)
List of Topics¶
Central to this process is the creation of a list of topics. These topics represent the various scientific or experimental subjects represented by this suite of proposals. Each of the reviewers on the panel will have a different strength in each of the topics. This strength is represented as a decimal topic value, ranging between zero (no strength) to 1.0 (expert or confident). The dot product of the topic values between any reviewer and proposal can be used as a measure of how appropriate is that reviewer to evaluate that proposal.
Initially, the list of topics is extracted from the proposals from the list of subjects selected for each proposal by the proposal’s author.
The list of topics may be edited to remove such nondescript names as “Other” and to add in instrument-specific techniques such as XPCS for which specific expertise may not be so common amongst the panel reviewers.
Once the list of topics has been modified, check both the list of reviewers and the list of proposals that the topic values are assigned properly. (New topics added will be given topic values of 0.0.)
Assign topic weights to each proposal¶
Similar to the reviewers, each proposal must be assigned a topic weight. The weight (a floating point decimal number) is the relevance of each topic to the proposal. The scale is 0 to 1 where 0 means not related and 1 mean related.
When the proposals XML file is imported, a list of topics is created from all the subjects as selected by the proposers. Each topic created is assigned a weight of 1.0. These topic assessments should be verified since some proposers put down lots of extraneous topics that are weakly related to the specifics of the proposal.
Assign Reviewers to Proposals¶
Choose a primary and a secondary reviewer for each proposal. Most likely, these will be reviewers with the strongest topic weighting for this proposal.
The rules are such that only one reviewer can be a primary and a different one can be secondary.
If a reviewer is ineligible to review a proposal (usually because they are named on the proposal team), their name will be grey and the checkboxes to select them as either primary or secondary reviewer will be disabled.
Refer to the summary and analysis grid reports when attempting to balance the number of proposals assigned to each reviewer.
Update the Assignments on the Web site¶
The assignments report gives a listing of the assigned reviewers for all proposals. Looking at this page, enter the same information into the APS Review web site. There is no upload capability possible for this data, you must enter it in from the web form.
Send emails to each reviewer listing the proposals on which they are primary or secondary reviewer. CC the APS User Office on each email.
To prepare the template for the emails, open the Edit Email Template ... from the Editors menu:
To prepare the email text for each reviewer, select Email Letters ... from the Reports menu. A new window will appear for each reviewer with details specific to that reviewer. Use this to create an email to each reviewer.