Note

Audience: Jury Manager, Reviewers

Assigning reviewer to candidates

Once (most of) the reviewers are registered in the application, you will want to assign reviewers to candidates. You can do it manually, but an algorithm have also be implemented to help you in this task.

The algorithm

The algorithm transforms the assignation of reviews into a mathematical problem (Integer linear programming optimization problem) to find an optimal assignation. As a jury manager, you specify an amount of reviews per candidate, and the algorithm assign reviewers while trying to minimize the overall number of reviews. In order to reduce the work for each reviewers, the algorithm tends to minimize the maximum of reviews each reviewer has to do.

The algorithm works in a best effort approach, it means that if 3 reviews are asked, but only two reviewers are available for a candidate, it will propose an assignation with only two reviews to do for this candidate, and three for the others.

The algorithm assign reviewers to a candidate only when they share at least one profile, although this feature can be disable.

The algorithm will never assign a reviewer to a candidate for which the reviewer have a conflict of interest.

The algorithm will never remove a review (i.e un-assign a candidate from a reviewer) if the review have been (partially) filled.

When there are multiple stages of review, the algorithm have to be run independently on each stage.

Running and re-running the auto assignation

To run the auto assignation, go to https://sincellte-workshop.pasteur.cloud/setup/review/overview/, clic on “Compute automatic assignation”. Indicate on which stage you are working and the amount of review. You can adjust the scope of reviewers and candidates.

Once clicked on “Compute automatic assignation”, the button is disabled and the algorithm runs. It can take some time to complete. You cannot stop the algorithm once started. See here-after in “Performance” section to have an idea of the time it can take.

You can re-run the automatic assignation, it is useful when new reviewers have joined, or when new conflict of interest have been provided. In such case the algorithm re-assign candidate to reviewers while preserving already provided reviews.

Performance

Here are some benchmark on time consumption.

How a review is protected, how it can be deleted

A review can be marked as manually assigned, or automatically assigned. When being manually assigned the algorithm will never delete the review. If a review is marked as automatically assigned and is not empty, then it cannot be removed either.

In other word: to be removed, a review have to be marked as automatically assigned AND be empty.

Example use cases where a review can be removed when re-running the auto assignation :

• If a reviewer indicate that zhe have a conflict of interest with a candidate and if the reviewer hasn’t started to fill it.

• If new reviewers have been added since the last automatic assignation then empty review can be re-assigned to these new reviewers.

• If a reviewer has changed its profils then empty review can be re-assign.

How To protect an automatic assignation

By going to the review list (https://sincellte-workshop.pasteur.cloud/review/) you can select some review, and use the action “Define as manually assigned”. They will thus be protected against re-assignation, even if they are empty.

Manual assignation

If you want to assign a reviewer to a candidate, either go in the review list (https://sincellte-workshop.pasteur.cloud/review/) or in the review assignation page (https://sincellte-workshop.pasteur.cloud/setup/review/overview/). Clic on “+ Review”, select the review phase, the reviewer and the candidate.

Manual assignation will not be remove/replace when re-running reviewer assignation. This ensure you that your manual tuning of the review process is respected. If you change your mind, you can set the review as manually assigned, and the review can be removed in a later auto assignation (while respecting rules explained here-before).