wiki_ghostguild/content/wiki/hub/reference/01-rubric-and-scoring-details.md

Rubric & Scoring Details

This page covers how scoring works end to end in Cohort-OS -- from how rubrics are structured, through per-criterion scoring by individual reviewers, to how consensus scores are calculated across the review panel.

Rubric Structure

Rubrics are defined in Flywheel (the external assessment engine) and linked to each review stage. A rubric contains:

• Criteria -- The individual dimensions being evaluated (e.g., "Mission Alignment", "Governance Readiness"). Each criterion has a name, description, and maximum point value.
• Thresholds (or levels) -- Descriptive scoring bands within each criterion. Each threshold has a title, a description of what that performance level looks like, and a point range. Reviewers select the threshold that best matches the applicant.
• Max points -- Each criterion defines its own maximum score (commonly 5 points). The rubric's total possible score is the sum of all criteria max points.

Each review stage in a cohort is linked to a specific Flywheel rubric via the stage's configuration. Different stages can use different rubrics.

How Reviewers Score

Reviewers use the guided scoring interface to evaluate applications criterion by criterion. For each criterion:

1. Read the criterion description and the applicant's relevant answers (shown side by side).
2. Select a threshold/level that matches the applicant's response. Selecting a threshold automatically assigns the corresponding point value.
3. Optionally add notes specific to that criterion.
4. All criteria must be scored before proceeding to the recommendation step.

After scoring all criteria, the reviewer sees their total score (sum of all criterion scores) and percentage (total divided by maximum possible), then selects a recommendation and writes overall notes.

:::info If a rubric does not have thresholds configured in Flywheel, reviewers see simple numeric score buttons (1 through the max points) instead of descriptive threshold options. :::

Individual Review Scores

Each completed review stores:

• Per-criterion scores -- The point value and max value for each criterion, plus optional notes.
• Total score -- Sum of all criterion scores.
• Percentage -- Total score divided by total max points, expressed as a percentage.
• Recommendation -- One of advance, reject, hold, or undecided.
• Overall notes -- Free-text assessment from the reviewer.

A review starts as draft (editable, can be saved and returned to) and becomes completed when submitted. Submitting locks the scores and notes. The recommendation can still be changed after submission with a recorded reason.

Consensus Calculation

When multiple reviewers have completed their reviews for the same application and stage, the consensus view aggregates their scores. The consensus endpoint calculates:

• Average score -- Mean of all reviewers' total scores.
• Average percentage -- Mean of all reviewers' percentage scores.
• Per-criterion averages -- For each criterion, the mean score across all reviewers.
• Score spread -- The difference between the highest and lowest score for each criterion. A criterion is flagged when the spread is 2 or more points, or 40% or more of the max points. Flagged criteria indicate significant disagreement between reviewers.
• Recommendation tally -- Count of advance, hold, reject, and undecided recommendations.

:::warning Only completed reviews with review permission are included in consensus calculations. View-only assignments and draft reviews are excluded. :::

Grade Bands

Cohorts can define grade bands that map percentage ranges to letter grades:

Band	Range	Label
A	90-100%	Exceptional
B	80-89%	Strong
C	70-79%	Good
D	60-69%	Acceptable
F	0-59%	Needs Work

Grade bands are configured per cohort in the Flywheel integration settings. They provide a quick way to categorize application strength during committee discussions.

Blind Review

When blind review is enabled on a stage (the default), reviewer identities are hidden from other reviewers viewing the consensus. Reviewers appear as "Reviewer 1", "Reviewer 2", etc. Admins always see full reviewer names and emails regardless of the blind review setting.

Blind review affects the consensus view only -- it does not change what reviewers see while scoring. Each reviewer always scores independently without seeing other reviewers' scores until their own review is submitted.