An evaluation finds where a building block breaks
A check tells whether one run was faithful to its inputs. An evaluation tells whether a building block can be trusted, under which conditions, and at which point it fails. Developing and proposing this protocol is the centrepiece of the development stage.
Benchmark tests
Compare a step or template with known answers, for instance an interview corpus that human coders have already coded.
Reproduction tests
Run a template on a published study and compare its results with the original.
Stress tests
Vary document length, corpus, data quality or model to locate the point at which output degrades.
Expert assessments
Researchers rate a building block against a shared rubric and report failures. RISE already scores research systems against a published rubric.
Real records
1 real recordEvaluation records
Example with invented dataThe records below concern hypothetical building blocks. They show what a researcher would read before relying on a block.
Compare two versions of a block
ExampleThe same stress test on two versions of the hypothetical qualitative-coding template. A researcher can see where the new version moved the breaking point before switching.
How a stress test reports a breaking point
Example with invented dataA hypothetical coding template, evaluated on transcripts of increasing length. The evaluation record states the length at which agreement with human coders falls below the threshold the template declares.
An evaluation record
Example formatblock: qualitative-coding@0.3.1 test: stress / document length material: interview corpus, 240 transcripts, coded by two humans model: named model and version runs: all runs, incl. failed and aborted ones result: agreement by length band; breaking point where it falls below threshold evaluator: name, ORCID, contributor role "validation" scope: valid for this material; other corpora need their own record
Records include failed and aborted runs, because reporting only successful runs overstates performance.
Evaluate a building block
Evaluators author a citable record with the contributor role "validation". Reporting a failure earns credit in the block's change history and in the evaluation it triggers.