Package org.egothor.methodatlas.eval

Class AccuracyEvaluator

java.lang.Object
org.egothor.methodatlas.eval.AccuracyEvaluator
public final class AccuracyEvaluator extends Object
Joins AI predictions with human-reviewed labels and scores them with AccuracyMetrics.

Ground truth

The labels are a ClassificationOverride document: the same human-reviewed YAML that -override-file applies to scans. Resolution follows the override rules: a method-level entry wins over a class-level entry for the same class, and the last duplicate wins. An entry yields:

  • a relevance label when securityRelevant is present;
  • a tag label when tags is present, or — as a deliberate convention — an empty tag set when securityRelevant: false is present without tags (a reviewer who says "not security-relevant" asserts that no security tag applies). securityRelevant: true without tags labels relevance only.

An entry that carries neither (for example a note-only entry) is not a label and is ignored.

Predictions

Predictions are the ai_security_relevant and ai_tags columns of a MethodAtlas scan CSV produced with -ai. The scan used to produce them must not have been run with -override-file: overrides replace the AI values and would make the evaluation score the human's labels against themselves.

Coverage bookkeeping

Missing and extra entries never fail the evaluation; they are counted and reported so that a reader can judge how representative the numbers are: predictions without a label (AccuracyEvaluator.Evaluation.unlabelledPredictions()), labelled methods without an AI prediction (AccuracyEvaluator.Evaluation.unpredictedLabels()), and label entries that match no scanned method or class (AccuracyEvaluator.Evaluation.staleLabelEntries()).

This class is stateless and thread-safe.

Since:
6.1.0
See Also:

  • Method Details

    • evaluate

      public static AccuracyEvaluator.Evaluation evaluate(Path labelsFile, Path predictionsCsv) throws IOException
      Loads the label file and the predictions CSV and evaluates them.
      Parameters:
      labelsFile - override-format YAML holding the human-reviewed labels; must exist
      predictionsCsv - MethodAtlas scan CSV produced with -ai
      Returns:
      the evaluation; never null
      Throws:
      IOException - if either file cannot be read or parsed
      IllegalArgumentException - if the CSV lacks the required fqcn or method columns

    • evaluate

      public static AccuracyEvaluator.Evaluation evaluate(org.egothor.methodatlas.emit.ClassificationOverride labels, List<org.egothor.methodatlas.api.ScanRecord> predictions)
      Evaluates already-loaded labels against already-loaded predictions.
      Parameters:
      labels - human-reviewed labels; must not be null
      predictions - scan records carrying the AI columns; must not be null
      Returns:
      the evaluation; never null