Skip to content

Improve JsonSchemaCheckIT metrics reporting for defensible compatibility statistics #31

@simbo1905

Description

@simbo1905

Problem

The current JsonSchemaCheckIT test output is not defensible upon detailed examination. While it shows individual test failures and skips, it lacks comprehensive metrics that would allow us to make credible claims about compatibility percentages.

Current Issues

  1. No aggregate statistics: We can see individual failures but have no running totals
  2. Unclear skip categorization: All skips are treated the same without distinguishing between:
    • Unsupported schema groups (compile-time failures)
    • Test execution exceptions
    • Validation mismatches in lenient mode
  3. No structured output: Results can't be easily consumed by tools or CI systems
  4. Manual percentage calculation: The ~70% compatibility claim is an estimate, not measured

Proposed Solution

Implement comprehensive metrics collection in JsonSchemaCheckIT with:

Metrics to Track

  • Groups discovered: Total test groups found in test suite
  • Tests discovered: Total individual tests found
  • Validations run: Actual validation attempts made
  • Passed/Failed: Clear success/failure counts
  • Skipped categories:
    • unsupportedSchemaGroup: Whole groups skipped at compile time
    • testException: Individual tests that threw exceptions
    • lenientMismatch: Expected≠actual in lenient mode

Output Format

Console summary line:

JSON-SCHEMA SUITE (LENIENT): groups=142 testsScanned=1234 run=987 passed=701 failed=0 skipped={unsupported=81, exception=35, lenientMismatch=225}

Optional Artifacts

When -Djson.schema.metrics=json or -Djson.schema.metrics=csv:

  • Write target/json-schema-compat.{json|csv} with structured data
  • Include per-file breakdowns for detailed analysis

Acceptance Criteria

  1. Preserve existing behavior: Strict vs lenient semantics unchanged
  2. Thread-safe: Use concurrent data structures for parallel execution
  3. Zero dependencies: No additional libraries required
  4. Backwards compatible: Existing test runs work exactly as before
  5. Actionable metrics: Enable data-driven compatibility improvements

Implementation Approach

The solution involves:

  1. Adding a SuiteMetrics class with LongAdder counters
  2. Hooking metrics collection at existing decision points
  3. Adding @AfterAll method for final reporting
  4. Optional JSON/CSV export functionality

This will provide defensible, repeatable metrics for compatibility claims and help prioritize implementation efforts based on actual test coverage data.
EOF < /dev/null

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions