Rule Violation Score Measures What Accuracy Misses: Logical Compliance

Two predictive models can produce the exact same accuracy on a benchmark while one quietly violates a hospital's treatment protocol and the other follows every rule - and until now, no standard metric captured that difference.

Accuracy, ranking quality, and prediction error all measure how tightly predictions match ground truth. None of them check whether the model respects logical constraints you actually care about. For high-stakes domains like healthcare, finance, and autonomous systems, that blind spot is a liability.

Why Accuracy Alone Is a Dangerous Metric

The authors behind arXiv:2606.20208 introduce the Rule Violation Score (RVS), a complementary metric that quantifies logical compliance independently of predictive accuracy. RVS separates hard rules - strict constraints that must never be broken - from soft rules, which represent statistical regularities where minor violations are tolerable. This distinction matters: a model that occasionally flips a soft rule might still be safe, but one that breaks a hard rule even once could be catastrophic.

RVS works on any dataset and any predictive model expressed over a relational vocabulary. The clever part: it automatically generates SQL queries for Horn rules, letting you compute violations directly on your database without custom code per rule set.

How RVS Reveals Hidden Model Behavior

The team tested RVS on three benchmarks covering knowledge graph link prediction and relational regression. They compared rule-based, embedding-based, and neuro-symbolic predictive models. Results show two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance. Standard metrics flatten these distinctions; RVS surfaces them.

This isn't an academic curiosity. If you deploy a model in a clinical decision support system, the one that matches patient outcomes equally well but ignores a dosage constraint is dangerous. RVS gives you a concrete number to catch that before it reaches production.

Beyond evaluating models, RVS can also score the logical consistency of training datasets and help you spot poorly defined rules. That means it doubles as a data quality tool.

RVS is what should sit next to your loss curve and validation accuracy from day one - especially when the cost of a rule violation is measured in more than just points on a leaderboard.

Source: Beyond Accuracy: Measuring Logical Compliance of Predictive Models
Domain: arxiv.org