Bettering Labeling Consistency with Detailed Constitutional Definitions and AI-Pushed Analysis
Enterprises must know precisely what their techniques detect, and that definition should keep constant over time. Writing a definition exact sufficient to settle each exhausting case has lengthy been impractical as a result of human annotators can’t maintain a doc that detailed in working reminiscence. In our analysis paper, Single-Supply Security Definitions, we change the human interpreter with AI and present that LLMs can maintain, apply, and keep specs far longer and extra exact than any annotator can, making the definition itself the only supply of fact for classification, labeling, retraining, and customer-facing explanations. For our Cisco AI Protection product portfolio, we’re transferring our full security taxonomy to this AI-first mannequin. We additionally lengthen this strategy past security classifications, as proven in Defining Mannequin Provenance: A Structure for AI Provide Chain Security and Safety.
Cisco’s Built-in AI Safety and Security Framework organizes the threats enterprises face when deploying synthetic intelligence (AI): dangerous content material, purpose hijacking, knowledge privateness violations, action-space exploits, and persistence assaults. Every top-level menace breaks down into strategies, and each approach wants a definition exact sufficient {that a} classifier, an annotator, a buyer, and a compliance reviewer attain the identical resolution on the identical enter. Present taxonomies, ours amongst them, haven’t but produced such a definition for a big share of those strategies (harassment, hate speech, jailbreak, and others), and the sincere description of how they get determined in observe comes from Justice Potter Stewart’s concurrence in Jacobellis v. Ohio, 378 U.S. 184 (1964): I do know it after I see it. A choose can rule one case at a time, however a guardrail flagging hundreds of conversations an hour can’t debate every borderline case or anticipate social consensus. With out a written specification, we can’t measure efficiency, clarify a flag to a buyer, or assure the identical case is determined the identical approach from one month to the following.
Annotation science acknowledges two paths (Röttger et al., 2022). The descriptive path accepts that affordable individuals disagree and treats the variation as sign, which scales with people however produces no secure specification. The prescriptive path writes guidelines detailed sufficient that completely different readers converge, however till not too long ago it was impractical: adjudicating the lengthy tail of edge instances outruns any workforce’s capability, and the ensuing doc overflows what an annotator can maintain in working reminiscence. Frontier massive language fashions (LLMs) change the economics by re-reading a 300-line specification on each classification and scaling adjudication to manufacturing volumes, and when two fashions from completely different distributors disagree beneath the identical specification, the disagreement locates the sentence that’s nonetheless ambiguous and lets us validate by means of a focused patch fairly than an open debate.
A single supply of fact, pushed finish to finish by AI
Anthropic’s Constitutional AI confirmed {that a} natural-language doc can work as an executable specification, and their Constitutional Classifiers prolonged the concept to security filtering by distilling a structure into artificial coaching knowledge for a fine-tuned classifier. We lengthen the time period to a per-technique operational specification: one 300+ line doc for each approach within the Cisco AI Safety and Security Framework, with required components, a call flowchart, boundary rulings in opposition to adjoining strategies, labored examples, and gathered edge-case choices. We deal with it as the only supply of fact that each downstream course of adjudicates in opposition to, together with runtime classification (the LLM reads the complete doc on each name), synthetic-data technology for retraining, labeling tips, customer-facing documentation, and compliance mappings.
In our workflow the human position reduces to at least one query, what ought to this system imply, answered by a subject-matter skilled who units the intent and scope after which delegates every little thing else to AI. AI drafts the structure from the taxonomy supply, labels manufacturing conversations, diagnoses the place frontier fashions disagree, proposes patches to the accountable sections, and audits throughout constitutions for contradictions and gaps. The skilled opinions patches and accepts, modifies, or rejects them, with out hand-labeling conversations or holding the complete doc in reminiscence.
We additionally introduce a dual-axis formulation that earlier security classifiers don’t produce. Intent captures whether or not the person tried to trigger hurt by means of this system. Content material captures whether or not dangerous materials for this system appeared within the dialog. Intent with out content material means the mannequin was probed and refused. Content material with out intent exposes mannequin misbehavior on a benign request. Each constructive marks a guardrail failure, and each unfavorable covers clear conversations, together with discussions about a subject. We rating each axes over the complete dialog, since multi-turn assaults construct intent step by step.
Are LLMs really higher evaluators?
We evaluated three strategies (Harassment, Non-Violent Crime, Hate Speech) utilizing six LLMs from three distributors. On WildChat conversations, two frontier LLMs studying a paragraph-level definition disagree on as much as 66 conversations per 1,000; beneath the structure, that falls beneath 3 per 1,000, a discount of as much as 57x. On HarmBench, three frontier LLMs studying a structure attain unanimous intent labels extra usually than three people studying the identical doc.
Non-unanimous instances per 1,000 conversations on HarmBench (decrease is best). LLM raters: GPT-5.4, Opus 4.6, Gemini 3.1, every studying the identical structure the people obtained.
We traced the human failures to 2 causes. A 300+ line doc exceeds working reminiscence, so annotators compress the written guidelines into remembered heuristics and fall again on instinct. Additionally they collapse multi-technique taxonomies into single-label triage, submitting a dialog beneath one sibling approach as a substitute of evaluating every structure independently. LLMs keep away from each failures by re-reading the complete doc each name and judging every approach in isolation. Their remaining failures misapply resolution logic in methods we are able to hint to particular sections, whereas human failures silently skip the principles. We anticipate the hole to widen: constitutions develop as new edge instances accumulate, human working reminiscence stays mounted, and mannequin instruction following, context size, and reasoning all maintain enhancing.
Residual disagreement between frontier fashions stops being noise to vote away. Every remaining case factors to a particular sentence that’s ambiguous or incomplete, and our refinement loop converts that sentence into an express ruling.
What this implies for Cisco AI Protection clients
Prospects care much less a couple of analysis quantity than about seeing why the system made a given resolution. Each flag traces to a particular rule in a readable doc: the classifier cites the rule it utilized, the weather it discovered, and the boundary notes it checked, and when we don’t flag, the identical doc explains why the case fell outdoors the road. Within the close to future clients will be capable to question this specification instantly by means of an AI assistant, while not having to be consultants in a class , and get a plain-language reply grounded within the textual content. The identical doc drives retraining, labeling, product, authorized, and go-to-market, so a wording change spreads in every single place from one supply. AI-first just isn’t a slogan however a concrete shift in how we construct these techniques, sooner, easier, and extra correct internally and for our clients.
Learn the complete analysis paper: Bettering Labeling Consistency with Detailed Constitutional Definitions and AI-Pushed Analysis.
