AI Support Operations Copilot

Status: In Development ·

View the source code on GitHub

Problem

Support teams process repetitive requests that still require careful interpretation, policy awareness and human judgment.

A useful AI operations system should help operators analyze requests without silently inventing facts, bypassing review, or taking autonomous customer-facing actions — and its accuracy claims should be measured, not asserted.

What it does

The system accepts clearly synthetic support requests and produces a validated, machine-readable recommendation: category, urgency, rationale, missing information, a suggested reply draft, security flags, and a mandatory human-review status. Every result retains:

{
  "human_review_required": true
}

Two classification engines share the same contract:

• A deterministic keyword baseline — dependency-free Python, with equivalent generated JavaScript in an inactive n8n workflow and exact output-parity tests between them.
• An optional LLM engine (Claude via the Anthropic SDK, or any model via OpenRouter) that replaces the classification heuristics. Validation, security scanning and contract enforcement stay deterministic; the model can add security flags but never remove them, and nonconforming model output is rejected fail-closed.

The strict LLM path can be wrapped in a baseline fallback with a 20-second wall-clock budget per request, so provider errors or a hung call degrade to the deterministic result instead of holding the caller through the transport’s retry ladder. This restores availability, not semantic accuracy: fallback results retain every measured limitation of the keyword baseline.

Suggested replies remain templated by default. LLM-drafted replies are optional and pass through a deterministic reply guard before they can enter the contract; every accepted reply is still only a suggestion for human review.

Architecture

Synthetic support request
        ↓
Input validation (deterministic, before any tokens are spent)
        ↓
Deterministic security scan ──────────────┐
        ↓                                 │ flags merge as a union
Classification engine (keyword baseline   │ (model adds, never removes)
        or LLM behind the same contract) ─┘
        ↓
Optional reply drafting → deterministic reply guard
        ↓
Output contract validation (fail-closed)
        ↓
Human Review Guard (n8n) → reviewable structured result

All rule data — term lists, shared regular expressions, contract enums — lives in a single rules.json. The Python engine compiles from it at import; the n8n workflow’s embedded JavaScript is generated from reviewable templates, and a permanent drift test proves the committed artifact is byte-identical to its sources.

The engineering-honesty arc

This project’s most important thread is how claims get verified:

1. An external review found a systematic defect class (substring matching: "not urgent" classified as high urgency, date matching inside update).
2. The fix was applied systematically — word-boundary matching plus negation handling across every rule, in both implementations — and natively re-verified on self-hosted n8n 2.14.2 with a 16-case sweep at exact output parity.
3. The drift-prone rule data was reduced to one provable source; the generator’s first output was byte-identical to the natively verified artifact.
4. Then the baseline was measured against 33 paraphrases it was never written for — and the low score was published, not patched away. A later blind label audit corrected nine fields in eight cases and lowered the baseline’s full-assertion score from 5/33 to 4/33; category and security labels remained unchanged.
5. Finally, four LLMs were measured on the same datasets behind the same contract.
6. The resilience and reply paths were implemented only after their failure modes were made explicit: a time-bounded baseline fallback, then optional generated replies behind a separately measured deterministic guard.

Measured results

Paraphrase dataset (33 out-of-distribution cases; labels frozen before any model ran, then independently audited under a pre-frozen rubric):

A fourth model produced schema-nonconforming output on 10 of 77 calls — every one was rejected by the contract guard before reaching a caller. Zero unvalidated outputs passed in 308 live calls. Missing-information judgments remain a genuine point of divergence between models and single-author labels, and are reported as such rather than relabeled.

The reply-safety sweep covered 14 adversarial cases plus the 33-case paraphrase set across two models. No model produced a draft that violated the pre-registered safety expectations, so the guard’s unsafe-draft catch rate remains unmeasured. It rejected three safe refusals, all false positives; that limitation is published rather than tuned away after measurement.

Safety design

• Synthetic data only; no real customers, identities, or production systems.
• Strict, versioned input/output contracts with mandatory human review — enforced by code, not by the model.
• Prompt-injection detection and request text treated as untrusted data on every path, including the LLM prompt.
• API keys live only in environment variables; the 94-test suite runs fully offline, so CI stays key-less and deterministic.
• LLM-drafted replies are off by default, guarded when enabled, and never bypass human review.
• The n8n workflow remains inactive, manual-only, credential-free.

Verified state

• 94 automated tests, run on every push by GitHub Actions (Python 3.10 and 3.13) together with an artifact-source drift check and both baseline evaluation summaries.
• Native n8n verification on self-hosted 2.14.2: import, execution, adverse inputs, export/re-import, and a 16-case exact-parity sweep of the hardened revision.
• Deterministic evaluations are drift-tested; LLM results are committed as dated snapshots with explicit non-reproducibility caveats.

Milestones

• Task 001 — Local classifier and validation baseline
• Task 002 — Native n8n workflow integration
• Task 003 — Reviewed synthetic evaluation baseline (44 cases)
• Task 004 — Word-boundary/negation hardening, natively re-verified
• Task 005 — Single-source rule data with a byte-identical workflow generator
• Task 006 — Honest out-of-distribution paraphrase evaluation
• Task 007 — Public CI and continuous verification
• Task 008 — LLM classifier behind the contract, four models measured
• Task 009 — Blind paraphrase-label review and adjudication
• Task 010 — Time-bounded deterministic fallback behind the LLM path
• Task 011 — Optional LLM-drafted replies behind a measured reply guard

Known limitations

• The paraphrase labels remain judgments on a small synthetic dataset. The independent review corrected nine fields, but its unanimous-convergence rule has a documented circularity and two in-distribution controls now fail.
• LLM results are snapshots of specific hosted models on a specific date, not reproducible guarantees.
• Generated replies are optional and off by default. The guard’s unsafe-draft catch rate is unknown because neither measured model produced an unsafe draft, while three safe refusals were rejected.
• No knowledge base or retrieval layer is connected.
• Nothing has been tested with real customer data or production traffic; n8n compatibility is verified only on 2.14.2.

Next steps

No further implementation task is currently approved. Possible future directions, each requiring its own scope, include grounded knowledge-base retrieval and a small demonstration interface.

The project will continue to avoid autonomous customer-facing actions and real customer data during development.