I Built a Local, Privacy-First GRC Analysis Tool and Measured It Honestly
As promised, it’s live!
Mapping security controls to frameworks is slow, repetitive, judgment-heavy work. You read a control statement, decide which NIST CSF subcategory it actually satisfies, check whether it also touches CIS or the AI RMF, note what's missing, and write it up. Then you do it a few hundred more times. Anyone who has built or maintained a control library knows the particular fatigue of it. The work is genuinely expert, but a lot of it is expert pattern-matching, and pattern-matching is exactly the kind of thing a model might help with.
So I spent the last few months finding out. The result is Lacunae ControlSense, and as of this week it's public.
What it is
In its current state, ControlSense ingests a security control statement, or an entire policy document and does four things: it maps the control to recognized frameworks (NIST CSF 2.0, CIS Controls v8, NIST AI RMF, and the OWASP LLM Top 10), it identifies coverage gaps, it suggests remediation, and it exports an auditable gap register you can actually hand to someone. For my friends in the DoD, this can be used as a POA&M for any accreditation work that you are doing. As a note, NIST 800-53 will be in the next large release.
The design constraint that shaped everything: it runs entirely on your own hardware after install. It's a fine-tuned Mistral-7B served locally through Ollama. No API calls, no telemetry, nothing about your policies leaving the machine. You don’t want that and I most certainly don’t want that. For GRC work that touches sensitive internal documents, "local-only" wasn't a nice-to-have it was the whole point. A tool that ships your control environment to a third-party API is a tool most security teams can't use on their real documents.
I'm not a developer and that's part of the story
I should be upfront: I come to this as a security governance practitioner, not a career software engineer. I have developed in the past, I instruct developer courses but it’s not my day to day job. This was a deliberate, hands-on project and I wanted to learn the modern AI stack by building something real with it, not by reading about it.
I leaned on AI-assisted development heavily, and I'm not going to pretend that I didn’t. I was also testing that for my growth. What I brought was the part I actually have: the domain judgment. The training data is 168 examples spanning 12 GRC domains, from access management to AI governance to business continuity that I authored myself, because knowing what a correct control-to-framework mapping looks like is the expertise, and that's mine. The model learned from a practitioner's judgment, even if I needed help wiring up the QLoRA fine-tune around it.
If you're a domain expert who has wondered whether you can build the thing instead of waiting for a vendor to build it for you: you can. It's more achievable than it looks. It is also, as I'll get to, harder to do honestly than it looks. I expected this though.
The part most launch posts skip: I measured it, and the number was humbling
Here's where I want to be different from the usual "look what I shipped" post.
I built an evaluation harness which included 100 hand-curated control statements, each with expected framework mappings I'd adjudicated by hand. Then I ran the model against it. The headline result:
53.1% required-mapping accuracy, scoped to NIST CSF 2.0 where the framework my training data covered most densely.
Not ninety-something. Fifty-three.
I sat with that number for a while, because the temptation to make it look better was real and immediate. I think this is a bit of my perfectionist coming through. I could have loosened the scoring. I could have rewritten my answer key to match whatever the model happened to prefer. Instead I went the other direction: I reviewed every single miss by hand to confirm it was a genuine model error and not a flaw in my key. It mostly held and what I learned from the misses was more useful than a flattering aggregate would have been. To understand why that matters, it helps to know how these frameworks are organized. NIST CSF doesn't list controls as a flat pile, instead it groups closely related ones into families. The access-control family, for instance, holds several neighboring subcategories that all deal with identity and authentication, differing in emphasis rather than topic. So when a model has to choose, "right family, wrong member" and "completely wrong area" are two very different kinds of error and they carry very different risk for the person relying on the output. The model's errors are largely defensible-but-imprecise: it reaches for an adjacent subcategory in the right family rather than producing nonsense. That's a known, characterizable weakness, not random failure.
I also measured something I cared about more than raw accuracy: how often does it fabricate framework IDs or cite a category code that doesn't actually exist? At the model level, that was 5.2%, almost entirely retired NIST CSF 1.1 numbers bleeding through from the base model's pretraining. So I built a runtime validator that checks every ID against the official framework enumerations and rejects anything that isn't real before it ever reaches the user. At the product level, the fabricated-ID rate is 0.0%. A control the model can't confidently and validly map gets surfaced as "no confident mapping" never as a confident wrong answer.
That distinction, 5.2% at the model level, 0.0% at the product level, is the actual engineering of this thing. The honest move wasn't to hide the model's hallucinations. It was to measure them, then make them structurally unable to mislead someone.
How it works, briefly
For the technically curious and those hoping to learn from this as I did: Mistral-7B-Instruct, fine-tuned with QLoRA on those 168 examples, merged and quantized to Q8_0 GGUF, served through Ollama on a consumer GPU (it runs comfortably on an RTX 4080-class card). ChromaDB and a small embedding model provide organizational document context in "org mode." A Gradio UI ties it together, and the whole stack comes up as two containers under Docker Compose. Model output is schema-constrained and validated before display — and that validator is where the 0.0% fabrication rate comes from.
The full source, the evaluation methodology, and a build path I verified end-to-end from a clean clone are all on GitHub. (Getting it to actually reproduce from scratch surfaced seven separate bugs that only existed in the gap between "works on my machine" and "works on a stranger's" which is its own lesson, and maybe its own future post.)
What it is not
It is not an oracle, and I'd rather say so plainly than let anyone find out the hard way. For anyone who knows me, knows that I’m a direct, transparent person.
ControlSense is a decision-support tool. Its NIST AI RMF and OWASP LLM coverage is thinner than its CSF coverage, because I had fewer training examples there, so I report those as emerging rather than measured-and-trusted, and I'm honest about that in the model card. The model is non-deterministic. And every mapping it produces is meant to be reviewed by a qualified practitioner before it informs any audit, attestation, or compliance decision. Just like a ‘chatbot’ this is a tool.
The goal was never to replace your judgment. It was to take the blank-page first pass off your plate so your judgment starts from a draft instead of from nothing.
Where to find it
It's open source under a noncommercial license. The fine-tuned adapter is on Hugging Face; the full source, eval harness, and documentation are on GitHub.
- GitHub: github.com/mrBenL/controlsense (release v1.0.0)
- Hugging Face: huggingface.co/MrBenL/controlsense
- License: PolyForm Noncommercial 1.0.0 — commercial inquiries: support@benluthy.com
If you do GRC work and want to run it against your own controls, I'd genuinely like to know how it does, including where it falls short. Please provide feedback! And if you're a practitioner building something similar, I'm always up to compare notes. This started as a question about whether the tedious first pass could be automated honestly. I think the answer is a qualified yes, emphasis on honestly.
