A passed policy check isn’t a fixed weakness

A system passes every check on the list and the report is green. Weeks later, a review finds a weak key protecting the same system. The check wasn’t wrong. It was answering a different question from the one the reviewer asked, and the report presented its answer as if it covered both.

Two questions that sound like one

A policy check asks whether a rule is met: is encryption configured, is the key at least the required length, are keys rotated on schedule. A security assessment asks whether something is weak: could this key be broken, has it been exposed, will this method stay safe.

The answers usually agree, which is why they get treated as one. When they disagree, they can disagree in either direction. A rule can require action where the technical risk is low, such as a rotation deadline on a key that’s perfectly strong. And passing a rule doesn’t erase a weakness found another way, such as a key that meets the minimum length but was copied to places it shouldn’t be.

How one score hides the gap

Roll the two into a single score and a pass on one can cancel a fail on the other without anyone noticing. A dashboard that averages compliance and risk will show a comfortable number while a real weakness sits underneath it.

The gap tends to surface at the worst moment: in an audit, after an incident, or in a regulator’s review, when someone asks a precise question and the single number can’t answer it. By then the conversation is about why the report was green, which is harder than fixing the key would have been.

Keep the answers apart

Report each answer on its own evidence. The rules result names the requirement, the check under it and the evidence found. The risk result names what is weak and why.

Treat missing evidence as its own result. If a rule needs proof that keys are rotated and the rotation logs weren’t collected, the honest answer is “not enough evidence”. That isn’t a pass, and it isn’t a fail; it’s a gap to close, with a clear next step.

Read a requirement as open until every check under it has an answer. A requirement to protect stored data might include one check that’s met and another that’s missing evidence, and the requirement stays open until both are settled.

What changes for your teams

Keeping the answers separate changes how work is planned. A low-risk finding that breaks a rule may still need action by a deadline. A high-risk weakness that no rule mentions still needs fixing. Seeing both side by side lets your teams schedule both, instead of chasing whichever one the latest report happened to show.

It also makes audits faster. Assessors decide compliance, and they work from evidence. When each conclusion arrives with its evidence and its gaps already named, there’s less to rebuild and fewer surprises, a theme covered in how to stop rediscovering the same gaps every audit.

A worked example

Illustrative: a requirement says stored customer data must be protected with strong encryption and well-managed keys. Two checks sit under it. The first confirms that encryption is switched on for the customer database, and it passes. The second asks for evidence that the database’s keys are rotated on schedule, and the rotation logs were never collected.

A single score marks the requirement 90% met. Kept apart, the answers say something more useful: encryption is in place, key rotation is unproven, and the requirement stays open until the logs are collected or rotation is fixed. Meanwhile, a separate security assessment finds that one backup copy of the database uses an older key that’s too short. No rule check mentioned it, and it needs fixing anyway.

Questions to ask of a compliance report

• Which requirement does each result belong to, and which checks sit under it?
• For each check, what evidence was found, and when was it collected?
• Where evidence was missing, is that shown as a gap or counted as a pass?
• Are security weaknesses that no rule mentions reported somewhere?

Where Arcrix™ helps

Arcrix™ by QTrino Labs answers three questions separately: is it safe today, will it stay safe, and does it meet the rules you answer to. A rules result stands on its own evidence and is never implied by the other two, and where the evidence a rule needs wasn’t collected, the result says so.

Reviewing evidence for selected card-payment requirements is not the same as being certified compliant. Your assessor still decides.