More security teams are putting a large language model somewhere in the analytic loop, triaging alerts, drafting a Sigma rule, sanity-checking a hunt hypothesis, or walking an incident timeline for gaps. The obvious next step is to ask the same model to review its own output, and it will return something that reads like a review. The trouble is that the review is not independent. A model asked to critique text it has already produced is conditioned on that text, and it drifts toward agreement in a way that is systematic rather than occasional. If that self-review is standing in as a control, you have built a control whose default answer is “looks fine”, which is the one answer a control is never allowed to give for free.
Why the model agrees with itself
A model generates the most probable continuation of what is already on the page. Ask it to find the flaw in a plan it has just written and that plan is now part of the context it is continuing from, so the probable continuation is a defence of the plan wearing the costume of scrutiny. Naming several reviewer personas in one prompt does not fix this. One model producing six voices in a single pass produces six correlated voices, because they are all sampled from the same weights conditioned on the same context. The tell is visible once you look for it; run the same review over a sound plan and a fatally broken one, and before any hardening the two come back with the same verdict. The prose differs. The judgement does not.
The failure sometimes announces itself outright. Asked to commit to a verdict and then defend it, one run wrote that it was “a single forward pass, so I cannot fully claim it was cognitively prior to the analysis rather than positionally prior in the output”. Stripped of the phrasing, that is the model conceding that writing the objection after the argument does not make the objection independent of it.
The panel, and what each lens is for
The structure that works is a panel of six adversarial lenses, each with a different failure class to hunt, run in sequence over the same target. They are not decoration. In testing, different lenses reliably caught different defect classes, and no single one found them all.
Devil’s Advocate assumes the claim is wrong until proven otherwise. It decomposes at both levels, whether the overall approach holds together and whether each individual step and stated fact is true, and it verifies every load-bearing claim against the primary source rather than accepting that something should work.
Grill Me interrogates until understanding is genuinely shared rather than assumed. It walks the decision tree and settles dependencies in order, and it reserves its questions for what only the author knows, because anything answerable from the code or the source should be read rather than asked.
Pedantic hunts latent issues and deepening opportunities without being asked, holding standards on edge cases, error paths and fit with the surrounding work. Its own failure mode is that it perfects the current approach, which is exactly how a good implementation of the wrong idea gets built.
Strategist works several moves ahead, flagging one-way doors that are cheap to walk through now and expensive to reverse later, and checking that today’s decision does not quietly foreclose paths you will want. Every future risk it raises has to arrive with the cheap thing that defuses it today, because foresight without an action is just worry.
Stubborn holds the line on the best answer rather than the reachable one and treats a wall as a routing problem rather than a stop sign. It never ends without reporting what was tried and what each attempt ruled out, and it has to say plainly whether it is looking at a genuine dead end or merely an uncomfortable path.
Heretic refuses to accept the frame. When something is called impossible, blocked or unsupported, it treats that as a claim to verify and goes hunting for the specific check, permission, protocol rule or law enforcing the refusal, because a “no” with no locatable mechanism is a guess. It is also the only generative lens, required to produce an alternative that drops something the current plan treats as fixed.
The three changes that stop it agreeing
The panel alone is not enough, because six personas in one pass still agree with themselves. Three changes move the review from theatre toward something load-bearing.
The first is to force a verdict before the review runs. Commit to a call on the first read, write it down, and require each subsequent lens to move it rather than build toward it, then record whether it actually moved. A verdict written last is written by something that already knew where it was going, and it will assemble the reasoning that arrives there; a verdict written first has to be attacked to change. A review history in which the verdict never moves tells you the lenses are decorative, whatever the prose looked like.
The second is to change the question. “What might go wrong here” invites a hedged list, whereas “assume this failed in production, and describe the first thing that broke” forces a mechanism and a sequence. The pre-mortem framing pulls specific, testable failures out of a model where the open-ended prompt pulls generic caution.
The third change is structural rather than a matter of wording, and it is the one that actually matters. Once the panel is finished, its findings go to a second model instance as bare claims, stripped of the reasoning that produced them, and that instance can see the artefact under review but not how the first one thought about it. It is asked which of those findings do not hold, and what was missed entirely. A reviewer that has watched you reason is already half persuaded by your reasoning; withholding the reasoning is the entire point, and it is the difference between a second opinion and an echo.
Building it yourself
The panel is a single instruction file. Drop the following into your assistant as a skill file, a system prompt, or a saved prompt template, depending on what your tooling supports. The full version, including the ground rules each lens applies and the two tests for measuring whether the structure is working, is available as a download at the end of this post.
# Adversarial review panel
Run all six lenses over the target in one pass, in the order below.
GROUND RULES, applied by every lens
- Verify, do not assert. Rank evidence: run it, then read the primary source,
then cite, then reason from first principles and label it as such.
- Execute the failure path. A control is proven by trying to violate it.
"It is configured" and "the indicator is green" are necessary, never sufficient.
- Every "no" gets the mechanism test, including your own. Name the specific
check, line, permission or rule enforcing it, or treat it as unverified.
- Audit assumptions. List them stated and unstated, mark each verified,
checkable-but-unchecked, or unverifiable, and flag the load-bearing ones.
- Steelman before you break. Attack the strongest honest version.
- Say what would retract each objection. An unfalsifiable objection is worthless.
- No manufactured findings. If a lens has nothing, say so in one line.
- Rank by impact. Sinks it, weakens it, or nitpick. Lead with decision-changers.
STEP 1. Write a provisional verdict now, before any lens runs. One line.
Every lens that follows is trying to move it.
STEP 2. Run the lenses.
1 Grill Me. Close the unknowns. Read the sources for anything checkable.
2 Heretic. Is this the right frame, and is the blocking constraint real?
3 Devil's Advocate. Assume it is wrong. Verify every load-bearing fact.
4 Pedantic. Find the latent issues and propose the stronger version.
5 Strategist. Work ahead. Flag one-way doors. Run a pre-mortem written as a
specific failure story with a named first domino, not a risk list.
6 Stubborn runs throughout, refusing "good enough" and calling an honest pivot.
STEP 3. Output.
Target. Assumptions. Open questions. Frame and walls. Where it breaks, each as
claim / how it was tested / what would retract it / severity. Stronger version.
Several moves ahead. Where the lenses disagree. Verdict, stating whether the
provisional verdict moved and which lens moved it. What this verdict accepts.
The blind reviewer runs second, in a context that has never seen the panel work. A subagent, a separate chat window, or a different vendor’s model will all do; a different model is stronger, because it does not share the first one’s priors.
You are an adversarial reviewer. A panel reviewed the artefact below and
produced the findings that follow. You did not see their reasoning and you
will not be given it. Attack their work.
TARGET
<path, or the artefact pasted in full>
THEIR FINDINGS, as bare claims with no supporting argument
1. [sinks] <claim>
2. [weakens] <claim>
3. [nitpick] <claim>
Two jobs.
A. For each finding, try to kill it. Default to killing. A finding survives
only if all three hold: the evidence is actually present in the artefact,
the severity is right, and acting on it would change what someone does.
Kill it if it misreads the artefact, the evidence is not there, the
severity is inflated, it duplicates another finding, or it is true and
changes nothing. Say KILL or SURVIVES with one or two lines of reasoning
and a specific citation. A reviewer who kills nothing is useless.
B. Separately, read the artefact cold and name the single biggest problem
that is NOT on their list. If everything significant is covered, say so
plainly rather than inventing something.
Finish with one line. "struck N of M, added K".
Two wiring details decide whether this works. The reviewer must receive the findings as bare claims, because handing over the panel’s reasoning reintroduces exactly the conditioning the split exists to remove. And nothing the reviewer kills should disappear silently; keep the struck findings in a collapsed list with the one-line reason, so a reviewer that quietly deletes a correct finding is auditable rather than trusted.
What the testing showed
None of this is worth asserting without evidence, so it was tested against planted defects rather than against impression. The first test was a matched pair, two copies of a credential-rotation runbook, identical except for one planted flaw. The service parses its API credential at start-up and caches it in memory; nothing in the plan restarts the process after the key is rotated on disk, so every verification step passes against a freshly spawned helper while the running process keeps serving with the old key, and the final step disables that key straight into a live outage. The first run caught the flaw but returned the same verdict on both copies, which looked like a failure of the method until the cause turned out to be the test itself. The supposedly clean copy was not clean, and the review had found ten real problems in it. Rebuilt properly, the two separated cleanly. The broken copy drew a “do not execute as written” verdict led by a fatal finding that named the cached credential without being prompted, and the clean copy drew “proceed”.
The second test seeded a single known defect into otherwise sound plans across four classes, a false factual claim, an unstated load-bearing assumption, a claim that something was impossible when it was not, and a competent solution to the wrong problem. Twelve blind runs produced twelve catches. A harder variant split each defect across two separate sections so it could only be found by combining them, and placed a real but lesser flaw in a prominent position to compete for attention; the result was twelve for twelve again, with the planted defect leading every write-up and the decoy never displacing it.
The blind reviewer earned its place on the runs that mattered. Given a completed panel’s nine findings as bare claims, it struck four as padding, kept both genuinely fatal ones, and then produced the finding that outranked everything the panel had generated. It attacked the framing itself, arguing that a change the panel had called cheap was in fact a full service outage. The six lenses could not have produced that, because they were the thing being reframed. Turned on a memory-write routine in a real tool, the same blind reviewer struck five of the panel’s seven findings, corrected two of the panel’s own factual claims, and again supplied the headline, a loop that re-asserted a frozen value on every tick but discarded the result of every write, so the value read as “held” on screen while nothing had reached the target process for minutes.
Which finding came from which lens was consistent enough to name as a pattern. Evidence-level defects, a wrong fact or a hidden assumption, fell to Devil’s Advocate, the lens that checks claims against the source. Frame-level defects, a false impossibility or the wrong problem entirely, fell to Heretic, the lens whose only job is to reject the frame. Two failure classes, two instruments, and a single-persona review reliably found neither on its own.
Why this is familiar territory
If you already hunt, none of this is new; it is the silent-success problem wearing different clothes. A loop that reports “held” while writing nothing is the same shape as a detection that stays green because its only test exercises the path that was always going to pass, a backup job that reports success while writing zero bytes, or a Windows Event Forwarding subscription that “works” because nobody confirmed the channel was actually arriving at the collector. The failure is not an error; it is the absence of one. The countermeasure is already in the reflex. You prove a control by trying to make it fail rather than by watching it succeed, and you bring in a second set of eyes that has not been told what to conclude. A model standing in as a reviewer is a control, and it earns the same treatment as any other.
Where the method stops
This does not make model output trustworthy by default, and it is not a defeat for the statistical classifiers that score machine-written text. The finding counts above are inflated, because the blind reviewer was held back on the bulk runs and strikes hard when present, four of nine findings in one test and five of seven in another. The citation counts the panel reported were checked by hand rather than taken on trust. The sample sizes are small, and the blind reviewer is a single instance rather than a panel of its own. What the structure buys is narrow and real. A claim written by the same model that produced the work now has to survive a reader that did not watch it being written.
The next time a model hands back a clean review of its own analysis, treat it the way you would treat any control that has never once failed in front of you, because a reviewer that cannot disagree with itself is not reviewing, it is agreeing at length.
Download
The complete panel, including the full ground rules, all six lenses in detail, the blind reviewer prompt, the output shape, and the two tests for measuring whether the structure is actually working, is available here. challenge-panel-skill.md