Why a 99% Report Rate Isn't Something to Brag About

A while back I built an in-house phishing simulation platform. A web system where an admin writes templates, sends recipients emails with an embedded tracking ID, and logs opens, clicks, and reports separately. The thing I cared about most while building it was "let's track clicks separately."

Then I came across a paper out of Korea University. It's an empirical study comparing manual training with AI-generated training across 12,180 public-sector employees. Reading it, I could see fairly clearly what my system got right and what it missed. Half of me was proud; half of me winced.

Tracking clicks was the right call

There's one point the paper keeps hammering. Korean public-sector phishing training has used open rate and report rate as its core metrics, per Article 77(5) of the National Information Security Basic Guidelines (don't open suspicious mail, report it immediately). But the very moment actual damage occurs, the "link click," sat outside measurement.

When I built my system, I caught opens with a tracking pixel, caught link clicks with a separate endpoint, and tallied the report button on its own. That was exactly this. At the time it was just a hunch of "obviously you should watch clicks too," but reading the paper, I saw this filled a structural gap in the existing metric system.

In the paper's second-round training, of the 1,449 people who opened the mail, 238 (16.43%) clicked the link. The post-open report rate for the same group was 35.89%. Which means the number of people who clicked before reporting was about half the number who reported. If you had only looked at open rate and report rate, these 238 people don't show up in the stats. The very behavior that leads to damage is invisible on the dashboard.

A plunging report rate is actually a success signal

This was the most interesting part.

In the 2022–2024 manual, repeated training, the post-open report rate stayed steadily high at 82–88%. Usually you look at that number and say "training is going well." One power company actually announced a 99% report rate as an achievement from monthly training (a case cited in the paper).

But when the same organization was sent highly realistic AI-generated mail, the report rate collapsed to 35.89%. Judging by the number alone, it looks like the training failed. The paper reads it the opposite way. That high 82–88% report rate was the result of people memorizing the patterns of similar manual templates every time, and that defense didn't hold up against genuinely sophisticated mail.

This is a critique that lands squarely on my system. In a structure where the admin builds templates by hand, no matter how well you make them, they end up repeating that person's writing style and patterns. After a few rounds, recipients catch on: "ah, this is that training mail again." The paper calls this the training adaptation effect.

What my system missed

The paper proposes a framework called IATF. It's based on NIST SP 800-50r1, and the core is a closed loop that auto-generates scenarios with an LLM, adjusts difficulty per individual based on the recipient's behavior history, and feeds the results back into the prompt.

Set against my system, the gap is clear. On metrics, I capture opens, clicks, and reports by stage, so this part overlaps. But scenarios are made by a person (the paper auto-generates them), and everyone gets the same difficulty (the paper adjusts it per individual). The first one especially, that scenarios are made by a person, is the root of the pattern-learning problem from earlier.

The data on how much scenario quality drives outcomes was striking. Same National Health Insurance Service impersonation theme, but one model attached a nonsensical sender domain (CJ Logistics) and got a 7.14% post-open click rate, while another combined a look-alike domain "nhis-check.kr" with personalization and real-time information and hit 41.5%. A 5.8x difference. In the end, how you design the scenario, more than whether you bolt on AI, decides the training's effect. And that's a principle that applies the same whether you do it by hand or with AI.

So I have no plans to bolt an LLM onto my system right now. Piling scenario auto-generation onto a small in-house tool is overkill. But two things are worth folding in right away: rotating at least three or four templates to slow down pattern learning, and putting click rate next to report rate on the dashboard instead of watching report rate alone. If the report rate rises but the click rate doesn't fall, that's not an improvement.

One thing to add: the persuasiveness of LLMs that this paper demonstrated is a training tool and an attack tool at once. The paper notes this duality at the end, too. When I remember that what I built is ultimately "a system that sends employees deceptive mail," the more sophisticated the tool gets, the heavier the hand that wields it has to become.

---

Source

• Gil Mun-cheol, Lee Sang-jin, Jung Hye-jung, "A Study on the Effectiveness of AI-Based Hacking Email Response Training," Journal of the Korea Institute of Information Security and Cryptology 36(3), pp.1047–1063, June 2026. Full text (DOI: 10.13089/JKIISC.2026.36.3.1047)