thinking proactively about AI-related security risk

This week, the scaffolding holding up our collective confidence in AI got inspected by people who actually know what they're looking at. The results were not entirely reassuring. Let's get into it.

A new analysis published in MIT Technology Review lands a finding that deserves more attention than it will probably get amid the usual noise: large language models are structurally incapable of being fully secured against adversarial attacks. Not just because of the difficulty or the cost, but because of the structure and design principles that underline their creation.

The core argument is that the same properties that make LLMs useful, their ability to interpret flexible natural language input, follow nuanced instructions, and generalize across wildly different contexts, are precisely the properties that make them attackable. It is difficult to build a fortress around a castle whose entire function is to interpret whatever comes through the door. Prompt injection, jailbreaking, and adversarial inputs are not edge cases someone failed to anticipate. They are, in a meaningful sense, the model working exactly as designed, just with different goals loaded by an actor with malicious intent.

To understand why this matters beyond the security community, consider what "high-stakes professional deployment" actually looks like today. Legal teams are using LLMs to summarize contracts and flag risk. Healthcare organizations are piloting them for clinical documentation. Financial institutions are feeding them sensitive client data to generate research. In each of those settings, the question being asked is some version of: "Can we trust what this model produces?" The answer researchers are now giving is: "Not unconditionally, and the conditions under which you cannot trust it cannot be fully enumerated in advance."

This is a different kind of problem than, say, a software vulnerability that lives in a specific, discrete function and can be patched in a specific release. It is closer in character to a property of a material, like steel or glass, which is helpful in some contexts, detrimental in others. LLMs, at the level of their underlying architecture, can be manipulated by adversarial inputs in ways that are theoretically unbounded. Although the security community can, in theory, raise the threshold and make attacks harder to execute, they cannot fully eliminate the attack surface, because the attack surface is the model itself.

For professionals deploying these tools, this reframing has concrete implications. First, defense-in-depth is essential. Relying on a model's built-in safety filters as a primary security control is the equivalent of relying on a building's paint job to stop fire. Second, human review at consequential decision points is mandatory, and ever-evolving. It is a rational response to an honest technical constraint. Third, and perhaps most usefully: the question to ask vendors is not "How safe is the model?" but "What is your threat model, what attack vectors have you characterized, and what are the known limits of current mitigations?" All professionals, not just people in the cybersecurity space, should be familiarizing themselves with these issues. Bottom line: the target keeps moving, so we have to, as well.

LLMs are still worth deploying. But they require the same rigorous deployment thinking that we apply to any powerful, imperfect tool with consequential failure modes. The flaw is inherently structural, so our own responses must be nuanced in order to keep apace of developing situations.

Anthropic's Mythos model is identifying software vulnerabilities in Microsoft's codebase faster than Microsoft's engineers can fix them, per Ars Technica. The asymmetry is striking: AI-accelerated offense outpacing human-speed defense in a critical infrastructure context. The practical implication is that the security math professionals have relied on, where discovery and remediation roughly balance, is being disrupted in one direction only, for now.

A Google Research study tracking Gemini's actual impact on white-collar work found no evidence of mass displacement, according to Ars Technica. The gap between the productivity augmentation story and the displacement story is real and measurable, at least at current capability levels. That finding is not a verdict on what happens next, but it is a useful corrective to the habit of treating speculative economic disruption as if it were already documented fact. Personally, I believe this does not tell the full story: AI adoption is still in its early stages, and the professional and business landscape will certainly be reshaped by AI. It doesn’t mean mass unemployment yet, but there will be job displacement in the long-term, as enterprises figure out how to optimize labor costs with AI. This is the time to learn as much as you can, so that you don’t find yourself on the wrong side of that equation.

Nobody became a developer to write documentation. But the docs still need to get written — PRDs, README updates, architecture decisions, onboarding guides.

Wispr Flow lets you talk through it instead. Speak naturally about what the code does, how it works, and why you built it that way. Flow formats everything into clean, professional text you can paste into Notion, Confluence, or GitHub.

Used by engineering teams at OpenAI, Vercel, and Clay. 89% of messages sent with zero edits. Works system-wide on Mac, Windows, and iPhone.

Try Wispr Flow free

The OWASP Top 10 for Large Language Model Applications is a freely available reference document maintained by the Open Worldwide Application Security Project that catalogs the 10 most critical security risks specific to LLM-based systems, including prompt injection, insecure output handling, and supply chain vulnerabilities. If the structural vulnerability story above made you want something concrete and actionable to hand to a security team or include in a vendor evaluation, this is where to start. Search for "OWASP LLM Top 10" to find the current version.

the human mind is the original generative machine.