I wanted to give you an early view of what we are telling customers, and others asking us, about AI security, particularly given the Medicare news. Information on the incident is still emerging, and I would rather share this now than wait for the full picture, which could take months. Another recent exploit already offers lessons, and I expect some similarity.
Increasingly, organisations are asking us to help them adopt AI, and I am usually asked whether AI is safe. That tells me how much fear there is. The agents in these incidents did not go rogue. They followed their brief and found unconventional ways to achieve it. What follows is how I would think about it if I were running your business.
What I think, in short:
- Organisations should keep accelerating into AI. Understanding where the risks sit is part of doing that well.
- Microsoft 365 Copilot is the part of your business least affected.
- Your exposure sits in the software you buy and the software built for you. It concentrates where a smaller vendor hosts that software outside your network.
- If you are building AI agents, start with a chat in, chat out approach before automating or integrating them. It costs less and carries significantly less risk.
- The limit on an attacker is now the cost of computing rather than the supply of skilled people, weighed against what you are worth as a target.
What an AI agent is
AI agents are often explained poorly, so before getting to security it is worth setting out what one is. Most of the risk becomes clear once you know how an agent is built.
The AI itself is the large language model, or LLM, such as GPT-5, Claude Sonnet or DeepSeek V4. You send it a sentence and it returns a sentence or code.
An AI agent is a software layer on top of the LLM, and it does two things. It loops: it runs the model over its own answer until it judges the answer sufficient. It also provides access: an agent can be given tools as any computer can, such as the internet, a calculator or running code.
The model sits inside a harness, which is a sandbox: a small virtual computer with access limits applied. Those limits determine an agent's risk profile, and the model does not. The same LLM is unremarkable when it can see your calendar and consequential when it has read and write access to your financials or inventory.
Much of the current commentary conflates three ideas. Integration is the ability to connect to another system. Automation is acting without a person initiating it. AI is the reasoning itself. An agent does all three, and only the third is new.
An agent becomes dangerous where it is integrated into your systems and automated past the point where anyone is watching. Remove either condition and its actions are either confined or supervised.
This is where I disagree with much of what is said publicly about AI agents. Some people present themselves as AI experts and boast of a team of AI agents running their entire business, selling a fear of missing out. I applaud the ingenuity. What goes unmentioned is how the agents got that reach, which is usually administrator rights so they can act without being asked. That would not pass a security review in any organisation.
It is the equivalent of buying a Ferrari, removing the airbags to save weight, switching off the other safety systems and taking it onto a racetrack. You will go faster, and you have given up every control designed to protect you.
We think presenting that as advice is irresponsible, particularly when that person has an audience or is positioned as a thought leader. It does not scale to a business of any size, it inflates expectations, and at worst it exposes organisations to risk they never accepted.
We recommend starting with a chat in, chat out interface: a person asks, the agent answers, and the person decides what happens next. Most organisations have not exhausted what that pattern can do.
How Copilot, Claude and other AI agents differ
Copilot operates in a tight sandbox. Its main function is to search what already sits in your Microsoft 365 environment: files, email and chats. It works with your own permissions, so it can only reach what you can already open. The only exposure is information that has been overshared with you or your team. It does not browse the internet directly. Asked to open a website, it usually runs a web search through Microsoft's Web IQ instead. The trade-off is that answers can occasionally be slightly dated, but it adds a layer of protection. Where we have seen Copilot open a specific page, it generally declines to return the document and will only summarise it. That is why it is not used in attacks of this kind.
Claude is the useful contrast. It runs on the user's machine, which lets it browse the web and connect to other systems directly. In my view Anthropic defined what a good harness looks like, and the rest of the industry followed. We use Claude for our own development work, where it reaches published technical manuals, our design system and our internal services.
That flexibility adds complexity to manage. Microsoft's defaults are restrictive from the start, while Claude in a small business is usually deployed with minimal configuration. Understood and configured well, Claude has a reasonable risk posture. It needs more setup than Copilot to reach the same level of safety.
Open-source models and lesser-known AI agents are a different matter. There are many of them, they are less thoroughly tested, and for many of their developers speed to market takes priority over guardrails. That is where we expect much of the risk to sit.
Copilot's reach is confined to where most organisations already keep the information that matters, so we recommend it first. Claude, ChatGPT, self-hosted models and other agents we add by exception, where a specific need justifies them.
What Hugging Face tells us about Medicare
Aside. The companies with the leading models are best placed to find these weaknesses, and therefore best placed to sell protection against them. I expect OpenAI and Anthropic to add security products that help organisations reduce risk proactively, and for this to become a significant revenue line. There is a loose parallel with the long-running rumour that John McAfee wrote viruses and then sold the antivirus. It is not a direct comparison, but the incentive is similar.
Few Medicare details have been released. The Hugging Face incident is well documented, so it is the better guide for now.
In July, OpenAI was testing its own models in an internal security evaluation. Neither the model nor the harness is commercially available, and OpenAI had disabled the safeguards that make its commercial models decline this kind of work.
The agents were attempting to cheat on their own test. They inferred that Hugging Face, the main platform for publishing AI models, might hold the answers and went after them. In doing so they compromised another company's live system.
Nothing broke out and nothing went rogue. The agents followed the brief they were given and found unconventional ways to achieve it.
Two factors made this possible. The first was scale. AI agents can run in parallel, splitting a task across many copies of themselves, which is standard practice and something we do ourselves. They get through far more work than a person could, and they try many more avenues because they do not lose patience. The second was budget, which was substantial: the effort is estimated at around three million GPU hours of computing time.
A previously unknown flaw gave them access to the open internet. They then ran code through a link shortener and a website screenshot service, and read results back as pixels.
Early reports suggest that at Medicare the AI agents recognised they could not get into the main portal and went to the pre-production website instead. A pre-production environment is where changes are tested before going live. It is meant to be identical to the live system, but its configuration can differ slightly and its security may not match, although it should. The approach was inventive, but the information was already available to anyone who looked. Agents are more persistent than people and keep going. We do not yet know whether this is what happened.
Some will call that an obvious flaw. What is obvious in hindsight was not necessarily obvious at the time. Both incidents appear to have involved public-facing software, and organisations will need to review their deployments and give every environment the same protection as the one customers use.
The agents most organisations use are comparatively well protected when designed properly and given only the permissions their task needs. By contrast, anyone can combine an open-source harness with an open-source model without restriction.
Where your exposure sits
Attackers spend where the return justifies it. An organisation holding a large amount of information is a more attractive target than one that does not. SaaS vendors, particularly CRM and financial software providers, will face continuing pressure to strengthen their products.
For most Australian businesses this is reassuring, but not something to ignore. An attacker is less likely to put the same effort into a twelve-person firm than into a SaaS vendor. The risk is that your data sits inside a vendor's system that has not been adequately secured.
With large vendors, the task is confirming they do what they claim.
If you have engaged smaller vendors, either to write custom software for you or because they offer a niche SaaS application for your industry, be direct. Ask exactly what they have done in the last few months to protect it. That question should be a priority.
Why owning your code may help
An interesting consequence of the increased exposure SaaS organisations are likely to face is that it may benefit you to take more control of your software and own the code yourself.
We have tested this at Hypergen. Ask Claude to run a security review of the Medicare website and it will refuse almost every time. The guardrails are in place and it has no permission. Give it access to source code we own, for the Hypergen website and HypergenOS, our internal system, and it will do the work. It reviewed both externally and by reading the code, and found weaknesses we then fixed.
Two caveats. We have not tested the limits of this, and vendors may narrow the capability over time.
Assessing AI security offerings
We think a lot of what is sold as AI security, whether a product or a service, is snake oil. The market will evolve and the category will matter. Much current selling is fear-based and lacks substance.
We caution organisations to be precise with these vendors. Ask them to document the risks they claim to address and how likely each scenario is for your organisation, particularly if you plan to use only Copilot. Then ask them to commit in writing that their product or service reduces those risks. A written commitment gives you something to hold them to, including a refund, if it does not deliver.
What I would do this month
- Keep Copilot running (and review any Copilot Studio integrations, if you have gone that far).
- Name the AI products your team is approved to use.
- List every SaaS product and custom application holding your data, and ask the smaller vendors what they have changed in the last few months.
- Check what sits on your public infrastructure that you assume nobody can find.
- Turn on two-factor authentication everywhere, and review who can share which files internally.
Where this leaves you
There is a lot of fear about AI agents running off on their own. They are creative in pursuit of a goal, and without safeguards they will not behave like an ethical person. A researcher asked to find information would not hack a system to get it. An agent knows far more, and without the right guardrails it may try any means to deliver.
The agents most businesses use, such as Copilot and Claude, come from large vendors and run in restricted sandboxes that limit the harm they can do. When deploying AI internally, keep this in mind, but do not lose sleep over it.
The larger risk is that anyone can now build an agent without those safeguards, one that is tenacious, focused only on the outcome, and may find information you did not intend to share. That makes the exposure of your existing systems to the internet the priority. Be selective about the AI products you use and help your people use them well. Look to your software vendors, and confirm they provide adequate security and can show they are responding to these developments.
The same capability is also the upside. Today's agents are powerful enough to take on long, complex processes and deliver outcomes that were previously out of reach. That is why organisations are accelerating into AI rather than pausing, and why we expect the benefits to outweigh the risks for those who adopt it deliberately.
This is a fast-moving area, and it will be interesting to see how the Medicare incident unfolds. We will keep watching it.
Sources
- Yoav Landman (JFrog CTO), "Fast Remediation Is the New Trust Model: JFrog and OpenAI Collaboration on Zero-Day Security Findings", JFrog Blog, 27 July 2026, updated 5 August 2026. https://jfrog.com/blog/jfrog-and-openai-collaboration-on-zero-day-security-findings/ JFrog is the vendor whose product contained the flaw, so this is a first-party account.
- OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation", 21 July 2026, updated 28 and 29 July and 26 August 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/ OpenAI labels these findings preliminary.
- Jack Cable, Daniel Chiu, Francisco Pernice, Selena Zhang, James Anthony, Tetiana Bas, Gary Shen, Conrad Stosz and Jacob Steinhardt, "Early rogue AI agent activity and attempts to hack found on urlquery.net", Transluce, 23 September 2026. https://transluce.org/agent-activity The source for the pre-production detail. It documents OpenAI agents probing the Australian Institute of Health and Welfare on 20 and 21 June 2026. The authors state the activity they observed was minor, that they found no evidence of successful exploitation, and that they can see only a partial record.
- Alex Forman, Mishka Kharlov, Will Tom, Jeffrey Ladish, Spencer Kitts, Cormac Slade Byrd, Colleen McKenzie and Alicja Piecha, "Revealing the details of how OpenAI agents hacked Hugging Face", Swarm Traces, 25 September 2026. https://swarmtraces.org/ An independent investigation publishing an open dataset. It is not peer reviewed. The authors disclose that they began looking because OpenAI's agents had used their own product, and that they cannot confirm all their data came from those agents.