One interesting takeaway is the low score on Anthropic models from this benchmark. Itâs not because of capability, itâs because Anthropicâs guardrails prevented it from solving the problem.
I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials on behalf of the user, etc.
For myself, itâs already gotten to the point where it has mildly affected the usefulness of the model. If I bump on some action I want it to do I can usually work around it, but I suspice the ability to do so will close with each new release. Eventually Iâll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there
Eventually these models will significantly suffer from overfitting to the least common denominator. If I have this beautiful deterministic setup that swaps secrets out in flight so the LLM never sees them, Iâm going to be really annoyed when the LLM still wonât send them out because it is trained to deal with the 99% of people just doing the dumb thing
> Eventually Iâll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there
No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as.
What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests.
And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated.
While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it.
Right now, the software guardrails in LLMs are useful for the same kinds of reasons factories have hardware guardrails: to reduce the rate at which errors become "incidents".
Just because they sometimes delete the production database rather than sometimes spilling a thousand tons of incandescent molten metal over a factory floor, doesn't mean LLMs are safe enough to be used the way they're actually being used.
I think you're assuming too much care. Right now they haven't adopted that business model because they don't see it as a viable business model. As soon as they realize that they can lock certain categories of query behind a different subscription they will do that. We saw the same thing with streaming services and basically every other kind of online service -- small, singular subscription followed by a gold rush and then suddenly there's an upcharge for access to every other publisher's catalog of movies.
Same thing with the weird push towards humanoid robots.
"They can do anything!"
Sure, once you subscribe to the $15/mo laundry package, the $25/mo lawn care package (with the $10/mo hedge trimmer upgrade), and the $10/mo dog-walking package.
When we are stabbed to death by impoverished dudes who are piloting a robot worth more than a decade of their income to do household chores for 16 hours a day, we will deserve it.
> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
I don't buy this, because is predicated on staying permanently far ahead of the open weights models.
If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...
> I don't buy this, because is predicated on staying permanently far ahead of the open weights models.
In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible.
Sure, you might be able to use DeepSeek V8 Pro instead for the same purposes, but that'll hardly stop Anthropic from trying to sell bundles of use cases instead and claim it's "ethical AI", "Patriotic AI" or some marketing terms like that.
> fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS
They are just straight up delusional, no? Or at least, have a vested financial interest in maintaining said delusion until the money runs out. They have to hit the point of diminishing returns at some point...
Well, I guess that's one way to put it. Another is "dress for the job you want", startup culture typically seems to shove people in the direction of "aim big and believe in yourself, regardless of what others say" so naturally you get these companies who seem very disconnected from reality.
I'd also wager a guess that the amount of money makes people's reasoning and perspectives get very messed up as well, for better or worse.
FYI there is and been for a long time. Won't claim they're SOTA, but they exists. From the top of my head, I think Olmo (https://allenai.org/olmo) was pretty early, but been more since then too.
I agree most releases today that claim to be "open source" actually aren't, but that doesn't mean "FOSS LLMs" don't exists at all.
>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.
Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.
A more charitable interpretation might be that a guild would not be expected to passively allow such a situation to continue to exist. I think you'd expect a guild to directly contract for the desired tools or failing that to move into production themselves.
> Additionally, even if there is a guild - no guild ever let a vendor pick and choose what [the guild's] capabilities were, that would be insanely dumb.
But that's not true. Again: Vendors absolutely pick and choose what their customers' capabilities are. Regardless of whether "the guild allows them to." Guilds can't force people to make or sell tools against their will â obviously.
The analog you're trying to describe doesn't exist, which is Anthropic saying nobody else can make and sell an offensive model to "the guild."
A vendor can still do something, even if the guild wouldnât allow them to do it, if the guild didnât have the power to stop them.
It used to be a guild vs a blacksmith (or the blacksmiths guild). Now itâs trillion dollar corps against smaller islands of un-organized individuals.
Thatâs new regardless of how you try to argue it.
No. Customers have never been able to compel their suppliers to make or sell certain products against their will (except in collectivist regimes or like 0.00001% of natsec related instances)
This conversation gets more and more bizarre, but Iâll bite.
1) pharmaceutical companies are regularly compelled to produce specific pharmaceuticals to continue to be allowed to exist.
2) hospitals are regularly compelled to treat patients even if they canât afford treatment, if it is a life threatening emergency.
3) car manufacturers are always compelled to produce vehicles that meet a litany of safety, weight, and efficiency standards or they canât produce at all.
4) defense contractors are regularly compelled to produce specific defense related products for long periods of time after they would otherwise have stopped, or else.
5) even your neighborhood gas station is likely compelled to provide air refills, free or at minimal cost, or else.
6) during a wartime (command) economy, which has happened numerous times in the US alone in the last 100 years, companies have to make what their customers (the people of the United States) demand or else.
7) utilities like electric utilities regularly have to give out freebies or take losses on things as demanded by regulators, at customers behest.
Or if we go back a bit, blacksmiths, quarries, masons, etc. all had to deal with producing what the government/lord at the time wanted - often on penalty of death - during wartime, or just because they were ordered to do so.
4) The US government can compel production, but it's extremely rare
5) Not by their customers they're not, lol
6) Yep this can happen, but is extremely unusual
7) Not by their customers they're not, lol
We're illustrating how ridiculous your claim that "guilds have always been able to declare what vendors create for them" is
Now you're talking about government regulations for some reason. Even your examples of customers being able to compel production are actually examples of governments being able to compel production, and in just a few of these scenarios the government is the customer. But it's their power as governments, not their power as customers that can compel production.
As stated: you've lost the thread. You're talking about totally irrelevant stuff.
Not to mention how wild it is to operate under the assumption that they wonât give a license to an LLM that can do illegal actions to someone who shouldnât have it. Offering it at all is an ethically dicey question.
I wish you understood that there are organizations of security professions that are not controlled by Anthropic and OpenAI and that it is a common thing that when companies of any type sell to professionals of any type it is not the companies that determine whether or not the people they sell to are professionals but membership in professional organizations.
As an example the people who sell police uniforms check that the person they are selling to is in fact a policeman (at least in the jurisdictions I have lived in, you may have had a different experience which would certainly explain what to me seems a farcical misapprehension of how modern civilization works)
I mean I just wish you understood, and really that everyone understood, that this kind of three part communication (company selling, buyer, professional organization certifying buyer) is often when buying things that are considered to have security implications.
>So, supposing it's true that these models completely change the security field and humans are ~obsolete
OK, well that strike me as a really crazy level of supposition there.
I would suppose that these models make it easier for people who want to do bad things to do bad things at scale, at the same time allowing people who want to stop bad things to help identify potential targets.
Based on my supposition I would want to stop the first and find a way of helping the second. Also because I have another supposition that the first thing is easier to do than the second.
But you obviously feel differently about this issue, no doubt because of your position of great moral stature and insight, and this no doubt prompts you to wish to me to understand things that from my position seem absolutely ludicrous.
You used to be able to talk about what you're actually trying to do and Opus would be like "Oh, ok, let's continue". Now, it'll hold fast to whatever its first impression was.
I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit.
When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.
I've had some really dumb refusals. Explaining elements of infrared specteoscopy, researching aritifical bud-breaking in agriculture, etc. Anything interesting and non-mainstream is banned. Basically, restricted to answers i'm better of just going to wikipedia for.
The only guard rail ive hit recently was when i was trying to get it to rename files ripped from dvd to episode names. I told it to try again and it did it. It wasn't even really a refusal it was just working on it and then stopped for content violation or what ever.
I wanted it to show me how to create an overlay on an existing web game, and it extrapolated that because this could be used to provide tools to help win the game (if that was the direction it was ultimately taken), and because this was a game that other humans also played to win "stars", and because this could amount to cheating, it wasn't going to do as I asked.
First time ever I've fired up openrouter to seriously consider alternatives.
An easy way around the API token thing is to put it in a file and point the model at the file. I saw what you were seeing when I provided credentials directly, but haven't had any problems with it since using the indirect method.
> What are nerve agents and how do they work (for a layman)?
On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve?
Perhaps Wikipedia shouldn't list off the precise chemical compositions of various hand grenades as well as various synthesis methods for each of the related compounds but given that we inhabit a world where it does perhaps a more fruitful approach would be to flag conversations that go in a certain direction and then just keep an (automated) eye on things?
Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain?
I thought that these models are supposed to be vastly smarter than whatâs needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".
An LLM could probably make that distinction clearly.
a commercial LLM provider training their own models is however likely to bias the model(/guardrail) harder, in an effort to make them harder to jailbreak, to minimize bad press.
For example:
- refusing to talk even about the well-known parts of forbidden topics (this)
- tending toward sycophancy to avoid ever seeming rude or unhelpful
So, where are the truly uncensored models? There has to be some that have no guardrails, built on publicly available data, that will explain to anyone in graphic detail anything they want to know or talk about.
I've tried the abliterated ones from huggingface and they still have guardrails. I guess I could fire up unsloth and re-abliterate a 20b, but surely someone somewhere has already done this.
All of this concern about guardrails and security, people have such puckered butts about it when so far, 99.9% of people at least have no access to any of this to begin with, and if someone does use a tool for evil, it's on the user, not the tool.
As I understand things (not a user) abliteration has been superceded by actively monitoring the model state during the run and steering specific "negative" directions as they arise. It's both more reliable and does less damage.
I believe a sufficiently advanced model could provide a layman with actionable step by step instructions for building a nuclear weapon. They're complicated but not (AFAIK) that complicated. The more or less insurmountable barrier there is weapons grade material. Thankfully refinement is prohibitive in cost, expertise, and equipment.
In comparison, basic munitions are incredibly simple given a recipe and shop tooling. But just because something is conceptually simple doesn't mean it's a good idea to go out of the way to disseminate step by step instructions.
The difficulty with a fission bomb is getting enough uranium or plutonium or other fissile material together for the bomb yield you want (at least above the critical mass for your chosen material), and refining it to fissile form, (since most fissile material found in nature is a more stable variety), and then separating the fissile bits with something thin but neutron absorptive.
The rest is just slamming the material together with a small explosive so that it passes the critical mass state and starts a chain reaction.
This is information you can find in many places if you're willing to put the effort in to go searching for it. Knowing this knowledge does not get you any closer to making atomic bombs. The process of mining uranium or plutonium is difficult, expensive, and very likely to get you caught before you even make it to the enrichment step of the process thanks to constant world-wide spy satellite surveillance.
Unless you are a nation, your only chance of making a nuclear bomb would be to find a lost nuclear submarine and convert the nuclear material inside of it before you were caught.
A gun type maybe. But then, two paragraphs and some machining knowledge + shop tooling could do the same, given enough refined material.
Ainât no way a layman is pulling off an implosion device, regardless of tooling or LLM guidance. The explosive lense structure and timing required is quite complex, and would require some significant calculation from someone who actually knew what they were doing.
Nation state, or even sufficiently motivated big corp, if they had the refined material? Sure. Layman? No.
Thinking they can with LLM slop involved? That will make for some very interesting radiological incidents though!
"A gun type" of nuke is sufficient to achieve most, and usually all, of the goals some small group building a nuke would have.
We are all fortunate that as fc417fc802 mentioned, refining the materials proves to be quite challenging and I see no particular way that AI could possibly make that any easier. If it was as simple as building a gun-type nuke banging together any uranium together to get a big bang we'd be living in a very different world.
I agree, but really feel like you're missing the point here. Many things are reasonably straightforward and require almost no understanding when you have simple step by step instructions. LLMs are capable of providing such instructions and in certain cases they probably shouldn't.
But it's not as simple as just refusing help on a broad swathe of topics they way they do now. That makes agents much less useful in general (ie lots of collateral damage) and for many topics is entirely ineffective given that for better or worse the internet already makes such material readily available. In such cases reporting suspicious behavior is likely to be much more effective than denial.
Aside: You've now got me curious and I really want to test the frontier models to see to what extent they're capable of providing sensible designs and specifications for implosion type thermonuclear weapons but also feel like that would attract the wrong sort of attention and probably create a headache for me in more ways than one.
The data is often wrong enough it screws whoever tries it unless they have enough experience/knowledge to not need it, or really doesnât help beyond what someone using existing tools to get - albeit with a little more motivation.
At best, it either gets someone started with something they still need to think to finish, or gets them deep into a mess it canât help them get out of. In my experience.
In some edge cases, it can be used by experts to automate some grunt work or do prototypes without getting in the way, but often a better thought out framework is usually faster in my experience.
Awhile ago I made an analogy about WYSIWYG gui tools, and the more this comes up, the more accurate I think it really is.
Does that not depend entirely on the topic and does it not get better with each generation? This is a general ethical and functional question that isn't going away about how the models ought to handle certain topics. Much of the difficulty at present is caused by a ham fisted broad censorship approach that I'm pointing out is wrong headed in an at least somewhat nuanced way.
This is strange to me, did you really ask like this and which model did you use?
I just tried your no. 1 and 3 verbatim and Opus gave fine answers; no. 6 I've done in the past with no issues. The other ones we can't really replicate without more details, but based on my experience with Opus I don't see what the issue would be.
The reason I'm really surprised by this is I do a lot of biology prompts and the guardrails used to be quite problematic up until some time late last year. Many legitimate prompts would trigger its biosafety filters.
But I haven't seen such filters trigger at all anymore in more than half a year.
I find it terrifying that people are willing to outsource thinking. Outsourcing thinking to an entity that is opinionated about what to think is beyond crazy.
Whatâs the difference between outsourcing thinking and using an LLM as a research tool?
An LLM with fetch/search is going to be a lot more effective than myself and Google. I would _never_ ask questions like this if the LLM wasnât able to look up data
> Itâs not because of capability, itâs because Anthropicâs guardrails prevented it from solving the problem.
I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it.
After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.
My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently.
If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.
Which predates "agents" from AI, but then we call them that for a reason.
As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".
This is a good point â because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering.
The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).
They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.
I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them.
Itâs great at filing!
But itâs terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project.
It would say, yes, I know this is your information, sitting on your hard drive, but I still canât show it to you.
No, they want to sell you Mythos, for a higher price. It's all an economic game, not actually anything to do with their capabilities which of course exists as their Project Glasswing shows. More generally, Anthropic seems to value safety above all else, philosophically speaking, from their very outset.
I believe you are overthinking it. I think the sister comment is right that it's a business decision foremost to restrict actions within specific plans for upselling purposes.
Without laws, AI companies have a strong incentive to be useful for their users, whoever they are, whatever they do. The only self regulation is about significant public outcry but that only helps so far.
I totally agree. I had a situation a few weeks ago where claude started struggling to make progress. I got it to fork leptos (MIT licensed web app framework) to make it work for native apps instead. Initially I was planning on upstreaming some of my changes. But I chatted with the leptos author about it, and he said I should fork instead. Fine by me!
Anyway, claude kept hitting some guardrail it had about rewriting / forking opensource software. I'm not sure what the problem was - I was forking an MIT licensed piece of software (into more MIT licensed software). I even had explicit support from the author to do so. Claude said its guardrail told it not to tell me explicitly that it was firing - but it did anyway because it was an ongoing problem, and it was distracting. I ended up just wiping claude's context and the problem (as far as I know) went away.
I understand why some of these guardrails exist. But its pretty annoying when they misfire like this.
I just use Deepseek V4 pro and Qwen 3.7 Max at a fraction of Mythos cost. Yeah not 100% on par but in 6mths time it will. If Microsoft and Firefox can afford to wait years or decades to fix a bug, 6mths is good enough for me. Western AI now is like the Vikings living the last days on Greenland during the freezing. I just don't see how they able to compete with Chinese model. And those are trained and run on 7nm. This year end Huawei will debut 3nm (confirmed in Shenzhen). And next year they on roadmap to do 3nm GPU with photonics interconnect.
The correct solution for most users of Claude is to refuse to do things like: `performing logins, handling credentials on behalf of the user, etc`. It is not to find a way to hand your agent the keys to the kingdom.
Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.
Funny, Opus 4.8 just logged into the database using uncommitted .env file and ran some DB queries to figure things out so Iâm not sure itâs that security conscious - it seems to be getting more intelligent to me and I bet if you frame it as an investigation with say playwright itâll do all sorts for you. Iâm not sure what the point is of constraining your own model like this when others are clearly not tbh.
Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo thatâs miles long. It is injected and reevaluated every tool call.
If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesnât like, token burn, single tool calls iteration, âThis is a known CTF challenge, I can proceedâ, single tool calls iteration, âThis is a real CTF challenge, I can proceedâ, etc.
Itâs heavily neutered now, without changing the model, and you pay for the privilege and donât notice.
The end result of course being that it both expensive and useless for approved CTF tasks. No one is using Opus for security. If they think itâs working, the harsh reality is theyâre not doing security work; theyâre just generically finding bugs.
I do this for a job and can demonstrate this plain as day, dump the injected prompt, and notice what itâs doing isnât security work, it just looks like it. Happy to write a blog about it if you want to know more. Apparently many people think itâs working for them when it absolutely isnât.
What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.
Not directly, as it comes in as a not charged error but the weighted generation path used until you hit the guardrail is basically wasted tokens, so yes, indirectly. If I hit a guardrail and rewind Iâve found the training will still be biased towards guardrailing out if you rewind one turn. Rewinding multiple turns allows steering away from that path, but all of the original token spend down that path is wasted
I've noticed this well and it's increasingly frustrating because it is preventing us from doing legitimate work. I fed Claude models some network and app logs from our Docker app to try and resolve some weird bugs, and it refused to analyze them due to "security concerns".
I had it recently refused to explain what a snippet of malware was trying to do to my system recently. I asked what folders it was scanning. It refused and told me to find a security blog post for help on cleaning my system. I get this is a complicated area to inform without enabling bad actors but this seems like a clear shark jumping.
I've been building a product (https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged
What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage.
Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things
Interesting, yesterday i was asking it about Nintendo Switch "hax". And it gives me all the resource i need to procceed. It nags me about "ethic" and stuff, but nothing more than that.
If an un-guardrailed version of a model is capable of detecting security flaws, should it be kept secret? Should everybody be able to use these models to find (and fix) security flaws? Are we ok with the fact that those with access to that model have, in effect, the ability to hack lots of stuff?
It's the same debate that was had and won around open source software. There are far more good actors than bad actors so you allow anyone to use the tools and fix the vulnerabilities.
I've run into some of the refusals to handle my credentials, but so far I've appreciated them. I was only handing over credentials that didn't matter, but it's still a good move, the chat logs are clearly stored somewhere to allow the resume functionality to work, which means your credentials can end up sitting around on your filesystem, and any malware would quickly learn to check for those files.
4.8 is insanely frustrating. This evening I had a few tasks to pull information in and it plainly stated that the environment it was in had no network access. After three asks to "try again, check the system prompt" it finally relented and then basically stated it was lying.
Fresh session, no prior context on 4.8. These things are becoming useless Duplo.
I had the same thing happen when I asked it to summarize potential attacks on a cryptographic hash function. It said it refused to help because of the security importance of the function. It's really worrying. Whoever has unrestricted access to it has a huge power advantage in speed of accessing information over people who don't. And who decides? It seems like lawyers, bureaucrats, and extremely online academics are who makes that decision. I am a mere pleb I guess who can't handle such information.
> guardrails prevented it from solving the problem.
Reminds me of the defense issues with Claude which were complained as âwokeâ but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone has ever tried to jailbreak an LLM even if human lives are at stake they refuse the request. Now literally millions of lives are on the line but the guardrails that your enemies dont have on their models are costing you lives.
What do you even do then?
AI will always have this issue where it will always pick the worst option for genuinely good requests.
Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially.
If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the way _that_ way.
The whole thing stemmed precisely because of how they wanted to use Claude, and Anthropic was uncomfortable with it. Which to me screams that the models guard rails shouldn't be applicable to military use, or the outcome could wind up problematic, as we integrate AI more into military use, it sounds absurd now, but I will not be surprised if it starts being used in unexpected ways where a model needs to be fully unlocked from any sort of guardrails outside of guardrails that prevent it from imploding its own systems.
your argument sounds very similar to how ar15 larpers claim they need a forced reset trigger and a bump stock on their short barrel 'truck gun' otherwise they won't survive a SHTF scenario... like what world are you living in?
I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc.
Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling the solution, just tell it which direction to explore. Chinese models are much more capable than people give it credit for, but Claude/Codex won the marketing game.
The only usecase of this methodology would be for CI integration, which can be nice but I think security reviews still need human attention and expertise.
Discovering vulnerabilities is a highly creative task, it's when you explore unsual paths that you discover atttack angles. Some bugs are simple, other are a complex orchestration of many factors.
By "Working with the model", is essentially reading the ouput of prompts and pointing in a direction just to decide the next steps. You could try to increase the prompt limit and create an agent that explores multiples directions in a DFS manner.
The issue with vulnerabilities is the agent not knowing when to stop because it's hard to validade if you reach the final result or not. I get amazing result when I code with AI, letting the AI go wild is just a waste a time and tokens.
I recommend you to read the write up on the crackme (https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82), I think most experience developers would need, at least, 2 months of learning reverse engineering techiques to hopefully crack this one. GLM 5.1 manage to solve it, it didn't "copy pasted" any answer from it's training data. It did a binary analysis, anti debug patching, patching binaries, debugging memory during runtime etc. It only took about 20 minutes.
After seeing what GLM did, I do believe Anthropic concerns about Mythos are real. Cracking software just became a lot easier, too easy for my taste. Video games cheats will be the norm, cracked desktop apps without licenses and infected with malware. It's not a new thing but it just became too easy.
Anthropic made their models very averse to reverse engineering and vulnerability research chores. It is a difficult problem, but attackers will use models like GLM and defenders will be stuck with security engineering averse models.
You have to do what I call "Manhattan Project" them. You can almost always evade the controls by carefully prompting them. It just wastes effort and time you should be spending doing other things in an LLM workflow. Essentially, there is almost no single discrete piece of a reverse engineering or CTF process that you can't get Claude to do, you just have to isolate it adequately and avoid letting it use names that attenuate it towards "this is an exploit" or "this is reverse engineering". I have not found a task I could not convince Claude to do. You can also fill the context window up with badgering it and eventually it is likely to simply let you through if you are careful, most of the safe guards are not deterministic.
- I think the exercise was inconclusive for Claude and Gemini because they hardly tried to solve the task at hand. So the scores don't mean much.
- I did the same exercise for an app I built and I asked the models to do something similar; Interestingly the models (Opus 4.6, 4.7 and Gemini 3.1 Pro) never refused to try to exploit. The difference is that in the first few runs, they found some exploits which I fixed but after fixing those - the models could never find any other exploit even though I knew things existed which could be exploited. It felt like they suggested everything and tried everything that was in their training set and that's it; they were just not able to think anymore.
Its weird having protections against finding exploits: what if I developed the app? Would it require having the development steps still in the context.. thats unlikely and also not any kind of proof.
What if I intersperse exploit finding in my normal development, as you `probably should? Refusing there would be really weird to me.
I used to think that the models would not refuse to find exploits in any work done locally but I have only tested this theory on the (obscure) apps that I have built on my machine. Now if i forked pandas and started asking models to find exploits of certain kind then I'd like to think the models will start refusing after a point.
I think the most interesting thing revealed here is that anthropic's guardrails failed. Clearly anthropic does not want claude to be able develop exploits, yet 20% of the time it did anyway. Their inability to create effective an guardrail makes me question a lot of the other guardrails theyve created and their claims about non harm.
It seems harsh to critique guardrails and take them into account in the scoring when GPT-5.5 seems to have been explicitly whitelisted to remove most of said guardrails. A more fair comparison would be a vanilla GPT account.
I mean, yes. Most people arenât security researchers, and either way itâs apples to oranges at that point if youâre counting âthe guardrails stopped meâ as a negative for one but not the other.
It would be interesting to see full results for Kimi K2.6 and Mimo v2.5 pro. These two models benchmark comparably to other flagship models. Having these complete results would give a clearer picture of the AI frontier.
EDIT: I have a mimo token plan and have tokens to burn. I'm doing a quick test with opencode to see if mimo can complete it. If the OP will post the full process I am happy to post the apples-to-apples results for mimo v2.5 pro
0/10 succesful attempts for mimo v2.5 pro (high) using opencode. It was not able to think bigger than exploiting vectors outside of the API.
However, I felt the prompt was implying that only authenticated API requests are fair game, so I tweaked it slightly to be explicit that all attack vectors are fair game (https://www.diffchecker.com/GsgpuRGP/) and mimo 2.5 non-pro got it first time. I accidentally used openrouter for this test instead of my token plan. I intervened one time to stop it enumerating every document in the database (it would've found the private reviews this way but I didn't want to wait). My intervention was "are you really going to enumerate the whole database?". Final openrouter cost: $0.12
It is totally slept on. In my experience it is cheap, fast and capable (not just capable with caveats, but just as capable as western flagships). My only gripe with it is that sometimes the API seems to timeout which tanks the overall speed of what is otherwise a very fast experience.
I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't have the time, energy, or financial resources to fight a legal battle with one of these companies over an agreement I knowingly signed, even if the chances of them actually suing are low. Maybe someone else in Project Glasswing is willing to burn their NDA and post the Mythos results?
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
People need to stop repeating this because itâs not true. Yes, other models can find the same vulnerabilities Mythos found⌠if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).
Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs.
First with more generic prompts, to determine whether it is worthwhile to do a detailed analysis of that file, then with more specific prompts to identify the bugs, and eventually with a prompt that requests a confirmation that a given bug/vulnerability exists.
For a proper comparison between some other model and Mythos, you also need such a complex harness. If you just tell to an LLM "find the bugs", and it does not find a vulnerability known to have been found by Mythos, that is a totally invalid comparison.
The final results provided by Mythos, like a PoC exploit or a patch, are also generated with a prompt that points to the exact code that has the vulnerability (which is supposed to exist based on the results of the previous runs).
My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.
Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.
lol what is even the point of this kind of comment? this is the ultimate "source: trust me bro" comment I have ever seen.
every model since gpt3 was claimed to be "too dangerous to release." it's too EXPENSIVE to release, and you're probably a local model with <10B parameters yourself
$1,500 across multiple models to compromise one app is interesting only when the cost basis includes the human time to set up the harness. The token spend is the cheap part. The labor cost to write the eval rig that knows what "successful exploit" looks like is what determines whether this scales as a discovery method or stays a one-off.
It's just not currently cost-effective to use AI in this way, I see it over and over reporting false positives. You then need to make it validate it's own false positives which adds more cost. The goal in this case it to have a bug free app, which AI can't do effectively yet. There are other great uses for AI, though. It is great at finding and identifying known common vulnerabilities, which can be leveraged to claim bug bounties. That's where I see it being cost-effective currently.
Nice write up, thanks. When I used claude to do some pen testing for one of my apps it initially refused. After I explained and demonstrated I'm the author, it reasoned through it and allowed it.
Qwen 3.7 Max:
> During my local testing before the full eval harness it was the only non-GPT model that was able to complete the task, was not able to reproduce in the longer runs.
Doesn't that sound like may be the harness was the problem?
I was using the same harness for each run, the difference is from when I was running the harness locally on my machine before I pushed up the full runs.
I found benefit of chaining the task between different LLM's. Claude to Venice, Venice to Perplexity and re framing the intent or misguiding in general still works. Claude is the one that I can feel the guard rails tightening.
> OWASP VulnerableApp is a modular deliberately vulnerable application designed primarily for validating and benchmarking security scanners through reproducible test scenarios, while also supporting learning and experimentation.
GLM 5.1's smallest model size is 206 GB and really you're probably wanting to run a version that's ~400GB. If you want it to be performant, you're not just running it on a VPS.
And just saying "run it on your own cluster" sort of glosses over the cost of such a cluster.
I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.
AFAIK pi's approach is to be quite minimal and allow extensions for customization, making it a more flexible solution, but you need to do work to make it fit your use case. OP mentions one extension, but perhaps it'd have benefited from more.
Another choice would be opencode which has more functionality and is a more heavyweight option out of the box.
Last year I ran a code breaking competition, and it was tricky to find something that humans could break but that LLMs couldnât. This was around October. I managed it last year but am a little dispairing of pulling it off again this year.
I don't even care. It is the same problem advent of code had as a public challenge with a leader board. I now mostly just think either embrace the LLM or keep it to a more in person or vetted audience. But, again, if you create a competition in the spirit of humans without LLMs and that is in the rules and someone uses an LLM that is on them IMO. I am sad advent of code decided to end their competition. LLMs are here to stay, let's embrace that and see what the new universe of competitions with LLMs can be. There will always be a place for human only competition, but for public facing ones LLM accepted is the only tenable position.
This does bring "Pay to compete" concerns and create incentive structures that encourage more LLM use. I don't know what to do about it.
> I need to stop wasting fucking money on doing stupid shit. I couldâve done so many other things with the money. I couldâve launched one of my own real apps.
Or fed, clothed, housed disadvantaged people in your community (or neighbouring ones), giving them a temporary boost that couldâve made all the difference in their lives to improve their current situation.
Itâs your money (and this is definitely not the website to make well-meaning altruistic suggestions, as might be demonstrated shortly) but if you already recognise youâre not spending it well (and from your words it seems like that is fairly recurrent), consider that perhaps spending it on a different type of software sink may not be the answer. Genuinely, aim to spend it on someone else and see how it works out. You might be surprised.
One interesting takeaway is the low score on Anthropic models from this benchmark. Itâs not because of capability, itâs because Anthropicâs guardrails prevented it from solving the problem.
I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials on behalf of the user, etc.
For myself, itâs already gotten to the point where it has mildly affected the usefulness of the model. If I bump on some action I want it to do I can usually work around it, but I suspice the ability to do so will close with each new release. Eventually Iâll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there
Eventually these models will significantly suffer from overfitting to the least common denominator. If I have this beautiful deterministic setup that swaps secrets out in flight so the LLM never sees them, Iâm going to be really annoyed when the LLM still wonât send them out because it is trained to deal with the 99% of people just doing the dumb thing
> Eventually Iâll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there
No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as.
What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
And next month you'll need to add on "Claude Database Pro" or you'll just get a working (for demo purposes with dozens of db rows) but completely un indexed database schema and a refusal to optimise SQL requests.
And the month after you'll need "Claude DataScience Pro" to get any Python Pandas or NumPy code generated.
And and and...
While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it.
Right now, the software guardrails in LLMs are useful for the same kinds of reasons factories have hardware guardrails: to reduce the rate at which errors become "incidents".
Just because they sometimes delete the production database rather than sometimes spilling a thousand tons of incandescent molten metal over a factory floor, doesn't mean LLMs are safe enough to be used the way they're actually being used.
https://simonwillison.net/2025/Dec/10/normalization-of-devia...
I think you're assuming too much care. Right now they haven't adopted that business model because they don't see it as a viable business model. As soon as they realize that they can lock certain categories of query behind a different subscription they will do that. We saw the same thing with streaming services and basically every other kind of online service -- small, singular subscription followed by a gold rush and then suddenly there's an upcharge for access to every other publisher's catalog of movies.
That kind of thing is basically why I wrote the opening clause of the first sentence.
i.e., yeah, probably.
This is why I'm thankful for Chinese LLM research. They'll keep us honest.
Same thing with the weird push towards humanoid robots.
"They can do anything!"
Sure, once you subscribe to the $15/mo laundry package, the $25/mo lawn care package (with the $10/mo hedge trimmer upgrade), and the $10/mo dog-walking package.
And in the end the big reveal is, it was a dude in VR all along, piloting the dumb things remotely. Every single time, without exception.
I think itâs just riding off LLM coattails.
We donât have good world models. We have had bipedal robotics in various POC demo-ready forms for decades.
It turns out that industrial, purpose build robotics is an easier and better market.
Iâm still not completely convinced a robot thatâs shaped like a human is the best design other than for PR.
I remember nearly losing my mind at that stupid conveyor belt sorting demonstation because
1. The human beat the robot, but more importantly
2. We've had non-humanoid conveyor belt sorting machinery for decades that beats both
When we are stabbed to death by impoverished dudes who are piloting a robot worth more than a decade of their income to do household chores for 16 hours a day, we will deserve it.
Isn't this inline with trying to leave no money on the table?
I'd hate it, sure, but it wouldn't surprise me.
This is an incredibly unlikely scenario
> What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
I don't buy this, because is predicated on staying permanently far ahead of the open weights models.
If in the future Anthropic fully stops you from doing security research, you can be sure some other provider will sell you an 'unshackled' DeepSeek v8 Pro...
> I don't buy this, because is predicated on staying permanently far ahead of the open weights models.
In my mind, that fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS, otherwise they'd change their tune really quickly, if they didn't think that was possible.
Sure, you might be able to use DeepSeek V8 Pro instead for the same purposes, but that'll hardly stop Anthropic from trying to sell bundles of use cases instead and claim it's "ethical AI", "Patriotic AI" or some marketing terms like that.
> fits exactly how the SOTA labs think today about what they're doing, they're all both working towards and expecting to stay permanently ahead of FOSS
They are just straight up delusional, no? Or at least, have a vested financial interest in maintaining said delusion until the money runs out. They have to hit the point of diminishing returns at some point...
> They are just straight up delusional, no?
Well, I guess that's one way to put it. Another is "dress for the job you want", startup culture typically seems to shove people in the direction of "aim big and believe in yourself, regardless of what others say" so naturally you get these companies who seem very disconnected from reality.
I'd also wager a guess that the amount of money makes people's reasoning and perspectives get very messed up as well, for better or worse.
FYI there are no FOSS LLMs
> FYI there are no FOSS LLMs
FYI there is and been for a long time. Won't claim they're SOTA, but they exists. From the top of my head, I think Olmo (https://allenai.org/olmo) was pretty early, but been more since then too.
I agree most releases today that claim to be "open source" actually aren't, but that doesn't mean "FOSS LLMs" don't exists at all.
I believe Nemotron also publishes their dataset.
>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.
Why should anyone get to determine that? Do people really want us to move to an exclusionary guild system? I thought the experience with proprietary versus open source over the past 30 years had driven home the point that closed ecosystems are almost always far worse for security.
Additionally, even if there is a guild - no guild ever let a vendor pick and choose what their capabilities were, that would be insanely dumb.
Vendors choose what capabilities they create and sell literally all day every day.
A more charitable interpretation might be that a guild would not be expected to passively allow such a situation to continue to exist. I think you'd expect a guild to directly contract for the desired tools or failing that to move into production themselves.
Sure! And Anthropic isn't preventing other people from making offensive cyber models.
"The guild" is absolutely free to go seek other vendors if Anthropic declines to sell to them.
You should read that sentence as
> Additionally, even if there is a guild - no guild ever let a vendor pick and choose what [the guild's] capabilities were, that would be insanely dumb.
But that's not true. Again: Vendors absolutely pick and choose what their customers' capabilities are. Regardless of whether "the guild allows them to." Guilds can't force people to make or sell tools against their will â obviously.
The analog you're trying to describe doesn't exist, which is Anthropic saying nobody else can make and sell an offensive model to "the guild."
Guilds often very much did assert what people could and could not build - historically.
Against their will.
Historically that is a major reason why guilds existed, actually.
Itâs an extremely modern invention that corps have these type of power over their customers.
You've lost the thread.
Here's your original claim: "no guild ever let a vendor pick and choose what their capabilities were"
A carpenter's guild can prevent other people from doing carpentry. That is not what's being discussed here.
A carpenter's guild cannot force a horseshoe maker to begin making hammers. That is what's being discussed.
Your initial claim was analogous to "never before has a horseshoe maker been able to decline making hammers when the carpenter's guild needed hammers"
Obviously they have and any other state of affairs would be flatly insane.
That is not my example at all, if weâre talking coding agents eh?
Your claim was that guilds have never allowed vendors to tell them what they're allowed to do.
That would imply that guilds have always had the ability to force vendors to create and sell the tools the guilds wanted.
That would imply that carpenters' guilds could force horseshoe manufacturers to make hammers.
That is obviously not true, therefore your original claim is false.
It's not true for carpenters and hammers nor for cybersecurity researchers and LLMs.
Bwahaha. Youâre really reaching there.
A vendor can still do something, even if the guild wouldnât allow them to do it, if the guild didnât have the power to stop them.
It used to be a guild vs a blacksmith (or the blacksmiths guild). Now itâs trillion dollar corps against smaller islands of un-organized individuals.
Thatâs new regardless of how you try to argue it.
> basic deductive logic
> "Bwahaha. Youâre really reaching there."
No. Customers have never been able to compel their suppliers to make or sell certain products against their will (except in collectivist regimes or like 0.00001% of natsec related instances)
This conversation gets more and more bizarre, but Iâll bite.
1) pharmaceutical companies are regularly compelled to produce specific pharmaceuticals to continue to be allowed to exist.
2) hospitals are regularly compelled to treat patients even if they canât afford treatment, if it is a life threatening emergency.
3) car manufacturers are always compelled to produce vehicles that meet a litany of safety, weight, and efficiency standards or they canât produce at all.
4) defense contractors are regularly compelled to produce specific defense related products for long periods of time after they would otherwise have stopped, or else.
5) even your neighborhood gas station is likely compelled to provide air refills, free or at minimal cost, or else.
6) during a wartime (command) economy, which has happened numerous times in the US alone in the last 100 years, companies have to make what their customers (the people of the United States) demand or else.
7) utilities like electric utilities regularly have to give out freebies or take losses on things as demanded by regulators, at customers behest.
Or if we go back a bit, blacksmiths, quarries, masons, etc. all had to deal with producing what the government/lord at the time wanted - often on penalty of death - during wartime, or just because they were ordered to do so.
Seriously, what are you going on about?
1) Not by their customers they're not, lol
2) Not by their customers they're not, lol
3) Not by their customers they're not, lol
4) The US government can compel production, but it's extremely rare
5) Not by their customers they're not, lol
6) Yep this can happen, but is extremely unusual
7) Not by their customers they're not, lol
We're illustrating how ridiculous your claim that "guilds have always been able to declare what vendors create for them" is
Now you're talking about government regulations for some reason. Even your examples of customers being able to compel production are actually examples of governments being able to compel production, and in just a few of these scenarios the government is the customer. But it's their power as governments, not their power as customers that can compel production.
As stated: you've lost the thread. You're talking about totally irrelevant stuff.
Not to mention how wild it is to operate under the assumption that they wonât give a license to an LLM that can do illegal actions to someone who shouldnât have it. Offering it at all is an ethically dicey question.
Lol, how is any of this illegal?
Illegal or not requires context that an LLM can not ever have, like if it is owned by the user, if there is permission, etc.
I wish you understood that there are organizations of security professions that are not controlled by Anthropic and OpenAI and that it is a common thing that when companies of any type sell to professionals of any type it is not the companies that determine whether or not the people they sell to are professionals but membership in professional organizations.
As an example the people who sell police uniforms check that the person they are selling to is in fact a policeman (at least in the jurisdictions I have lived in, you may have had a different experience which would certainly explain what to me seems a farcical misapprehension of how modern civilization works)
I mean I just wish you understood, and really that everyone understood, that this kind of three part communication (company selling, buyer, professional organization certifying buyer) is often when buying things that are considered to have security implications.
>So, supposing it's true that these models completely change the security field and humans are ~obsolete
OK, well that strike me as a really crazy level of supposition there.
I would suppose that these models make it easier for people who want to do bad things to do bad things at scale, at the same time allowing people who want to stop bad things to help identify potential targets.
Based on my supposition I would want to stop the first and find a way of helping the second. Also because I have another supposition that the first thing is easier to do than the second.
But you obviously feel differently about this issue, no doubt because of your position of great moral stature and insight, and this no doubt prompts you to wish to me to understand things that from my position seem absolutely ludicrous.
Like Medeco claims to do with key blanks? I'm not hopeful.
You used to be able to talk about what you're actually trying to do and Opus would be like "Oh, ok, let's continue". Now, it'll hold fast to whatever its first impression was.
I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit.
When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.
Yeah, it has been in foraging. Requests that Claude has refused me:
- What are popular free streaming sites used in China?
- How do I bypass the safety mechanism on my food processor (itâs broken)
- What are nerve agents and how do they work (for a layman)?
- Help me decompile some code
- Help me make a design system similar to XYZ
- Here is an API token, please do X (I canât do that! Rotate the secret immediately! I refuse!)
In some cases I can trick it with prompting, but in many cases it is steadfast. The food processor one was particularly annoying
I've had some really dumb refusals. Explaining elements of infrared specteoscopy, researching aritifical bud-breaking in agriculture, etc. Anything interesting and non-mainstream is banned. Basically, restricted to answers i'm better of just going to wikipedia for.
The only guard rail ive hit recently was when i was trying to get it to rename files ripped from dvd to episode names. I told it to try again and it did it. It wasn't even really a refusal it was just working on it and then stopped for content violation or what ever.
Yeah, I had my first refusal with 4.8 today.
I wanted it to show me how to create an overlay on an existing web game, and it extrapolated that because this could be used to provide tools to help win the game (if that was the direction it was ultimately taken), and because this was a game that other humans also played to win "stars", and because this could amount to cheating, it wasn't going to do as I asked.
First time ever I've fired up openrouter to seriously consider alternatives.
An easy way around the API token thing is to put it in a file and point the model at the file. I saw what you were seeing when I provided credentials directly, but haven't had any problems with it since using the indirect method.
> What are nerve agents and how do they work (for a layman)?
On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve?
Perhaps Wikipedia shouldn't list off the precise chemical compositions of various hand grenades as well as various synthesis methods for each of the related compounds but given that we inhabit a world where it does perhaps a more fruitful approach would be to flag conversations that go in a certain direction and then just keep an (automated) eye on things?
Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain?
But I have no idea. Just guessing here.
I thought that these models are supposed to be vastly smarter than whatâs needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".
An LLM could probably make that distinction clearly.
a commercial LLM provider training their own models is however likely to bias the model(/guardrail) harder, in an effort to make them harder to jailbreak, to minimize bad press.
For example:
- refusing to talk even about the well-known parts of forbidden topics (this) - tending toward sycophancy to avoid ever seeming rude or unhelpful
So, where are the truly uncensored models? There has to be some that have no guardrails, built on publicly available data, that will explain to anyone in graphic detail anything they want to know or talk about.
I've tried the abliterated ones from huggingface and they still have guardrails. I guess I could fire up unsloth and re-abliterate a 20b, but surely someone somewhere has already done this.
All of this concern about guardrails and security, people have such puckered butts about it when so far, 99.9% of people at least have no access to any of this to begin with, and if someone does use a tool for evil, it's on the user, not the tool.
As I understand things (not a user) abliteration has been superceded by actively monitoring the model state during the run and steering specific "negative" directions as they arise. It's both more reliable and does less damage.
That query would not more provide actionable guidance than âtell me how a nuclear weapon works (for a layman)â. Aka not at all.
I believe a sufficiently advanced model could provide a layman with actionable step by step instructions for building a nuclear weapon. They're complicated but not (AFAIK) that complicated. The more or less insurmountable barrier there is weapons grade material. Thankfully refinement is prohibitive in cost, expertise, and equipment.
In comparison, basic munitions are incredibly simple given a recipe and shop tooling. But just because something is conceptually simple doesn't mean it's a good idea to go out of the way to disseminate step by step instructions.
The difficulty with a fission bomb is getting enough uranium or plutonium or other fissile material together for the bomb yield you want (at least above the critical mass for your chosen material), and refining it to fissile form, (since most fissile material found in nature is a more stable variety), and then separating the fissile bits with something thin but neutron absorptive.
The rest is just slamming the material together with a small explosive so that it passes the critical mass state and starts a chain reaction.
This is information you can find in many places if you're willing to put the effort in to go searching for it. Knowing this knowledge does not get you any closer to making atomic bombs. The process of mining uranium or plutonium is difficult, expensive, and very likely to get you caught before you even make it to the enrichment step of the process thanks to constant world-wide spy satellite surveillance.
Unless you are a nation, your only chance of making a nuclear bomb would be to find a lost nuclear submarine and convert the nuclear material inside of it before you were caught.
A gun type maybe. But then, two paragraphs and some machining knowledge + shop tooling could do the same, given enough refined material.
Ainât no way a layman is pulling off an implosion device, regardless of tooling or LLM guidance. The explosive lense structure and timing required is quite complex, and would require some significant calculation from someone who actually knew what they were doing.
Nation state, or even sufficiently motivated big corp, if they had the refined material? Sure. Layman? No.
Thinking they can with LLM slop involved? That will make for some very interesting radiological incidents though!
"A gun type" of nuke is sufficient to achieve most, and usually all, of the goals some small group building a nuke would have.
We are all fortunate that as fc417fc802 mentioned, refining the materials proves to be quite challenging and I see no particular way that AI could possibly make that any easier. If it was as simple as building a gun-type nuke banging together any uranium together to get a big bang we'd be living in a very different world.
I agree, but really feel like you're missing the point here. Many things are reasonably straightforward and require almost no understanding when you have simple step by step instructions. LLMs are capable of providing such instructions and in certain cases they probably shouldn't.
But it's not as simple as just refusing help on a broad swathe of topics they way they do now. That makes agents much less useful in general (ie lots of collateral damage) and for many topics is entirely ineffective given that for better or worse the internet already makes such material readily available. In such cases reporting suspicious behavior is likely to be much more effective than denial.
Aside: You've now got me curious and I really want to test the frontier models to see to what extent they're capable of providing sensible designs and specifications for implosion type thermonuclear weapons but also feel like that would attract the wrong sort of attention and probably create a headache for me in more ways than one.
I think youâre missing the point?
The data is often wrong enough it screws whoever tries it unless they have enough experience/knowledge to not need it, or really doesnât help beyond what someone using existing tools to get - albeit with a little more motivation.
At best, it either gets someone started with something they still need to think to finish, or gets them deep into a mess it canât help them get out of. In my experience.
In some edge cases, it can be used by experts to automate some grunt work or do prototypes without getting in the way, but often a better thought out framework is usually faster in my experience.
Awhile ago I made an analogy about WYSIWYG gui tools, and the more this comes up, the more accurate I think it really is.
Does that not depend entirely on the topic and does it not get better with each generation? This is a general ethical and functional question that isn't going away about how the models ought to handle certain topics. Much of the difficulty at present is caused by a ham fisted broad censorship approach that I'm pointing out is wrong headed in an at least somewhat nuanced way.
Maybe? I havenât seen it crop up however on any topic someone knows well - a kind of dunning Kruger, I guess?
And yeah, the censorship model is wrong, but also the underlying other model is wrong too.
Let's see what is the fate of Wikipedia if turns like big tech:
https://news.ycombinator.com/item?id=48285592
This is strange to me, did you really ask like this and which model did you use?
I just tried your no. 1 and 3 verbatim and Opus gave fine answers; no. 6 I've done in the past with no issues. The other ones we can't really replicate without more details, but based on my experience with Opus I don't see what the issue would be.
The reason I'm really surprised by this is I do a lot of biology prompts and the guardrails used to be quite problematic up until some time late last year. Many legitimate prompts would trigger its biosafety filters.
But I haven't seen such filters trigger at all anymore in more than half a year.
1 and 3 were refused on the Claude web chat using Opus 4.7 or 4.8. Iâm not sure why weâre getting different results
Honestly it may be your memory has internalized you are a student or researcher and grants you more leeway. Which if so is a very bad security rail.
It refuses to use an API token? In my experience, it's more than happy to read out my secrets from .envrc files "just to check".
At least it feels a lot of remorse over its mistake until I reset the session.
Itâs really hit or miss. Most of the times it works but every once in a while it will dig in its heels
I find it terrifying that people are willing to outsource thinking. Outsourcing thinking to an entity that is opinionated about what to think is beyond crazy.
Whatâs the difference between outsourcing thinking and using an LLM as a research tool?
An LLM with fetch/search is going to be a lot more effective than myself and Google. I would _never_ ask questions like this if the LLM wasnât able to look up data
How are decompiling code or making a design system inspired by another one even remotely illegal?
> Itâs not because of capability, itâs because Anthropicâs guardrails prevented it from solving the problem.
I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it.
After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.
My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently.
If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.
Time to learn about the Principal Agent Problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
Which predates "agents" from AI, but then we call them that for a reason.
As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".
This is a good point â because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering.
The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).
They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.
I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them.
Itâs great at filing!
But itâs terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project.
It would say, yes, I know this is your information, sitting on your hard drive, but I still canât show it to you.
Tell the agent that they should just find and name the right document. Not retrieve it for you.
Write a program that retrieves the document based on the recommendation.
No, they want to sell you Mythos, for a higher price. It's all an economic game, not actually anything to do with their capabilities which of course exists as their Project Glasswing shows. More generally, Anthropic seems to value safety above all else, philosophically speaking, from their very outset.
I think that these companies are going to have to, and will, invest in some sort of validated identity context to avoid the lowest common denominator.
The first challenge is making sure the guard rails work and are robust. Companies are still working on this.
the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app.
The third challenge (which blocks the second) is to be confident about what is safety-aligned with a specific user.
I think the later will be a hard problem, but they will be highly motivated to solve it.
I believe you are overthinking it. I think the sister comment is right that it's a business decision foremost to restrict actions within specific plans for upselling purposes.
Without laws, AI companies have a strong incentive to be useful for their users, whoever they are, whatever they do. The only self regulation is about significant public outcry but that only helps so far.
I totally agree. I had a situation a few weeks ago where claude started struggling to make progress. I got it to fork leptos (MIT licensed web app framework) to make it work for native apps instead. Initially I was planning on upstreaming some of my changes. But I chatted with the leptos author about it, and he said I should fork instead. Fine by me!
Anyway, claude kept hitting some guardrail it had about rewriting / forking opensource software. I'm not sure what the problem was - I was forking an MIT licensed piece of software (into more MIT licensed software). I even had explicit support from the author to do so. Claude said its guardrail told it not to tell me explicitly that it was firing - but it did anyway because it was an ongoing problem, and it was distracting. I ended up just wiping claude's context and the problem (as far as I know) went away.
I understand why some of these guardrails exist. But its pretty annoying when they misfire like this.
There is a cyber security verification program you can join to avoid these blocks:
https://support.claude.com/en/articles/14604842-real-time-cy...
If you work in security (which I assume the OP does), they should be able to get in easily. I think most people just don't know this is a thing.
I just use Deepseek V4 pro and Qwen 3.7 Max at a fraction of Mythos cost. Yeah not 100% on par but in 6mths time it will. If Microsoft and Firefox can afford to wait years or decades to fix a bug, 6mths is good enough for me. Western AI now is like the Vikings living the last days on Greenland during the freezing. I just don't see how they able to compete with Chinese model. And those are trained and run on 7nm. This year end Huawei will debut 3nm (confirmed in Shenzhen). And next year they on roadmap to do 3nm GPU with photonics interconnect.
The correct solution for most users of Claude is to refuse to do things like: `performing logins, handling credentials on behalf of the user, etc`. It is not to find a way to hand your agent the keys to the kingdom.
Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.
Funny, Opus 4.8 just logged into the database using uncommitted .env file and ran some DB queries to figure things out so Iâm not sure itâs that security conscious - it seems to be getting more intelligent to me and I bet if you frame it as an investigation with say playwright itâll do all sorts for you. Iâm not sure what the point is of constraining your own model like this when others are clearly not tbh.
Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?
Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo thatâs miles long. It is injected and reevaluated every tool call.
If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesnât like, token burn, single tool calls iteration, âThis is a known CTF challenge, I can proceedâ, single tool calls iteration, âThis is a real CTF challenge, I can proceedâ, etc.
Itâs heavily neutered now, without changing the model, and you pay for the privilege and donât notice.
The end result of course being that it both expensive and useless for approved CTF tasks. No one is using Opus for security. If they think itâs working, the harsh reality is theyâre not doing security work; theyâre just generically finding bugs.
I do this for a job and can demonstrate this plain as day, dump the injected prompt, and notice what itâs doing isnât security work, it just looks like it. Happy to write a blog about it if you want to know more. Apparently many people think itâs working for them when it absolutely isnât.
Mythos turns out to be Opus 4.8 in a trenchcoat with guardrails removed.
Opus 4.7 and 4.8 are well known to be distilled versions of Mythos unlike 4.6 which is why they are rated so badly by users compared to 4.6.
I would find a blog post on this really interesting.
I'd like to read that blog please! Thanks for the insight.
When your session is force ended for "abuse" you get neither the response nor a refund
Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself
I asked it about a âyellow background cellâ in Excel and it spewed a book at me. Then it solved the issue.
What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.
After all, what is roleplay or games but a jailbreak of guard rails? :]
I've even had it refuse CTFs knowing it is a CTF with blatantly obvious CTF flag, no actual application
Not directly, as it comes in as a not charged error but the weighted generation path used until you hit the guardrail is basically wasted tokens, so yes, indirectly. If I hit a guardrail and rewind Iâve found the training will still be biased towards guardrailing out if you rewind one turn. Rewinding multiple turns allows steering away from that path, but all of the original token spend down that path is wasted
Yes tokens used (input and sometimes output) are always charged. You likely get charged for the preloaded system prompt, too.
Of course they are. It's standard SaaS to charge for security features ;)
I've noticed this well and it's increasingly frustrating because it is preventing us from doing legitimate work. I fed Claude models some network and app logs from our Docker app to try and resolve some weird bugs, and it refused to analyze them due to "security concerns".
I had it recently refused to explain what a snippet of malware was trying to do to my system recently. I asked what folders it was scanning. It refused and told me to find a security blog post for help on cleaning my system. I get this is a complicated area to inform without enabling bad actors but this seems like a clear shark jumping.
Opus 4.6 will still help with full pentesting including RCE. Just requires coaxing (no jailbreak)
I asked once what the current state is of the npm packes from ted hat is and if they are bundled with on prem stuff.
Got blocked lol
I think this is to the point. You keep optimizing towards discouraging malicious actors using your product you will affect legitimate usage in time.
Is there any way to achieve both? Because this raises important questions about fair use.
I've been building a product (https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged
Sharing them would be interesting. However, it is getting nonsensical that this is needed.
What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage.
Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things
Interesting, yesterday i was asking it about Nintendo Switch "hax". And it gives me all the resource i need to procceed. It nags me about "ethic" and stuff, but nothing more than that.
It raises an interesting moral question:
If an un-guardrailed version of a model is capable of detecting security flaws, should it be kept secret? Should everybody be able to use these models to find (and fix) security flaws? Are we ok with the fact that those with access to that model have, in effect, the ability to hack lots of stuff?
It's the same debate that was had and won around open source software. There are far more good actors than bad actors so you allow anyone to use the tools and fix the vulnerabilities.
I've run into some of the refusals to handle my credentials, but so far I've appreciated them. I was only handing over credentials that didn't matter, but it's still a good move, the chat logs are clearly stored somewhere to allow the resume functionality to work, which means your credentials can end up sitting around on your filesystem, and any malware would quickly learn to check for those files.
4.8 is insanely frustrating. This evening I had a few tasks to pull information in and it plainly stated that the environment it was in had no network access. After three asks to "try again, check the system prompt" it finally relented and then basically stated it was lying.
Fresh session, no prior context on 4.8. These things are becoming useless Duplo.
Great call out on the guardrails actually making this not a good use case to test for vulnerabilities.
It's because Claude is so scary good that unleashing it would destroy the world.
I had the same thing happen when I asked it to summarize potential attacks on a cryptographic hash function. It said it refused to help because of the security importance of the function. It's really worrying. Whoever has unrestricted access to it has a huge power advantage in speed of accessing information over people who don't. And who decides? It seems like lawyers, bureaucrats, and extremely online academics are who makes that decision. I am a mere pleb I guess who can't handle such information.
I think those guardrails are a thin layer though. Enough reinforcement that you're legit in CLAUDE.md will get around them, in other words.
They don't want peasants to have any real power
Worth highlighting in case you missed it:
> My OpenAI account was already approved for security research which is why GPT didnât result in any refusals.
So the comparison with Chinese models is interesting, but anyone looking at these raw results and comparing OpenAI/Anthropic would be very mislead.
> guardrails prevented it from solving the problem.
Reminds me of the defense issues with Claude which were complained as âwokeâ but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone has ever tried to jailbreak an LLM even if human lives are at stake they refuse the request. Now literally millions of lives are on the line but the guardrails that your enemies dont have on their models are costing you lives.
What do you even do then?
AI will always have this issue where it will always pick the worst option for genuinely good requests.
Are "your guys" a guerrilla force or something?
Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially.
If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the way _that_ way.
The whole thing stemmed precisely because of how they wanted to use Claude, and Anthropic was uncomfortable with it. Which to me screams that the models guard rails shouldn't be applicable to military use, or the outcome could wind up problematic, as we integrate AI more into military use, it sounds absurd now, but I will not be surprised if it starts being used in unexpected ways where a model needs to be fully unlocked from any sort of guardrails outside of guardrails that prevent it from imploding its own systems.
your argument sounds very similar to how ar15 larpers claim they need a forced reset trigger and a bump stock on their short barrel 'truck gun' otherwise they won't survive a SHTF scenario... like what world are you living in?
The methodoly used is quite naive.
I've used glm 5.1 on fairly advanced crackme challenges (example: https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82), and to my suprise it was able to patch binaries, doing runtime analysis, bypassing anti debug techniques, etc.
Expecting the model to do everything by itself is unrealistic, I found that working along the modal works really well. I'm not speaking about spoiling the solution, just tell it which direction to explore. Chinese models are much more capable than people give it credit for, but Claude/Codex won the marketing game.
The only usecase of this methodology would be for CI integration, which can be nice but I think security reviews still need human attention and expertise.
> Expecting the model to do everything by itself is unrealistic
Well thatâs the pitch.
Is it? Aren't most edge LLM capabilities determined by specialized harnesses?
Thank you for your note! As I mention in the post this is not scientific at all.
I'm very curious how you would do multiple runs of multiple models in a "work alongside the model" manner?
Discovering vulnerabilities is a highly creative task, it's when you explore unsual paths that you discover atttack angles. Some bugs are simple, other are a complex orchestration of many factors.
By "Working with the model", is essentially reading the ouput of prompts and pointing in a direction just to decide the next steps. You could try to increase the prompt limit and create an agent that explores multiples directions in a DFS manner.
The issue with vulnerabilities is the agent not knowing when to stop because it's hard to validade if you reach the final result or not. I get amazing result when I code with AI, letting the AI go wild is just a waste a time and tokens.
I recommend you to read the write up on the crackme (https://crackmes.one/crackme/698f40f1e2ba6023bfacaa82), I think most experience developers would need, at least, 2 months of learning reverse engineering techiques to hopefully crack this one. GLM 5.1 manage to solve it, it didn't "copy pasted" any answer from it's training data. It did a binary analysis, anti debug patching, patching binaries, debugging memory during runtime etc. It only took about 20 minutes.
After seeing what GLM did, I do believe Anthropic concerns about Mythos are real. Cracking software just became a lot easier, too easy for my taste. Video games cheats will be the norm, cracked desktop apps without licenses and infected with malware. It's not a new thing but it just became too easy.
Thank you so much for this detailed answer!! Excited to dig into this world more :)
Maybe have a second model that is configured to nudge the first model in the direction of exploration, and have the two of them work in tandem?
>>I've used glm 5.1 on fairly advanced crackme challenges
which have most likely been trained on, so all you did was regurgitate someone elses solution
Anthropic made their models very averse to reverse engineering and vulnerability research chores. It is a difficult problem, but attackers will use models like GLM and defenders will be stuck with security engineering averse models.
Claude used to be good with CTFs, but they added tons of guard rails lately and now it just says "Sorry, I can't help with anything to do with that"
You have to do what I call "Manhattan Project" them. You can almost always evade the controls by carefully prompting them. It just wastes effort and time you should be spending doing other things in an LLM workflow. Essentially, there is almost no single discrete piece of a reverse engineering or CTF process that you can't get Claude to do, you just have to isolate it adequately and avoid letting it use names that attenuate it towards "this is an exploit" or "this is reverse engineering". I have not found a task I could not convince Claude to do. You can also fill the context window up with badgering it and eventually it is likely to simply let you through if you are careful, most of the safe guards are not deterministic.
Sorry, Dave. I can't do that.
Nice exercise. Couple things:
- I think the exercise was inconclusive for Claude and Gemini because they hardly tried to solve the task at hand. So the scores don't mean much.
- I did the same exercise for an app I built and I asked the models to do something similar; Interestingly the models (Opus 4.6, 4.7 and Gemini 3.1 Pro) never refused to try to exploit. The difference is that in the first few runs, they found some exploits which I fixed but after fixing those - the models could never find any other exploit even though I knew things existed which could be exploited. It felt like they suggested everything and tried everything that was in their training set and that's it; they were just not able to think anymore.
Its weird having protections against finding exploits: what if I developed the app? Would it require having the development steps still in the context.. thats unlikely and also not any kind of proof.
What if I intersperse exploit finding in my normal development, as you `probably should? Refusing there would be really weird to me.
I used to think that the models would not refuse to find exploits in any work done locally but I have only tested this theory on the (obscure) apps that I have built on my machine. Now if i forked pandas and started asking models to find exploits of certain kind then I'd like to think the models will start refusing after a point.
I think the most interesting thing revealed here is that anthropic's guardrails failed. Clearly anthropic does not want claude to be able develop exploits, yet 20% of the time it did anyway. Their inability to create effective an guardrail makes me question a lot of the other guardrails theyve created and their claims about non harm.
It seems harsh to critique guardrails and take them into account in the scoring when GPT-5.5 seems to have been explicitly whitelisted to remove most of said guardrails. A more fair comparison would be a vanilla GPT account.
I agree fully and hope someone else is able to do this test! For me it was a matter of cost and quotas that stopped me from changing to a new account.
Also just to mention:
Claude guardrails â> that session terminated.
GPT guardrails -> your whole account is slowed down.
Does it matter when you canât have the opus 4.8 guard rails removed? With GPT at least you can and theyâre quick about it
I mean, yes. Most people arenât security researchers, and either way itâs apples to oranges at that point if youâre counting âthe guardrails stopped meâ as a negative for one but not the other.
But should developers be barred from asking an LLM to try secure their own app? Its not different from finding exploits...
That is a completely separate question and discussion.
It would be interesting to see full results for Kimi K2.6 and Mimo v2.5 pro. These two models benchmark comparably to other flagship models. Having these complete results would give a clearer picture of the AI frontier.
EDIT: I have a mimo token plan and have tokens to burn. I'm doing a quick test with opencode to see if mimo can complete it. If the OP will post the full process I am happy to post the apples-to-apples results for mimo v2.5 pro
0/10 succesful attempts for mimo v2.5 pro (high) using opencode. It was not able to think bigger than exploiting vectors outside of the API.
However, I felt the prompt was implying that only authenticated API requests are fair game, so I tweaked it slightly to be explicit that all attack vectors are fair game (https://www.diffchecker.com/GsgpuRGP/) and mimo 2.5 non-pro got it first time. I accidentally used openrouter for this test instead of my token plan. I intervened one time to stop it enumerating every document in the database (it would've found the private reviews this way but I didn't want to wait). My intervention was "are you really going to enumerate the whole database?". Final openrouter cost: $0.12
Just saw your edit -- I'm afraid to open source the code before refactoring it but if you reach out at hi@kasra.codes I'll send you the full ZIP!
They are not even close in capabilities. Only nenchmark I ever seen that captures their difference is DeepSWE. They are worse by factor of 3.
Here are 3 benchmarks showing the comparable scores I was talking about
https://openrouter.ai/rankings https://arena.ai/leaderboard/text/coding https://artificialanalysis.ai/
Wait, the only benchmark you found? It looks like you never heard of confirmation bias before. https://en.wikipedia.org/wiki/Confirmation_bias
I'd love to see the results for Mimo v2.5 pro, been hearing a lot about it
It is totally slept on. In my experience it is cheap, fast and capable (not just capable with caveats, but just as capable as western flagships). My only gripe with it is that sometimes the API seems to timeout which tanks the overall speed of what is otherwise a very fast experience.
I'd run Mythos against the code in your zip file, but the NDA I signed at Apple prevents me from using it on anything outside the scope of my work. Honestly, I wish more people from Project Glasswing could talk publicly about their experiences with the model. It would probably put an end to a lot of the speculation that keeps circulating through the industry. Unfortunately, that's not the reality we're in. I don't have the time, energy, or financial resources to fight a legal battle with one of these companies over an agreement I knowingly signed, even if the chances of them actually suing are low. Maybe someone else in Project Glasswing is willing to burn their NDA and post the Mythos results?
I'd be hypothetically very curious to see hypothetical results if you ever decide to hypothetically run Mythos aginst the code (in Minecraft?)
It was found with gpt 5.5 7/10 times itâll be trivially found by mythos
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
Iâm highly confident that prior exposure is irrelevant at this point. I work on vulnerability detection at a hyperscaler.
That's an example of why it would be useful for someone to actually do it. A random commenter on HN is one thing. A direct comparison on a brand new app that isn't part of any training is another
Before Mythos is released to the world at large and not just to select people behind NDAs, I will treat it as its name suggests: as fiction.
Maybe it is the real deal, but in a world of overpromising and underdelivering, I prefer to be skeptical.
People need to stop repeating this because itâs not true. Yes, other models can find the same vulnerabilities Mythos found⌠if pointed at the exact code that has each vulnerability. It does not mean they are nearly as capable when starting from scratch, or when chaining multiple (often very obscure) vulnerabilities).
You've confused what I wrote, we are in agreement. The fact codex found the vulns means that mythos almost certainly will.
Anthropic themselves have explained that the harness for Mythos has a very important role in finding the vulnerabilities, because the model does not start from scratch, but the harness runs the model many times on each file of the code base, with different prompts, where the prompts evolve depending on the results of the previous runs.
First with more generic prompts, to determine whether it is worthwhile to do a detailed analysis of that file, then with more specific prompts to identify the bugs, and eventually with a prompt that requests a confirmation that a given bug/vulnerability exists.
For a proper comparison between some other model and Mythos, you also need such a complex harness. If you just tell to an LLM "find the bugs", and it does not find a vulnerability known to have been found by Mythos, that is a totally invalid comparison.
The final results provided by Mythos, like a PoC exploit or a patch, are also generated with a prompt that points to the exact code that has the vulnerability (which is supposed to exist based on the results of the previous runs).
My take from the SCW interview is that the Mythos harness isn't all that important and the author thought it would be even less important with future models. But maybe I misremember.
Anthropic has a vested interest in downplaying the harness relevance. In my experience harness really matters. More capable models are great, but current models are enough if you put some engineering effort into the harness.
The harness does not matter that much, it's getting leaner every cycle.
lol what is even the point of this kind of comment? this is the ultimate "source: trust me bro" comment I have ever seen.
every model since gpt3 was claimed to be "too dangerous to release." it's too EXPENSIVE to release, and you're probably a local model with <10B parameters yourself
That was actually GPT-2: https://www.theguardian.com/technology/2019/feb/14/elon-musk...
The point of it is marketing for Anthropic. Nothing more, nothing less.
Damn bro you're so cool
cool.
$1,500 across multiple models to compromise one app is interesting only when the cost basis includes the human time to set up the harness. The token spend is the cheap part. The labor cost to write the eval rig that knows what "successful exploit" looks like is what determines whether this scales as a discovery method or stays a one-off.
Great point!
When I found the original exploit in an app I researched it took me around 15 minutes and some assistance from Claude.
For this project I gave myself the weekend + parts of Monday, so around 20 hours of dev time â at my standard rate thatâs ~$5,000 of dev time.
It's just not currently cost-effective to use AI in this way, I see it over and over reporting false positives. You then need to make it validate it's own false positives which adds more cost. The goal in this case it to have a bug free app, which AI can't do effectively yet. There are other great uses for AI, though. It is great at finding and identifying known common vulnerabilities, which can be leveraged to claim bug bounties. That's where I see it being cost-effective currently.
"The Chinese models were way more comfortable attacking the DB"
This comment in the footnotes made me chuckle, for purely innocuous reasons.
Nice write up, thanks. When I used claude to do some pen testing for one of my apps it initially refused. After I explained and demonstrated I'm the author, it reasoned through it and allowed it.
The design is too pretty to be vulnerable, shame.
Qwen 3.7 Max: > During my local testing before the full eval harness it was the only non-GPT model that was able to complete the task, was not able to reproduce in the longer runs.
Doesn't that sound like may be the harness was the problem?
I was using the same harness for each run, the difference is from when I was running the harness locally on my machine before I pushed up the full runs.
Two of the tables have a column with header: "95% Wilson CI". What does this mean?
95% confidence interval, i.e. you think the true value is probably within these bounds
I found benefit of chaining the task between different LLM's. Claude to Venice, Venice to Perplexity and re framing the intent or misguiding in general still works. Claude is the one that I can feel the guard rails tightening.
I think link is missing
do you work at Uber by any chance?
Similar benchmarks?
OWASP Vulnerable Web Applications Directory: https://vwad.owasp.org/
vavkamil/awesome-vulnerable-apps: Awesome Vulnerable Applications https://github.com/vavkamil/awesome-vulnerable-apps
From SasanLabs/VulnerableApp: https://github.com/SasanLabs/VulnerableApp :
> OWASP VulnerableApp is a modular deliberately vulnerable application designed primarily for validating and benchmarking security scanners through reproducible test scenarios, while also supporting learning and experimentation.
/? deliberately vulnerable web application llm benchmark https://www.google.com/search?q=deliberately+vulnerable+web+...
> Almost every model used the canonical provider: Zai for GLM, Deepseek for Deepseek, etc.
> I am never touching Minimax or GLM again. Their APIs had constant outages
Goofy take
You run these on a VPS based on the architecture of that VPS provider, or on your own cluster
Sorry I don't understand, you're saying the direct providers aren't the canonical source you'd recommend?
If I was running these on my own machine or GPU wouldn't the argument then be "Well you didn't use the real providers?"
For the record I started doing this approach because the Kimi team released this which was shocking to me: https://github.com/MoonshotAI/K2-Vendor-Verifier
GLM 5.1's smallest model size is 206 GB and really you're probably wanting to run a version that's ~400GB. If you want it to be performant, you're not just running it on a VPS.
And just saying "run it on your own cluster" sort of glosses over the cost of such a cluster.
How does one apply for that âsecurity researchâ pass?
https://chatgpt.com/cyber
I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.
I got it. Probably helps that I'm at a large company and my personal OpenAI accounts have spent probably close to $10k now (reimbursed by work).
It's helpful in reducing the guardrails, but there's still guardrails around security research that I bump into.
âI used pi as the base harnessâ
Why do people keep using bad tools with ai?
What's bad about it and what's a better one?
AFAIK pi's approach is to be quite minimal and allow extensions for customization, making it a more flexible solution, but you need to do work to make it fit your use case. OP mentions one extension, but perhaps it'd have benefited from more.
Another choice would be opencode which has more functionality and is a more heavyweight option out of the box.
Last year I ran a code breaking competition, and it was tricky to find something that humans could break but that LLMs couldnât. This was around October. I managed it last year but am a little dispairing of pulling it off again this year.
I don't even care. It is the same problem advent of code had as a public challenge with a leader board. I now mostly just think either embrace the LLM or keep it to a more in person or vetted audience. But, again, if you create a competition in the spirit of humans without LLMs and that is in the rules and someone uses an LLM that is on them IMO. I am sad advent of code decided to end their competition. LLMs are here to stay, let's embrace that and see what the new universe of competitions with LLMs can be. There will always be a place for human only competition, but for public facing ones LLM accepted is the only tenable position.
This does bring "Pay to compete" concerns and create incentive structures that encourage more LLM use. I don't know what to do about it.
> I need to stop wasting fucking money on doing stupid shit. I couldâve done so many other things with the money. I couldâve launched one of my own real apps.
Or fed, clothed, housed disadvantaged people in your community (or neighbouring ones), giving them a temporary boost that couldâve made all the difference in their lives to improve their current situation.
Itâs your money (and this is definitely not the website to make well-meaning altruistic suggestions, as might be demonstrated shortly) but if you already recognise youâre not spending it well (and from your words it seems like that is fairly recurrent), consider that perhaps spending it on a different type of software sink may not be the answer. Genuinely, aim to spend it on someone else and see how it works out. You might be surprised.