The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
Disabling http2 can be a big deal. My service is primarily made up of gRPC APIs. Not an option for me to disable http2.
Luckily I'm not using nginx at the moment - but I easily could have been.
If you're using grpc, http2, etc you're so much better off using Envoy proxy
You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.
The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.
I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
Envoy probably has the advantage that itâs only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.
Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]
In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.
There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee.
I know Google spends a lot of manpower on making envoy secure.
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.
I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.
In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.
Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.
There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
Not saying that thie image is affected (I don't know), but small footprint does not guarantee security either. It can hit systems of any complexity these days.
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
One idea that might work for you is client certificates if those are only internal APIs. I don't know about you, but I have a lot more confidence in openssl than the people who use openssl.
> this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week
And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.
It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
I'm sure things will stabilise in the near future.
That said, I just moved from a Linux based job to a Windows based job and it's been nice not having to be panicked about having update most of the technologies that my last job used. As an aside I did try in the past to make sure we were always updating dependencies but my manager was bothered by the amount of work that created. I'm watching that team now take forever to deal with these issues because they're too far behind to just version number and rebuild.
Havenât figured out how to deal with contributions going forwardâŚ
The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesnât break in interesting ways is still pretty hard.
Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
How about a strict anti LLM use policy for contributors? It adds what's basically a "proof of human work" challenge to opening new PRs, and effort required to review human PRs usually roughly scales in proportion to the amount of effort required to make human PRs. And human contributors who take time to make a PR manually are probably more likely to become longer term community members and not just drive-by contributors
It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
There's plenty of people who contribute to non-AI open source projects, you just don't get the lazy people who don't want to be a serious part of the project or the people who don't know how to program anymore
Until we don't replace UNIX clones, Khronos APIs, LLVM or GCC with something else.
C and C++ will stay around for a very long time, even with AI powered systems.
If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Rust is on its way. Vulkan is no GL, and Gallium it's something else over classical MESA under SGI/Sun.
Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
Fun fact, Russian soldiers ride on top of their tanks and APCs because its safer than being inside of one during an assault. Doesn't mean you are wrong, but there will always be trade-offs. There is no such thing as perfectly secure.
Not exactly true. BMPs are death traps, sure, and sitting on top of those instead of in them is rational given the risk of burning to death inside of one or being obliterated by a mine splitting the hull in half.
But the only reason to sit outside a tank rather than in one is because riding on a vehicle is safer than walking through an open field. Tank crews are not sitting outside of their tanks, and they don't have seats inside for passengers.
Tank desant is practiced because there isnât room in the tank to fit extra people. Itâs very much not safer outside of a tank than inside! Itâs actually the worst of both worlds, because you donât have the armor protection but are highly visible. The only advantage it gives over infantry assault on foot is mobility. Which is why itâs generally only practiced when there arenât sufficient APCs/IFVs available
Fun fact, these specific types of problems aren't possible on a JVM. Yet somehow Java isn't the right language to write a webserver in. These problems are possible in Rust and C/C++. At least its "hard" to have this type of problem in Rust.
Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
How bad are these CVEs?
Could any of these be related to some of the recent outages (not claiming, asking) in the last week, coinciding with when this bug was fixed in the repo? Monitoring such repos seems a given for nation states.
While that is true, you should also account for exploitability. These exploits are demonstrated in lab environments which are simple and controlled. Random legit traffic will interfere with the exploit logic, so these are very hard to get in practice. Not impossible, but it's also not immediately broken.
According to https://www.rapid7.com/blog/post/ve-cve-2026-123007-critical... it requires 128GB to be transferred in a single connection. "a network slower than approximately 300 Mbit/s will take over the one hour connection limit threshold and will not be able to succeed"
Not all of the bugs disclosed today in nginx require that much traffic to trigger, the reverse proxy HTTP/2 bug found by Xint needs just KBs of traffic.
Thankfully most of these bugs coming out seem very difficult to exploit (likely why they took so long to find). They usually become dangerous when you can build an entire chain of bugs to exploit.
But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..
:) If the battering ram is 128gb big, it'll be a long time unless someone has reversed those changes in Nginx to get at the exploit. After that, yeah, it's no contest
And nginx is so widely used many people will be running it and not really know that. Itâs ubiquitous reliable invisible infra for many issues. Like preventing the internet from getting to your RCE riddled application servers, terminating SSL, free software LB, etc. if you have cloud native from the beginning you might not know where AWS uses it but otherwise you have it. Cloud flare I think uses a Rust rewrite.
These days even the "disable X" is enough to get everybody looking in the right direction though. i.e. such hints are (given enough tokens) already RCEs
In my experience, asking an LLM to "Do a security audit" doesn't get you much more than an overview of the most obvious config issues while "look for bugs specifically related to HTTP/2" gives much better results.
I took a quick look at The Anarchist's Tool Chest and noticed in Chapter 20 a short vignette about an expert woodworker named Troy that seemed... oddly familiar.
According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.
"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.
So no matter what Troy did, the prices for work (good or otherwise) declined. Ikeaâs well-designed kitchens and Value Cityâs cheap junk made it impossible to survive and live well.
So Troy struggled with his business. And thatâs nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time
we talked, Troy was working for his wifeâs successful tax business.
Who is to blame for this? We are.
We have become a culture that is obsessed with price more than any
other attribute of the things we buy. It doesnât matter if the item is ugly,
poorly made or constructed of materials that cannot be recycled. All that
matters is if the price is low enough.
Because the price of our household objects has hit rock bottom, if an
item breaks or starts to look dated, we can throw it away and buy some-
thing else. For the first time in human history, manufactured furniture is
shockingly inexpensive.
So itâs no wonder that artisans are exiting the craft. Itâs difficult to
compete against furniture that costs less than what you pay for your raw
materials.
Lesson: People cannot imitate machines and defeat them."
The lesson is that most people aren't discerning consumers in multiple fields.
Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to naturally get a critical mass of people to care about that quality curve. You have to sell them on it.
Troy struggled because he failed at marketing, not production.
Or even single fields. I've purchased my parents the "nicer" version of things and they inevitably end up back on some crappier version. Japanese knives? Nope, still use the ones that barely cut. Ubiquiti wifi system that covered their whole house? Nope, dad unplugged something, got frustrated, and the cable company came over and put them back on the old crappy system. Nice speakers? Nope, they listen to music on their crappy TV speakers.
I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them improperly and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
On the other hand, there's an accessibility to products of higher median quality to more people than would have existed even only five generations ago.
I felt your post was off topic. But maybe it was on purpose because we are getting bored of so many vulnerabilities being discovered by AI security tools. So your post is important because of that.
I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
Becoming a carpenter, and sentiments of "I'm going to live in the forest" or "I'm going to be a farmer" are usually expressions of fatigue with technology, and a desire to be free of their problems; thus, an expressed urge to pursue a career where technology is not a major component. It can also sometimes carry a desire to have one's labor output be tangible and physical (i.e. a worked wood object) rather than abstract, as so much programming/operations/etc. work often is.
The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Haha. The image of shouting voice instructions to Claude while trying to herd a flock of chickens to their coop is the dream, very tony stark'esque only with chickens!
Well, I am usually the person in the room who does not get a joke. I am so scared of get-togethers that I actively try to avoid it. Also, they ask me to dance. I am terrible at that. I am not autistic but donât get the drift of many jokes. I am human and not a bot. Guess thatâs what a bot would say. But still I am human.
I didnât get the joke myself either, so thanks for asking. I first thought âwoodworkingâ was used as a synonym for hacking and clicked on the links with the book recommendations just to see theyâre about actual woodworking.
Not sure why others downvoted you, but you have my upvote on it.
Just take a second to check out the account if you're going to say stuff like this. It's not just you either, so this goes for everyone making these comments.
The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.
> Hey so, does anyone know of a good tutorial on getting in to woodwork?
Some people really don't get these type of things.
Ha, sadly there's a huge long line of developers who pivoted this way already. Plus now that you can fire up an MCP for a CnC machine, that art will be "ruined" too!
Are we going to discover in a year that every server online this month had a rootkit installed? Thereâs got to be thousands who are too small time to wipe and reinstall.
AI is securing the internet. This is great. You should assume professional hacking gangs and state actors discovered (at least some of) these vulnerabilities months and years ago.
Both are true. The problem is security is a much harder job than attacking.
Worse patching adds instability into a system, especially when you don't have much time to test.
Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.
Things are going to get much worse before they get better.
Yes, things will get worse before they get better. Attackers have the advantage initially and get to pick a lot of low-hanging fruit, but the advantage will shift to defenders as code becomes more heavily tested and verified.
One hopes, but the models are also getting smarter and deeper too. I don't know if anyone really has a clear vision of when the future plateau state is where equilibrium is achieved on this.
I find this kind unreasonable: software has finite bugs, no matter how smart your attackers are. Current CVEs are pretty subtle, and I see no reason to think that they're actually low-hanging fruit.
I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.
Software has finite bugs, but there is infinite software, especially with "vibecoding", everyone, every organization can vibecode a "nginx" for in house using.
>Worse patching adds instability into a system, especially when you don't have much time to test.
Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
Except, human in the loop in control - blue team is not defending red team fast enough that most the CVSS v4.0 Critical CVEs aren't easily patched in coordinated fashion. Many times, almost impractical or impossible to patch at all, with decades of intertwined dependency tree and their ecosystem.
Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
AI provides the capabilities to secure the internet, maybe, but the internet is secured when site owners choose to secure it.
That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.
Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
Yes but the barrier is infinitely lower. If you were used to running servers just for fun, that kind of threat was probably not too much of a concern. Now it very much is because it can be exploited at a much larger scale.
Yep. In recent discussion this came up, and my analogy was that LLM will cause a similar effect to widesread proliferation of TLS. It was easy to snitch on people before TLS had been deployed everywhere. Not so much after that.
No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.
But in this day and age, whatever is happening, the better question is:
Is it good for me and, beyond me, for the world in general?
This internal models stuff remains marketing BS afa I am concerned. Smart enough to be a junior engineer and steered toward the right tools and behaviors within a context is more than enough. Places with "super models" do nothing to demonstrate they wouldn't have done better to use the resources on relative brute force within the security domain. They have no sales pitch for good results that just needed Legos and cash.
Man, is it aggravating to read fantasies like this, as someone who's trying to juggle a full-time job and hosted public services for other people at the same time.
There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.
The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".
The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."
We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.
Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.
When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
I wonder whether part of the solution is not using the same software as big tech. For them rapid development is worth extra maintenance. Things like the efficiencies of http2 over 1.1 actually matter. Maybe the rest of us should be using BSDs or different Linux distros (Alpine? Void? Slackware?), different web servers, turn off features we do not need (e.g. not disable http2, or at least not allow both 1.1 and 2 as that is clearly extra attack surface).
My sites are now sat behind my own custom reverse proxy. It's missing tons of features, which I guess I can now call a security benefit, and it pretends to be nginx just because I thought that would be quite funny. I've debated open-sourcing it but I a) don't want to maintain a project right now, and b) security-by-obscurity is back in action.
> When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
Appreciate your take. Question in my mind is if the less-commercial internet will have the resources to defend against attacks made easier and easier by AI.
The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.
I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Poor security practices are burning down the internet not AI. Many people warned and got ignored, by developing a memory safe language, introduce large scale fuzzing, utilize verification tools and other things.
There's definitely an issue of scale, but a wooden door at least lives up to its own "contract". There's no bugs in a wooden door, everyone can see plainly it doesn't withstand a chainsaw.
A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.
The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.
Itâs for the best, IMO. We must now design every system on the assumption that every vulnerability will eventually be exploited and every bug will be found. Relying on security by obscurity or following poor security practices will be costly! In the long run, I believe this will improve software quality.
For eight years, MongoDB shipped a default configuration that left it opened to the internet with no password.
Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
This is some insane framing. This vulnerability exists already. Even if there were no LLM, this would still exist and might even be actively exploited right now. We just wouldn't have a chance to know it.
I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
If a vulnerability is practically impossible for a human, but possible for AI, then it's not security through obscurity. Technology just advanced past a line where new things can be exploited.
There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
None of these vulns was impossible for a human. There just wasn't that many humans clever enough to find them, and those that did we're busy not hacking random things. There isn't a single type of new exploit that AI has found that I know about.
We have a billon hacking experts now is the big change.
I donât know any systems programming language that doesnât ignore integer overflow/underflow in release builds. IMHO this is a CPU design flaw - integer operations should carry a flag indicating if the operation is allowed to wrap or if it should generate a trap. Then there wouldnât be a performance penalty for this and languages like Rust could have it be trapped by default.
No. Memory safe languages still have integer handling issues, but the consequences are different. In a memory-unsafe language, integer mishandling often nets out to bad copies and memory corruption.
I wouldnât be sure that a Rust equivalent would be immune because itâs not hard to imagine an underflow/overflow from reaching an unsafe block thatâs used for performance (the implicated TLS path would be a hotspot worthy of such a choice).
This is the thing about `unsafe` blocks: they're not memory safe. It's right there in the name.
I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.
Android had an integer overflow vulnerability where it could be tricked into verifying the signing of one file but install another for execution. Boeing 787 Dreamliner had a bug where after 248 it's 32-bit counter overflowed triggering a "fail-safe" mode that shut down all electrical power to the aircraft.
You're right it's more rare but the results can still be catastrophic.
Isn't "memory corruption" typically the fast path to RCE? The attacker gets to overwrite a function pointer, or a regular pointer that then lets them overwrite something else, or the length field that will then cause a buffer overrun even in a "safe" language because the run-time length is now different than the one in the compile-time check, etc.
X86 had the INTO instruction to trigger a fault if the Overflow flag is set, so not quite a per-instruction bit but more of a suffix. JO is also fairly straightforward for compilers to add on important arithmetic operations. We can probably blame C entirely for this; only in C23 are there intrinsics for overflow detection on arithmetic.
C23 intrinsics are so ugly that nobody is going to use them. They should have made ordinary "+" to trap on overflow, and an ugly instirnsic to wrap around.
JavaScript and Python arenât systems languages. Swift often pays the performance penalty for trapping everywhere and hasnât seen adoption as a systems language for that reason.
We're after a way to go fast without having your things broken into. "Language allows you to specify whether you want wrap around or an exception and hardware makes implementing that efficient" is better than both inefficient runtime checks and silent data corruption.
Yes, so "+" should trap, and there should be a function that wraps around because wraparound addition is used very rarely (mostly in crypto). It seems that you agree that C is designed the wrong way.
i still remember win xp pre service pack 1 days where you could get p0wned by a worm just by leaving machine on the internet. this is nothing compared to that.
I am having a hard time understanding this position. AI is uncovering flaws that "state-level actors" are exploiting since decades, for some of the recently found issue.
Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.
Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.
An attacker wins when they can breach any defender's security.
This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
"Cattle, not pets" has been industry best practice for a long time now. Those that don't follow it will eventually pay the price for one reason or another.
Think of it as the opposite of security through obscurity: at some point the hackers won't be able to keep track of it all. They'll start paying for Jira accounts and need to hire PMs. They'll be patching servers to drive away the competition and will need experienced IT people. Someone is going to have to build apps to keep track of it all.
I'm pretty solidly in the AI skeptic camp. But adversarial use of AI to improve software security is a good thing in my mind.
I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
At the end of the day it will be both. In fact I wouldn't be surprised if we started seeing offensive defense. That is already compromised boxes counter hacked and erased or otherwise made unbootable.
Iâm generally not in favor of government meddling with private property, but I feel like I could be on board with the NSA or similar covertly patching vulnerable machines.
>> Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
Yes, AIs are fallible. Yes, humans are fallible. Nobody said differently.
That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.
Really? AI is catching all the easy bugs Iâd catch, and way more subtle ones as well. Maybe it was the case theyâd miss obvious things often a year ago, but thatâs not todayâs frontier models.
It's going to be rough sledding for a while, but, so long as you're on FOSS, at some point, the bugs and vulns should run out, and best practices trained into AI should hopefully reduce new bugs being introduced.
All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
Also in the few years prior to 1995 most computers that could run a C compiler weren't big enough to run an Ada compiler. Heck most computers today can't run the rust toolchain.
The bigger problem is lack of liability: we knew C sucked in the 90s, too, but we had three decades of over-confident dudes saying that was user error and _they_ were smart enough to use C correctly.
Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or itâd be too much work to rewrite things.
Iâm hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
It's also valid to say that Rust's memory safety is good but the rest of the language sucks. Just as annoying as the over confident c-heads is the rust community that made even countenancing working on an alternative memory safe system a moral sin, which stifled innovation in memory safety. Hell, people barely know that Ada/spark is memory safe (or that sel4 is memory safe C)
Aside from the overhead of borrow checking, what about the rest of the language sucks, in your view?
To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.
Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.
Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.
You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
> what about the rest of the language sucks, in your view
I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
I imagine they are referring to the cognitive overhead of learning to work with it.
Whether that is âsomething natural to youâ, âsomething you learn and adapt toâ or âsomething you fight with foreverâ seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
This really isn't true. It's true that those people saying those things existed, but much more recently. Back when all the C/C++ legacy code we depend on today was being written it was the performance penalty imposed by safety that forced developers down that path, not some lack of interest in other options. We were also transitioning from a world where a fair bit of systems code was written in even lower level languages than C (assembler, BLISS, BCPL). The world we're in today effectively began around 1975 when people figured out how to make a C compiler run on an 11/45. All the decisions were pretty much locked in from there on.
What a strange takeaway. This is exactly how a server at a restaurant thinks about AI. Just the shortest term goals of keeping the status quo. They think that if they bury their head in the sand, everything will continue as normal.
what do you mean it's a non issue? Surely no one is going to take the trouble to take down my toy website, but if there's something valuable enough, a motivated actor can/will try to exploit
That first link sounded like a scanner that'll actually verify your fix or workaround but it's just a long version of "check yourself if you run a vulnerable version and have http2 enabled". To save others a click
It's becoming mentally costly to maintain personal VPS. Beside the time you have to put into it, now it's also worries of a vulnerability you missed, and boom, you're hacked.
Hardly anyone talking about the Linux SCTP bug that they used to LPE out of the container. Apparently that's a zero-day without a public CVE number yet? Presumably they're coordinating with the kernel team and distros to get it patched first. Of course, one might just assume that there are a bazillion LPEs available to even a properly confined rootless container, but ... yuck.
I notice a phenomena that when older software has a critical vulnerability, like say Apache, you get quite a few comments of "Why does anyone use that? Use Nginx"
But when newer software has the same vulnerability, you do not see the same kind of comments.
There always seems to be a latent stance in IT that newer is better.
> But when newer software has the same vulnerability, you do not see the same kind of comments.
This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
Having RCEs in general is fine. It's the rate and severity that matter.
We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.
In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
CVE-2026-98432 also applies to non-MPTCP connections. That's why disabling MPTCP was actively discouraged as a workaround (low level firewalls that clear all options do work, though, and they'd also work fine with Caddy).
Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
surprised there hasn't been an effort to rewrite nginx or other critical networking infrastructure in Rust. We just went straight from C (nginx) to Go (Caddy/Traefik) which are not interchangeable (garbage collection).
I scan new domain names but for a different reason. There is an insane amount of nonsense domains registered everyday. Would be hard to tell just based on that.
Sitting on flaws is dead, the responsible disclosure people can get lost. AI finds and exploits flaws in hours. We also have little knowledge if the people finding the exploits have already intentionally or unintentionally leaked said exploits to AI providers.
If you want to promote responsible anything, go for responsible writing of software in the first place.
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
Rust has a compile time option, though to my immense disappointment, it defaults to on only in debug mode. (You can opt into it for release, if desired.)
Python simply can't overflow, since its integral type is not a fixed width.
But it is the sort of correctness thing that I do wish languages would pay more attention to.
In Rust's case it just matters a lot less since bounds checks are still present. But I do think int under/overflows are probably one of the scarier issues in the language.
Overflows are dangerous by itself. Assume you keep the order total price as uint32. Someone orders a spaceship for 2.2B credits, and then adds another one which is not an expected scenario. The total counter overflows and now you have sold 2 expensive spaceships at 99% discount.
While I do think that was one of (maybe the only) really great decision Python made... There are so many other things wrong with it that it's a poor choice for almost anything in my book.
HTTP/2 again, not surprisingly. The curse of complexity strikes again. When I first saw the spec, I knew it would be far more difficult to implement correctly than 1.1 or 1.0. While 1.x is not ideal either (parsing a protocol that involves free-form text with no fixed limits is always a bit scary), at least it has a much smaller attack surface.
On the bright side, perhaps "security review" of code which involves throwing it at an AI and fixing what it finds before releasing will become a normal practice.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
given the state of our weapons industry I think it's more like
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
And search space in software can be absolutely staggeringly large. Just a little bit of directing and you can shrink the problem from all the entropy in the universe to a few hours of compute needed to find an exploit.
No (well maybe no), because they are lobotomizing the public models on cyber tasks.
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
The observation was "if one person found the bug with LLMs, other people are also going to find the bug". Whether it was due to communication between instances of the agent or just due to the predictability of an individual agent is not clear and there is not much evidence for the former.
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: https://nginx2root.com/
Recent high-profile vulns involved CEOs of a security company breaking embargo on a Hacker News account, and anime dancing girl videos that somehow leaked; so I think itâs barely any coordination anymore.
Consider a world in which the only people that get to know about it are the people who discovered it in the first place.
Consider a world in which corporations like the Ubuntu Foundation or Microsoft experience zero public pressure to fix such vulnerabilities but instead fix it only when a major customer notices their servers getting hit.
Imagine if instead of "you've got 30/60/90 days to fix this thing I noticed before I post it publically" it was "okay, well, take your time I guess, I just hope North Korea doesn't notice the same issue I noticed, which is especially likely because they have far more resources than I or my small team does to point at discovering this kind of thing, while you're dragging your feet fixing it."
The reason people want shorter disclosure windows is because the shorter the window is the faster things get fixed for the average users and smaller companies.
Xint.ioâs homepage says âXint finds real vulnerabilities, and proves every finding with a working exploit.â, which for defenders is probably helpful, since automated scanning tools have a LOT of noise.
Marketing, baby!
It's the same small handful of firms pushing these same AI-generated spam attention grabbing vulnerabilities, quirky name and domain on top for maximum fear mongering.
Why do you consider it fear mongering? The exploit is clearly real, and TFA even has a big disclaimer about how it's difficult to exploit "in the wild".
So who's the judge of where to draw the line between exploits that are relevant to publish, and which are "spam". You?
Or do you just think that companies shouldn't be allowed to show the work they're doing for marketing?
I could be wrong (my sed is rusty) but I don't think that sed command will help if http2 is enabled at the http block level, it just disables it at the server level which will only take effect if not overridden elsewhere.
Everyone's worried about security right now, even non-tech people, but hasn't half the nation's personal data been leaked twice a month for the last decade or two?
I know times are changing, it's just curious that nobody seemed to care until now.
I think all airplanes in the entire country being grounded, banks not able to process transfers, or grocery store freezers being disabled hit differently than data leakingâŚ
The freezers were, at least to some extent. One could argue that they should never have been on the internet and so the vulns wouldn't matter ... but a hidden, insecure admin password based only on a static key and today's date is definitely a vuln in my opinion.
I disagree. They may not be because of exploited vulns, but the rate updates and patches are happening there is a very long tail of instability after patching due to extremely fast turn arounds.
I'm not sure yet, but my guess is that the FAA grounding was not really related to vuln patches. I thought there was a copper to fiber network upgrade gone wrong involved there. It seems like the bigger problem is aging infra that needs replacement.
Yes. Thatâs the confidentiality part of the CIA triad. But now you have a whole lot of things suddenly impacting availability in really visible ways, so yes, a whole different cross-section of people are really perking up.
I wonder when all of this die down. I assume at some tipping point most software will be written/patched with LLMs and so hopefully (fingers crossed) more resistent to these bugs.
Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
It's possible that AI is going to be better at finding and exploiting systems than at securing them. It's not really an even playing field. On the one hand you have to somehow ensure that you've patched everything when there's no definite list of everything, the other side just needs to find one overlooked way in.
At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
It's not possible. It generally requires more work to find an exploit than to patch the (known) exploit.
Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
How do you know when the "exploit finding" pass is done? A hacker group would happily burn a month or more of tokens trying to break a critical piece of software.
The better LLMs get at finding vulnerabilities, the more obvious it will be to bolt them into software development processes to find the bugs prior to release.
With enterprise frontier LLM models, there is no longer an âother side.â Everyone is just using what they can rent from a few big suppliers. You can too.
But are LLM patches exhaustive? Cant a new model find even more obscure bugs, including bugs introduced by the old LLM patch? Is it really realistic that we such complex projects as some of these critical dependencies are can be truly 100% bug free?
If the software is too complex to understand, maybe it should be made simpler. Nginx does a million things currently but what most people need is something that parses http1 strings and sends back a file or calls a second program based on the path. Every component can be thought of in this way, with the most complexity perhaps in being sure the interfaces don't do something unexpected
We dont need to be bug free, so much as the juice needs to be not worth squeeze. I think i am also making an assumption that we "level" in terms of frontier model progress at some point, and we have months or years between upgrades.
>so much as the juice needs to be not worth squeeze
I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
Reading news like this makes me remember the ship hacking scene from GITS Innocence. In the end there will be AI attacking vs AI defending. Humans will be too slow to participate. Defender AIs should be coming soon - maybe some sort of proactive memory and usage anomaly pattern monitoring AI systems?
Nobody seems to notify major distros before publishing their latest vulnerability anymore. Coordinated patches are a thing of the past, now we just hope that somebody at Red Hat and/or Canonical is reading HN today.
Who is sane enough in 2026 to expose distro default services to the Internet?
I wouldn't expect any distro maintainers to react in hours. Distros are too complex to dump updates in rush.
Start to use Docker/Podman/systemd-nspawn.. and isolate public services to necessary needs (static binaries, drop capabilities, default drop rule firewall between containers, r/o storage, access logs bind mounted as rw file, memory temp dir with noexec). Container images are patched the quickest.
And there were a couple of other http2 CVEs in nginx before the Apache one.
Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
Obviously... the default configuration also doesn't enable TLS, which is a de-facto prerequisite for HTTP 2. How many websites do you (or your "AI gods") think run nginx in its default configuration?
The point is that it must have been deliberately enabled. TLS will generally be enabled out of necessity, whereas the chances for HTTP/2 are much lower, and your NGINX configuration will explicitly indicate if itâs enabled. If you donât see âhttp2â in your configuration, then you know you are fine.
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches https://github.com/kubernetes/ingress-NGINX
As much as everyone is getting allergic to Rust migrations that sometimes make little sense, keeping network-facing daemons in C is just asking for it at this point. It's time to start seriously migrating to web servers like ferron.sh or https://xinproxy.com or whatnot.
I now run all my network facing C code via fil-c. I have limited needs (ssh, static file server), but for those it works very well. I personally am more in favour of making existing battle tested C safer via fil-c or formal verification, than I am of rewriting the world in rust.
New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
In theory, scpptool provides another solution with different performance, compatibility and portability trade-offs. Unlike Fil-C, it is not "fanatically compatible" with existing C code. But in theory, with maybe some minor adjustments to the code (and maybe less-minor adjustments to the build system), you can auto-translate your existing C code base to an essentially memory-safe subset of C++ at build-time [1].
While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.
In many cases the performance is not that important, for example, SSH is often used to transmit mere kilobytes of text per second and I would prefer a safe SSH to a faster unsafe SSH. Or take mDNS, it servers several requests per minute max.
This sounds like a fantastic solution, especially for non-high-volume stuff. Would you consider writing down your setup and experiences? I'm sure it'll be a hit on this site!
I've reached the same conclusion myself. Many of the recent exploits seem like edge-cases that the author acknowledged 10-20 years ago but didn't predict that tools would eventually be able to chain together 100k network requests in order to trigger a buffer overflow.
Nobody is saying that it needs to be rushed. Cloudflare started moving towards Rust more than a decade ago and (shortly after heartbleed happened) and progressively rolled bits out over the course of years. I'm sure there were problems here and there but it's hardly been a drag on them.
And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception weâve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
Given the volume and depth of testing possible with AI these days I'd say the decades of testing isn't that expensive to reproduce, especially if it's an OSS project with a public bug tracker.
But rust is not a silver bullet. Rust compiler still has memory corruption bugs. Rust is only memory safe by design not by implementation. Borrow checker allows unsafe code without using unsafe keyword.
> Borrow checker allows unsafe code without using unsafe keyword.
is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
this is just a perfection fallacy. silver bullets donât need to be perfect in every single way to be silver bullets. and rust is as much of one as anything.
It is true that mileage is a good proxy for finding bugs, but this is isomorphic to the fundamental differences between testing v. verification. You can prove absence of classes of bugs with verification. Obviously there can be other logic bugs in newer software. No question. The issue is the memory-corruption bugs in socket-facing daemons can have catastrophic consequences, especially in light of chaining kernel vulnerabilities which defeats the privilege drop that nginx does.
So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
A more holistic risk-analysis would also need to consider the fact that a high-performance webserver written in Rust may also have memory safety issue due to use of "unsafe" features or unsafe dependencies, may be affected by memory safety issues in the standard library, may also have catastrophic logic errors, may be more affect by supply chain risks due to the use of a language-level package manager, and may have a larger patch gap.
Which is like complaining that Python is unsafe because it is written in C.
That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
Not in the slightest. It's like pointing out that a Python script calling out to binaries written in C doesn't have the memory safety of Python's GC. Which is obviously the case, and if you're a serious developer it's a failure point you're very mindful of.
On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.
Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.
Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
What specifically distinguishes the empirical research presented in Googleâs blog posts, for example the one linked below, from your definition of research?
Research is not just making some plots and posting them on the internet. A proper scientific studies would need to be done much more carefully, if you want to have any chance to get meaningful results, because there is a lot which can go wrong at any level. This is something learned people learned the hard way in empirical sciences. In particular, in this setting I do not see how one could ever properly control for confounding factors. Such a study should also be peer reviewed. Then repeated by others. Also if there are commercial interests mixed-up this needs to be questioned carefully. Finally, I just note the data presented seems mostly about C++ not C, so is not clear how this applies to C. Not separating this would also already be an obvious methodological problem. This is also connected to the problem that the underlying data is not made available, and not properly described in sufficient detail. Etc. (Not a full list, just to make it clear this is not even remotely science).
A more holistic risk-analysis would also need to consider the fact that wearing seatbelts in motor vehicles still leaves drivers vulnerable to drive-by shootings, may be affected by massive boulders crashing on top of the vehicle, driving off a bridge, or gas explosions, and costs more to produce.
In theory, that could be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception weâve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
If we're suggesting alternative web servers, I have had a pleasant experience with caddyserver (Go). It's been around 10+ years, has essentially transparent TLS certificate provisioning, and its config syntax is easy to understand. Even using AI to help vet configs, it's readable. It's packaged everywhere and has lots of plugins for things, though I only need the base package for my frontend proxy usecase.
See Morris worm, and when it took place, afterwards check Multics security assement done by the DoD versus UNIX.
Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.
Yet, some folks are more attached to C than the inventors themselves.
As someone that was a developer for a long time what is the problem with C? How does going to rust mitigate this? Why is having the stuff in C a security challenge?
It's very easy to accidentally write buffer overflows, use-after-frees or other memory-safety bugs in C. Modern tooling can probably catch some of these better than in the days of yore but the fundamentals of the language are still the same.
If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.
But C in itself has no guardrails against something like
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.
SQLite is probably the most thoroughly tested and carefully developed C library in history and they still found a handful of memory safety bugs every year.
Rust is memory safe at the cost of being harder to write, but with AI, it's actually really easy to write it because AI's loove strict compilers. So now that AIs can just find 0-days before things are even released (so I'm told), it makes a lot of sense to do bun[1]-style rewrites to Rust with AI.
If a bug is found before its released... Then that's not a 0-day?
The name refers to the number of days since public release. "Exploitable 0 days since release". If you found a bug before release, then that's... Just a bug picked it up in normal development lifecycle.
I read 0 days like "Exploitable 0 days since _discovery_". Also, it's only a bug in the normal lifecycle if the discoverer tells the developer making it. If they discover it and then have hours and days to perfect a program that exploits it, then starts running immediately when it is deployed or released, that's a new kind of "zero day". Makes you want to start differentiating what "0 days" mean, though.
The ai told the hacker. The hacker discovered it with his AI before the developer released by examining his git branch.
I think that's the fundamental disconnect in the conversation here. You think I'm talking about the dev because it's not released yet but just because it isn't tagged and released doesn't mean it's not done in the open where hackers can see it and their AI can scan it.
This looks like someone prompted "rewrite nginx in rust, make no mistakes" in an afternoon and is now distributing it as a closed-source binary. Might as well just do it yourself.
xin seems to be closed source and ai generated. Before even worrying about exploits, how is someone supposed to ascertain that xin's implementation is correct? Sure Rust might not have any memory safety issues, but other logic vulnerabilities may exist. 'Claude make no mistakes?'
Caddy, Traefik, or even Envoy may be better choices.
I'd hate to interrupt all the Rust fans sucking each other off again, but no one has yet mentioned the RCE relies on ASLR being disabled, which is basically non-existent nowadays. Another nothingburger CVE with a silly name and it's only Wednesday.
Haven't you heard? These vulnerabilities are just AI marketing stunts. And also, the Bun rewrite to rust was a marketing stunt, those things don't work. There's no way Bun is running on Rust from a couple month AI powered rewrite, that would shatter my preconceived worldview about what the future looks like. /s
On the other hand I've never been so proactive about maintaining security on my homelab! I did a full audit of my self-hosted services with claude code, strengthened the isolation of all my docker services, I'm running coding agents in sandboxed microVMs now, my firewall settings are better, I set up separate VLAN for IoT, I get security notifications on my dependencies, etc. etc.
A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
Yeah I'm having to do attack surface review and threat modeling like I'm a full enterprise now...
Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
HTTP is used for a lot more things than web pages, and most non-browser clients in production donât support HTTP/3. The current LTS Java version (25) for example doesnât support it.
Because any HTTP spec after 1.1 is incredibly complicated. You can implement HTTP/1 in half a day, and HTTP/1.1 in a day. But HTTP/2 will probably take you a month and HTTP/3 might take something like 6 months.
It boggles my mind that people are patting themselves on the back for staying on old tech stacks and using a discovered vulnerability as further justification of their decision to stay on h1.1. The response is to patch and move on if your environment requires h2; many large orgs these days use microservices which use h2 (grpc) as its network.
There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).
If you donât need these features and just serve vanilla text webpages thatâs also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing whatâs best for your environment.
Do you realize that no one writes toy HTTP implementations from scratch? I get that you might have a "suckless" mindset but that's just not how the real world works.
That's not the point. The point is that the simpler the protocol is to implement, the simpler it is to validate and harden the implementations. A HTTP/1.1 implementation can be 10x to 30x smaller than an equivalently well-rounded HTTP/3 implementation.
It's debuggable. I can wireshark it and understand everything. I can even speak to HTTP servers directly with a telnet client and fetch pages. I can write microcontroller code in C that issues REST API requests. Pure TCP, nice to work with, no multiplexed UDP hex craps.
Oh no, how annoying, yes I can see how that could be the case if Discourse is running its own copy of nginx with its own configuration inside its container. That's unfortunate - I don't know much about Discourse but I hope that had a good reason to bundle in nginx, seems unnecessary at first glance. I really hope a Discourse developer gets to the issue soon and releases an update, that would be the fastest and easiest way!
I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Is this the reason why those banks were down the other day? Did they get advanced knowledge so they could patch their instances? Kind of questionable that the AI labs are creating a two-tiered system for system security if so.
I wonder how much of this exploitation will push the industry towards closed-source. Previously the advantage of OSS was trust and collaborative improvement, but that is getting eroded quite heavily now by the barrage of attacks and insidious discovery.
The first step was closing contributions down to known contributors and maintainers. Maybe thereâll be a similar step where only a select group will be able to access â and attest to â the code?
Almost like âRestricted Access Sourceâ or âWithheld Sourceâ
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down.
maybe not an RCE but a DOS in the same http/tls code.
well it's a good time to pitch tech debt and infrastructure cleanup. If you aren't making use of least-privilege permissions, service-mesh, netpols etc. this is the time to add all of that.
Because clearly the patching treadmill is not sustainable, you will need a a second line of defense and reduce the blast radius of every individual vulnerability.
One, as the RCEs are coming out, admins are on edge and paying more attention, so more likely to catch the annoucement on their platform. "Oh, that's on PHP, we don't use this. Oh, wait, another one?"
Two, all of these hitting at once, while they may expand the surface for attackers, it also spreads their attention. They only have so much time and bandwidth. When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
So, in the sense, it can actually slow down the actual potential impact of the revelations.
We can only hope this gives users time to get their systems up to date before they come knocking.
Ops is not fun these days ⌠used to pinned versions and an auditing approval process with SecDevOps ⌠how are you mostly automate the process in a you build it you run it environment ?
Yeah, if it matters for your workload, then don't do this :) And obviously test outside of production first if you're doing anything important. But if all is happy with http/1.1 then this is likely much easier than making a config change or waiting for a patch that doesn't seem to exist yet for some downstream consumers of nginx (like apisix for example).
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
This fixes the integer underflow for CVE-2026-123007, and it's true that the underflow can happen in any language, notwithstanding whether it's UB or not. The issue is what happens as a consequence of this underflow, in this case: out of bounds copy, which is specific to C.
Some languages make it significantly easier than C to just forbid integer overflow/underflow entirely.
Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
Sure, I usually use `overflow-checks = true` in Rust for this reason. However, it is still true that the overflow is not what caused the issue here. In Rust it would've paniced on the out of bounds copy, even with `overflow-checks = false`.
It is not specific to C. You could have bounds checking also in C, and you can have non-checked indexing also in many other languages, including Rust. I would agree though that many other language make it harder to not have the bounds checking, but the bigger question is why this is not done C (and the answer that this is very hard to do is false).
C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it the good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.
Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
It wasn't necessary, at least for people who kept hearing that there is no need for languages like Go or Rust, since it is possible to write memory safe programs in C, you just have to not make any bugs.
Quite the opposite, it looks like there has been no responsible disclosure here. No update available for anyone running something based on Debian (Ubuntu etc.), only if you go and fetch binaries from the source and do the system integration yourself (easy in this case, but in general that's the point of a distro). For php I could just update at some point that day
Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
HTTP1.1 has a fundamental flaw related to request smuggling. Fixing specific variants of that problem is a game of whack a mole, as issues arise most often from larger server composition, not individual parts. Fixing the protocol flaw requires a major version increment and a phase out of the old one. Today the hesitation to do that gives us a mitigation option, but in general an http downgrade is a bad thing.
There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
Both this vuln and the java4shell vuln from yesterday have very narrow attack vectors. It's not impossible for a human to figure out, but it does feel like an AI was used to discover both of these vulns.
That's to say that this is our new normal. Something that would have taken months or years to figure out can now take an AI hours if not minutes if you have enough compute.
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
I think the problem is that while what you say is true, AI is also capable of generating the special conditions required. And if it canât do so reliably, it can sit there hammering the server for as long as it takes until the conditions are met - as an example hitting a bunch of load balanced servers until 2 requests hit the same workhorse sequentially.
It is completely realistic that we start seeing a lot of bugs a human canât (reasonably) exploit, but AI can.
I was talking with someone about buffer overflows due to HTTP requests this morning. I think I said something like "but we don't need to worry about that, just put nginx in front of it." Sigh. Off to update some VMs. Again.
Yeah I'm not quite sure what is going on with Debian, I would have hoped that this vulnerability would have been disclosed in collaboration with Linux distros to get a patch out?
Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
Based on <https://news.ycombinator.com/item?id=50547678>, a bunch of people reported this to Nginx in parallel. Wouldn't surprise me if they went "Igor, you wont believe, this rce is the same as the five previous I reviewed this week! Let's ship your patches before malicious attacker discovers in parallel also"
"Both vulnerabilities are memory-safety issues in how NGINX handles HTTP/2 traffic."
HTTP/2 again. First Apache, now NGINX
I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1
I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
Even the US federal government (CISA and NSA) have been strongly advising companies to transition to memory safe languages for a couple of years now. https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre... (one of the goals of such advisories is it acts as a signal to the industry that in several years this will become a mandatory requirement around purchasing decisions)
We've literally decades of evidence that we cannot write software securely in languages that aren't memory safe, for all sorts of reasons (and it's wrong to look at it as a developer competence issue, even though sometimes that's going to be the case, even extremely experienced and highly competent engineers write memory safety bugs in critical code we rely on day in day out.). Not leaving yourself open to such problems is an entirely logical conclusion. We know a large percentage of vulnerabilities in software are caused by it[1]
Pick a language that doesn't have these risk factors, or pick an enforceable subset of one that makes such impossible.
"Insanity is doing the same thing over and over and expecting different results."
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
It's now been battle-tested in production, and also supports websockets, etc.
(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
This doesnât seem very HTTP/2 specific, though, and it seems more like the conclusion should be âdonât use C for things which matterâ. Itâs not like HTTP pipelining and keepalives didnât have plenty of edge cases to worry about.
True, but offering three different protocols with different code paths for the same thing is a lot more attack surface. If you start factoring in different compression methods things look even worse.
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.
Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.
This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Well, you can't because the LLMs are all nerfed and censored w.r.t cybersecurity.
I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
The refusals do not make that big of a difference! Holy cope. No you still need to study and become an expert. Ill give you access to unlocked fable and you still wont be able to do much with it if you dont have a ton of experience hunting bugs.
Keep telling yourself that! The refusals arent that big of a deal. I have access to all the models without refusals. Your average joe still would not be able to hack something with it.
So much cope on this site. Llms havent replaced the requirement of studying and becoming an expert.
Isreael btw everyone at Wiz is a veteran of Unit 8200.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
Disabling http2 can be a big deal. My service is primarily made up of gRPC APIs. Not an option for me to disable http2. Luckily I'm not using nginx at the moment - but I easily could have been.
If you're using grpc, http2, etc you're so much better off using Envoy proxy
You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.
The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.
I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
Envoy probably has the advantage that itâs only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
You can implement custom http filters for envoy or use this for general web serving: https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/h...
The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.
Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.
Envoy is only attractive at a higher scale.
Does Envoy support things like rate limiting?
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]
In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.
There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.
[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...
[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...
[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...
[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee. I know Google spends a lot of manpower on making envoy secure.
I wonder if being prepared to switch the reverse proxy you use in whim is a good strategy in this climate.
A while ago I started releasing our stuff with dual stacks (originally nginx and haproxy, more recently added traefik to the list).
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.
I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.
In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
I am using Envoy :)
If you're using httpd just fyi - there was a recent RCE vuln for apache that was also http2 specific.
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.
Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.
There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
At least for Debian, http2 is disabled by default, and so not affected.
You would have had to enable http2 and have a buggy Linux kernel to be vulnerable.
You can also add nginx.org repos and install the updated versions from there. https://nginx.org/en/linux_packages.html
Another reason not to couple your software versioning to Ubuntu and Debians' package managers' schedule
Their package manager and security response is the only reason to use (put up with?) their OS.
If I wanted to build my own software from source every time I'd be running Solaris like a virile man from 2004.
NixOS is probably the best modern option for those who crave building the whole world from source.
yep, already deployed fixes to our fleet (NixOS for the win)
Or just use it in a podman quadlet?
stable-alpine-slim docker iamge is only 14MB?
Alpine recently had https://sshhole.com/
Not saying that thie image is affected (I don't know), but small footprint does not guarantee security either. It can hit systems of any complexity these days.
...or you could use Gentoo.
Keep the flame alive!
Embrace true chaos and let an LLM do the config file merges for you in Portage.
LLMs love gentoo they go hog wild.
Which may be good or bad.
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
Debian has it in unstable now. https://tracker.debian.org/news/1825662/accepted-nginx-1306-...
Good to see
It makes more sense to install services like nginx via the official containers. In the modern world, the OS is for base services only.
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
You are aware of the massive vulnerabilities coming out across software eco-systems recently right?
Ubuntu and other distros have not kept up.
Podman container auto-updates to the latest upstream. It is a minimal surface area given alpine images. Security first containment.
In this day and age with AI attackers and zero-days it is absolutely daft to still treat servers as the old-school pets model.
> Podman container auto-updates to the latest upstream
Doesn't this make the system more susceptible to supply chain attacks?
Agreed. At a previous job all the important stuff came from repos that always shipped latest.
This looks like it was patched in Ubuntu on the 12th?
https://ubuntu.com/security/notices/USN-9090-1
This Ubuntu security notice is about an Apache vulnerability, not nginx.
> Several security issues were fixed in Apache HTTP Server.
Um, what does this have to do with NGINX?
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
I told me agent to do that - similar situation.
> me agent
Did you talk to it like a pirate?
There are large areas of the English speaking world where this is common speech.
New caveman version just dropped! /Pirate
Definitely, but unfortunately I use Debian for my servers so.
Oof.
One idea that might work for you is client certificates if those are only internal APIs. I don't know about you, but I have a lot more confidence in openssl than the people who use openssl.
I turned off http2 a few years ago https://news.ycombinator.com/item?id=41486452 and absolutely did not regret it.
I do regret every experience I've had with gRPC though. I feel for you.
> this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week
And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.
It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
How's that alignment going...
As much as I like open weight models.. what alignment?
Thereâs no good solution, itâs a minefield and the only choices are what youâre willing to lose to maybe make it out of it
This is the reason why the oss project i work for is not publishing commits anymore outside of releases but people are freaking out over this.
Cant win.
Seriously reconsidering if open source has a future nowadays
I'm sure things will stabilise in the near future.
That said, I just moved from a Linux based job to a Windows based job and it's been nice not having to be panicked about having update most of the technologies that my last job used. As an aside I did try in the past to make sure we were always updating dependencies but my manager was bothered by the amount of work that created. I'm watching that team now take forever to deal with these issues because they're too far behind to just version number and rebuild.
The future is in closed source ecosystems on a modified Harvard architecture
WTF but I get the point
What happens when somebody tries to contribute based on the last-published commit? Do you merge/rebase their contributions for them?
Havenât figured out how to deal with contributions going forwardâŚ
The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesnât break in interesting ways is still pretty hard.
Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
How about a strict anti LLM use policy for contributors? It adds what's basically a "proof of human work" challenge to opening new PRs, and effort required to review human PRs usually roughly scales in proportion to the amount of effort required to make human PRs. And human contributors who take time to make a PR manually are probably more likely to become longer term community members and not just drive-by contributors
It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
We do have a LLM disclosure policy.
Iâm not sure how we could enforce an 0 LLM policy in practice thoughâŚ
You only need to enforce it enough to fulfill its goals.
https://xkcd.com/810/
Amounts to âwonât be getting contributionsâ which is maybe not that bad at this point on the timeline
There's plenty of people who contribute to non-AI open source projects, you just don't get the lazy people who don't want to be a serious part of the project or the people who don't know how to program anymore
I mean, win or not, it has nothing to do with some people freaking out, and that certainly have nothing to do with whether open source have a future.
Not really sure what youâre trying to say; my main point is that OSS is going to change drastically in the next couple years
> Seriously reconsidering if open source has a future nowadays
*In C or C++.
Until we don't replace UNIX clones, Khronos APIs, LLVM or GCC with something else.
C and C++ will stay around for a very long time, even with AI powered systems.
If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Rust is on its way. Vulkan is no GL, and Gallium it's something else over classical MESA under SGI/Sun.
Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
Rust being here matters,
https://www.khronos.org
https://vulkan.org/events/vulkanised-2027#program
The only thing closer to it is WSGL.
Also there are much more operating systems than those examples.
rust isn't invulnerable.
A tank isn't invulnerable either, but of someone starts shooting at me I'd rather be in a tank than on a 50 year old bicycle.
Fun fact, Russian soldiers ride on top of their tanks and APCs because its safer than being inside of one during an assault. Doesn't mean you are wrong, but there will always be trade-offs. There is no such thing as perfectly secure.
Not exactly true. BMPs are death traps, sure, and sitting on top of those instead of in them is rational given the risk of burning to death inside of one or being obliterated by a mine splitting the hull in half.
But the only reason to sit outside a tank rather than in one is because riding on a vehicle is safer than walking through an open field. Tank crews are not sitting outside of their tanks, and they don't have seats inside for passengers.
Tank desant is practiced because there isnât room in the tank to fit extra people. Itâs very much not safer outside of a tank than inside! Itâs actually the worst of both worlds, because you donât have the armor protection but are highly visible. The only advantage it gives over infantry assault on foot is mobility. Which is why itâs generally only practiced when there arenât sufficient APCs/IFVs available
Fun fact, these specific types of problems aren't possible on a JVM. Yet somehow Java isn't the right language to write a webserver in. These problems are possible in Rust and C/C++. At least its "hard" to have this type of problem in Rust.
Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
These commits weren't visible until today. Yesterday, the last public commit was Nov 9.
How is that done with git?
Just don't push upstream?
How bad are these CVEs? Could any of these be related to some of the recent outages (not claiming, asking) in the last week, coinciding with when this bug was fixed in the repo? Monitoring such repos seems a given for nation states.
RCE is about at the top of the stack for severity.
While that is true, you should also account for exploitability. These exploits are demonstrated in lab environments which are simple and controlled. Random legit traffic will interfere with the exploit logic, so these are very hard to get in practice. Not impossible, but it's also not immediately broken.
According to https://www.rapid7.com/blog/post/ve-cve-2026-123007-critical... it requires 128GB to be transferred in a single connection. "a network slower than approximately 300 Mbit/s will take over the one hour connection limit threshold and will not be able to succeed"
Not all of the bugs disclosed today in nginx require that much traffic to trigger, the reverse proxy HTTP/2 bug found by Xint needs just KBs of traffic.
Thankfully most of these bugs coming out seem very difficult to exploit (likely why they took so long to find). They usually become dangerous when you can build an entire chain of bugs to exploit.
But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..
Great context. Thank you for the add.
put up some bitcoin behind an nginx server and tell hackernews to steal it, see how long it takes
:) If the battering ram is 128gb big, it'll be a long time unless someone has reversed those changes in Nginx to get at the exploit. After that, yeah, it's no contest
This was also true with php2shell but AI got there in the end.
And nginx is so widely used many people will be running it and not really know that. Itâs ubiquitous reliable invisible infra for many issues. Like preventing the internet from getting to your RCE riddled application servers, terminating SSL, free software LB, etc. if you have cloud native from the beginning you might not know where AWS uses it but otherwise you have it. Cloud flare I think uses a Rust rewrite.
Source: https://github.com/nginx/nginx/commit/b89c4100bf6762a45e4a64...
These days even the "disable X" is enough to get everybody looking in the right direction though. i.e. such hints are (given enough tokens) already RCEs
In my experience, asking an LLM to "Do a security audit" doesn't get you much more than an overview of the most obvious config issues while "look for bugs specifically related to HTTP/2" gives much better results.
HTTP/2 is usually the source of most of these bugs in webservers of all kinds
Yes, fixing open source software is at most an obfuscated announcement.
Oh everybody worth worrying about is monitoring this on every major repo already.
The patch gap.
Small consolation but HTTP/2 is not enabled by default in Nginx and "only" 12% of servers run it (still huge at Internet scale) https://w3techs.com/technologies/segmentation/ce-http2/web_s...
Hey so, does anyone know of a good tutorial on getting in to woodwork?
As a matter of fact, I know three such books (all free): https://lostartpress.com/products/the-anarchist-s-trilogy
Well, actually four: https://lostartpress.com/products/the-stick-chair-book
I took a quick look at The Anarchist's Tool Chest and noticed in Chapter 20 a short vignette about an expert woodworker named Troy that seemed... oddly familiar.
According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.
"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.
So no matter what Troy did, the prices for work (good or otherwise) declined. Ikeaâs well-designed kitchens and Value Cityâs cheap junk made it impossible to survive and live well.
So Troy struggled with his business. And thatâs nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time we talked, Troy was working for his wifeâs successful tax business.
Who is to blame for this? We are.
We have become a culture that is obsessed with price more than any other attribute of the things we buy. It doesnât matter if the item is ugly, poorly made or constructed of materials that cannot be recycled. All that matters is if the price is low enough.
Because the price of our household objects has hit rock bottom, if an item breaks or starts to look dated, we can throw it away and buy some- thing else. For the first time in human history, manufactured furniture is shockingly inexpensive.
So itâs no wonder that artisans are exiting the craft. Itâs difficult to compete against furniture that costs less than what you pay for your raw materials.
Lesson: People cannot imitate machines and defeat them."
The lesson is that most people aren't discerning consumers in multiple fields.
Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to naturally get a critical mass of people to care about that quality curve. You have to sell them on it.
Troy struggled because he failed at marketing, not production.
Or even single fields. I've purchased my parents the "nicer" version of things and they inevitably end up back on some crappier version. Japanese knives? Nope, still use the ones that barely cut. Ubiquiti wifi system that covered their whole house? Nope, dad unplugged something, got frustrated, and the cable company came over and put them back on the old crappy system. Nice speakers? Nope, they listen to music on their crappy TV speakers.
I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them improperly and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
On the other hand, there's an accessibility to products of higher median quality to more people than would have existed even only five generations ago.
cf. John Henry [0]
https://en.wikipedia.org/wiki/John_Henry_(folklore)
I felt your post was off topic. But maybe it was on purpose because we are getting bored of so many vulnerabilities being discovered by AI security tools. So your post is important because of that.
I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
Becoming a carpenter, and sentiments of "I'm going to live in the forest" or "I'm going to be a farmer" are usually expressions of fatigue with technology, and a desire to be free of their problems; thus, an expressed urge to pursue a career where technology is not a major component. It can also sometimes carry a desire to have one's labor output be tangible and physical (i.e. a worked wood object) rather than abstract, as so much programming/operations/etc. work often is.
The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Piffle.
Preposterous, even.
That said, my RHCE CISSP self is going to go outside and spend some time with my flock of chickens and guinea fowl.
While Claude writes Ansible to update the fleet to use nginx's deb repo.
Haha. The image of shouting voice instructions to Claude while trying to herd a flock of chickens to their coop is the dream, very tony stark'esque only with chickens!
wood, unlike logic and machine, is firm, yet yielding
This is a beautifully autistic comment.
Well, I am usually the person in the room who does not get a joke. I am so scared of get-togethers that I actively try to avoid it. Also, they ask me to dance. I am terrible at that. I am not autistic but donât get the drift of many jokes. I am human and not a bot. Guess thatâs what a bot would say. But still I am human.
I didnât get the joke myself either, so thanks for asking. I first thought âwoodworkingâ was used as a synonym for hacking and clicked on the links with the book recommendations just to see theyâre about actual woodworking.
Not sure why others downvoted you, but you have my upvote on it.
Botistic more like it
Just take a second to check out the account if you're going to say stuff like this. It's not just you either, so this goes for everyone making these comments.
The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.
> Hey so, does anyone know of a good tutorial on getting in to woodwork?
Some people really don't get these type of things.
I think most LLMs would have gotten that woodworking joke.
This feels like an AI bot reply
As programmers, we do see ourselves as Jesus, yes.
Ha, sadly there's a huge long line of developers who pivoted this way already. Plus now that you can fire up an MCP for a CnC machine, that art will be "ruined" too!
I hear electrician and plumbing schools are overflowing with newly enrolled students, too
I would be suspicious of any plumbing school that is overflowing.
AI is burning down the internet. This is insane.
Are we going to discover in a year that every server online this month had a rootkit installed? Thereâs got to be thousands who are too small time to wipe and reinstall.
AI is securing the internet. This is great. You should assume professional hacking gangs and state actors discovered (at least some of) these vulnerabilities months and years ago.
Both are true. The problem is security is a much harder job than attacking.
Worse patching adds instability into a system, especially when you don't have much time to test.
Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.
Things are going to get much worse before they get better.
Yes, things will get worse before they get better. Attackers have the advantage initially and get to pick a lot of low-hanging fruit, but the advantage will shift to defenders as code becomes more heavily tested and verified.
One hopes, but the models are also getting smarter and deeper too. I don't know if anyone really has a clear vision of when the future plateau state is where equilibrium is achieved on this.
I find this kind unreasonable: software has finite bugs, no matter how smart your attackers are. Current CVEs are pretty subtle, and I see no reason to think that they're actually low-hanging fruit.
I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.
Software has finite bugs, but there is infinite software, especially with "vibecoding", everyone, every organization can vibecode a "nginx" for in house using.
Well that's up to them.
>but there is infinite software
And the attacker does not have infinite tokens..
Close to infinite that one may not care about it anymore, especially when they are backed by State.
This is the correct view. Look at blockchains.
Question is how big is that finite number? One a month for the next 40 years helps no one.
>Worse patching adds instability into a system, especially when you don't have much time to test.
Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
I once did work for a company that hadn't updated or patched anything in over 4 years. Systems with 1500+ day uptimes. Totally insane.
Attack the attackers.
Except, human in the loop in control - blue team is not defending red team fast enough that most the CVSS v4.0 Critical CVEs aren't easily patched in coordinated fashion. Many times, almost impractical or impossible to patch at all, with decades of intertwined dependency tree and their ecosystem.
Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
AI provides the capabilities to secure the internet, maybe, but the internet is secured when site owners choose to secure it.
That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.
Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
Yes but the barrier is infinitely lower. If you were used to running servers just for fun, that kind of threat was probably not too much of a concern. Now it very much is because it can be exploited at a much larger scale.
Yep. In recent discussion this came up, and my analogy was that LLM will cause a similar effect to widesread proliferation of TLS. It was easy to snitch on people before TLS had been deployed everywhere. Not so much after that.
> > AI is burning down the internet.
> AI is securing the internet.
No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.
But in this day and age, whatever is happening, the better question is:
Is it good for me and, beyond me, for the world in general?
AI is also making it much easier to attack, likely more than it's helping secure things
it's a timing thing - eventually we'll reach an equilibirium
it was previously expensive to attack, so people spent less money on defense.
It's cheaper to attack, and so people will increase their budget on defense. But it's a lagging cause & effect situation
You're a nanometer away from understanding why labs have powerhouse internal models that they only allow partners to use.
Even Kimi K3 was only shipped as a lobotomized version of it.
the idea that internal models can protect the Internet like this is so dumb IMHO.
A huge fraction of devices and servers are on old versions, not actively updated or managed in any way.
Then the best option for everyone's security is to hack into them and kill them.
If you're not going to take care of it you've created a public nuisance.
This internal models stuff remains marketing BS afa I am concerned. Smart enough to be a junior engineer and steered toward the right tools and behaviors within a context is more than enough. Places with "super models" do nothing to demonstrate they wouldn't have done better to use the resources on relative brute force within the security domain. They have no sales pitch for good results that just needed Legos and cash.
Man, is it aggravating to read fantasies like this, as someone who's trying to juggle a full-time job and hosted public services for other people at the same time.
There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.
The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".
The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."
We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.
Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.
When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
I wonder whether part of the solution is not using the same software as big tech. For them rapid development is worth extra maintenance. Things like the efficiencies of http2 over 1.1 actually matter. Maybe the rest of us should be using BSDs or different Linux distros (Alpine? Void? Slackware?), different web servers, turn off features we do not need (e.g. not disable http2, or at least not allow both 1.1 and 2 as that is clearly extra attack surface).
My sites are now sat behind my own custom reverse proxy. It's missing tons of features, which I guess I can now call a security benefit, and it pretends to be nginx just because I thought that would be quite funny. I've debated open-sourcing it but I a) don't want to maintain a project right now, and b) security-by-obscurity is back in action.
> When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
https://reticulum.network
Appreciate your take. Question in my mind is if the less-commercial internet will have the resources to defend against attacks made easier and easier by AI.
Usenet.
It's brilliant isn't it?
The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.
I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Well I didn't get into any trouble with the professional hacker gangs. The script kiddies on the other hand...
There's no honor among the thieves if everyone can be a thief.
One would think that the NGINX team would be banging on their code with AI agents trying to attack it for preemptive security.
"Hacker gangs"
What term do you prefer?
Organized Cybercrime?
Digimafia?
e2p3yHizAungrCeKum7ifrssu32lnzEnyRoGdcAo23m2dNapvoGmymid.onion?
Kim Jong Un's megayacht fund
Just like in the movie "Hackers".
Hey, that's a piece of art!
Poor security practices are burning down the internet not AI. Many people warned and got ignored, by developing a memory safe language, introduce large scale fuzzing, utilize verification tools and other things.
It's like blaming people for having wooden doors when robots with chainsaws start appearing in the street.
There's definitely an issue of scale, but a wooden door at least lives up to its own "contract". There's no bugs in a wooden door, everyone can see plainly it doesn't withstand a chainsaw.
A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.
It's been a culture decades in the making.
The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.
Now we are starting to understand it more.
When 85 percent of doors were installed before metal doors were available.
> robots with chainsaws start appearing in the street
Please stop willing things into existence.
What about ASLR and DEP? Guess they are useless?
Itâs for the best, IMO. We must now design every system on the assumption that every vulnerability will eventually be exploited and every bug will be found. Relying on security by obscurity or following poor security practices will be costly! In the long run, I believe this will improve software quality.
For eight years, MongoDB shipped a default configuration that left it opened to the internet with no password.
Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
web scale vulnerabilities
This comment is very dramatic and very hype-y. I don't think this is an appropriate comment.
This is some insane framing. This vulnerability exists already. Even if there were no LLM, this would still exist and might even be actively exploited right now. We just wouldn't have a chance to know it.
I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
If a vulnerability is practically impossible for a human, but possible for AI, then it's not security through obscurity. Technology just advanced past a line where new things can be exploited.
There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
None of these vulns was impossible for a human. There just wasn't that many humans clever enough to find them, and those that did we're busy not hacking random things. There isn't a single type of new exploit that AI has found that I know about.
We have a billon hacking experts now is the big change.
Not AI but poorly designed programming languages ignoring integer overflows.
I donât know any systems programming language that doesnât ignore integer overflow/underflow in release builds. IMHO this is a CPU design flaw - integer operations should carry a flag indicating if the operation is allowed to wrap or if it should generate a trap. Then there wouldnât be a performance penalty for this and languages like Rust could have it be trapped by default.
No. Memory safe languages still have integer handling issues, but the consequences are different. In a memory-unsafe language, integer mishandling often nets out to bad copies and memory corruption.
I wouldnât be sure that a Rust equivalent would be immune because itâs not hard to imagine an underflow/overflow from reaching an unsafe block thatâs used for performance (the implicated TLS path would be a hotspot worthy of such a choice).
This is the thing about `unsafe` blocks: they're not memory safe. It's right there in the name.
I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.
Android had an integer overflow vulnerability where it could be tricked into verifying the signing of one file but install another for execution. Boeing 787 Dreamliner had a bug where after 248 it's 32-bit counter overflowed triggering a "fail-safe" mode that shut down all electrical power to the aircraft.
You're right it's more rare but the results can still be catastrophic.
Isn't "memory corruption" typically the fast path to RCE? The attacker gets to overwrite a function pointer, or a regular pointer that then lets them overwrite something else, or the length field that will then cause a buffer overrun even in a "safe" language because the run-time length is now different than the one in the compile-time check, etc.
Memory corruption -> RCE.
Integer handling mistake -> memory corruption -> RCE.
That's the pattern. In a memory-safe language, the integer handling error has no link to memory corruption (though it will to other logic bugs!).
Zig traps arithmetic overflow/underflow when compiling in ReleaseSafe.
X86 had the INTO instruction to trigger a fault if the Overflow flag is set, so not quite a per-instruction bit but more of a suffix. JO is also fairly straightforward for compilers to add on important arithmetic operations. We can probably blame C entirely for this; only in C23 are there intrinsics for overflow detection on arithmetic.
C23 intrinsics are so ugly that nobody is going to use them. They should have made ordinary "+" to trap on overflow, and an ugly instirnsic to wrap around.
Lisp or Python have a proper numeric tower and don't default to machine types. Keyword being default, fast is still possible in CL.
"I don't know any decent language so let's continue using poorly designed languages". Please look at Swift, Javascript, Python.
Why would CPU designers make a trap if languages do not require it?
JavaScript and Python arenât systems languages. Swift often pays the performance penalty for trapping everywhere and hasnât seen adoption as a systems language for that reason.
How does that saying go?
"Go fast and get your things broken into"
We're after a way to go fast without having your things broken into. "Language allows you to specify whether you want wrap around or an exception and hardware makes implementing that efficient" is better than both inefficient runtime checks and silent data corruption.
Yes, so "+" should trap, and there should be a function that wraps around because wraparound addition is used very rarely (mostly in crypto). It seems that you agree that C is designed the wrong way.
i still remember win xp pre service pack 1 days where you could get p0wned by a worm just by leaving machine on the internet. this is nothing compared to that.
Why your standards are so low?
I am having a hard time understanding this position. AI is uncovering flaws that "state-level actors" are exploiting since decades, for some of the recently found issue.
Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Agents are simply faster.
Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.
Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.
An attacker wins when they can breach any defender's security.
AI is -> Forcing you to PAY UP your Tech Debt.
This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
"Cattle, not pets" has been industry best practice for a long time now. Those that don't follow it will eventually pay the price for one reason or another.
This is why I worry about upgrading in place between major Fedora versions. I think every six months you really need start from a fresh image.
Think of it as the opposite of security through obscurity: at some point the hackers won't be able to keep track of it all. They'll start paying for Jira accounts and need to hire PMs. They'll be patching servers to drive away the competition and will need experienced IT people. Someone is going to have to build apps to keep track of it all.
The circle of life.
Not quite
There is going to be hardships before this gets better. But it is hardship we must endure
I'm pretty solidly in the AI skeptic camp. But adversarial use of AI to improve software security is a good thing in my mind.
I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
At the end of the day it will be both. In fact I wouldn't be surprised if we started seeing offensive defense. That is already compromised boxes counter hacked and erased or otherwise made unbootable.
A dead box isn't part of any bot army.
and C is the tinderbox
The billions of lines of C and C++ code is like a square lightyear of agar for LLMs to find bugs in.
Iâm generally not in favor of government meddling with private property, but I feel like I could be on board with the NSA or similar covertly patching vulnerable machines.
And your first thought was the NSA? Patching and adding their own rootkit eh?
Don't you dare patching my honeypots!
Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
>> Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
> tried and true software
Like Java. Surely anyone can't mess up simple things like base64 decoding in it...
I would hope you know that no matter how many reviews, humans are fallible and miss things.
AI agents are also fallible? So, what do you think we should do?
Yes, AIs are fallible. Yes, humans are fallible. Nobody said differently.
That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.
The difference is that the AI doesn't get tired so you can review everything.
Except that it's very far from guaranteed to find everything, and especially everything that humans would find easily.
Really? AI is catching all the easy bugs Iâd catch, and way more subtle ones as well. Maybe it was the case theyâd miss obvious things often a year ago, but thatâs not todayâs frontier models.
Oh, if it's catching all the "bugs you'd catch" that's sure proof it will always catch all the bugs that people would catch....
Yes, we both posted anecdotes, very astute.
Well, I only said that it's far from guaranteed to find everything, while you seemed to argue that your anecdote guarantees it
Every now and then, I think about the old demonization against encryption⌠and how it completely went away.
Sure, there are niche things. You canât use US ciphers in China and vice versa.
But overall, one day, the intelligence communities world wide just decided to not raise a fuss about everyone having solid cryptography.
It makes me think if we did find out that every server out there had a route kit⌠it would make that part make sense.
It's going to be rough sledding for a while, but, so long as you're on FOSS, at some point, the bugs and vulns should run out, and best practices trained into AI should hopefully reduce new bugs being introduced.
All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
AI is just showing how foolish using such a unsafe language like C to write code that connects to every other computer on the internet was.
Imagine a alternate reality where Ada replaced C by 1995. We would be in a MUCH better place.
Unlikely. An Ada compiler would be an expensive additional purchase, a C compiler usually came as part of the installed copy of UNIX.
Also in the few years prior to 1995 most computers that could run a C compiler weren't big enough to run an Ada compiler. Heck most computers today can't run the rust toolchain.
The bigger problem is lack of liability: we knew C sucked in the 90s, too, but we had three decades of over-confident dudes saying that was user error and _they_ were smart enough to use C correctly.
Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or itâd be too much work to rewrite things.
Iâm hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
It's also valid to say that Rust's memory safety is good but the rest of the language sucks. Just as annoying as the over confident c-heads is the rust community that made even countenancing working on an alternative memory safe system a moral sin, which stifled innovation in memory safety. Hell, people barely know that Ada/spark is memory safe (or that sel4 is memory safe C)
Aside from the overhead of borrow checking, what about the rest of the language sucks, in your view?
To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.
Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.
Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.
You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
> what about the rest of the language sucks, in your view
I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
Ownership model is perfect for multithreading but then Rust goes all in on async like WTF?
lack of a stable ABI is a big one
What overhead from borrow checking?
I imagine they are referring to the cognitive overhead of learning to work with it.
Whether that is âsomething natural to youâ, âsomething you learn and adapt toâ or âsomething you fight with foreverâ seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
This really isn't true. It's true that those people saying those things existed, but much more recently. Back when all the C/C++ legacy code we depend on today was being written it was the performance penalty imposed by safety that forced developers down that path, not some lack of interest in other options. We were also transitioning from a world where a fair bit of systems code was written in even lower level languages than C (assembler, BLISS, BCPL). The world we're in today effectively began around 1975 when people figured out how to make a C compiler run on an 11/45. All the decisions were pretty much locked in from there on.
What a strange takeaway. This is exactly how a server at a restaurant thinks about AI. Just the shortest term goals of keeping the status quo. They think that if they bury their head in the sand, everything will continue as normal.
You need 128Gb of http/2 traffic and specific cpu instructions to get this to work.
https://x.com/Laughing_Mantis/status/2123146699396858260
Its a non issue, but people just want to be told AI is God or something and the world is ending.
Its a non issue.
There are two bugs here; CVE-2026-123633 does not require a large amount of data or AVX-512.
what do you mean it's a non issue? Surely no one is going to take the trouble to take down my toy website, but if there's something valuable enough, a motivated actor can/will try to exploit
At 1 Gbit/s (typical datacenter connection) how much time does it take to transmit 128 Gb of traffic?
If you want to check if your host might be vulnerable:
https://xint.io/blog/nginx2root-remote-code-execution#23-com...
Workarounds if you're vulnerable and can't update immediately:
https://xint.io/blog/nginx2root-remote-code-execution#33-eme...
That first link sounded like a scanner that'll actually verify your fix or workaround but it's just a long version of "check yourself if you run a vulnerable version and have http2 enabled". To save others a click
It's becoming mentally costly to maintain personal VPS. Beside the time you have to put into it, now it's also worries of a vulnerability you missed, and boom, you're hacked.
Hardly anyone talking about the Linux SCTP bug that they used to LPE out of the container. Apparently that's a zero-day without a public CVE number yet? Presumably they're coordinating with the kernel team and distros to get it patched first. Of course, one might just assume that there are a bazillion LPEs available to even a properly confined rootless container, but ... yuck.
Is there some context to this, I don't see a mention of SCTP in the blog post? Maybe edited out?
https://xint.io/blog/nginx2root-remote-code-execution#6-anal...
SCTP should be disabled in every distro.
Sadly it is not but one can disable it easily enough.
I notice a phenomena that when older software has a critical vulnerability, like say Apache, you get quite a few comments of "Why does anyone use that? Use Nginx"
But when newer software has the same vulnerability, you do not see the same kind of comments.
There always seems to be a latent stance in IT that newer is better.
> But when newer software has the same vulnerability, you do not see the same kind of comments.
This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
I can't speak for the others, but one advantage Caddy has over Nginx is that it isn't written in C.
Go isn't quite a memory safe language, but it provides fewer footguns.
On the other hand, Caddy has a lot more features, so there's more code that can contain exploits.
Having RCEs in general is fine. It's the rate and severity that matter.
We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.
In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
I haven't even heard of any of those other than caddy...
I'm actually moving from nginx to Caddy after this. I'll replace all C/C++ stuff within reason.
I do love Caddy. But it was vulnerable to CVE-2026-98432 a few weeks back, due to Go's MPTCP policy.
CVE-2026-98432 also applies to non-MPTCP connections. That's why disabling MPTCP was actively discouraged as a workaround (low level firewalls that clear all options do work, though, and they'd also work fine with Caddy).
Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
Me too. An agent was able to replace my nginx with caddy on my webserver in a few minutes.
All the remaining C code needs to die in a matter of months not years.
> All the remaining C code needs to die in a matter of months not years.
Why so? are most of these vulnerabilities from C's memory model?
Yes
surprised there hasn't been an effort to rewrite nginx or other critical networking infrastructure in Rust. We just went straight from C (nginx) to Go (Caddy/Traefik) which are not interchangeable (garbage collection).
Doesn't nginx count as "older software" at this point? It's over twenty years old...
Just use Pingora. It's in Rust and from Cloudflare so it is safe. (/s but not really I guess)
every major exploit now comes with a marketing website - nginx2root dot com in this case
the certificate for it was issued on the 2nd of November
if you were scanning CT logs you would have had a 16 day head start on RE'ing this
there are more examples in the current CT logs
https://api.certspotter.com/v1/issuances?domain=nginx2root.c...
Security companies should buy all this $tech2root domains as a diverting measure first and also as a pretty safe investment, given current trends.
there's one that squatted on 30+ domains a few weeks ago that i saw
but what about $tech4shell.com tho?
I scan new domain names but for a different reason. There is an insane amount of nonsense domains registered everyday. Would be hard to tell just based on that.
Someone registered ping2shellâ˘com on Nov 9th, nine days ago.
ping2rootâ˘com is available. :)
And irresponsible disclosure by a bunch of attention seeking tossers
Sitting on flaws is dead, the responsible disclosure people can get lost. AI finds and exploits flaws in hours. We also have little knowledge if the people finding the exploits have already intentionally or unintentionally leaked said exploits to AI providers.
If you want to promote responsible anything, go for responsible writing of software in the first place.
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
Rust has a compile time option, though to my immense disappointment, it defaults to on only in debug mode. (You can opt into it for release, if desired.)
Python simply can't overflow, since its integral type is not a fixed width.
But it is the sort of correctness thing that I do wish languages would pay more attention to.
In Rust's case it just matters a lot less since bounds checks are still present. But I do think int under/overflows are probably one of the scarier issues in the language.
Overflows are dangerous by itself. Assume you keep the order total price as uint32. Someone orders a spaceship for 2.2B credits, and then adds another one which is not an expected scenario. The total counter overflows and now you have sold 2 expensive spaceships at 99% discount.
>Python simply can't overflow, since its integral type is not a fixed width.
Python master race
While I do think that was one of (maybe the only) really great decision Python made... There are so many other things wrong with it that it's a poor choice for almost anything in my book.
Well I guess this was the vuln that was teased about on Mastodon some hours ago:
https://is-a.cat/@ar/117634569518544368
HTTP/2 again, not surprisingly. The curse of complexity strikes again. When I first saw the spec, I knew it would be far more difficult to implement correctly than 1.1 or 1.0. While 1.x is not ideal either (parsing a protocol that involves free-form text with no fixed limits is always a bit scary), at least it has a much smaller attack surface.
On the bright side, perhaps "security review" of code which involves throwing it at an AI and fixing what it finds before releasing will become a normal practice.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare
> Wiz Research, Ikotas Labs, Metnir, Rapid7, and Xint independently uncovered
Is this some kind of record for bug collision?
Itâs a reflection of the times.
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
It's never clear quite where you are in the technologically cycle.
But it's like inventing a hammer and finding you can smash whatever you want with it.
Soon you'll be building houses with it.
Maybe that's being glib but it's just a tool being applied to a world that is still largely prestine from its impact (pun unintended).
given the state of our weapons industry I think it's more like
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
The âhintâ is for the modern equivalent of script kiddies.
i.e fast followers of the researchers who donât have the resources to burn through checking all of a tool.
Cost scales with the size of your search space. Smarter questions get answers for less money & time.
And search space in software can be absolutely staggeringly large. Just a little bit of directing and you can shrink the problem from all the entropy in the universe to a few hours of compute needed to find an exploit.
Can I take this to mean a recent model release is able to find this bug, and these groups have access to that model?
What are the odds several humans independently uncovered a complex vulnerability like this simultaneously?
No (well maybe no), because they are lobotomizing the public models on cyber tasks.
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
That seems likely, but maybe it was being actively exploited so several people started investigating.
or maybe the model leaked the exploit from one to the other...
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
This was one of the takeaways from Greg Kroah-Hartmanâs recent Linux in the land of LLM keynote.
The observation was "if one person found the bug with LLMs, other people are also going to find the bug". Whether it was due to communication between instances of the agent or just due to the predictability of an individual agent is not clear and there is not much evidence for the former.
When everyone is using the same tools for the same purpose...
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: https://nginx2root.com/
How was this disclosure coordinated? Considering the website dropped before the official nginx patch was even released on Github...
Recent high-profile vulns involved CEOs of a security company breaking embargo on a Hacker News account, and anime dancing girl videos that somehow leaked; so I think itâs barely any coordination anymore.
Why post such exploits? What is the purpose?
Fame, money, and everything else those bring along for the ride.
Consider a world in which the only people that get to know about it are the people who discovered it in the first place.
Consider a world in which corporations like the Ubuntu Foundation or Microsoft experience zero public pressure to fix such vulnerabilities but instead fix it only when a major customer notices their servers getting hit.
Imagine if instead of "you've got 30/60/90 days to fix this thing I noticed before I post it publically" it was "okay, well, take your time I guess, I just hope North Korea doesn't notice the same issue I noticed, which is especially likely because they have far more resources than I or my small team does to point at discovering this kind of thing, while you're dragging your feet fixing it."
The reason people want shorter disclosure windows is because the shorter the window is the faster things get fixed for the average users and smaller companies.
Responsible disclosure has been a massive failure that has not made anything more secure.
Massive monetary loss, explosions, and dramatic failures will push computer security in a year where it should have been decades ago.
Xint.ioâs homepage says âXint finds real vulnerabilities, and proves every finding with a working exploit.â, which for defenders is probably helpful, since automated scanning tools have a LOT of noise.
> Will the exploit be published?
> Not yet. We are not publishing the exploit now because the risk is too high.
I agree, never post anything you find, keep all the bugs to yourself, write reliable exploits who knows when you would need one.
Marketing, baby! It's the same small handful of firms pushing these same AI-generated spam attention grabbing vulnerabilities, quirky name and domain on top for maximum fear mongering.
Why do you consider it fear mongering? The exploit is clearly real, and TFA even has a big disclaimer about how it's difficult to exploit "in the wild".
So who's the judge of where to draw the line between exploits that are relevant to publish, and which are "spam". You?
Or do you just think that companies shouldn't be allowed to show the work they're doing for marketing?
Because some people get deranged by AI in the opposite way AI lovers do.
Seems like you're a fan of security by obscurity over secure code?
FreeBSD is unpatched,
I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.
Now I sit and wait for an HaProxy announcement...
ps. if you are on FreeBSD and doing TLS directly
sed -i.bak -E 's/^([[:space:]]server[[:space:]]\{)$/\1\ http2 off;/' /usr/local/etc/nginx/nginx.conf ;service nginx restart
I don't know if direct access with no proxy is needed for all 5. CVE-2026-123634 and CVE-2026-113425 is plain HTTP.
I could be wrong (my sed is rusty) but I don't think that sed command will help if http2 is enabled at the http block level, it just disables it at the server level which will only take effect if not overridden elsewhere.
FreeBSD ports are now updated to nginx 1.30.7.
Everyone's worried about security right now, even non-tech people, but hasn't half the nation's personal data been leaked twice a month for the last decade or two?
I know times are changing, it's just curious that nobody seemed to care until now.
I think all airplanes in the entire country being grounded, banks not able to process transfers, or grocery store freezers being disabled hit differently than data leakingâŚ
None of those were due to vulnerabilities
The freezers were, at least to some extent. One could argue that they should never have been on the internet and so the vulns wouldn't matter ... but a hidden, insecure admin password based only on a static key and today's date is definitely a vuln in my opinion.
I disagree. They may not be because of exploited vulns, but the rate updates and patches are happening there is a very long tail of instability after patching due to extremely fast turn arounds.
I'm not sure yet, but my guess is that the FAA grounding was not really related to vuln patches. I thought there was a copper to fiber network upgrade gone wrong involved there. It seems like the bigger problem is aging infra that needs replacement.
Yes. Thatâs the confidentiality part of the CIA triad. But now you have a whole lot of things suddenly impacting availability in really visible ways, so yes, a whole different cross-section of people are really perking up.
I wonder when all of this die down. I assume at some tipping point most software will be written/patched with LLMs and so hopefully (fingers crossed) more resistent to these bugs.
Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
It's possible that AI is going to be better at finding and exploiting systems than at securing them. It's not really an even playing field. On the one hand you have to somehow ensure that you've patched everything when there's no definite list of everything, the other side just needs to find one overlooked way in.
At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
It's not possible. It generally requires more work to find an exploit than to patch the (known) exploit.
Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
> It generally requires more work to find an exploit than to patch the (known) exploit.
Even if that's true, one should not forget that:
When the hackers have breached something, awaits a reward.
E.g. a Bitcoin wallet, or learning something about a political competitor and getting rid of them.
(So the hackers can spend lots of ...)
How do you know when the "exploit finding" pass is done? A hacker group would happily burn a month or more of tokens trying to break a critical piece of software.
The better LLMs get at finding vulnerabilities, the more obvious it will be to bolt them into software development processes to find the bugs prior to release.
With enterprise frontier LLM models, there is no longer an âother side.â Everyone is just using what they can rent from a few big suppliers. You can too.
But are LLM patches exhaustive? Cant a new model find even more obscure bugs, including bugs introduced by the old LLM patch? Is it really realistic that we such complex projects as some of these critical dependencies are can be truly 100% bug free?
If the software is too complex to understand, maybe it should be made simpler. Nginx does a million things currently but what most people need is something that parses http1 strings and sends back a file or calls a second program based on the path. Every component can be thought of in this way, with the most complexity perhaps in being sure the interfaces don't do something unexpected
We dont need to be bug free, so much as the juice needs to be not worth squeeze. I think i am also making an assumption that we "level" in terms of frontier model progress at some point, and we have months or years between upgrades.
>so much as the juice needs to be not worth squeeze
I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
but at least sysadmins/ops still have work to do
Reading news like this makes me remember the ship hacking scene from GITS Innocence. In the end there will be AI attacking vs AI defending. Humans will be too slow to participate. Defender AIs should be coming soon - maybe some sort of proactive memory and usage anomaly pattern monitoring AI systems?
So work around is to disable http/2 and only allow http/1.1?
Yep, that or update to 1.31.9+ or 1.30.7+ if they are available in your distro
Nobody seems to notify major distros before publishing their latest vulnerability anymore. Coordinated patches are a thing of the past, now we just hope that somebody at Red Hat and/or Canonical is reading HN today.
Unless I am confused, it doesn't look like ubuntu server has made available the fix.
Itâs time to switch away from Ubuntu and go upstream to Debian.
While I wholeheartedly agree with you, Debian hasn't patched yet, either. See my sibling comment: https://news.ycombinator.com/item?id=50551026.
Lol no
Debian is unfixed too
Canonical hasn't patched yet:
https://ubuntu.com/security/CVE-2026-123633
https://ubuntu.com/security/CVE-2026-123007
Neither has Debian:
https://security-tracker.debian.org/tracker/CVE-2026-123633
https://security-tracker.debian.org/tracker/CVE-2026-123007
https://tracker.debian.org/news/1825662/accepted-nginx-1306-... it's been only accepted into unstable
Amazing that they don't disclose these CVEs to the major Linux distros before going public
There isn't even a mail about this on oss-security, as of now.
Who is sane enough in 2026 to expose distro default services to the Internet?
I wouldn't expect any distro maintainers to react in hours. Distros are too complex to dump updates in rush.
Start to use Docker/Podman/systemd-nspawn.. and isolate public services to necessary needs (static binaries, drop capabilities, default drop rule firewall between containers, r/o storage, access logs bind mounted as rw file, memory temp dir with noexec). Container images are patched the quickest.
My Apaches still have http2 disabled from the last cve, now its njinx time
And there were a couple of other http2 CVEs in nginx before the Apache one.
Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
The AI gods say that http/2 is disabled in a default installation fwiw.
Obviously... the default configuration also doesn't enable TLS, which is a de-facto prerequisite for HTTP 2. How many websites do you (or your "AI gods") think run nginx in its default configuration?
The point is that it must have been deliberately enabled. TLS will generally be enabled out of necessity, whereas the chances for HTTP/2 are much lower, and your NGINX configuration will explicitly indicate if itâs enabled. If you donât see âhttp2â in your configuration, then you know you are fine.
Yeah and park is the default gear in cars...
Nginx Proxy Manager isn't updated yet, I know a lot of people use this for their home server so just a reminder to disable http/2 here as well.
Disabling http/2 here is a bit of a pain: you have to go through each configured host and disable it, there's no global switch
Yes, but multi-file find and replace exists, and you can verify every replacement.
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches https://github.com/kubernetes/ingress-NGINX
edit noted it's about a Kubernetes controller
As much as everyone is getting allergic to Rust migrations that sometimes make little sense, keeping network-facing daemons in C is just asking for it at this point. It's time to start seriously migrating to web servers like ferron.sh or https://xinproxy.com or whatnot.
I now run all my network facing C code via fil-c. I have limited needs (ssh, static file server), but for those it works very well. I personally am more in favour of making existing battle tested C safer via fil-c or formal verification, than I am of rewriting the world in rust.
New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
Fil-C is also a great solution, albeit with different performance trade-offs.
In theory, scpptool provides another solution with different performance, compatibility and portability trade-offs. Unlike Fil-C, it is not "fanatically compatible" with existing C code. But in theory, with maybe some minor adjustments to the code (and maybe less-minor adjustments to the build system), you can auto-translate your existing C code base to an essentially memory-safe subset of C++ at build-time [1].
While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.
[1] https://github.com/duneroadrunner/SaferCPlusPlus-AutoTransla...
In many cases the performance is not that important, for example, SSH is often used to transmit mere kilobytes of text per second and I would prefer a safe SSH to a faster unsafe SSH. Or take mDNS, it servers several requests per minute max.
What good is a car that goes 1000 miles per hour if it has no brakes?
It's time for performance above everything else to die, especially on anything and everything that gets anywhere close to untrusted data.
Same, made a fil-c fork of nginx a while back, saved me from many RCEs https://github.com/remixer-dec/filc-nginx Don't have the tokens to maintain it though :(
This sounds like a fantastic solution, especially for non-high-volume stuff. Would you consider writing down your setup and experiences? I'm sure it'll be a hit on this site!
fv?
Formal verification.
formal verification?
I've reached the same conclusion myself. Many of the recent exploits seem like edge-cases that the author acknowledged 10-20 years ago but didn't predict that tools would eventually be able to chain together 100k network requests in order to trigger a buffer overflow.
Who exactly are the people behind xin...?
The project isn't currently open source and there's next to nothing on the site to reassure you who's behind the server gateway you'd be running.
Googling the company in the footer produces this, but it also doesn't have much: https://www.signeen.com
Nginx and C have been around for decades. Migrating to new, poorly tested (in the real world) software does not seem an improvement at all.
Nobody is saying that it needs to be rushed. Cloudflare started moving towards Rust more than a decade ago and (shortly after heartbleed happened) and progressively rolled bits out over the course of years. I'm sure there were problems here and there but it's hardly been a drag on them.
And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.
https://blog.cloudflare.com/20-percent-internet-upgrade/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception weâve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
I mean, a lot of nginx use cases can be handled by e.g caddy - which does have significant real world deployment.
Given the volume and depth of testing possible with AI these days I'd say the decades of testing isn't that expensive to reproduce, especially if it's an OSS project with a public bug tracker.
But rust is not a silver bullet. Rust compiler still has memory corruption bugs. Rust is only memory safe by design not by implementation. Borrow checker allows unsafe code without using unsafe keyword.
No one is expecting a silver bullet. The absence of a silver bullet shouldn't prevent people moving towards better solutions.
It isn't perfect. But it is still way better than c.
> Borrow checker allows unsafe code without using unsafe keyword.
is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
this is just a perfection fallacy. silver bullets donât need to be perfect in every single way to be silver bullets. and rust is as much of one as anything.
You can still die in a car accident while wearing a seatbelt, too.
xinproxy has a pretty restrictive license, but I'll give ferron.sh a closer look.
Software doesnât generally get checked for bugs until it has users. These new web servers mostly have that going for them rather than the language.
It is true that mileage is a good proxy for finding bugs, but this is isomorphic to the fundamental differences between testing v. verification. You can prove absence of classes of bugs with verification. Obviously there can be other logic bugs in newer software. No question. The issue is the memory-corruption bugs in socket-facing daemons can have catastrophic consequences, especially in light of chaining kernel vulnerabilities which defeats the privilege drop that nginx does.
So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
A more holistic risk-analysis would also need to consider the fact that a high-performance webserver written in Rust may also have memory safety issue due to use of "unsafe" features or unsafe dependencies, may be affected by memory safety issues in the standard library, may also have catastrophic logic errors, may be more affect by supply chain risks due to the use of a language-level package manager, and may have a larger patch gap.
Which is like complaining that Python is unsafe because it is written in C.
That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
Not in the slightest. It's like pointing out that a Python script calling out to binaries written in C doesn't have the memory safety of Python's GC. Which is obviously the case, and if you're a serious developer it's a failure point you're very mindful of.
On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.
Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.
Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
By research you mean blog posts by Google?
What specifically distinguishes the empirical research presented in Googleâs blog posts, for example the one linked below, from your definition of research?
https://blog.google/security/rust-in-android-move-fast-fix-t...
Research is not just making some plots and posting them on the internet. A proper scientific studies would need to be done much more carefully, if you want to have any chance to get meaningful results, because there is a lot which can go wrong at any level. This is something learned people learned the hard way in empirical sciences. In particular, in this setting I do not see how one could ever properly control for confounding factors. Such a study should also be peer reviewed. Then repeated by others. Also if there are commercial interests mixed-up this needs to be questioned carefully. Finally, I just note the data presented seems mostly about C++ not C, so is not clear how this applies to C. Not separating this would also already be an obvious methodological problem. This is also connected to the problem that the underlying data is not made available, and not properly described in sufficient detail. Etc. (Not a full list, just to make it clear this is not even remotely science).
A more holistic risk-analysis would also need to consider the fact that wearing seatbelts in motor vehicles still leaves drivers vulnerable to drive-by shootings, may be affected by massive boulders crashing on top of the vehicle, driving off a bridge, or gas explosions, and costs more to produce.
This argument is cope.
In theory, that could be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.
e.g. Cloudflare replaced Nginx with Pingora: https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception weâve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
If we're suggesting alternative web servers, I have had a pleasant experience with caddyserver (Go). It's been around 10+ years, has essentially transparent TLS certificate provisioning, and its config syntax is easy to understand. Even using AI to help vet configs, it's readable. It's packaged everywhere and has lots of plugins for things, though I only need the base package for my frontend proxy usecase.
is xinproxy open source? the website only has binaries and I couldn't find anything on github
Ferron is opensource. xin isn't yet.
If you're using NGINX to run your PHP applications, consider eliminating everything other than PHP itself:
https://news.ycombinator.com/item?id=50551504
See Morris worm, and when it took place, afterwards check Multics security assement done by the DoD versus UNIX.
Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.
Yet, some folks are more attached to C than the inventors themselves.
So what is the solution, rewrite in Rust?
Sure, using a closed source proxy from a company nobody has ever heard of... That's a good idea...
Ferron seems much better, but I'm also not convinced that it's really production ready
As someone that was a developer for a long time what is the problem with C? How does going to rust mitigate this? Why is having the stuff in C a security challenge?
It's very easy to accidentally write buffer overflows, use-after-frees or other memory-safety bugs in C. Modern tooling can probably catch some of these better than in the days of yore but the fundamentals of the language are still the same.
If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.
But C in itself has no guardrails against something like
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.Some people think it's fundamentally impossible for C to be correct or memory safe
Fundamentally impossible? No.
Practically impossible? Yes.
SQLite is probably the most thoroughly tested and carefully developed C library in history and they still found a handful of memory safety bugs every year.
Rust is memory safe at the cost of being harder to write, but with AI, it's actually really easy to write it because AI's loove strict compilers. So now that AIs can just find 0-days before things are even released (so I'm told), it makes a lot of sense to do bun[1]-style rewrites to Rust with AI.
1: https://bun.com/blog/bun-in-rust
If a bug is found before its released... Then that's not a 0-day?
The name refers to the number of days since public release. "Exploitable 0 days since release". If you found a bug before release, then that's... Just a bug picked it up in normal development lifecycle.
I read 0 days like "Exploitable 0 days since _discovery_". Also, it's only a bug in the normal lifecycle if the discoverer tells the developer making it. If they discover it and then have hours and days to perfect a program that exploits it, then starts running immediately when it is deployed or released, that's a new kind of "zero day". Makes you want to start differentiating what "0 days" mean, though.
If AI found it, then the discoverer has told the dev, though, right? So it _is_ in the normal lifecycle. So it can't be a zero-day.
The ai told the hacker. The hacker discovered it with his AI before the developer released by examining his git branch.
I think that's the fundamental disconnect in the conversation here. You think I'm talking about the dev because it's not released yet but just because it isn't tagged and released doesn't mean it's not done in the open where hackers can see it and their AI can scan it.
It's a bit political to say that C is the fault here, I'd venture the radical position that this was an Nginx issue.
This looks like someone prompted "rewrite nginx in rust, make no mistakes" in an afternoon and is now distributing it as a closed-source binary. Might as well just do it yourself.
xin seems to be closed source and ai generated. Before even worrying about exploits, how is someone supposed to ascertain that xin's implementation is correct? Sure Rust might not have any memory safety issues, but other logic vulnerabilities may exist. 'Claude make no mistakes?'
Caddy, Traefik, or even Envoy may be better choices.
No idea what the code is like, but the AI-generated landing page makes xinproxy an immediate "no thanks" for me.
I'd hate to interrupt all the Rust fans sucking each other off again, but no one has yet mentioned the RCE relies on ASLR being disabled, which is basically non-existent nowadays. Another nothingburger CVE with a silly name and it's only Wednesday.
Haven't you heard? These vulnerabilities are just AI marketing stunts. And also, the Bun rewrite to rust was a marketing stunt, those things don't work. There's no way Bun is running on Rust from a couple month AI powered rewrite, that would shatter my preconceived worldview about what the future looks like. /s
Running a homelab is quickly turning into a bigger and bigger liability. Just disabled http/2 on my cluster.
On the other hand I've never been so proactive about maintaining security on my homelab! I did a full audit of my self-hosted services with claude code, strengthened the isolation of all my docker services, I'm running coding agents in sandboxed microVMs now, my firewall settings are better, I set up separate VLAN for IoT, I get security notifications on my dependencies, etc. etc.
A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
The only port I expose is Wireguard.
Isn't the point of a homelab to not expose it to the internet?
Annoyingly I've switched to strict IP address whitelists for anyone I share services with. Slight hassle but massively reduced the attack service.
My box got hammered daily with port scans etc before.
Yeah I'm having to do attack surface review and threat modeling like I'm a full enterprise now...
Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
HTTP is used for a lot more things than web pages, and most non-browser clients in production donât support HTTP/3. The current LTS Java version (25) for example doesnât support it.
HTTP/1.1 is still way better and way simpler than HTTP/3.
HTTP desync attacks are reason enough to stay as far away from HTTP1.1 as you can:
https://portswigger.net/research/http-desync-attacks-request...
Unfortunately, there are still plenty of customers behind CDNs which don't support HTTP2 for backend origin connections.
Could you please share why?
Because any HTTP spec after 1.1 is incredibly complicated. You can implement HTTP/1 in half a day, and HTTP/1.1 in a day. But HTTP/2 will probably take you a month and HTTP/3 might take something like 6 months.
It boggles my mind that people are patting themselves on the back for staying on old tech stacks and using a discovered vulnerability as further justification of their decision to stay on h1.1. The response is to patch and move on if your environment requires h2; many large orgs these days use microservices which use h2 (grpc) as its network.
There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).
If you donât need these features and just serve vanilla text webpages thatâs also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing whatâs best for your environment.
> If you donât need these features and just serve vanilla text webpages thatâs also fine to just use h1.1.
Which, to be fair, is most sites.
You forgot about all the networking hardware and software those run on. The difference is negligible.
Do you realize that no one writes toy HTTP implementations from scratch? I get that you might have a "suckless" mindset but that's just not how the real world works.
That's not the point. The point is that the simpler the protocol is to implement, the simpler it is to validate and harden the implementations. A HTTP/1.1 implementation can be 10x to 30x smaller than an equivalently well-rounded HTTP/3 implementation.
A simpler protocol has less surface area for vulnerabilities. I think GP was just using that as an expression of this.
I've definitely done that before on a microcontroller.
HTTP 1.1 has enough complex behaviors at the edge that the idea of implementing it in a day, correctly, is wildly laughable.
It's debuggable. I can wireshark it and understand everything. I can even speak to HTTP servers directly with a telnet client and fetch pages. I can write microcontroller code in C that issues REST API requests. Pure TCP, nice to work with, no multiplexed UDP hex craps.
Simple is good.
~25 years of hardening.
well HTTP3 wouldn't have helped here (CVE-2026-121810)
Because HTTP3 was implemented without any errors? Citation needed I guess.
I assume OpenResty is affected as well? ("High Performance Web Platform Based on Nginx and LuaJIT")
I saw this and realised I'm running nginx in a docker container, very easy update, no waiting for the distro to make the update available
I'm self-hosting Discourse forum software which runs in a Docker container, but it sounds like I can't just easily update myself ? See https://meta.discourse.org/t/nginx2root-http2-is-on-by-defau...
Oh no, how annoying, yes I can see how that could be the case if Discourse is running its own copy of nginx with its own configuration inside its container. That's unfortunate - I don't know much about Discourse but I hope that had a good reason to bundle in nginx, seems unnecessary at first glance. I really hope a Discourse developer gets to the issue soon and releases an update, that would be the fastest and easiest way!
I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Is this the reason why those banks were down the other day? Did they get advanced knowledge so they could patch their instances? Kind of questionable that the AI labs are creating a two-tiered system for system security if so.
I wonder how much of this exploitation will push the industry towards closed-source. Previously the advantage of OSS was trust and collaborative improvement, but that is getting eroded quite heavily now by the barrage of attacks and insidious discovery.
The first step was closing contributions down to known contributors and maintainers. Maybe thereâll be a similar step where only a select group will be able to access â and attest to â the code?
Almost like âRestricted Access Sourceâ or âWithheld Sourceâ
OPs article mentions 2 CVEs, however this one claims that there are 5 new ones: https://news.ycombinator.com/item?id=50547423
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down. maybe not an RCE but a DOS in the same http/tls code.
Imagine there are 1 million things to know. Some of them are related but most of them aren't. Now imagine that you know 10 things.
Never turned http2 on in my side project pfew. But still, this is one is pretty rough...
This whole week is rough. And itâs only Wed.
well it's a good time to pitch tech debt and infrastructure cleanup. If you aren't making use of least-privilege permissions, service-mesh, netpols etc. this is the time to add all of that.
Because clearly the patching treadmill is not sustainable, you will need a a second line of defense and reduce the blast radius of every individual vulnerability.
Actually, this is good I think.
One, as the RCEs are coming out, admins are on edge and paying more attention, so more likely to catch the annoucement on their platform. "Oh, that's on PHP, we don't use this. Oh, wait, another one?"
Two, all of these hitting at once, while they may expand the surface for attackers, it also spreads their attention. They only have so much time and bandwidth. When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
So, in the sense, it can actually slow down the actual potential impact of the revelations.
We can only hope this gives users time to get their systems up to date before they come knocking.
I mean, I know its a reach, but...who knows.
> When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
Only a little bit. An attacker will just hand this to an LLM with a "ping me when this is ready to use..." prompt.
FYI, you need to send around 128GB of data for this bug to work, so not exactly discreet or quick.
How many hours before this is exploited in the wild ? Probably not all that long, that blog has a significant amount of details.
One tops
CVE-2026-1337: Race condition in compile time vs automated exploit
Seriously though, with how CVE disclosures are handled these days patching is a race against the clock
Most web servers are undergoing a security tsunami right now. It's important to switch to the ones that are on the fixed side of the tsunami, IMO.
Ops is not fun these days ⌠used to pinned versions and an auditing approval process with SecDevOps ⌠how are you mostly automate the process in a you build it you run it environment ?
Mandatory SBOM creation while building. Sending the SBOM into Dependency Track and then some automation flavors of your choice.
If you use Cloudflare to proxy traffic, you can disable HTTP/2 quickly and easily: https://developers.cloudflare.com/speed/optimization/protoco...
If Cloudflare is in front of an origin I'd expect that they absorb or block this attack in the first place.
Yes - that indeed seems to be the case! https://radar.cloudflare.com/zero-wire/nginx2root#cloudflare...
Please know the implications of this it's a much bigger thing to do than usually deserves a quick on off switch
Yeah, if it matters for your workload, then don't do this :) And obviously test outside of production first if you're doing anything important. But if all is happy with http/1.1 then this is likely much easier than making a config change or waiting for a patch that doesn't seem to exist yet for some downstream consumers of nginx (like apisix for example).
You mean moving your site behind the great internet firewall or turning off a second http implementation?
Mirror: https://archive.is/dUN0c
don't know why the mainline alpine have the empty image sha-256 : https://hub.docker.com/layers/library/nginx/1.31.9-alpine/im...
its crazy how exploits are being discovered in software nowadays, especially using AI
Thank you! I had to patch bunch of products I'm working on.
Pretty funny times. AI is making it faster to both write insecure code and to detect buffer overflows. Sell both the problem and the solution. Genius.
It'll take a few years for the market to adjust. If nothing else, businesses are good at figuring out ROIs, eventually.
"Check if our site is vulnerable to the latest nginx security advisory"
> Error: API Error: Opus 5.5's safeguards flagged this session
Literally the first message, lol.
Docker images are a bit delayed because of unstable GitHub runners: https://github.com/nginx/docker-nginx/pull/1075#issuecomment...
Cue xkcd
Sadly, the same happened last time a CVE hit.
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
The patch (or, one of them) seems very much like a bug that can happen independently of the programming language
https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...
This fixes the integer underflow for CVE-2026-123007, and it's true that the underflow can happen in any language, notwithstanding whether it's UB or not. The issue is what happens as a consequence of this underflow, in this case: out of bounds copy, which is specific to C.
Some languages make it significantly easier than C to just forbid integer overflow/underflow entirely.
Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
Sure, I usually use `overflow-checks = true` in Rust for this reason. However, it is still true that the overflow is not what caused the issue here. In Rust it would've paniced on the out of bounds copy, even with `overflow-checks = false`.
It is not specific to C. You could have bounds checking also in C, and you can have non-checked indexing also in many other languages, including Rust. I would agree though that many other language make it harder to not have the bounds checking, but the bigger question is why this is not done C (and the answer that this is very hard to do is false).
I think you forgot the /s
C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it the good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.
Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
I didn't think the /s was necessary
It wasn't necessary, at least for people who kept hearing that there is no need for languages like Go or Rust, since it is possible to write memory safe programs in C, you just have to not make any bugs.
Is it time to vibe code my whole stack so I'm secure through obscurity?
Only half joking.
Whatâs the point of these researchers if their autonomous system is now finding these vulnerabilities?
Making sure AI isn't hallucinating.
Just what I needed on a Wednesday evening after promising myself I'll go to bed before 10. Oh well, thanks for the disclosure.
So...what library/tool/language is next folks? Presumably Wiz is sitting on a bunch more.
Linux2shell?
Thankfully this time there are patches available right away, unlike that PHP bug not too long ago!
Quite the opposite, it looks like there has been no responsible disclosure here. No update available for anyone running something based on Debian (Ubuntu etc.), only if you go and fetch binaries from the source and do the system integration yourself (easy in this case, but in general that's the point of a distro). For php I could just update at some point that day
Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
HTTP1.1 has a fundamental flaw related to request smuggling. Fixing specific variants of that problem is a game of whack a mole, as issues arise most often from larger server composition, not individual parts. Fixing the protocol flaw requires a major version increment and a phase out of the old one. Today the hesitation to do that gives us a mitigation option, but in general an http downgrade is a bad thing.
There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
Everyday we are seeing new vulnerabilities. Thanks to AI
Thanks to "C is perfectly safe if you know what you are doing".
Did AI write this vuln?
Both this vuln and the java4shell vuln from yesterday have very narrow attack vectors. It's not impossible for a human to figure out, but it does feel like an AI was used to discover both of these vulns.
That's to say that this is our new normal. Something that would have taken months or years to figure out can now take an AI hours if not minutes if you have enough compute.
Right, but a human wrote the vulnerable code.
Was this discovered using an AI?
the new "hack" now is to monitor wide-spread repos for critical PRs
My firewall needs a firewall
Maybe you should not run in production a Russian reverse proxy slash web server commercialized by F5 written in C to begin with...
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
I think the problem is that while what you say is true, AI is also capable of generating the special conditions required. And if it canât do so reliably, it can sit there hammering the server for as long as it takes until the conditions are met - as an example hitting a bunch of load balanced servers until 2 requests hit the same workhorse sequentially.
It is completely realistic that we start seeing a lot of bugs a human canât (reasonably) exploit, but AI can.
At this point, Iâm not even going to check, so far all of these have been nothing burgers
Critical open source software will have to be shut down to the public
I was talking with someone about buffer overflows due to HTTP requests this morning. I think I said something like "but we don't need to worry about that, just put nginx in front of it." Sigh. Off to update some VMs. Again.
Uhm, am I missing something or are the patched releases not out yet? Why disclose this level of detail when the patches are not ready yet?
Edit: All good. They released the new versions within 30 mins.
Currently, the Ubuntu [1] & Debian [2] packages seem unpatched.
[1] https://launchpad.net/ubuntu/+source/nginx [2] https://tracker.debian.org/pkg/nginx
Yeah I'm not quite sure what is going on with Debian, I would have hoped that this vulnerability would have been disclosed in collaboration with Linux distros to get a patch out?
Everyone called me crazy for running Arch on my servers btw
Well the fix for this seemingly already landed in the repo last week: https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...
Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
> Given how common it's become to constantly scan popular repos for n-days
We have people doing this on my side project and then reporting the fixed issues to try and get CVE 'credit'. Tiring.
Upstream fixes are available in 1.30.7 stable and 1.31.9 mainline.
They're out now.
> Why disclose this level of detail when the patches are not ready yet?
Perhaps it was already being exploited?
Based on <https://news.ycombinator.com/item?id=50547678>, a bunch of people reported this to Nginx in parallel. Wouldn't surprise me if they went "Igor, you wont believe, this rce is the same as the five previous I reviewed this week! Let's ship your patches before malicious attacker discovers in parallel also"
https://security-tracker.debian.org/tracker/CVE-2026-123633
https://security-tracker.debian.org/tracker/CVE-2026-123007
Debian still vulnerable in all versions.
Nginx used to be in the OpenBSD base system between 2012-2015.
12 years on, OpenBSD's port of nginx still maintains its default chroot(2) patch.
https://raw.githubusercontent.com/rnagy/nginx_chroot_patch/m...
I recommend reading Reyk FlĂśter's AsianBSDcon 2015 paper about nginx and the thing that replaced it in base.
https://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf
> Complexity can no longer be relied upon as a barrier to discovery or exploitation.
I mean sure if we close our eyes and ears we can ignore the elephant in the room that is proprietary, closed source software.
"Both vulnerabilities are memory-safety issues in how NGINX handles HTTP/2 traffic."
HTTP/2 again. First Apache, now NGINX
I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1
I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
The beatings will continue until morale improves.
All system critical software must be rewritten in memory-safe languages before the end of the year.
Wow, that sounds eerily like leadership at my $dayjob! :-D lol Or, maybe that is default way of thinking for many leaders nowadays!?! :-(
Even the US federal government (CISA and NSA) have been strongly advising companies to transition to memory safe languages for a couple of years now. https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre... (one of the goals of such advisories is it acts as a signal to the industry that in several years this will become a mandatory requirement around purchasing decisions)
We've literally decades of evidence that we cannot write software securely in languages that aren't memory safe, for all sorts of reasons (and it's wrong to look at it as a developer competence issue, even though sometimes that's going to be the case, even extremely experienced and highly competent engineers write memory safety bugs in critical code we rely on day in day out.). Not leaving yourself open to such problems is an entirely logical conclusion. We know a large percentage of vulnerabilities in software are caused by it[1]
Pick a language that doesn't have these risk factors, or pick an enforceable subset of one that makes such impossible.
"Insanity is doing the same thing over and over and expecting different results."
[1] https://www.chromium.org/Home/chromium-security/memory-safet... 70% of Chrome vulnerabilities caused by it. Similar stories for Microsoft products https://github.com/microsoft/MSRC-Security-Research/blob/mas... and Mozilla: https://hacks.mozilla.org/2019/02/rewriting-a-browser-compon.... Curl is at 40%, and dropping because Daniel Stenberg has provided various helpers that avoid it (and I believe they've now migrated almost everything over to them) https://daniel.haxx.se/blog/2023/12/13/making-it-harder-to-d..., so there are definitely ways to do it, we just... don't.
Great work by wiz.
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
I'm sure this will happen much more than it should.
nginx is so pervasive, these CVEs are serious.
It seems like it always HTTP/2 (except when it's regex) with nginx. Glad I'm http/1.1 only.
Not surprising given how complex HTTP/2 is. Complexity is the enemy of security.
See also: https://queue.acm.org/doi/10.1145/2732266.2716278
If you are running a PHP site, you don't need an external webserver anymore, like Apache, NGINX or Caddy.
This webserver is not vulnerable to PHP2Shell nor this TLS issue, because it's completely userland PHP:
https://github.com/Qbix/webserver
It's now been battle-tested in production, and also supports websockets, etc.
(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
HTTP/2 is so complicated that the code that handles it tends to be vulnerable. What a surprise
This doesnât seem very HTTP/2 specific, though, and it seems more like the conclusion should be âdonât use C for things which matterâ. Itâs not like HTTP pipelining and keepalives didnât have plenty of edge cases to worry about.
True, but offering three different protocols with different code paths for the same thing is a lot more attack surface. If you start factoring in different compression methods things look even worse.
Caddy also supports HTTP/2 and it's not affected.
In fact in general there were waaay less CVEs for Caddy lately compared to nginx
I have used nginx for ages, but have been slowly moving things over to Traefik + Caddy.
I guess it was the right move.
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
It was clear a year ago already.
Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.
Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.
This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
Imagine using Nginx
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Well, you can't because the LLMs are all nerfed and censored w.r.t cybersecurity.
I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
The refusals do not make that big of a difference! Holy cope. No you still need to study and become an expert. Ill give you access to unlocked fable and you still wont be able to do much with it if you dont have a ton of experience hunting bugs.
Cope ping-pong is a bit ridiculous, but you need to believe weird things about the human brain or LLMs if you think that only humans can be experts.
if it wasn't AI then how do you explain 3 other companies co-discovering the issue? Also, didn't one of the companies discover it before Wiz?
The only people who have access to full throttle, no-rejection security pipeline are the OpenAI employees and very preferred security partners...
This is a monumentally dumb assertion that any Job Blow can just add a credit card and ram through these requests
Keep telling yourself that! The refusals arent that big of a deal. I have access to all the models without refusals. Your average joe still would not be able to hack something with it.
So much cope on this site. Llms havent replaced the requirement of studying and becoming an expert.
Isreael btw everyone at Wiz is a veteran of Unit 8200.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
i dont know what it is this but good work