Let's Encrypt is adding two additional remote perspectives for domain validation

(community.letsencrypt.org)

229 points | by gslin 11 hours ago ago

73 comments

  • nbadg 10 hours ago

    For anyone else who didn't have "remote perspectives" in their personal technical lexicon: this refers to performing the ACME challenge/response check when validating the domain of a certificate signing request from multiple physical locations, and therefore via multiple physical network routes, in order to make network-level attacks more difficult.

    For example, if you were validating example.com, you'd need to set up some response to the ACME challenge somewhere, either through direct DNS-based validation (DNS-01, DNS-PERSIST-01) or through a server running at the resolved A/AAAA IP address for example.com (HTTP-01, TLS-ALPN-01). In all cases, when a CA goes to validate your challenge response, it relies both upon DNS and upon the network integrity between the CA's datacenters and your DNS and/or app servers. Both of these can be subverted, eg via BGP attack, to allow an attacker to replace a DNS response, re-route the destination IP address to a different server, etc, at which point they can inject their own challenge response, and therefore issue their own (unauthorized) certificates.

    Multi-perspective validation means that the CA validates your challenge response from multiple datacenters on different networks instead of just one, such that a hypothetical network attack would have to target multiple independent networks in order to successfully issue their unauthorized certificate.

    More info:

    https://letsencrypt.org/2020/02/19/multi-perspective-validat...

    https://community.letsencrypt.org/t/multi-perspective-valida...

    • ratorx 7 hours ago

      Being pedantic, the risk/benefit of remote perspectives are theoretical and dependent on the destination network also having multiple independent links. Otherwise any “uplink” ISPs are all SPoF regardless of anything Let’s Encrypt does because the traffic must traverse them to get to the destination.

      But remote perspectives are still worth doing, Let’s Encrypt should not be the “less diverse” than the infrastructure of their typical certificate requestor.

      • muvlon 7 hours ago

        Sure, if you can MitM the connection to a single-homed destination network, you can get Let's Encrypt to sign you certificates for things in that network, regardless of how many perspectives Let's Encrypt uses.

        However, if you can MitM the uplinks at all of Let's Encrypt's perspectives, you can basically break the entire CA for everyone. Adding more perspectives reduces the risk of that.

      • dabraham1248 6 hours ago

        Let's encrypt only offers DV certs. For DV, if you can successfully MitM every connection to a destination, then you _do_ own the domain. From [DigiCert](https://www.digicert.com/difference-between-dv-ov-and-ev-ssl...):

        > Domain Validated (DV) certificates are the least-identity-validated SSL certificates and can be obtained quickly and easily—even by a malicious bot. These certificates are low-cost certificates that only require validation that a company or person can demonstrate control over a web domain for which they want to secure a certificate.

        This is arguably a flaw in allowing DV certs at all, but given that it's there, LE doing this raises the bar for spoofing about as far as it can be raised.

        • Dylan16807 21 minutes ago

          > For DV, if you can successfully MitM every connection to a destination, then you _do_ own the domain.

          That's a policy, not an inherent consequence of DV certs.

          If every verification method other than DNS was shut off, it would still be DV, but with care it could be immune to MitM. I'm not advocating for that, but it's an option.

        • dannyw 6 hours ago

          Without DV certs, much of the web might not be encrypted at all.

          I think few independent site owners would go through the more costly and complicated EV process.

          • failbuffer 4 hours ago

            Bingo... that was muchly the case prior to Let's Encrypt's introduction. They single handily changed prevailing practices outside of big tech.

            • duskwuff an hour ago

              That's a bit of an exaggeration. Domain-validated certificates already existed, and were fairly commonplace, before Let's Encrypt. What was different about LE was that their certificates were free.

          • duskwuff an hour ago

            Most independent site owners couldn't obtain an EV certificate even if they wanted to - the process effectively requires that the applicant be a business.

        • kbolino 4 hours ago

          Without major cross-national legal changes, OV/EV certificates on the open web are pointless at best and misleading to the point of being dangerous at worst. They remain quite useful in specific scenarios outside of normal web browsing, but that's not the issue at hand.

        • mwwaters 6 hours ago

          I liked the idea conceptually of CAA record only allowing DNS validation. Then use DNSSEC.

          I say “conceptually” since I know: 1. Actual attacks found with CT have used compromised TLDs, which kills the whole thing anyway, and 2. I haven’t really seen where the hosting company’s infrastructure (BGP ads and internal routing) is what’s compromised. Well certainly in the latter case they can probably also get in to the hosting machines directly.

      • sandeepkd 5 hours ago

        On the contrary I have personally seen the attacks and the impact. My understanding was that CAs are required to do that by spec and Let's Encrypt already does that.

        The Use case - Lot of SAAS based products allow/require the administrators to prove the ownership of a domain via ACME methods. The administrators then are allowed to manage the profiles for users that have email address with same domains in the SAAS product.

        Ironically the testing setup itself used the network hack for the successful ACME challenge

  • throw0101a 10 hours ago

    In August an attacker was able to get a certificate issued even with multi-perspective validation:

    > For 33 hours in late August, a rival network hijacked a block of Hetzner-hosted infrastructure serving Softaculous and Virtualizor, using it to issue a valid TLS certificate and push a malicious software update. The stolen route was RPKI-invalid the entire time, on file, checkable by anyone. It still won on most of the internet.

    > At 20:57 UTC on 28 August, a network nobody at Hetzner or Softaculous had dealt with before started announcing 162.55.80.0/24 to the internet. It was a small, unremarkable-looking slice of Hetzner’s own address space — the block hosting Softaculous’s client area, its billing pages, and the update endpoint that every Virtualizor-licensed control panel on the internet periodically calls home to. For the next 33 hours, with an 11-hour lull in the middle, a meaningful share of global traffic bound for that block went to the hijacker instead. Some of it was Let’s Encrypt’s own domain-control validation, which is how the attacker walked away with a certificate for virtualizor.com that no browser would ever flag.

    * https://cloudpunk.blog/post/bgp-hijack-rpki-gap/

    So it's all very well to have multiple network locations to prevent hijacking on the path (i.e., you have to attack multiple paths), but if the network as a whole thinks the destination is someplace else then all those paths will lead to the same (false) location.

    Perhaps the CAB should consider RPKI validation as a "MUST" rather than piling on multiple sources of validation who still have the same single point of failure (the spoofing of the destination).

    • toast0 8 hours ago

      The hijacker effectively established control over the domain. That's a properly issued cert in my book. There's no part of a domain control cert issuance where the CA is expected to establish the identity of the domain owner or obtain their consent.

      • organsnyder 7 hours ago

        That's not how it used to be, though. It used to be that you had to prove a much higher level of ownership, even for standard, non-"enhanced" certs. There was one time we discovered that the business hadn't had its articles of incorporation renewed in the state, and the validation process by the cert issuer caught it.

        I do prefer the current model (the trade-offs are worth it, IMHO), but it wasn't always this way.

        • throw0101c 6 hours ago

          I.e.:

          > An Extended Validation (EV) Certificate is a certificate conforming to X.509 that proves the legal entity of the owner and is signed by a certificate authority key that can issue EV certificates. EV certificates can be used in the same manner as any other X.509 certificates, including securing web communications with HTTPS and signing software and documents. Unlike domain-validated certificates and organization-validation certificates, EV certificates can be issued only by a subset of certificate authorities (CAs) and require verification of the requesting entity's legal identity before certificate issuance.

          * https://en.wikipedia.org/wiki/Extended_Validation_Certificat...

          • organsnyder 6 hours ago

            This was before Extended Validation was available. I don't remember the exact year, but I know for sure it was before 2007 (it was my dad's business, and he passed away in 2007).

        • tialaramex 5 hours ago

          Arguably in the extremely early days of SSL this is how it works, because the Certificate Authorities have never done this before and so sure, we fly our account excecutive out to your New York HQ and they sign (with a pen) a contract for $5000. Those early certificates are, by modern standards, worthless garbage, Chat GPT can probably teach you how to forge one though it might be too expensive for an amateur to do at home - but they do have street addresses and so on if you cared to read them. A web browser of course did not read those street addresses, and it has even less idea than you do whether the details correspond to reality.

          Within a few years at most the Race to the Bottom is well underway - if I spent $10 on my "streamlined" process to offer 1000 certificates for $10 each I make more money than a rival spending $1000 to offer 100 certificates for $100 each. As a result "Domain Verification" is normal by the turn of the century. If you weren't here for the Dotcom boom, you have never seen any of this. If you bought books on this stuff on Amazon, or worse you've never read any books about it because it was all online, you arrived after DV was everywhere.

          A peer mentions EV, that and OV are actually newcomers, the CAs realised that customers don't want to pay $100 for a "better" certificate when customers can't tell the difference so there began a meeting, the CA/Browser Forum sometimes just referred to as "cab" or "cabf". This meeting had two initial outputs. One of them endures to this day and only got stronger:

          1. Certificate Authorities must meet at least certain Baseline Requirements (the "CA/BF BRs") for all certificates. You can't issue certificates which do not meet these requirements, putting a floor on how terrible your verifications can be.

          2. Browser Vendors will add shiny UI for a new "Enhanced" verification certificate so that customers can see it's better and that's a new sales opportunity for CAs.

          In practice after some years (2) stopped making sense. A shiny UI works on a desktop PC but there's no room on an early smartphone, so you just don't add this. Soon enough you make Desktop browsers look the same as your smartphone product and all the time your UX experts are telling you users do not undertand this UI (no surprise it was just a sales gimmick) and Safety experts are telling you that it's a net negative for user safety. So, bye bye EV UI in browsers.

          But the other side of the bargain went very well. The BRs were a ratchet. In 2012 a ten year certificate issued based on some sales guy who met the director of a company at lunch was fine. By 2016 even five years is clearly too long and we should check some actual official records or have some kind of formal process. In 2012 it's fine to issue for mysql.test.internal because that's not a public name right? By 2016 uh, no, that's why it's not OK to issue that certificate, nobody could possibly own that name, do not issue bogus certificates, this is Serious Business. The BRs are how, gradually, we got where we are today.

        • mopsi 6 hours ago

          I went through the standard validation many times and the entire process felt like a security theater. For example, I had to upload a scan of my ID, but I know for a fact that in my jurisdiction no private entity can access government databases to validate that it is a genuine ID and not a photoshopped image. The government provides a SSO that third parties can use, but none of the main cert issuers implemented it, even for EV certs. They just trusted a JPEG.

          It's like those arcane verification checks used by US-based services that ask for utility bills, as if they had any way of verifying an invoice issued by a small utility provider in rural Finland.

    • lb_ 5 hours ago

      The route hijack was RPKI valid because they spoofed Hetzner's AS into the AS_PATH and Hetzner used a max length of /24 on the ROA (https://bgp.tools/rpki-history?cidr=162.55.0.0%2f16&asn=2494...).

      I personally believe the real fix would be to incentivise the use of CAA records, ideally including the accounturi parameter.

    • cowlevel 8 hours ago

      There's also the jabber.ru attack, the attacker physically intercepted the server's network cable and just issued themselves a cert.

    • woodruffw 9 hours ago

      70% adoption doesn't seem high enough or it to be a mandatory component of validation[1], that seems like it would be very disruptive to nearly a third of the Internet.

      [1]: https://ipinfo.io/blog/rpki-protects-70-internet-routes

    • traceroute66 4 hours ago

      > RPKI validation as a "MUST" rather than piling on multiple sources of validation

      And ASPA validation[1]

      [1] https://blog.apnic.net/2026/07/09/aspa-at-apnic-strengthenin...

    • Lex-2008 9 hours ago

      I think having multiple authoritative DNS servers in different networks might help with this.

      I'm not sure how CAs deal with situations when a domain has several authoritative DNS servers, but not all of them respond with expected reply. Would it would be a good idea for CAs to ask *all* authoritative DNS servers and require a coherent answer from them?

      • throw0101c 9 hours ago

        > I think having multiple authoritative DNS servers in different networks might help with this.

        For the case of the dns-01 challenge type, sure. But if you're using http-01 (or the attacker asks for it), then all those DNS servers will answer back with the same IP address for the A(AAA) query and you're back to the hijacking scenario.

        • Lex-2008 8 hours ago

          > http-01

          Good point, I forgot about that! But then CAA record might help, I guess, if you specify your accounturi?

        • codedokode 7 hours ago

          http-01 seems absolutely unreliable, any hosting provider or any government can issue fake certificates using it. What a garbage technology.

          • mixdup 5 hours ago

            The point is to prove you control a domain. If you are answering traffic to requests from clients at that domain, you control it. Whether you're supposed to have control over it is outside the scope of this validation

          • Lex-2008 5 hours ago

            Agreed. On the other hand, these DV certificates only claim to verify who _controls_ the domain. Probably, you can agree that hosting provider and government can also control the domain, no? :)

            Moreover, with dns-01, DNS provider can issue fake certificates - unless you self-host DNS - in which case, only your DNS registrar can... and a government.

            • throw0101c 3 hours ago

              > Moreover, with dns-01, DNS provider can issue fake certificates - unless you self-host DNS - in which case, only your DNS registrar can... and a government.

              Which is what DNSSEC is supposed to help with: the registrar/government would have to replace your DS/KSK with their own. Without DNSSEC queries/replies could potentially be altered over the wire (or you're in a race condition as to whether the real or attacker's reply get back first).

    • Ajedi32 8 hours ago

      Yeah, checking from multiple perspectives is a good idea, but still ultimately a band-aid over the top of a flawed system. A properly engineered solution would need to ultimately come down to cryptographic proof of domain ownership (whether through DNSSEC, or something similar). Even RPKI still seems a little too indirect for my tastes (it fixes BGP attacks but potentially leaves the door open to other forms of domain hijacking).

      Maybe in practice this is good enough, but the security nerd inside me rebels at how sloppy the whole thing feels.

      • tptacek 6 hours ago

        A lot of practical security is, in effect, band-aids over flawed systems. But I think it's useful to remember that the WebPKI doesn't actually want arbitrary secure DNS; it wants proof of domain ownership. There are ways to deliver that without forklifting in an entire new non-transparent PKI, especially since literally the only thing it will be used for is proving domain ownership.

        • Ajedi32 4 hours ago

          I don't see how you can have actual "proof of domain ownership" without secure DNS, or some kind of similar PKI. Right now they don't check for proof of domain ownership at all, just whether you have temporary, de facto control of the DNS record or the network it points at. This usually works well enough, but won't do anything if a MITM attack occurs on the server side rather than the client side.

          • tptacek 3 hours ago

            There's a whole design space of options here, one of the simpler ones being just checking via the registrar, using a protocol like RDAP. You could also use the DNS for this, but not require the entire zone to be encrypted (you'd be designing a new protocol, but that's what we have to do anyways, since DNSSEC is a dead letter).

            • Ajedi32 3 hours ago

              Yes. That would fall under the "some kind of similar PKI" category I mentioned, as you'd still need some kind of PKI to secure the RDAP connection. (Perhaps that PKI could be simpler than full DNSSEC, but it would have to solve a lot of the same problems.)

              • tptacek 2 hours ago

                For RDAP to work (for instance) you only need to bootstrap with secure connectivity to the registrars. Current DV challenges have a much harder problem: they have to work with every zone on the Internet. There's no real reason to accept that constraint!

                When we think about "sound engineering" this is the level we need to start at, not "what features could we bolt on to core Internet infrastructure to define the problem away".

                The people most avidly pushing DNSSEC for this problem aren't stupid. They're just smuggling in an additional premise: that the WebPKI itself is bad, and should be replaced with a DNS PKI. Seen through that lens, secure proof of domain ownership would be deeply problematic, as it would leave the DNS PKI with literally no reason to exist.

    • cowlevel 8 hours ago

      RPKI is, unfortunately, dual purpose as a government censorship mechanism. It would be trivial for the government of the Netherlands (or the USA, which the Netherlands is subservient to) to compel RIPE NCC to stop publishing RPKI records for Russian networks, and if RPKI was mandatory when this happened, this would cut Russia off the internet.

  • Schlagbohrer 10 hours ago

    It is wild to think that enabling HTTPS used to be difficult and expensive before Let's Encrypt came along in the wake of the Snowden revelations.

    • coldpie 9 hours ago

      Remember Firesheep? It used to be the normal thing to use plain HTTP over open wifi networks. Literally anyone else on that access point could read your whole network session, cookies and all. https://en.wikipedia.org/wiki/Firesheep

      • cowlevel 8 hours ago

        Firesheep and Snowden are the confluence of factors that happened around the same time and led to the current near-universal use of HTTPS.

      • shim__ 7 hours ago

        There was also an fun Android app which you could use to replace all YouTube videos with a different one

    • tcdent 6 hours ago

      CSRs that seemed like they had to be handled and approved by humans, and thousands of dollars in fees. We have come a long way.

      • EvanAnderson 5 hours ago

        > We have come a long way.

        Except for all the legacy systems and embedded devices I have to deal with that have no certificate automation functionality and are still within their budgeted lifetimes. Those dumb things still require manual enrollment (or, at the very least, ugly one-off scripting to hand-puppet their stupidity) and the decreasing certificate lifetimes makes the problem that much more excruciating.

    • PurpleRamen 6 hours ago

      Depending on the usecase, it's still difficult. Selfhosting can become really annoying because of this.

      • jandrese 4 hours ago

        Normally I think LetsEncrypt made self hosting a lot easier, at least if you wanted to support HTTPS.

        LetsEncrypt is what finally solved the problem with the previous systems were perfect was the enemy of the good. There used to be a lot of focus on trying to verify that the person getting the cert was who they said they are, but LetsEncrypt came and sad "that part isn't important, protecting the data is what really matters" and basically solved the real problem wholesale. Very few people inspect the certificates on every website they visit, so those protections were mostly useless and created a roadblock to widespread adoption of HTTPS, scoping the problem down to just making sure nobody can inspect or modify the data in transit made automation not only possible but easy.

      • DaSHacka 5 hours ago

        Caddy and Tailscale can at least alleviate many of the issues and all-but make it automatic and painless (though both are more geared for homelab use). I agree nginx and the more standard webservers/services oftentimes don't have good support and require quite frankly too much manual configuration though. There's no excuse in 2026 for not including acme support OOTB with a single checkbox/config value. Plus, I can't believe the default ACME client is a snap (literally WHO wants this?) and not a simple static binary to help adoption/integration.

    • onionisafruit 10 hours ago

      Before reading this I never considered Snowden could have been a motivator for Let’s Encrypt.

      • meuna 9 hours ago

        Same. Fact checking on Wikipedia seems to indicate that the letsencrypt initiative predates Snowden revelations.

        • onionisafruit 9 hours ago

          I did the same. I found the same about it predating Snowden, but I also found this quote from J Alex Halderman in a remembrance of Peter Eckersley.

          > That summer, Edward Snowden came forward with chilling revelations about NSA mass surveillance, and the need for ubiquitous encryption became clearer than ever.

          It’s not a claim that it was the instigator, but it seems like it was heavily on their minds.

    • educar 9 hours ago

      how are snowden and LE related?

      • iamnothere 9 hours ago

        The Snowden revelations did help drive the push towards having HTTPS everywhere, particularly the findings about snooping on internal Google traffic, which seemingly made Google obsessive about HTTPS afterwards. But I don’t think they are directly related.

      • Vvector 9 hours ago

        The Snowden revelations revealed just how vast the intercept of internet communications could be. There was already talk of needing HTTPS everywhere. Snowden's release got more support behind it, and brought the timeline forward.

  • Tepix 9 hours ago

    It's a shame that now, where S/MIME is a feature of pretty much every mail app, noone uses it outside of businesses. Opportunistic encryption between mail servers isn't good enough.

    We really need a CA that does for mail what Let's encrypt has done for HTTP:

    The CA needs to provider proper S/MIME certificates:

    - free like Let's Encrypt

    - duration of 1 year or longer and

    - users generate their own keys

    • kbolino 3 hours ago

      Why such a long lifetime?

      If you could actually solve the deeper problem of the sender being able to find the correct certificate to use for the recipient(s), then the lifetime could and should be as short as possible, as in days not years. The recipient can decrypt the message forever, as long as they continue to possess the private key, even if the associated certificate expires. Short-lived certificates are the only effective mechanism for dealing with key theft and mis-issuance that works at scale.

    • betterunix2 7 hours ago

      The problem with email encryption is not about CAs or verification of keys at all. The problem is key management and how encryption interferes with the common expectations that users have about email. In a corporate environment it is easier because the corporation itself can manage user keys and issue yubikeys etc., but for individual users there is no such support. It is also clunky to have to first receive someone's public key before you can send an encrypted message; for chat apps that is not a problem because chat is interactive and typically real-time, but with email there is no guarantee about interaction.

      Those issues are partially addressed with "identity based encryption," but IBE always has a "backdoor" because there is a party that issues the users' private keys (and hence can always read a users' messages). On the other hand IBE does allow you to "just send" an email to someone because the public keys are just the "identities" (e.g. email addresses) combined with a single master public key. Key management is not as much of a problem because of how keys are generated. IBE does have some users, but as you said it is mostly business/organizational use.

    • nottorp 9 hours ago

      > duration of 1 year or longer

      Let's Encrypt is bringing the duration of certs down not up. I believe their goal is 30 minutes validity.

      • throw0101c 8 hours ago

        > Let's Encrypt is bringing the duration of certs down not up. I believe their goal is 30 minutes validity.

        This is mandated by the CA/Browser Forum, which, as the name implies, is related to browsers. It says nothing about e-mail clients. In its Extended Key Usage filed (RFC 5280 § 4.2.1.12) a certificate could not have server or client authentication, but have email protection as an allowed use.

        Edit: there is an RFC for "Extensions to Automatic Certificate Management Environment for End-User S/MIME Certificates"

        * https://datatracker.ietf.org/doc/html/rfc8823

        • mixdup 5 hours ago

          Let's Encrypt has always had short-lived certificates, long before they were mandated by the CAB. They've been pushing the cutting edge here

      • doublepg23 10 minutes ago

        I say we go for one Planck time of validity.

      • FiloSottile 7 hours ago

        As far as I know 30 minutes validity is no one's goal, even long term.

        The most aggressive plans (not Let's Encrypt's specifically) target 7 day validities.

    • nazcan 8 hours ago

      Thoughts on why it hasn't been solved?

      • autoboy 7 hours ago

        Just look above here, people still want to do cert reneval manually with long timespans. The most radical thing LE did was not to provide free certs, or even automated certs, but short-lived certs.

        • tialaramex 20 minutes ago

          The short lifespans are to encourage people to actually automate, not just say they want to.

          If the cert lifetime is 5 years, Bill installs "just the first one" manually, solemnly swearing they will automate next time, and in five years nobody even knows who Bill was, Bill left three years ago. How do we replace the certificate?

          If the cert lifetime is 2 years. Bill is still there or maybe somebody on the team still knows Bill's email address, but most likely the decision is taken to "just do it manually once because it's urgent" and we just recurse. This would have been the "obvious" choice when ISRG lit up Let's Encrypt.

          But because the cert lifetime was 90 days, Bill gets told it's expiring at about the same time he's about to shove that "Actually automate certificates" work onto the back burner. Oh yeah, right, we should actually do that.

          Now, it may be that at a startup that "Automation" is initially "Put it on Natalie the intern's TODO list every Monday" but by week six Natalie has learned about cron, or Windows event scheduling, or a dozen other ways to make stuff you always do regularly happen without actual work. Today I expect her LLM will tell her how to do it.

          And Natalie is going to discover that some "permanent" solutions are more permanent than others, that you want monitoring and so on. But because Let's Encrypt is so popular unless you're doing something pretty esoteric you won't even be the first person to ask about it this month.

      • shim__ 7 hours ago

        How would you verify the users real name ?

        • kbolino 18 minutes ago

          You wouldn't. Let's Encrypt doesn't verify anything more than that the requestor of the HTTPS certificate currently has control over the domain. The equivalent would verify nothing more than that the requestor of the S/MIME certificate currently has control over the email address. This part is actually a fairly standard procedure already, as a lot of websites will send verification emails to fully activate new account signups.

  • dgrabla 6 hours ago

    Does anybody know if Let's Encrypt plans to support C2PA certificates?

  • superkuh 7 hours ago

    What we really need form Lets Encrypt is stability and lack of change. But LE is just following the megacorps down the current TLS road. We're already down to only 30 day certs and on the second iteration of acme. Acme original protocol setups no longer work, they were broken by the dropping of support.

    Addding new methods of validation does not help nearly so much as just supporting existing methods for more than a few years would. The race to the bottom will end up with an incredibly fragile system that breaks the moment automation does. Combine this with the trend towards HTTP-only and any given website will only have a lifetime of a couple years without direct human intervention. This is a drastic change from the status quo of websites simply lasting forever and being accessible till their webservers stop.

    It is 100% a corporate-for-profit-only mindset and architecture. LE is not designing or accomidating human person goals and needs.

    • tancop 3 hours ago

      You can set up a script that installs an updated web server next to the old one, tests it and switches the site to the new version. Run it on a timer every week or so. If any tests fail it either sends a message to you or tells a coding agent to fix it.

      It's practically zero overhead, low maintenance (most updates are not breaking) and you get the latest version of everything. If you're worried about supply chain attacks set a minimum release age.