Two years ago I wrote an agent in 25 lines of PHP [0]. It was surprisingly effective, even back then before tool calling was a thing and you had to coax the LLM into returning structured output. I think it even worked with GPT-3.5 for trivial things.
In my mind LLMs are just UNIX strong manipulation tools like `sed` or `awk`: you give them an input and command and they give you an output. This is especially true if you use something like `llm` [1].
It then seems logical that you can compose calls to LLMs, loop and branch and combine them with other functions.
Composing multiple smaller agents allows you to build more complex pipelines, which is a lot easier than getting a single monolithic agent to switch between contexts for different tasks. I also get some insight into how the agent performs (e.g via langfuse) because itâs less of a black box.
To use an example: I could write an elaborate prompt to fetch requirements, browse a website, generate E2E test cases, and compile a report, and Claude could run it all to some degree of success. But I could also break it down into four specialised agents, with their own context windows, and make them good at their individual tasks.
If you can get a specialized agent to work in its domain at 10% parameters of a foundation model, you can feasibly run locally, which opens up e.g. offline use cases.
Personally Iâd absolutely buy an LLM in a box which I could connect to my home assistant via usb.
Can you (or someone else) explain how to do that? How much does it typically cost to create a specialized agents that uses a local model? I thought it was expensive?
An agent is just a program which invokes a model in a loop, adding resources like files to the context etc. It's easy to write such a program and it costs nothing, all the compute cost is in the LLM call. What parent was referring to most likely is fine-tuning a smaller model which can run locally, specialized for whatever task. Since it's fine-tuned for that particular task, the hope is that it will be able to perform as well as a general purpose frontier model at a fraction of the compute cost (and locally, hence privately as well).
Absolutely, especially the part about just rolling your own alternative to Claude Code - build your own lightsaber. Having your coding agent improve itself is a pretty magical experience. And then you can trivially swap in whatever model you want (Cerebras is crazy fast, for example, which makes a big difference for these many-turn tool call conversations with big lumps of context, though gpt-oss 120b is obviously not as good as one of the frontier models). Add note-taking/memory, and ask it to remember key facts to that. Add voice transcription so that you can reply much faster (LLMs are amazing at taking in imperfect transcriptions and understanding what you meant). Each of these things takes on the order of a few minutes, and it's super fun.
On the other hand, I think that show or it didnât happen is essential.
Dumping a bit of code into an LLM doesnât make it a code agent.
And what Magic? I think you never hit conceptual and structural problems. Context window? History? Good or bad? Large Scale changes or small refactoring here and there? Sample size one or several teams? What app? How many components? Green field or not? Which programming language?
I bet you will color Claude and especially GitHub Copilot a bit differently, given that you can easily kill any self made Code Agent quite easily with a bit of steam.
Code Agents are incredibly hard to build and use. Vibe Coding is dead for a reason. I remember vividly the inflation of Todo apps and JS frameworks (Ember, Backbone, Knockout are survivors) years ago.
The more you know about agents and especially code agents the more you know, why engineers wonât be replaced so fast - Senior Engineers who hone their craft.
I enjoy fiddling with experimental agent implementations, but value certain frameworks. They solved in an opiated way problems you will run into if you dig deeper and others depend on you.
Kimi is noticeably better at tool calling than gpt-oss-120b.
I made a fun toy agent where the two models are shoulder surfing each other and swap the turns (either voluntarily, during a summarization phase), or forcefully if a tool calling mistake is made, and Kimi ends up running the show much much more often than gpt-oss.
I recently bought a mint-condition Alf phone, in the shape of Gordon Shumway of TV's "Alf", out of the back of an old auto shop in the south suburbs of Chicago, and naturally did the most obvious thing, which was to make a Gordon Shumway phone that has conversations in the voice of Gordon Shumway (sampled from Youtube and synthesized with ElevenLabs). I use https://github.com/etalab-ia/faster-whisper-server (I think?) as the Whisper backend. It's fine! Asterix feeds me WAV files, an ASI program feeds them to Whisper (running locally as a server) and does audio synthesis with the ElevenLabs API. Took like 2 hours.
Whisper.cpp/Faster-whisper are a good bit faster than OpenAI's implementation. I've found the larger whisper models to be surprisingly good in terms of transcription quality, even with our young children, but I'm sure it varies depending on the speaker, no idea how well it handles heavy accents.
I'm mostly running this on an M4 Max, so pretty good, but not an exotic GPU or anything. But with that setup, multiple sentences usually transcribe quickly enough that it doesn't really feel like much of a delay.
If you want something polished for system-wide use rather than rolling your own, I've been liking MacWhisper on the Mac side, currently hunting for something on Arch.
Agreed. I just launched https://voice-ai.knowii.net and am really a fan of Parakeet now. What it manages to achieve locally without hogging too much resources is awesome
The reason a lot of people donât do this is because Claude Code lets you use a Claude Max subscription to get virtually unlimited tokens. If youâre using this stuff for your job, Claude Max ends up being like 10x the value of paying by the token, itâs basically mandatory. And you canât use your Claude Max subscription for tools other than Claude Code (for TOS reasons. And theyâll likely catch you eventually if you try to extract and reuse access tokens).
Iâm saying if you try to use Wireshark or something to grab the session token Claude Code is using and pass it to another tool so that tool can use the same session token, theyâll probably eventually find out. All it would take is having Claude Code start passing an extra header that your other tool doesnât know about yet, suspend any accounts whose session token is used in requests that donât have that header and manually deal with any false positives. (If youâre thinking of replying with a workaround: That was just one example, there are a bajillion ways they can figure people out if they want to)
I imagine they can spot it pretty quick using machine learning to spot unlikely API access patterns. They're an AI research company after all, spotting patterns is very much in their wheelhouse.
I think this is the best way of putting it I've heard to date. I started building one just to know what's happening under the hood when I use an off-the-shelf one, but it's actually so straightforward that now I'm adding features I want. I can add them faster than a whole team of developers on a "real" product can add them - because they have a bigger audience.
The other takeaway is that agents are fantastically simple.
I also started building my own, it's fun and you get far quickly.
I'm now experimenting with letting the agent generate its own source code from a specification (currently generating 9K lines of Python code (3K of implementation, 6K of tests) from 1.5K lines in specifications (https://alejo.ch/3hi).
Agreed, and it's actually how I've been thinking about it, but it's also straight from the article, so can't claim credit. But it was fun to see it put into words by someone else.
And yeah, the LLM does so much of the lifting that the agent part is really surprisingly simple. It was really a revelation when I started working on mine.
A lot of it is just from HN osmosis, but /r/LocalLLaMA/ is a good place to hear about the latest open weight models, if that's interesting.
gpt-oss 120b is an open weight model that OpenAI released a while back, and Cerebras (a startup that is making massive wafer-scale chips that keep models in SRAM) is running that as one of the models they provide. They're a small scale contender against nvidia, but by keeping the model weights in SRAM, they get pretty crazy token throughput at low latency.
In terms of making your own agent, this one's pretty good as a starting point, and you can ask the models to help you make tools for eg running ls on a subdirectory, or editing a file. Once you have those two, you can ask it to edit itself, and you're off to the races.
No dependencies, and very easy to swap out for OpenRouter, Groq or any other API. (Except Anthropic and Google, they are special ;)
This also works on the frontend: pro tip you don't need a server for this stuff, you can make the requests directly from a HTML file. (Patent pending.)
You know how sometimes when you send a prompt to Claude, you just know itâs gonna take a while, so you go grab a coffee, come back, and itâs still working? With Cerebras itâs not even worth switching tabs, because itâll finish the same task in like three seconds.
It's interesting how much this makes you want to write Unix-style tools that do one thing and only one thing really well. Not just because it makes coding an agent simpler, but because it's much more secure!
One thing that radicalized me was building an agent that tested network connectivity for our fleet. Early on, in like 2021, I deployed a little mini-fleet of off-network DNS probes on, like, Vultr to check on our DNS routing, and actually devising metrics for them and making the data that stuff generated legible/operationalizable was annoying and error prone. But you can give basic Unix network tools --- ping, dig, traceroute --- to an agent and ask it for a clean, usable signal, and they'll do a reasonable job! They know all the flags and are generally better at interpreting tool output than I am.
I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now. But I do know that getting an agent across the 90% threshold of utility for a problem like this is much, much easier than building the good telemetry system is.
> I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now.
And that's why I won't touch 'em. All the agents will be abandoned when people realize their inherent flaws (security, reliability, truthfulness, etc) are not worth the constant low-grade uncertainty.
In a way it fits our times. Our leaders don't find truth to be a very useful notion. So we build systems that hallucinate and act unpredictably, and then invest all our money and infrastructure in them. Humans are weird.
I abandoned Claude Code pretty quickly, I find generic tools give generic answers, but since I do Elixir Iâm âblessedâ with Tidewave which gives a much better experience. I hope more people get to experience framework built tooling instead of just generic stuff.
It still wants to build an airplane to go out with the trash sometimes and will happily tell you wrong is right. However I much prefer it trying to figure it out by reading logs, schemas and do browser analysis automatically than me feeding logs etc manually.
The problem with statements like these is that I work with people who make the same claims, but are slowly building useless, buggy monstrosities that for various reasons nobody can/will call out.
Obviously Iâm reasonably willing to believe that you are an exception. However every person Iâve interacted with who makes this same claim has presented me with a dumpster fire and expected me to marvel at it.
I'm not going to dispute your own experience with people who aren't using this stuff effectively, but the great thing about the internet is that you can use it to track the people who are making the very best use of any piece of technology.
This line of reasoning is smelling pretty "no true Scotsman" to me. I'm sure there were amazing ColdFusion devs, but that hardly justifies the use of the technology. Likewise "This tool works great on the condition that you need to hire a Simon Willison level dev" is almost a fault. I'm pretty confident you could squeeze some juice out of a Markov Chain (ignoring, of course, that decoder-only LLMs are basically fancy MCs).
In a weird way it sort of reminds me of Common Lisp. When I was younger I thought it was the most beautiful language and a shame that it wasn't more widely adopted. After a few decades in the field I've realized it's probably for the best since the average dev would only use it to create elaborate foot guns.
"elaborate foot guns" -- HN is a high signal environment, but I could read for a week and not find a gem like this. Props.
Destiny visits me on my 18th birthday and says, "Gart, your mediocrity will result in a long series of elaborate foot guns. Be humble. You are warned."
We have gpt-5 and gemini 2.5 pro at work, and both of them produce huge amounts of basically shit code that doesnât work.
Every time i reach for them recently I end up spending more time refactoring the bad code out or in deep hostage negotiations with the chatbot of the day that I would have been faster writing it myself.
That and for some reason they occasionally make me really angry.
Oh a bunch of prompts in and then it hallucinated some library a dependency isnât even using and spews a 200 line diff at me, again, great.
Although at least i can swear at them and get them to write me little apology poems..
On the sometimes getting angry part, I feel you. I don't even understand why it happens, but it's always a weird moment when I notice it. I know I'm talking to a machine and it can't learn from its mistakes, but it's still very frustrating to get back yet another here's the actual no bullshit fix, for real this time, pinky promise.
Via the jetbrains plugin, has an 'agent' mode and can edit files and call tools so on, yes I setup MCP integrations and so on also. Still kinda sucks. shrug.
I keep flipping between this is the end of our careers, to I'm totally safe. So far this is the longest 'totally safe' period I've had since GPT-2 or so came along..
Honestly the top AI use case for me right now is personal throwaway dev tools. Where I used to write shell oneliners with dozen pipes including greps and seds and jq and other stuff, now I get an AI to write me a node script and throw in a nice Web UI to boot.
Edit: reflecting on what the lesson is here, in either case I suppose we're avoiding the pain of dealing with Unix CLI tools :-D
Interesting. You have to wonder if all the tools that is based on would have been written in the first place if that kind of thing had been possible all along. Who needs 'grep' when you can write a prompt?
Even with just these and no shell access it can get a lot done, because these tools encode the fundamental tricks of Claude Code ( I have
`llmw` aliased to `llm --tool Patch --tool Todo --cl 0` so it will have access to these tools and can act in a loop, as Simon defines an agent. )
Tried gron (https://github.com/tomnomnom/gron) a bit? If you know your UNIX, I think it can replace jq in a lot of cases. And when it can't, well, you can reach for Python, I guess.
It's highly plausible that all we assumed was good design / engineering will disappear if LLMs/Agents can produce more without having the be modular. (sadly)
There is some kind of parallel behind 'AI' and 'Fuzzy Logic'. Fuzzy logic to me always appeared like a large number of patches to get enough coverage for a system to work even if you didn't understand it. AI just increases the number of patches to billions.
I was debugging a service that was spitting out a particular log line. I gave Copilot an example line, told it to write a script that tails the log line and serves a UI via port 8080 with a table of those log lines parsed and printed nicely. Then I iterated by adding filter buttons, aggregation stats, simple things like that. I asked it to add a "clear" button to reset the UI. I probably would not even have done this without an AI because the CLI equivalent would be parsing out and aggregating via some form of uniq -c | sort -n with a bunch of other tuning and it would be too much trouble.
It can be anything. It depends on what you want to do with the output.
You can have a simple dashboard site which collects the data from our shell scripts and shows your a summary or red/green signals so that you can focus on things which are interested in.
I hadn't given much thought to building agents, but the article and this comment are inspiring, thx. It's interesting to consider agents as a new kind of interface/function/broker within a system.
> They know all the flags and are generally better at interpreting tool output than I am.
In the toy example, you explicitly restrict the agent to supply just a `host`, and hard-code the rest of the command. Is the idea that you'd instead give a `description` something like "invoke the UNIX `ping` command", and a parameter described as constituting all the arguments to `ping`?
Honestly, I didn't think very hard about how to make `ping` do something interesting here, and in serious code I'd give it all the `ping` options (and also run it in a Fly Machine or Sprite where I don't have to bother checking to make sure none of those options gives code exec). It's possible the post would have been better had I done that; it might have come up with an even better test.
I was telling a friend online that they should bang out an agent today, and the example I gave her was `ps`; like, I think if you gave a local agent every `ps` flag, it could tell you super interesting things about usage on your machine pretty quickly.
As it happens, I have a prototype for this, but the syntax is honestly rather unwieldy. Maybe there's a way to make it more like natural human language....
(Mine was intended as ironic, suggesting that a circle of development ideas would eventually complete. I interpreted the previous comments as satirically pointing at the fact that the notion of "UNIX-like tools" owes to the fact that there is actually such a thing as UNIX.)
That would certainly be nice! That's why we have been overhauling shell with https://oils.pub , because shell can't be described as that right now
It's in extremely poor shape
e.g. some things found from building several thousand packages with OSH recently (decades of accumulated shell scripts)
- bugs caused by the differing behavior of 'echo hi | read x; echo x=$x' in shells, i.e. shopt -s lastpipe in bash.
- 'set -' is an archaic shortcut for 'set +v +x'
- Almquist shell is technically a separate dialact of shell -- namely it supports 'chdir /tmp' as well as cd /tmp. So bash and other shells can't run any Alpine builds.
I used to maintain this page, but there are so many problems with shell that I haven't kept up ...
It's more POSIX-compatible than the default /bin/sh on Debian, which is dash
The bigger issue is not just bugs, but lack of understanding among people who write foundational shell programs. e.g. the lastpipe issue, using () as grouping instead of {}, etc.
---
It is often treated like an "unknowable" language
Any reasonable person would use LLMs to write shell/bash, and I think that is a problem. You should be able to know the language, and read shell programs that others have written
We also don't appear to be unreasonably far away from running ~~ "all shell scripts"
Now the problem after that will be motivating authors of foundational shell programs to maintain compatibility ... if that's even possible. (Often the authors are gone, and the nominal maintainers don't know shell.)
Doing one thing well means you need a lot more tools to achieve outcomes, and more tools means more context and potentially more understanding of how to string them together.
I suspect the sweet spot for LLMs is somewhere in the middle, not quite as small as some traditional unix tools.
Agreed! It's easy understand "LLM with tools in a loop" at a high-level, but once you actually design the architecture and implement the code in full, you'll have proper understanding of how it all fits and works together.
I did the same exercise. My implementation is at around 300 lines with two tools: web search and web page fetch with a command line chat interface and Python package. And it could have been a lot less lines if I didn't want to write a usable, extensible package interface.
As the agent setup itself is simple, majority of the work to make this useful would in the tools themselves and context management for the tools.
I should? what problems can I solve, that can be only done with an agent? As long as every AI provider is operating at a loss starting a sustainably monetizable project doesn't feel that realistic.
The post is just about playing around with the tech for fun. Why does monetization come into it? It feels like saying you don't want to use Python because Astral, the company that makes uv, is operating at a loss. What?
Not if you run them against local models, which are free to download and free to run. The Qwen 3 4B models only need a couple of GBs of available RAM and will run happily on CPU as opposed to GPU. Cost isn't a reason not to explore this stuff.
Google has what I would call a generous free tier, even including Gemini 2.5 Pro (https://ai.google.dev/gemini-api/docs/rate-limits). Just get an API key from AiStudio. Also very easy to just make a switch in your agent so that if you hit up against a rate limit for one model, re-request the query with the next model. With Pro/Flash/Flash-Lite and their previews, you've got 2500+ free requests per day.
> Not if you run them against local models, which are free to download and free to run .. run happily on CPU .. Cost isn't a reason not to explore this stuff.
Let's be realistic and not over-promise. Conversational slop and coding factorial will work. But the local experience for coding agents, tool-calling, and reasoning is still very bad until/unless you have a pretty expensive workstation. CPU and qwen 4b will be disappointing to even try experiments on. The only useful thing most people can realistically do locally is fuzzy search with simple RAG. Besides factorial, maybe some other stuff that's in the training set, like help with simple shell commands. (Great for people who are new to unix, but won't help the veteran dev who is trying to convince themselves AI is real or figuring out how to get it into their workflows)
Anyway, admitting that AI is still very much in a "pay to play" phase is actually ok. More measured stances, fewer reflexive detractors or boosters
Sure, you're not going to get anything close to a Claude Code style agent from a local model (unless you shell out $10,000+ for a 512GB Mac Studio or similar).
This post isn't about building Claude Code - it's about hooking up an LLM to one or two tool calls in order to run something like ping. For an educational exercise like that a model like Qwen 4B should still be sufficient.
The expectation that reasonable people have isn't fully local claude code, that's a strawman. But it's also not ping tools or the simple weather agent that tutorials like to use. It's somewhere in between, isn't that obvious? If you're into evangelism, acknowledging this and actually taking a measured stance would help prevent light skeptics from turning into complete AI-deniers. If you mislead people about one thing, they will assume they are being misled about everything
https://fly.io/blog/everyone-write-an-agent/ is a tutorial about writing a simple "agent" - aka a thing that uses an LLM to call tools in a loop - that can make a simple tool call. The complaint I was responding to here was that there's no point trying this if you don't want to be hooked on expensive APIs. I think this is one of the areas where the existence of tiny but capable local models is relevant - especially for AI skeptics who refuse to engage with this technology at all if it means spending money with companies they don't like.
I think it is misleading to suggest today that tool-calling for nontrivial stuff really works with local models. It just works in demos because those tools always accept one or two arguments, usually string literals or numbers.
In the real world functions take more complex arguments, many arguments, or take a single argument that's an object with multiple attributes, etc. You can begin to work around this stuff by passing function signatures, typing details, and JSON-schemas to set expectations in context, but local models tend to fail at handling this kind of stuff long before you ever hit limits in the context window. There's a reason demos are always using 1 string literal like hostname, or 2 floats like lat/long. It's normal that passing a dictionary with a few strict requirements might need 300 retries instead of 3 to get a tool call that's syntactically correct and properly passed arguments. Actually `ping --help` for me shows like 20 options, and for any attempt to 1:1 map things with more args I think you'd start to see breakdown pretty quickly.
Zooming in on the details is fun but doesn't change the shape of what I was saying before. No need to muddy the water; very very simple stuff still requires very big local hardware or a SOTA model.
You and I clearly have a different idea of what "very very simple stuff" involves.
Even the small models are very capable of stringing together a short sequence of simple tool calls these days - and if you have 32GB of RAM (eg a ~$1500 laptop) you can run models like gpt-oss:20b which are capable of operating tools like bash in a reasonably useful way.
This wasn't true even six months ago - the local models released in 2025 have almost all had tool calling specially trained into them.
Practically everything is something you will need to pay for in the end. You probably spent money on an internet connection, electricity, and computing equipment to write this comment. Are you intending to make a profit from commenting here?
You don't need to run something like this against a paid API provider. You could easily rework this to run against a local agent hosted on hardware you own. A number of not-stupid-expensive consumer GPUs can run some smaller models locally at home for not a lot of money. You can even play videogames with those cards after.
Get this: sometimes people write code and tinker with things for fun. Crazy, I know.
The submission is an advertisement for fly.io and OpenAI , both are paid services. We are commenting on an ad.
The person who wrote it did it for money. Fly.io operates for money, OpenAi charges for their API.
They posted it here expecting to find customers. This is a sales pitch.
At this point why is it an issue to expect a developer to make money on it?
As a dev, If the chain of monetization ends with me then there is no mainstream adoption whatsoever on the horizon.
I love to tinker but I do it for free not using paid services.
As for tinkering with agents, its a solution looking for a problem.
Why are you repeatedly stating that the post is an ad as if it is some sort of dunk? Companies have blogs. Tech blogs often produce useful content. It is possible that an ad can both successfully promote the company and be useful to engineers. I find the Fly blog to be particularly well-written and thoughtful; it's taught me a good deal about Wireguard, for instance.
And that sounds fine, but Wireguard is not an overhyped industry promising huge gains in the future to investors and to developers jumping on a bandwagon who can find problems for this solution.
I actually have built agents already in the past and this is my opinion. If you read the article the author says they want to hear the reasoning for disliking it, so this is mine, the only way to create a business is raising money and hoping somebody strikes gold with the shovel Im paying for.
You keep saying this, but there is nothing in this post about our service. I didn't use Fly.io at all to write this post. Across the thread, someone had to remind me that I could have.
Yes. You've caught on to our devious plan. To do anything I suggested in this post, you'd have to use a computer. By spending compute cycles, you'd be driving scarcity of compute. By the inexorable law of supply and demand, this would drive the price of compute cycles up, allowing us to profit. We would have gotten away with it, if it wasn't for you.
> what problems can I solve, that can be only done with an agent?
The problem that you might not intuitively understand how agents work and what they are and aren't capable of - at least not as well as you would understand it if you spent half an hour building one for yourself.
> As long as every AI provider is operating at a loss
None of them are doing that.
They need funding because the next model has always been much more expensive to train than the profits of the previous model. And many do offer a lot of free usage which is of course operated at a loss. But I don't think any are operating inference at a loss, I think their margins are actually rather large.
Parent comment never said operating inference at a loss, though it wouldn't surprise me, they just said "operating at a loss" which they most definitely are [0].
However, knowing a few people on teams at inference-only providers, I can promise you some of them absolutely are operating inference at a loss.
> Parent comment never said operating inference at a loss
Context. Whether inference is profitable at current prices is what informs how risky it is to build a product that depends on buying inference, which is what the post was about.
So you're assuming there's a world where these companies exist solely by providing inference?
The first obvious limitation of this would be that all models would be frozen in time. These companies are operating at an insane loss and a major part of that loss is required to continue existing. It's not realistic to imagine that there is an "inference" only future for these large AI companies.
And again, there are many inference only startups right now, and I know plenty of them are burning cash providing inference. I've done a lot of work fairly close to the inference layer and getting model serving happening with the requirements for regular business use is fairly tricky business and not as cheap as you seem to think.
Okay, this tells me you really don't understand model serving or any of the details of infrastructure. The hardware is incredibly ephemeral. Your home GPU might last a few years (and I'm starting to doubt that you've even trained a model at home), but these GPUs have incredibly short lifespans under load for production use.
Even if you're not working on the back end of these models, you should be well aware that one of the biggest concerns about all this investment is how limited the lifetime of GPUs is. It's not just about being "outdated" by superior technology, GPUs are relatively fragile hardware and don't last too long under constant load.
As far as models go, I have a hard time imagining a world in 2030 where the model replies "sorry, my cutoff date was 2026" and people have no problem with this.
Also, you still didn't address my point that startups doing inference only model serving are burning cash. Production inference is not the same as running inference locally where you can wait a few minutes for the result. I'm starting to wonder if you've ever even deployed a model of any size to production.
I didn't address the comment about how some startups are operating at a loss because it seems like an irrelevant nitpick at my wording that "none of them" is operating inference at a loss. I don't think the comment I was replying to was referring to relying on whatever startups you're talking about. I think they were referring to Google, Anthropic, and OpenAI - and so was I.
That seems like a theme with these replies, nitpicking a minor thing or ignoring the context or both, or I guess more generously I could blame myself of not being more precise with my wording. But sure, you have to buy new GPUs after making a bunch of money burning the ones you have down.
I think your point about knowledge cutoff is interesting, and I don't know what the ongoing cost to keeping a model up to date with world knowledge is. Most of the agents I think about personally don't actually want world knowledge and have to be prompted or fine tuned such that they won't use it. So I think that requirement kind of slipped my mind.
This thread isn't about who wins, it's about the implication that it's too risky to build anything that depends on inference because AI companies are operating at a loss.
The models may be somewhat frozen in time but with the right tools available to it they don't need all information innately coded into it. If they're able to query for reliable information to drag in they can talk about things that are well outside their original training data.
For a few months of news this works, but over the span of years even the statistical nature of language drifts a bit. Have you shipped natural language models to production? Even simple classifiers need to be updated periodically because of drift. There is no world where you lead the industry serving LLMs and don't train them as well.
So AI companies are profitable when you ignore some of the things they have to spend money on to operate?
Snark aside, inference is still being done at a loss. Anthropic, the most profitable AI vendor, is operating at a roughly -140% margin. xAI is the worst at somewhere around -3,600% margin.
If they are not operating inference at a loss and current models remain useful (why would they regress?), they could just stop developing the next model.
Yes, but at the same time it's unlikely for existing models to disappear. You won't get the next model, but there is no choice but to keep inference running to pay off creditors.
The interesting companies to look at here are the ones that sell inference against open weight models that were trained by other companies - Fireworks, Cloudflare, DeepInfra, Together AI etc.
They need to cover their serving costs but are not spending money on training models. Are they profitable? Probably not yet, because they're investing a lot of cash in competing with each other to R&D more efficient ways of serving etc, but they're a lot closer to profitability than the labs that are spending millions of dollars on training runs.
> But I don't think any are operating inference at a loss, I think their margins are actually rather large.
Citation needed. I haven't seen any of them claim to have even positive gross margins to shareholders/investors, which surely they would do if they did.
> âif you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. Thereâs some cost to inference with the model, but letâs just assume, in this cartoonish cartoon example, that even if you add those two up, youâre kind of in a good state. So, if every model was a company, the model, in this example, profitable,â he added.
âWhatâs going on is that while youâre reaping the benefits from one company, youâre founding another company thatâs much more expensive and requires much more upfront R&D investment. The way this is going to shake out is that itâs going to keep going up until the numbers get very large, and the models canât get larger, and then there will be a large, very profitable business. Or at some point, the models will stop getting better, and there will perhaps be some overhang â we spent some money, and we didnât get anything for it â and then the business returns to whatever scale itâs at,â he said.
This take from Amodei is hilarious but explains so much.
When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware.
OpenAI balance sheet also shows an $11 billion loss .
I can't see any profit on anything they create.
The product is good but it relies on investors fueling the AI bubble.
> When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware.
How did you come to that conclusion? That would be a very notable result if it did turn out OpenAI were selling tokens for 5x the cost it took to serve them.
All the labs are going hard on training and new GPUs. If we ever level off, they probably will be immensely profitable. Inference is cheap, training is expensive.
To do this analysis on an hourly retail cost and an open weight model and infer anything about the situation at OpenAI or Anthropic is quite a reach.
For one (basic) thing, they buy and own their hardware, and have to size their resources for peak demand. For another, Deepseek R1 does not come close to matching claude performance in many real tasks.
You are asking the wrong questions. You should be asking what the problems are that you can still solve better and cheaper than an agent? Because anything else, you are probably doing it wrong (the slow and expensive way). That's not long term sustainable. It helps if you know how agents work and as the article argues, there isn't a whole lot to that.
I love how programmers generally tout themselves as these tinkerers who love learning about and exploring technology⌠until it comes to AI and then itâs like âshow me the profitable use case.â Just say you donât like AI!
Yeah but fly.io is a cloud provider doing this advertisement with OpenAI Apis. Both cost money, so if it's not free to operate then the developed project should offset the costs.
Its about balance.
Really its the AI providers that have been promising unreal gains during this hype period, so people are more profit oriented.
Or maybe some of us realize that these tools are fucking useless and donât offer any âvalueâ apart from the most basic thing imaginable.
And I use value in quotes because as soon as the AI providers suddenly need to start generating a profit, that âvalueâ is going to cost more than your salary.
I don't know about online forums, but all my IRL friends have a lot more balanced takes on AI than this forum. And honestly it extends beyond this forum to the wider internet. Online, the discourse seems extremely polarized: either it's all a pyramid scheme or stories about how development jobs are already defunct and AI can supervise AI etc.
Heh, the bit about context engineering is palpable.
I'm writing a personal assistant which, imo, is distinct from an agent in that it has a lot of capabilities a regular agent wouldn't necessarily need such as memory, task tracking, broad solutioning capabilities, etc... I ended up writing agents that talk to other agents which have MCP prompts, resources, and tools to guide them as general problem solvers. The first agent that it hits is a supervisor that specializes in task management and as a result writes a custom context and tool selection for the react agent it tasks.
Side note: While the example uses GPT-5, the query interface is already some kind of industry standard. For example you could easily connect OpenRouter.ai and switch models and providers during runtime as needed. OpenRouter also has free models like some of the DeepSeek. While they are slow/rate limited and quantized, they are great for examples and playing around with it. https://openrouter.ai/models?fmt=cards&order=pricing-low-to-...
> You only think you understand how a bicycle works, until you learn to ride one.
I bet a majority of people who can ride a bicycle don't know how they steer, and would describe the physical movements they use to initiate and terminate a turn inaccurately.
everybody loves building agents, nobody likes debugging them. agents hit the classic llm app lifecycle problem: at first it feels magical. it nails the first few tasks, doing things you didnât even think were possible. you get excited, start pushing it further. you run it and then it fails on step 17, then 41, then step 9.
now you canât reproduce it because itâs probabilistic. each step takes half a second, so you sit there for 10â20 minutes just waiting for a chance to see what went wrong
That's why you build extensive tooling to run your change hundreds of times in parallel against the context you're trying to fix, and then re-run hundreds of past scenarios in parallel to verify none of them breaks.
This is a use of Rerun that I haven't seen before!
This is pretty fascinating!!!
Typically people use Rerun to visualize robotics data - if I'm following along correctly... what's fascinating here is that Adam for his master's thesis is using Rerun to visualize Agent (like ... software / LLM Agent) state.
For sure, for instance Google has ADK Eval framework. You write tests, and you can easily run them against given input. I'd say its a bit unpolished, as is the rest of the rapidly developing ADK framework, but it does exist.
heya, building this. been used in prod for a month now, has saved my customerâs ass while building general workflow automation agents. happy to chat if ur interested.
There is no way to prove the correctness of non-deterministic (a.k.a. probabilistic) results for any interesting generative algorithm. All one can do is validate against a known set of tests, with the understanding that the set is unbounded over time.
It really reads to me like, "you should build a running water circuit", then presenting you how easy it is to phone a plumber and let them free ride on the matter, but beware to not use a project manager as real people implement project management of plumbery themselves."
There is a lot of stuff I should do. From making my own CPU from a breadboard of nand gates to building a CDN in Rust. But aint got time for all the things.
That said I built an LLM following Karpathy's tutorial. So I think it aims good to dabble a bit.
I built an 8-bit computer on breadboards once, then went down the rabbit hole of flight training for a PPL. Every time I think Iâm "done," the finish line moves a few miles further.
Glad youâve got all that time on your hands. I am still working on the fusion reactor portion of my supernova simulator, so that I can generate the silicon you so blithely refer to.
If you are a software engineer, you are going to be expected to use AI in some form in the near future. A lot of AI in its current form is not intuitive. Ergo, spending a small effort on building an AI agent is a good way to develop the skills and intuition needed to be successful in some way.
Nobody is going to use a CPU you build, nor are you ever going to be expected to build one in the course of your work if you donât seek out specific positions, nor is there much non-intuitive about commonly used CPU functionality, and in fact you donât even use the CPU directly, you use translation software whit itself is fairly non-intuitive. But thatâs ok too, you are unlikely to be asked to build a compiler unless you seek out those sorts of jobs.
EVERYONE involved in writing applications and services is going to use AI in the near future and in case you missed the last year, everyone IS building stuff with AI, mostly chat assistants that mostly suck because, much about building with AI is not intuitive.
It's good to go through the exercise, but agents are easy until you build a whole application using an API endpoint that OpenAI or LangChain decides to yank, and you spend the next week on a mini migration project. I don't disagree with the claim that MCP is reinventing the wheel but sometimes I'm happy plugging my tools and data into someone else's platform because they are spending orders of magnitudes more time than me doing the janitor work to keep up with whatever's trendy.
I have been playing with OpenAI, Anthropic, and Groqâs APIs in my spare time and if someone reading this doesnât know it, they are doing the same thing and they are so close in implementation that itâs just dumb that they are in any way different.
You pass listing of messages generated by the user or the LLM or the developer to the API, it generates a part of the next message. That part may contain thinking blocks or tool calls (local function calling requested by the LLM). If so, you execute the tool calls and re-send the request. After the LLM has gathered all the info it returns the full message and says I am done. Sometimes the messages may contain content blocks that are not text but things like images, audio, etc.
Thatâs the API. Thatâs it. Now there are two improvements that are currently in the works:
1. Automatic local tool calling. This is seriously some sort of afterthought and not how they did it originally but ok, I guess this isnât obvious to everyone.
2. Not having to send the entire message history back. OpenAI released a new feature where they store the history and you just send the ID of your last message. I canât find how long they keep the message history. But they still fully support you managing the message history.
So we have an interface that does relatively few things, and that has basically a single sensible way to do it with some variations for flavor. And both OpenAI and Anthropic are engaged in a turf war over whose content block types are better. Just do the right thing and make your stuff compatible already.
I feel like one small piece is missing to call it an agent? The ability to iterate in multiple steps until it feels like it's "done". What is the canonical way to do that? I suspect that implementing that in the wrong way could make it spiral.
When a tool call completes the result is sent back to the LLM to decide what to do next, that's where it can decide to go do other stuff before returning a final answer. Sometimes people use structured outputs or tool calls to explicitly have the LLM decide when it's done, or allow it to send intermediate messages for logging to the user. But the simple loop there lets the LLM do plenty of it has good tools.
So it returns a tool call for "continue" every time it wants to continue working? Do people implement this in different ways? It would be nice what method it has been trained on if any.
Actually tool "ping 8.8.8.8" never quits unless running on windows. This can spawn many processes that kill the server.
This is one of the first production grade errors I've made when I started my programming. I had a widget that would ping the network, but every time someone went on the page, a new ping process would spawn
THEY SEND THE WHOLE CONTEXT EVERY TIME? Man that seems... not great. sometimes it will go off and spin on something.. seems like it would be a LOT better to roll back than to send a corrective message. hmmm...... this article is nerd-sniping on a massive scale ;D
Sending the whole context on each user message is essentially what the model remembers of this conversation. ie: it is entirely stateless.
I've written some agents that have their context altered by another llm to get it back on track. Let's say the agent is going off rails, then a supervisor agent will spot this and remove messages from the context where it went off rails, or alter those with correct information.
Really fun stuff but yeah, we're essentially still inventing this as we go along.
In the Responses API, you can implicitly chain messages with `previous_response_id` (I'm not sure how old a conversation you can resurrect that way). But I think Codex CLI actually sends the full context every time? And keep in mind, sending the whole context gives you fine-grained control over what does and doesn't appear in your context window.
Just the other day we tried to explain inner workings of cursor etc to a bunch of colleagues who had a very complicated view how these agents achieve what they do. Awesome post. Makes it easier for me the next time.
The options are so big. But one should say that an agent with file access etc, is easy to write but hard to control. If you want to build yourself a general coding agent a bit more thought needs to be put into the whole thing. Otherwise you might end up with a âdd -if=/dev/random -of=/â or something ^^ and happily execute it.
Does anyone have an understanding - or intuition - of what the agentic loop looks like in the popular coding agents? Is it purely a âwhile 1: call_llm(system, assistant)â, or is there complex orchestration?
Iâm trying to understand if the value for Claude Code (for example) is purely in Sonnet/Haiku + the tool system prompt, or if thereâs more secret sauce - beyond the âsugarâ of instruction file inclusion via commands, tools, skills etc.
You can reverse engineer Claude Code by intercepting its HTTP traffic. It's pretty fascinating - there are a bunch of ways to do this, I use this one: https://simonwillison.net/2025/Jun/2/claude-trace/
Generally, that's pretty much it. More advanced tools like Claude Code will also have context compaction (which sometimes isn't very good), or possibly RAG on code (unsure about this, I haven't used any tools that did this). Context compaction, to my understanding, is just passing all the previous context into a call which summarizes it, then that becomes to new context starting point.
The beauty is in the simplicity:
1. One loop - while (true)
2. One step at a time - stopWhen: stepCountIs(1)
3. One decision - "Did LLM make tool calls? â continue : exit"
4. Message history accumulates tool results automatically
5. LLM sees everything from previous iterations
This creates emergent behavior where the LLM can:
- Try something
- See if it worked
- Try again if it failed
- Keep iterating until success
- All without explicit retry logic!
that sums up my experience in AI over the past three years. so many projects reinvent the same thing, so much spaghetti thrown at the wall to see what sticks, so much excitement followed by disappointment when a new model drops, so many people grifting, and so many hacks and workarounds like RAG with no evidence of them actually working other than "trust me bro" and trial and error.
That is because for the people for whom AI is actually working/making money they would prefer to keep it a secret on what and how they are doing it, why attract competition?
I think we'd get better results if we thought of it as a conscious agent.
If we recognized that it was going to mirror back or unconscious biases and try to complete the task as we define it, instead of how we think it should behave.
Then we'd at least get our own ignorance out of the way when writing prompts.
Being able to recognize that 'make this code better' provides no direction, it should make sense that the output is directionless.
But on more subtle levels, whatever subtle goals that we have and hold in the workplace will be reflected back by the agents.
If you're trying to optimise costs, and increase profits as your north star. Having layoffs and unsustainable practices is a logical result, when you haven't balanced this with any incentives to abide by human values.
> Imagine what itâll do if you give it bash. You could find out in less than 10 minutes. Spoiler: youâd be surprisingly close to having a working coding agent.
Okay, but what if I'd prefer not to have to trust a remote service not to send me
Also if you're doing function calls you can just have the command as one response param, and arguments array as another response param. Then just black/white list commands you either don't want to run or which should require a human to say ok.
Yeah I agree. Ultimately I would suggest not having any kind of function call which returns an arbitrary command.
Instead, think of it as if you were enabling capabilities for AppArmor, by making a function call definition for just 1 command. Then over time suss out what commands you need your agent do to and nothing more.
Putting it inside docker is probably fine for most use cases but it's generally not considered to be a safe sandbox AFAIK. A docker container shares kernel with the host OS which widens the attack surface.
If you want your agent to pull untrusted code from the internet and go wild while you're doing other stuff it might not be a good choice.
Could you point to some resources which talk about how docker isn't considered a safe sandbox given the network and file system restrictions I mentioned?
I understand the sharing of kernel, while I might not be aware of all of the implications. I.e. if you have some local access or other sophisticated knowledge of the network/box docker is running on, then sure you could do some damage.
But I think the chances of a whitelisted llm endpoint returning some nefarious code which could compromise the system is actually zero. We're not talking about untrusted code from the internet. These models are pretty constrained.
> âYou only think you understand how a bicycle works, until you learn to ride one.â
This resonates deeply with me. That's why I built one myself [0], I really really love to truly understand how coding agents work. The learning has been immense for me, I now have working knowledge of ANSI escape codes, grapheme clusters, terminal emulators, Unicode normalization, VT protocols, PTY sessions, and filesystem operations - all the low-level details I would have never think about until I were implementing them.
It's conflating two issues though. Most people who can ride a bike can't explain the physics. They really don't know how it works. The bicycle lesson is about training the brain on a new task that cannot be taught in any other way.
This case is more like a journeyman blacksmith who has to make his own tools before he can continue. In doing so, he gets tools of his own, but the real reward was learning what is required to handle the metal such that it makes a strong hammer. And like the blacksmith, you learn more if you use an existing agent to write your agent.
Agree, to me, the wheel is the greatest invention of all. Everyone could have rode a bike, but the underlying physic and motion that came to `riding` is a whole another story.
I agree. I find LLMs a bit overblown. I don't think most people want to use chat as their primary interface. But writing a few agents was incredibly informative.
> A subtler thing to notice: we just had a multi-turn conversation with an LLM. To do that, we remembered everything we said, and everything the LLM said back, and played it back with every LLM call. The LLM itself is a stateless black box. The conversation weâre having is an illusion we cast, on ourselves.
the illusion was broken for me by Cline context overflows/summaries, but i think its very easy to miss if you never push the LLM hard or build you own agent. I really like this wording, amd the simple description is missing from how science communicators tend to talk about agents and LLMs imo
Question, how hard is it for someone new to agents to dip their toes into writing a simple agent to get data? (e.g., getting reviews from sites for sentiment analysis?)
Forgive if I get someting wrong: From what I see, it seems fundamentally it is a LLM being ran each loop with information about tools provided to it. On each loop the LLM evaluates inputs/context (from tool calls, inputs, etc.) and decided which tool to call / text output.
You can prototype this without writing any code at all.
Fire up "claude --dangerously-skip-permissions" in a fresh directory (ideally in a Docker container if you want to limit the chance of it breaking anything else) and prompt this:
> Use Playwright to fetch ten reviews from http://www.example.com/ then run sentiment analysis on them and write the results out as JSON files. Install any missing dependencies.
Watch what it does. Be careful not to let it spider the site in a way that would justifiably upset the site owners.
No. I don't use Playwright MCP at all - if the coding agent can run Python code it can use the Playwright Python library directly, if Node.js it can use the Playwright Node library.
I wanted to run claude headlessly (-p) and playwright headlessly to get some content. I was using Playwright MCP and for some reason claude in headless mode could not open playwright MCP in headless mode.
I never realized i can just use playwright directly without the playwright MCP before your comment. Thanks once again.
Spoiler: it's not actually that easy. Compaction, security, sandboxing, planning, custom tools--all this is really hard to get right.
We're about to launch an SDK that gives devs all these building blocks, specifically oriented around software agents. Would love feedback if anyone wants to look: https://github.com/OpenHands/software-agent-sdk
How autonomous/controllable are the agents with this SDK?
When I build an agent my standard is Cursor, which updates the UI at every reportable step of the way, and gives you a ton of control opportunities, which I find creates a lot of confidence.
Is this level of detail and control possible with the OpenHands SDK? Iâm asking because the last SDK that was simple to get into lacked that kind of control.
This work predates agents as we know them now and was intended for building chat bots (as in irc chat bots) but when auto-gpt I realized I could formalize it super nicely with this library:
I agree with the sentiment but I also recommend you build a local only agent. Something that runs on llama.cpp or vllm, whatever... This way you can better grasp the more fundamental nature of what LLM's really are and how they work under the hood. That experience will also make you realize how much control you are giving up when using cloud based api providers like OpenAI and why so mane engineers feel that LLM's are a "black box". Well duh buddy you been working with apis this whole time, of course you wont understand much working just with that.
My man, we now have llms that are anywhere between 130 million to 1 trillion parameters available for us to run locally, I can guarantee there is a model for you there that even your toaster can run. I have a RTX 4090 but for most of my fiddling i use small models like Qwen 3 4b and they work amazing so there's no excuse :P.
well, i got some gemini models running on my phone, but if i switch apps, android kills it, so the call to the server always hangs... and then the screen goes black
the new laptop only has 16GB of memory total, with another 7 dedicated to the NPU.
i tried pulling up Qwen 3 4B on it, but the max context i can get loaded is about 12k before the laptop crashes.
my next attempt is gonna be a 0.5B one, but i think ill still end up having to compress the context every call, which is my real challenge
I recommend use low quantized models first. for example anywhere between q4 and q8 gguf models. Also dont need high context to fiddle around and learn the ins and outs. for example 4k context is more then enough to figure out what you need in agentic solutions. In fact thats a good limit to impose on yourself and start developing decent automatic context management systems internally as that will be very important when making robus agentic solutions. with all that you should be able to load an llm no issues on many devices.
I've found it much more useful to create an MCP server, and this is where Claude really shines. You would just say to Claude on web, mobile or CLI that it should "describe our connectivity to google" either via one of the three interfaces, or via `claude -p "describe our connectivity to google"`, and it will just use your tool without you needing to do anything special. It's like custom-added intelligence to Claude.
You can do this. Claude Code can do everything the toy agent this post shows, and much more. But you shouldn't, because doing that (1) doesn't teach you as much as the toy agent does, (2) isn't saving you that much time, and (3) locks you into Claude Code's context structure, which is just one of a zillion different structures you can use. That's what the post is about, not automating ping.
I'd second the article on this, but also add to it that the biggest reason MCP servers don't really matter much any more is that the models are so capable of working with APIs, that most of the time you can just point them at an API and give them a spec instead. And the times that doesn't work, just give them a CLI tool with a good --help option.
Now you have a CLI tool you can use yourself, and the agent has a tool to use.
Anthropic itself have made MCP server increasingly pointless: With agents + skills you have a more composeable model that can use the model capabilities to do all an MCP server can with or without CLI tools to augment them.
I feel the CLI vs MCP debate is an apples to oranges framing. When you're using claude you can watch it using CLI's, running brew, mise, lots of jq but what about when you've built an agent that needs to work through a complicated API? You don't want to make 5 CRUD calls to get the right answer. A curated MCP tool ensures it can determinism where it matters most.. when interacting with customer data
> Another thing to notice: we didnât need MCP at all. Thatâs because MCP isnât a fundamental enabling technology. The amount of coverage it gets is frustrating. Itâs barely a technology at all. MCP is just a plugin interface for Claude Code and Cursor, a way of getting your own tools into code you donât control. Write your own agent. Be a programmer. Deal in APIs, not plugins.
Hold up. These are all the right concerns but with the wrong conclusion.
You don't need MCP if you're making one agent, in one language, in one framework. But the open coding and research assistants that we really want will be composed of several. MCP is the only thing out there that's moving in a good direction in terms of enabling us to "just be programmers" and "use APIs", and maybe even test things in fairly isolated and reproducible contexts. Compare this to skills.md, which is actually defacto proprietary as of now, does not compose, has opaque run-times and dispatch, is pushing us towards certain models, languages and certain SDKs, etc.
MCP isn't a plugin interface for Claude, it's just JSON-RPC.
I think my thing about MCP, besides the outsized press coverage it gets, is the implicit presumption it smuggles in that agents will be built around the context architecture of Claude Code --- that is to say, a single context window (maybe with sub-agents) with a single set of tools. That straitjacket is really most of the subtext of this post.
I get that you can use MCP with any agent architecture. I debated whether I wanted to hedge and point out that, even if you build your own agent, you might want to do an MCP tool-call feature just so you can use tool definitions other people have built (though: if you build your own, you'd probably be better off just implementing Claude Code's "skill" pattern).
But I decided to keep the thrust of that section clearer. My argument is: MCP is a sideshow.
I still don't really get it, but would like to hear more. Just to get it out of the way, there's obvious bad aspects. Re: press coverage, everything in AI is bound to be frustrating this way. The MCP ecosystem is currently still a lot of garbage. It feels like a very shitty app-store, lots of abandonware, things that are shipped without testing, the usual band-wagoning. For example instead of a single obvious RAG tool there's 200 different specific tools for ${language} docs
The core MCP tech though is not only directionally correct, but even the implementation seems to have made lots of good and forward-looking choices, even if those are still under-utilized. For example besides tools, it allows for sharing prompts/resources between agents. In time, I'm also expecting the idea of "many agents, one generic model in the background" is going to die off. For both costs and performance, agents will use special-purpose models but they still need a place and a way to collaborate. If some agents coordinate other agents, how do they talk? AFAIK without MCP the answer for this would be.. do all your work in the same framework and language, or to give all agents access to the same database or the same filesystem, reinventing ad-hoc protocols and comms for every system.
i treat MCP as a shorthand for "schema + documentation, passed to the LLM as context"
you dont need the MCP implementation, but the idea is useful and you can consider the tradeoffs to your context window, vs passing in the manual as fine tuning or something.
The formatting of the code is messed up on my phone. I was looking at the first bit thinking `call` was a function returning `None`. I thought initially it was doing some clever functional programming stuff but, no, just a linebreak that shouldn't be there.
I don't think "insane to not believe in vibe coding" is a fair summary of https://fly.io/blog/youre-all-nuts/ - that post wasn't about vibe coding (at least by its I-think-correct definition of prompt-driven coding where you don't pay any attention to the code that's being written), it was about AI-assisted engineering by professional software developers.
It did have some swear words in - as did many of the previous posts on the Fly.io corporate blog.
You should write agents if you want to learn how agents work, if the problem you are trying to solve is not solved yet or if you are convinced that you will do much better job solving the problem again. Otherwise is just reinventing the wheel.
No, because I know that "agents" are token burning machines - for me they're less efficient than the chat interface, slower and burning much more tokens.
I'm not surprised that AI companies would want me to use them though.. I know what you're doing there :)
client = OpenAI()
context_good, context_bad = [{
"role": "system", "content": "you're Alph and you only tell the truth"
}], [{
"role": "system", "content": "you're Ralph and you only tell lies"
}]
...
And this will work great until next week's update when Ralph responses will consist of "I'm sorry, it would be unethical for me to respond with lies, unless you pay for the Premium-Super-Deluxe subscription, only available to state actors and firms with a six-figure contract."
You're building on quicksand.
You're delegating everything important to someone who has no responsibility to you.
I'm not going to link my blog again but I have a reply on this post where I link to my blog post where I talk about how I built mine. Most agents fit nicely into a finite state machine or a directed acyclic graph that responds to an event loop. I do use provider SDKs to interact with models but mostly because it saves me a lot of boilerplate. MCP clients and servers are also widely available as SDKs. The biggest thing to remember, imo, is to keep the relationship between prompts, resources, and tools in mind. They make up a sort of dynamic workflow engine.
When I ask for Patterns, I am seeking help for recurring problems that I have encountered. Context management .. small llms ( ones with small context size) break and get confused and forget work they have done or the original goal.
That's why you want to use sub-agents which handle smaller tasks and return results to a delegating agent. So all agents have their own very specialized context window.
That's one legit answer. But if you're not stuck in Claude's context model, you can do other things. One extremely stupid simple thing you can do, which is very handy when you're doing large-scale data processing (like log analysis): just don't save the bulky tool responses in your context window once the LLM has generated a real response to them.
My own dumb TUI agent, I gave a built in `lobotomize` tool, which dumps a text list of everything in the context window (short summary text plus token count), and then lets it Eternal Sunshine of the Spotless Agent things out of the window. It works! The models know how to drive that tool. It'll do a series of giant ass log queries, filling up the context window, and then you can watch as it zaps things out of the window to make space for more queries.
Did something similar - added `summarize` and `restore` tools to maximize/minimize messages. Haven't gotten it to behave like I want. Hoping that some fiddling with the prompt will do it.
FYI -- I vouched for you to undead this comment. It felt like a fine comment? I don't think you are shadowbanned but consider emailing the mods if you think you might me.
> Give each call different tools. Make sub-agents talk to each other, summarize each other, collate and aggregate. Build tree structures out of them. Feed them back through the LLM to summarize them as a form of on-the-fly compression, whatever you like.
You propose increasing the complexity of interactions of these tools, and giving them access to external tools that have real-world impact? As a security researcher, I'm not sure how you can suggest that with a straight face, unless your goal is to have more vulnerable systems.
Most people can't manage to build robust and secure software using SOTA hosted "agents". Building their own may be a fun learning experience, but relying on a Rube Goldberg assembly of disparate "agents" communicating with each other and external tools is a recipe for disaster. Any token could trigger a cascade of hallucinations, wild tangents, ignored prompts, poisoned contexts, and similar issues that have plagued this tech since the beginning. Except that now you've wired them up to external tools, so maybe the system chooses to wipe your home directory for whatever reason.
People nonchalantly trusting nondeterministic tech with increasingly more real-world tasks should concern everyone. Today it's executing `ping` and `rm`; tomorrow it's managing nuclear launch systems.
Write an agent, it's easy! You will learn so much!
... let's see ...
client = OpenAI()
Um right. That's like saying you should implement a web server, you will learn so much, and then you go and import http (in golang). Yeah well, sure, but that brings you like 98% of the way there, doesn't it? What am I missing?
No, it's saying "let's build a web service" and starting with a framework that just lets you write your endpoints. This is about something higher level than the nuts and bolts. Both are worth learning.
The fact you find this trivial is kind of the point that's being made. Some people think having an agent is some kind of voodoo, but it's really not.
An agent is more like a web service in your metaphor. Yes, building a web server is instructive, but almost nobody has a reason to do it instead of using an out of the box implementation once itâs time to build a production web service.
Two years ago I wrote an agent in 25 lines of PHP [0]. It was surprisingly effective, even back then before tool calling was a thing and you had to coax the LLM into returning structured output. I think it even worked with GPT-3.5 for trivial things.
In my mind LLMs are just UNIX strong manipulation tools like `sed` or `awk`: you give them an input and command and they give you an output. This is especially true if you use something like `llm` [1].
It then seems logical that you can compose calls to LLMs, loop and branch and combine them with other functions.
[0] https://github.com/dave1010/hubcap
[1] https://github.com/simonw/llm
I love hubcap so much. It was a real eye-opener for me at the time, really impressive result for so little code. https://simonwillison.net/2023/Sep/6/hubcap/
> a small Autobot that you can't trust
That gave me a hearty chuckle!
what's the point of specialized agents when you just have one universal agent that can do anything e.g. Claude
Composing multiple smaller agents allows you to build more complex pipelines, which is a lot easier than getting a single monolithic agent to switch between contexts for different tasks. I also get some insight into how the agent performs (e.g via langfuse) because itâs less of a black box.
To use an example: I could write an elaborate prompt to fetch requirements, browse a website, generate E2E test cases, and compile a report, and Claude could run it all to some degree of success. But I could also break it down into four specialised agents, with their own context windows, and make them good at their individual tasks.
If you can get a specialized agent to work in its domain at 10% parameters of a foundation model, you can feasibly run locally, which opens up e.g. offline use cases.
Personally Iâd absolutely buy an LLM in a box which I could connect to my home assistant via usb.
Can you (or someone else) explain how to do that? How much does it typically cost to create a specialized agents that uses a local model? I thought it was expensive?
An agent is just a program which invokes a model in a loop, adding resources like files to the context etc. It's easy to write such a program and it costs nothing, all the compute cost is in the LLM call. What parent was referring to most likely is fine-tuning a smaller model which can run locally, specialized for whatever task. Since it's fine-tuned for that particular task, the hope is that it will be able to perform as well as a general purpose frontier model at a fraction of the compute cost (and locally, hence privately as well).
Absolutely, especially the part about just rolling your own alternative to Claude Code - build your own lightsaber. Having your coding agent improve itself is a pretty magical experience. And then you can trivially swap in whatever model you want (Cerebras is crazy fast, for example, which makes a big difference for these many-turn tool call conversations with big lumps of context, though gpt-oss 120b is obviously not as good as one of the frontier models). Add note-taking/memory, and ask it to remember key facts to that. Add voice transcription so that you can reply much faster (LLMs are amazing at taking in imperfect transcriptions and understanding what you meant). Each of these things takes on the order of a few minutes, and it's super fun.
I agree with you mostly.
On the other hand, I think that show or it didnât happen is essential.
Dumping a bit of code into an LLM doesnât make it a code agent.
And what Magic? I think you never hit conceptual and structural problems. Context window? History? Good or bad? Large Scale changes or small refactoring here and there? Sample size one or several teams? What app? How many components? Green field or not? Which programming language?
I bet you will color Claude and especially GitHub Copilot a bit differently, given that you can easily kill any self made Code Agent quite easily with a bit of steam.
Code Agents are incredibly hard to build and use. Vibe Coding is dead for a reason. I remember vividly the inflation of Todo apps and JS frameworks (Ember, Backbone, Knockout are survivors) years ago.
The more you know about agents and especially code agents the more you know, why engineers wonât be replaced so fast - Senior Engineers who hone their craft.
I enjoy fiddling with experimental agent implementations, but value certain frameworks. They solved in an opiated way problems you will run into if you dig deeper and others depend on you.
Kimi is noticeably better at tool calling than gpt-oss-120b.
I made a fun toy agent where the two models are shoulder surfing each other and swap the turns (either voluntarily, during a summarization phase), or forcefully if a tool calling mistake is made, and Kimi ends up running the show much much more often than gpt-oss.
And yes - it is very much fun to build those!
What are you using for transcription?
I tried Whisper, but it's slow and not great.
I tried the gpt audio models, but they're trained to refuse to transcribe things.
I tried Google's models and they were terrible.
I ended up using one of Mistral's models, which is alright and very fast except sometimes it will respond to the text instead of transcribing it.
So I'll occasionally end up with pages of LLM rambling pasted instead of the words I said!
I recently bought a mint-condition Alf phone, in the shape of Gordon Shumway of TV's "Alf", out of the back of an old auto shop in the south suburbs of Chicago, and naturally did the most obvious thing, which was to make a Gordon Shumway phone that has conversations in the voice of Gordon Shumway (sampled from Youtube and synthesized with ElevenLabs). I use https://github.com/etalab-ia/faster-whisper-server (I think?) as the Whisper backend. It's fine! Asterix feeds me WAV files, an ASI program feeds them to Whisper (running locally as a server) and does audio synthesis with the ElevenLabs API. Took like 2 hours.
https://github.com/rhulha/Speech2Speech
https://github.com/rhulha/EchoMate
Whisper.cpp/Faster-whisper are a good bit faster than OpenAI's implementation. I've found the larger whisper models to be surprisingly good in terms of transcription quality, even with our young children, but I'm sure it varies depending on the speaker, no idea how well it handles heavy accents.
I'm mostly running this on an M4 Max, so pretty good, but not an exotic GPU or anything. But with that setup, multiple sentences usually transcribe quickly enough that it doesn't really feel like much of a delay.
If you want something polished for system-wide use rather than rolling your own, I've been liking MacWhisper on the Mac side, currently hunting for something on Arch.
Parakeet is sota
Agreed. I just launched https://voice-ai.knowii.net and am really a fan of Parakeet now. What it manages to achieve locally without hogging too much resources is awesome
I use Willow AI, which I think is pretty good
But it's way more expensive since most providers won't give you prompt caching?
The reason a lot of people donât do this is because Claude Code lets you use a Claude Max subscription to get virtually unlimited tokens. If youâre using this stuff for your job, Claude Max ends up being like 10x the value of paying by the token, itâs basically mandatory. And you canât use your Claude Max subscription for tools other than Claude Code (for TOS reasons. And theyâll likely catch you eventually if you try to extract and reuse access tokens).
> catch you eventually if you try to extract and reuse access tokens
What does that mean?
Iâm saying if you try to use Wireshark or something to grab the session token Claude Code is using and pass it to another tool so that tool can use the same session token, theyâll probably eventually find out. All it would take is having Claude Code start passing an extra header that your other tool doesnât know about yet, suspend any accounts whose session token is used in requests that donât have that header and manually deal with any false positives. (If youâre thinking of replying with a workaround: That was just one example, there are a bajillion ways they can figure people out if they want to)
How do they know your requests come from Claude Code?
I imagine they can spot it pretty quick using machine learning to spot unlikely API access patterns. They're an AI research company after all, spotting patterns is very much in their wheelhouse.
>build your own lightsaber
I think this is the best way of putting it I've heard to date. I started building one just to know what's happening under the hood when I use an off-the-shelf one, but it's actually so straightforward that now I'm adding features I want. I can add them faster than a whole team of developers on a "real" product can add them - because they have a bigger audience.
The other takeaway is that agents are fantastically simple.
I also started building my own, it's fun and you get far quickly.
I'm now experimenting with letting the agent generate its own source code from a specification (currently generating 9K lines of Python code (3K of implementation, 6K of tests) from 1.5K lines in specifications (https://alejo.ch/3hi).
Agreed, and it's actually how I've been thinking about it, but it's also straight from the article, so can't claim credit. But it was fun to see it put into words by someone else.
And yeah, the LLM does so much of the lifting that the agent part is really surprisingly simple. It was really a revelation when I started working on mine.
Whatâs a good staring point for getting into this? I donât even know what Cerebras is. I just use GitHub copilot in VS Code. Is this local models?
A lot of it is just from HN osmosis, but /r/LocalLLaMA/ is a good place to hear about the latest open weight models, if that's interesting.
gpt-oss 120b is an open weight model that OpenAI released a while back, and Cerebras (a startup that is making massive wafer-scale chips that keep models in SRAM) is running that as one of the models they provide. They're a small scale contender against nvidia, but by keeping the model weights in SRAM, they get pretty crazy token throughput at low latency.
In terms of making your own agent, this one's pretty good as a starting point, and you can ask the models to help you make tools for eg running ls on a subdirectory, or editing a file. Once you have those two, you can ask it to edit itself, and you're off to the races.
Here is ChatGpt in 50 lines of Python:
https://gist.github.com/avelican/4fa1baaac403bc0af04f3a7f007...
No dependencies, and very easy to swap out for OpenRouter, Groq or any other API. (Except Anthropic and Google, they are special ;)
This also works on the frontend: pro tip you don't need a server for this stuff, you can make the requests directly from a HTML file. (Patent pending.)
Cerebras now has glm 4.6. Still obscenely fast, and now obscenely smart, too.
Aren't there cheaper providers of GLM 4.6 on Openrouter? What are the advantages of using Cerebras? Is it much faster?
You know how sometimes when you send a prompt to Claude, you just know itâs gonna take a while, so you go grab a coffee, come back, and itâs still working? With Cerebras itâs not even worth switching tabs, because itâll finish the same task in like three seconds.
It's astonishingly fast.
Ooh thanks for the heads up!
It's interesting how much this makes you want to write Unix-style tools that do one thing and only one thing really well. Not just because it makes coding an agent simpler, but because it's much more secure!
One thing that radicalized me was building an agent that tested network connectivity for our fleet. Early on, in like 2021, I deployed a little mini-fleet of off-network DNS probes on, like, Vultr to check on our DNS routing, and actually devising metrics for them and making the data that stuff generated legible/operationalizable was annoying and error prone. But you can give basic Unix network tools --- ping, dig, traceroute --- to an agent and ask it for a clean, usable signal, and they'll do a reasonable job! They know all the flags and are generally better at interpreting tool output than I am.
I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now. But I do know that getting an agent across the 90% threshold of utility for a problem like this is much, much easier than building the good telemetry system is.
> I'm not saying that the agent would do a better job than a good "hardcoded" human telemetry system, and we don't use agents for this stuff right now.
And that's why I won't touch 'em. All the agents will be abandoned when people realize their inherent flaws (security, reliability, truthfulness, etc) are not worth the constant low-grade uncertainty.
In a way it fits our times. Our leaders don't find truth to be a very useful notion. So we build systems that hallucinate and act unpredictably, and then invest all our money and infrastructure in them. Humans are weird.
Some of us have been happily using agentic coding tools (Claude Code etc) since February and we're still not abandoning them for their inherent flaws.
I abandoned Claude Code pretty quickly, I find generic tools give generic answers, but since I do Elixir Iâm âblessedâ with Tidewave which gives a much better experience. I hope more people get to experience framework built tooling instead of just generic stuff.
It still wants to build an airplane to go out with the trash sometimes and will happily tell you wrong is right. However I much prefer it trying to figure it out by reading logs, schemas and do browser analysis automatically than me feeding logs etc manually.
Cursor can read logs and schemas and use curl to test API responses. It can also look into the database.
But then you have to use Cursor. Tidewave runs as a dependency in the framework and you just navigate to a url, itâs quite refreshing actually.
The problem with statements like these is that I work with people who make the same claims, but are slowly building useless, buggy monstrosities that for various reasons nobody can/will call out.
Obviously Iâm reasonably willing to believe that you are an exception. However every person Iâve interacted with who makes this same claim has presented me with a dumpster fire and expected me to marvel at it.
I'm not going to dispute your own experience with people who aren't using this stuff effectively, but the great thing about the internet is that you can use it to track the people who are making the very best use of any piece of technology.
This line of reasoning is smelling pretty "no true Scotsman" to me. I'm sure there were amazing ColdFusion devs, but that hardly justifies the use of the technology. Likewise "This tool works great on the condition that you need to hire a Simon Willison level dev" is almost a fault. I'm pretty confident you could squeeze some juice out of a Markov Chain (ignoring, of course, that decoder-only LLMs are basically fancy MCs).
In a weird way it sort of reminds me of Common Lisp. When I was younger I thought it was the most beautiful language and a shame that it wasn't more widely adopted. After a few decades in the field I've realized it's probably for the best since the average dev would only use it to create elaborate foot guns.
"elaborate foot guns" -- HN is a high signal environment, but I could read for a week and not find a gem like this. Props.
Destiny visits me on my 18th birthday and says, "Gart, your mediocrity will result in a long series of elaborate foot guns. Be humble. You are warned."
We have gpt-5 and gemini 2.5 pro at work, and both of them produce huge amounts of basically shit code that doesnât work.
Every time i reach for them recently I end up spending more time refactoring the bad code out or in deep hostage negotiations with the chatbot of the day that I would have been faster writing it myself.
That and for some reason they occasionally make me really angry.
Oh a bunch of prompts in and then it hallucinated some library a dependency isnât even using and spews a 200 line diff at me, again, great.
Although at least i can swear at them and get them to write me little apology poems..
On the sometimes getting angry part, I feel you. I don't even understand why it happens, but it's always a weird moment when I notice it. I know I'm talking to a machine and it can't learn from its mistakes, but it's still very frustrating to get back yet another here's the actual no bullshit fix, for real this time, pinky promise.
Are you using them via a coding agent harness such as Codex CLI or Gemini CLI?
Via the jetbrains plugin, has an 'agent' mode and can edit files and call tools so on, yes I setup MCP integrations and so on also. Still kinda sucks. shrug.
I keep flipping between this is the end of our careers, to I'm totally safe. So far this is the longest 'totally safe' period I've had since GPT-2 or so came along..
Honestly the top AI use case for me right now is personal throwaway dev tools. Where I used to write shell oneliners with dozen pipes including greps and seds and jq and other stuff, now I get an AI to write me a node script and throw in a nice Web UI to boot.
Edit: reflecting on what the lesson is here, in either case I suppose we're avoiding the pain of dealing with Unix CLI tools :-D
Interesting. You have to wonder if all the tools that is based on would have been written in the first place if that kind of thing had been possible all along. Who needs 'grep' when you can write a prompt?
My long running joke is that the actual good `jq` is just the LLM interface that generates `jq` queries; 'simonw actually went and built that.
https://github.com/simonw/llm-jq for those following along at home
https://github.com/simonw/llm-cmd is what i use as the "actually good ffmpeg etc front end"
and just to toot my own horn, I hand Simon's `llm` command lone tool access to its own todo list and read/write access to the cwd with my own tools, https://github.com/dannyob/llm-tools-todo and https://github.com/dannyob/llm-tools-patch
Even with just these and no shell access it can get a lot done, because these tools encode the fundamental tricks of Claude Code ( I have `llmw` aliased to `llm --tool Patch --tool Todo --cl 0` so it will have access to these tools and can act in a loop, as Simon defines an agent. )
Tried gron (https://github.com/tomnomnom/gron) a bit? If you know your UNIX, I think it can replace jq in a lot of cases. And when it can't, well, you can reach for Python, I guess.
It's highly plausible that all we assumed was good design / engineering will disappear if LLMs/Agents can produce more without having the be modular. (sadly)
There is some kind of parallel behind 'AI' and 'Fuzzy Logic'. Fuzzy logic to me always appeared like a large number of patches to get enough coverage for a system to work even if you didn't understand it. AI just increases the number of patches to billions.
true, there's often a point where your system becomes a blurry miracle
Could you give some examples? I'm having the AI write the shell scripts, wondering if I'm missing out on some comfy UIs...
I was debugging a service that was spitting out a particular log line. I gave Copilot an example line, told it to write a script that tails the log line and serves a UI via port 8080 with a table of those log lines parsed and printed nicely. Then I iterated by adding filter buttons, aggregation stats, simple things like that. I asked it to add a "clear" button to reset the UI. I probably would not even have done this without an AI because the CLI equivalent would be parsing out and aggregating via some form of uniq -c | sort -n with a bunch of other tuning and it would be too much trouble.
It can be anything. It depends on what you want to do with the output.
You can have a simple dashboard site which collects the data from our shell scripts and shows your a summary or red/green signals so that you can focus on things which are interested in.
I hadn't given much thought to building agents, but the article and this comment are inspiring, thx. It's interesting to consider agents as a new kind of interface/function/broker within a system.
> They know all the flags and are generally better at interpreting tool output than I am.
In the toy example, you explicitly restrict the agent to supply just a `host`, and hard-code the rest of the command. Is the idea that you'd instead give a `description` something like "invoke the UNIX `ping` command", and a parameter described as constituting all the arguments to `ping`?
Honestly, I didn't think very hard about how to make `ping` do something interesting here, and in serious code I'd give it all the `ping` options (and also run it in a Fly Machine or Sprite where I don't have to bother checking to make sure none of those options gives code exec). It's possible the post would have been better had I done that; it might have come up with an even better test.
I was telling a friend online that they should bang out an agent today, and the example I gave her was `ps`; like, I think if you gave a local agent every `ps` flag, it could tell you super interesting things about usage on your machine pretty quickly.
What is Sprite in this context?
I'm guessing the Fly Machine they're referring to is a container running on fly.io, perhaps the sprite is what the Spritely Institute calls a goblin.
Also to be clear: are the schemas for the JSON data sent and parsed here specific to the model used? Or is there a standard? (Is that the P in MCP?)
Its JSON schema, well standardized, and predates LLMs: https://json-schema.org/
Ah, so I can specify how I want it to describe the tool request? And it's been trained to just accommodate that?
Most LLMs have tool patterns trained into them now, which are then managed for you by the API that the developers run on top of the models.
But... you don't have to use that at all. You can use pure prompting with ANY good LLM to get your own custom version of tool calling:
Before LLMs had tool calling we called this the ReAct pattern - I wrote up an example of implementing that in March 2023 here: https://til.simonwillison.net/llms/python-react-patternYou could even imagine a world in which we create an entire suite of deterministic, limited-purpose tools and then expose it directly to humans!
Half my use of LLM tools is just to remember the options for command line tools, including ones I wrote but only use every few months.
I wonder if we could develop a language with well-defined semantics to interact with and wire up those tools.
As it happens, I have a prototype for this, but the syntax is honestly rather unwieldy. Maybe there's a way to make it more like natural human language....
I can't tell whether any comment in this thread is a parody or not.
When in doubt, there's always the option of rewriting an existing interactive shell in Rust.
(Mine was intended as ironic, suggesting that a circle of development ideas would eventually complete. I interpreted the previous comments as satirically pointing at the fact that the notion of "UNIX-like tools" owes to the fact that there is actually such a thing as UNIX.)
> language with well-defined semantics
That would certainly be nice! That's why we have been overhauling shell with https://oils.pub , because shell can't be described as that right now
It's in extremely poor shape
e.g. some things found from building several thousand packages with OSH recently (decades of accumulated shell scripts)
- bugs caused by the differing behavior of 'echo hi | read x; echo x=$x' in shells, i.e. shopt -s lastpipe in bash.
- 'set -' is an archaic shortcut for 'set +v +x'
- Almquist shell is technically a separate dialact of shell -- namely it supports 'chdir /tmp' as well as cd /tmp. So bash and other shells can't run any Alpine builds.
I used to maintain this page, but there are so many problems with shell that I haven't kept up ...
https://github.com/oils-for-unix/oils/wiki/Shell-WTFs
OSH is the most bash-compatible shell, and it's also now Almquist shell compatible: https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/sp...
It's more POSIX-compatible than the default /bin/sh on Debian, which is dash
The bigger issue is not just bugs, but lack of understanding among people who write foundational shell programs. e.g. the lastpipe issue, using () as grouping instead of {}, etc.
---
It is often treated like an "unknowable" language
Any reasonable person would use LLMs to write shell/bash, and I think that is a problem. You should be able to know the language, and read shell programs that others have written
I love it how you went from 'Shell-WTFs' to 'let's fix this'. Kudos, most people get stuck at the first stage.
Thanks! We are down to 14 disagreements between OSH and busybox ash/bash on Alpine Linux main
https://op.oils.pub/aports-build/published.html
We also don't appear to be unreasonably far away from running ~~ "all shell scripts"
Now the problem after that will be motivating authors of foundational shell programs to maintain compatibility ... if that's even possible. (Often the authors are gone, and the nominal maintainers don't know shell.)
As I said, the state of affairs is pretty sorry and sad. Some of it I attribute to this phenomenon: https://news.ycombinator.com/item?id=17083976
Either way, YSH benefits from all this work
Hmmm but how would you name that? Agent skills? Meta cognition agentic tooling? Intelligence driven self improving partial building blocks?
Oh... oh I know how about... UNIX Philosophy? No... no that'd never work.
/s
Doing one thing well means you need a lot more tools to achieve outcomes, and more tools means more context and potentially more understanding of how to string them together.
I suspect the sweet spot for LLMs is somewhere in the middle, not quite as small as some traditional unix tools.
Agreed! It's easy understand "LLM with tools in a loop" at a high-level, but once you actually design the architecture and implement the code in full, you'll have proper understanding of how it all fits and works together.
I did the same exercise. My implementation is at around 300 lines with two tools: web search and web page fetch with a command line chat interface and Python package. And it could have been a lot less lines if I didn't want to write a usable, extensible package interface.
As the agent setup itself is simple, majority of the work to make this useful would in the tools themselves and context management for the tools.
I wrote an agent from scratch in Ruby several months back. Was fun!
These 4 lines wound up being the heart of it, which is surprisingly simple, conceptually.
I should? what problems can I solve, that can be only done with an agent? As long as every AI provider is operating at a loss starting a sustainably monetizable project doesn't feel that realistic.
The post is just about playing around with the tech for fun. Why does monetization come into it? It feels like saying you don't want to use Python because Astral, the company that makes uv, is operating at a loss. What?
Agents use Apis that I will need to pay for and generally software dev is a job for me that needs to generate income.
If the Apis I call are not profitable for the provider then they won't be for me either.
This post is a fly.io advertisement
"Agents use Apis that I will need to pay for"
Not if you run them against local models, which are free to download and free to run. The Qwen 3 4B models only need a couple of GBs of available RAM and will run happily on CPU as opposed to GPU. Cost isn't a reason not to explore this stuff.
Google has what I would call a generous free tier, even including Gemini 2.5 Pro (https://ai.google.dev/gemini-api/docs/rate-limits). Just get an API key from AiStudio. Also very easy to just make a switch in your agent so that if you hit up against a rate limit for one model, re-request the query with the next model. With Pro/Flash/Flash-Lite and their previews, you've got 2500+ free requests per day.
> Not if you run them against local models, which are free to download and free to run .. run happily on CPU .. Cost isn't a reason not to explore this stuff.
Let's be realistic and not over-promise. Conversational slop and coding factorial will work. But the local experience for coding agents, tool-calling, and reasoning is still very bad until/unless you have a pretty expensive workstation. CPU and qwen 4b will be disappointing to even try experiments on. The only useful thing most people can realistically do locally is fuzzy search with simple RAG. Besides factorial, maybe some other stuff that's in the training set, like help with simple shell commands. (Great for people who are new to unix, but won't help the veteran dev who is trying to convince themselves AI is real or figuring out how to get it into their workflows)
Anyway, admitting that AI is still very much in a "pay to play" phase is actually ok. More measured stances, fewer reflexive detractors or boosters
Sure, you're not going to get anything close to a Claude Code style agent from a local model (unless you shell out $10,000+ for a 512GB Mac Studio or similar).
This post isn't about building Claude Code - it's about hooking up an LLM to one or two tool calls in order to run something like ping. For an educational exercise like that a model like Qwen 4B should still be sufficient.
The expectation that reasonable people have isn't fully local claude code, that's a strawman. But it's also not ping tools or the simple weather agent that tutorials like to use. It's somewhere in between, isn't that obvious? If you're into evangelism, acknowledging this and actually taking a measured stance would help prevent light skeptics from turning into complete AI-deniers. If you mislead people about one thing, they will assume they are being misled about everything
I don't think I was being misleading here.
https://fly.io/blog/everyone-write-an-agent/ is a tutorial about writing a simple "agent" - aka a thing that uses an LLM to call tools in a loop - that can make a simple tool call. The complaint I was responding to here was that there's no point trying this if you don't want to be hooked on expensive APIs. I think this is one of the areas where the existence of tiny but capable local models is relevant - especially for AI skeptics who refuse to engage with this technology at all if it means spending money with companies they don't like.
I think it is misleading to suggest today that tool-calling for nontrivial stuff really works with local models. It just works in demos because those tools always accept one or two arguments, usually string literals or numbers. In the real world functions take more complex arguments, many arguments, or take a single argument that's an object with multiple attributes, etc. You can begin to work around this stuff by passing function signatures, typing details, and JSON-schemas to set expectations in context, but local models tend to fail at handling this kind of stuff long before you ever hit limits in the context window. There's a reason demos are always using 1 string literal like hostname, or 2 floats like lat/long. It's normal that passing a dictionary with a few strict requirements might need 300 retries instead of 3 to get a tool call that's syntactically correct and properly passed arguments. Actually `ping --help` for me shows like 20 options, and for any attempt to 1:1 map things with more args I think you'd start to see breakdown pretty quickly.
Zooming in on the details is fun but doesn't change the shape of what I was saying before. No need to muddy the water; very very simple stuff still requires very big local hardware or a SOTA model.
You and I clearly have a different idea of what "very very simple stuff" involves.
Even the small models are very capable of stringing together a short sequence of simple tool calls these days - and if you have 32GB of RAM (eg a ~$1500 laptop) you can run models like gpt-oss:20b which are capable of operating tools like bash in a reasonably useful way.
This wasn't true even six months ago - the local models released in 2025 have almost all had tool calling specially trained into them.
> software dev is a job for me that needs to generate income
sir, this is a hackernews
> This post is a <insert-startup-here> advertisement
same thing you said but in a different context... sir, this is a hackernews
No, we are not an LLM provider.
Practically everything is something you will need to pay for in the end. You probably spent money on an internet connection, electricity, and computing equipment to write this comment. Are you intending to make a profit from commenting here?
You don't need to run something like this against a paid API provider. You could easily rework this to run against a local agent hosted on hardware you own. A number of not-stupid-expensive consumer GPUs can run some smaller models locally at home for not a lot of money. You can even play videogames with those cards after.
Get this: sometimes people write code and tinker with things for fun. Crazy, I know.
The submission is an advertisement for fly.io and OpenAI , both are paid services. We are commenting on an ad. The person who wrote it did it for money. Fly.io operates for money, OpenAi charges for their API.
They posted it here expecting to find customers. This is a sales pitch.
At this point why is it an issue to expect a developer to make money on it?
As a dev, If the chain of monetization ends with me then there is no mainstream adoption whatsoever on the horizon.
I love to tinker but I do it for free not using paid services.
As for tinkering with agents, its a solution looking for a problem.
Why are you repeatedly stating that the post is an ad as if it is some sort of dunk? Companies have blogs. Tech blogs often produce useful content. It is possible that an ad can both successfully promote the company and be useful to engineers. I find the Fly blog to be particularly well-written and thoughtful; it's taught me a good deal about Wireguard, for instance.
And that sounds fine, but Wireguard is not an overhyped industry promising huge gains in the future to investors and to developers jumping on a bandwagon who can find problems for this solution.
I actually have built agents already in the past and this is my opinion. If you read the article the author says they want to hear the reasoning for disliking it, so this is mine, the only way to create a business is raising money and hoping somebody strikes gold with the shovel Im paying for.
How would you feel about this post if the exact same content was posted on a developer's personal blog instead?
I ask because it's rare for a post on a corporate blog to also make sense outside of the context of that company, but this one does.
They're mentioning WireGuard because we do in fact do WireGuard, unlike LLM agents, which we do not offer as a service.
You keep saying this, but there is nothing in this post about our service. I didn't use Fly.io at all to write this post. Across the thread, someone had to remind me that I could have.
Sorry, I assumed a service offering Virtual machines shares python code with the intent to get people to run that python on their infra.
Yes. You've caught on to our devious plan. To do anything I suggested in this post, you'd have to use a computer. By spending compute cycles, you'd be driving scarcity of compute. By the inexorable law of supply and demand, this would drive the price of compute cycles up, allowing us to profit. We would have gotten away with it, if it wasn't for you.
Yeah we have open source models too that we can use, and itâs actually more fun than using cloud providers in my opinion.
> what problems can I solve, that can be only done with an agent?
The problem that you might not intuitively understand how agents work and what they are and aren't capable of - at least not as well as you would understand it if you spent half an hour building one for yourself.
Seems like it would be a lot easier for everyone if we knew the answer to his/her question.
> As long as every AI provider is operating at a loss
None of them are doing that.
They need funding because the next model has always been much more expensive to train than the profits of the previous model. And many do offer a lot of free usage which is of course operated at a loss. But I don't think any are operating inference at a loss, I think their margins are actually rather large.
Parent comment never said operating inference at a loss, though it wouldn't surprise me, they just said "operating at a loss" which they most definitely are [0].
However, knowing a few people on teams at inference-only providers, I can promise you some of them absolutely are operating inference at a loss.
0. https://www.theregister.com/2025/10/29/microsoft_earnings_q1...
> Parent comment never said operating inference at a loss
Context. Whether inference is profitable at current prices is what informs how risky it is to build a product that depends on buying inference, which is what the post was about.
So you're assuming there's a world where these companies exist solely by providing inference?
The first obvious limitation of this would be that all models would be frozen in time. These companies are operating at an insane loss and a major part of that loss is required to continue existing. It's not realistic to imagine that there is an "inference" only future for these large AI companies.
And again, there are many inference only startups right now, and I know plenty of them are burning cash providing inference. I've done a lot of work fairly close to the inference layer and getting model serving happening with the requirements for regular business use is fairly tricky business and not as cheap as you seem to think.
> So you're assuming there's a world where these companies exist solely by providing inference?
Yes, obviously? There is no world where the models and hardware just vanish.
> and hardware just vanish.
Okay, this tells me you really don't understand model serving or any of the details of infrastructure. The hardware is incredibly ephemeral. Your home GPU might last a few years (and I'm starting to doubt that you've even trained a model at home), but these GPUs have incredibly short lifespans under load for production use.
Even if you're not working on the back end of these models, you should be well aware that one of the biggest concerns about all this investment is how limited the lifetime of GPUs is. It's not just about being "outdated" by superior technology, GPUs are relatively fragile hardware and don't last too long under constant load.
As far as models go, I have a hard time imagining a world in 2030 where the model replies "sorry, my cutoff date was 2026" and people have no problem with this.
Also, you still didn't address my point that startups doing inference only model serving are burning cash. Production inference is not the same as running inference locally where you can wait a few minutes for the result. I'm starting to wonder if you've ever even deployed a model of any size to production.
I didn't address the comment about how some startups are operating at a loss because it seems like an irrelevant nitpick at my wording that "none of them" is operating inference at a loss. I don't think the comment I was replying to was referring to relying on whatever startups you're talking about. I think they were referring to Google, Anthropic, and OpenAI - and so was I.
That seems like a theme with these replies, nitpicking a minor thing or ignoring the context or both, or I guess more generously I could blame myself of not being more precise with my wording. But sure, you have to buy new GPUs after making a bunch of money burning the ones you have down.
I think your point about knowledge cutoff is interesting, and I don't know what the ongoing cost to keeping a model up to date with world knowledge is. Most of the agents I think about personally don't actually want world knowledge and have to be prompted or fine tuned such that they won't use it. So I think that requirement kind of slipped my mind.
If the game is inference the winners are the cloud mega scalers, not the ai labs.
This thread isn't about who wins, it's about the implication that it's too risky to build anything that depends on inference because AI companies are operating at a loss.
The models may be somewhat frozen in time but with the right tools available to it they don't need all information innately coded into it. If they're able to query for reliable information to drag in they can talk about things that are well outside their original training data.
For a few months of news this works, but over the span of years even the statistical nature of language drifts a bit. Have you shipped natural language models to production? Even simple classifiers need to be updated periodically because of drift. There is no world where you lead the industry serving LLMs and don't train them as well.
So AI companies are profitable when you ignore some of the things they have to spend money on to operate?
Snark aside, inference is still being done at a loss. Anthropic, the most profitable AI vendor, is operating at a roughly -140% margin. xAI is the worst at somewhere around -3,600% margin.
If they are not operating inference at a loss and current models remain useful (why would they regress?), they could just stop developing the next model.
They could, but thatâs a recipe for going out of business in the current environment.
Yes, but at the same time it's unlikely for existing models to disappear. You won't get the next model, but there is no choice but to keep inference running to pay off creditors.
The interesting companies to look at here are the ones that sell inference against open weight models that were trained by other companies - Fireworks, Cloudflare, DeepInfra, Together AI etc.
They need to cover their serving costs but are not spending money on training models. Are they profitable? Probably not yet, because they're investing a lot of cash in competing with each other to R&D more efficient ways of serving etc, but they're a lot closer to profitability than the labs that are spending millions of dollars on training runs.
Where do those numbers come from?
Sounds quite a bit like pyramid scheme "business model": how is it different?
If a company stops training new models until they can fund it out of previous profits, do we only slow down or halt altogether? If they all do?
> But I don't think any are operating inference at a loss, I think their margins are actually rather large.
Citation needed. I haven't seen any of them claim to have even positive gross margins to shareholders/investors, which surely they would do if they did.
https://officechai.com/ai/each-individual-ai-model-can-alrea...
> âif you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. Thereâs some cost to inference with the model, but letâs just assume, in this cartoonish cartoon example, that even if you add those two up, youâre kind of in a good state. So, if every model was a company, the model, in this example, profitable,â he added.
âWhatâs going on is that while youâre reaping the benefits from one company, youâre founding another company thatâs much more expensive and requires much more upfront R&D investment. The way this is going to shake out is that itâs going to keep going up until the numbers get very large, and the models canât get larger, and then there will be a large, very profitable business. Or at some point, the models will stop getting better, and there will perhaps be some overhang â we spent some money, and we didnât get anything for it â and then the business returns to whatever scale itâs at,â he said.
This take from Amodei is hilarious but explains so much.
When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware.
OpenAI balance sheet also shows an $11 billion loss .
I can't see any profit on anything they create. The product is good but it relies on investors fueling the AI bubble.
> When comparing the cost of an H100 GPU per hour and calculating cost of tokens, it seems the OpenAI offering for the latest model is 5 times cheaper than renting the hardware.
How did you come to that conclusion? That would be a very notable result if it did turn out OpenAI were selling tokens for 5x the cost it took to serve them.
I am reading it as OpenAI selling them for 20% of the cost to serve them (serving at the equivalent token/s with cloud pay-per-use GPUs).
You're right, I misunderstood.
it seems to me they are saying the opposite
https://martinalderson.com/posts/are-openai-and-anthropic-re...
All the labs are going hard on training and new GPUs. If we ever level off, they probably will be immensely profitable. Inference is cheap, training is expensive.
To do this analysis on an hourly retail cost and an open weight model and infer anything about the situation at OpenAI or Anthropic is quite a reach.
For one (basic) thing, they buy and own their hardware, and have to size their resources for peak demand. For another, Deepseek R1 does not come close to matching claude performance in many real tasks.
> None of them are doing that.
Can you point us to the data?
You are asking the wrong questions. You should be asking what the problems are that you can still solve better and cheaper than an agent? Because anything else, you are probably doing it wrong (the slow and expensive way). That's not long term sustainable. It helps if you know how agents work and as the article argues, there isn't a whole lot to that.
You can be your own AI provider.
For internal software maybe, but for a client facing service the incentives are not right when the norm is to operate at a loss.
>starting a sustainably monetizable project doesn't feel that realistic.
and
>You can be your own AI provider.
Not sure that being your own AI provider is "sustainably monetizable"?
Show me where TFA even implied that you should start a sustainably monetizable project with agents?
I love how programmers generally tout themselves as these tinkerers who love learning about and exploring technology⌠until it comes to AI and then itâs like âshow me the profitable use case.â Just say you donât like AI!
Yeah but fly.io is a cloud provider doing this advertisement with OpenAI Apis. Both cost money, so if it's not free to operate then the developed project should offset the costs.
Its about balance.
Really its the AI providers that have been promising unreal gains during this hype period, so people are more profit oriented.
What does "cloud provider" even have to do with this post?
Or maybe some of us realize that these tools are fucking useless and donât offer any âvalueâ apart from the most basic thing imaginable.
And I use value in quotes because as soon as the AI providers suddenly need to start generating a profit, that âvalueâ is going to cost more than your salary.
It doesn't have to be profitable. Elegant and clever would suffice.
I don't think hn is reflective of where programmers are today, culturally. 10 years ago, sure, it probably was.
what place is more reflective today?
I don't know about online forums, but all my IRL friends have a lot more balanced takes on AI than this forum. And honestly it extends beyond this forum to the wider internet. Online, the discourse seems extremely polarized: either it's all a pyramid scheme or stories about how development jobs are already defunct and AI can supervise AI etc.
Heh, the bit about context engineering is palpable.
I'm writing a personal assistant which, imo, is distinct from an agent in that it has a lot of capabilities a regular agent wouldn't necessarily need such as memory, task tracking, broad solutioning capabilities, etc... I ended up writing agents that talk to other agents which have MCP prompts, resources, and tools to guide them as general problem solvers. The first agent that it hits is a supervisor that specializes in task management and as a result writes a custom context and tool selection for the react agent it tasks.
All that to say, the farther you go down this rabbit hole the more "engineering" it becomes. I wrote a bit on it here: https://ooo-yay.com/blog/building-my-own-personal-assistant/
https://github.com/mem0ai/mem0?tab=readme-ov-file
Is this useful for you?
This sounds really great.
If you want to play with this stuff without spending a lot of money, what are your best options?
Most cloud providers, like Azure have free credits at the start. On azure you can deploy your own model and pay with the free credits.
You can just stick a tenner in OpenAI though and it won't charge anymore than the credit you've put in
and sorry, forgot you can also run local models aswell :)
Side note: While the example uses GPT-5, the query interface is already some kind of industry standard. For example you could easily connect OpenRouter.ai and switch models and providers during runtime as needed. OpenRouter also has free models like some of the DeepSeek. While they are slow/rate limited and quantized, they are great for examples and playing around with it. https://openrouter.ai/models?fmt=cards&order=pricing-low-to-...
> You only think you understand how a bicycle works, until you learn to ride one.
I bet a majority of people who can ride a bicycle don't know how they steer, and would describe the physical movements they use to initiate and terminate a turn inaccurately.
https://en.wikipedia.org/wiki/Countersteering
Relevant interesting tangent:
âMost People Don't Know How Bikes Workâ
https://www.youtube.com/watch?v=9cNmUNHSBac
Seriously, what is the advantage of tools at all. Why not implement custom string based triggers.
First of all, the call accuracy is much higher.
Second, you get more consistent results across models.
everybody loves building agents, nobody likes debugging them. agents hit the classic llm app lifecycle problem: at first it feels magical. it nails the first few tasks, doing things you didnât even think were possible. you get excited, start pushing it further. you run it and then it fails on step 17, then 41, then step 9.
now you canât reproduce it because itâs probabilistic. each step takes half a second, so you sit there for 10â20 minutes just waiting for a chance to see what went wrong
That's why you build extensive tooling to run your change hundreds of times in parallel against the context you're trying to fix, and then re-run hundreds of past scenarios in parallel to verify none of them breaks.
In the event this comment is slathered in sarcasm:
Do you use a tool for this? Is there some sort of tool which collects evals from live inferences (especially those which fail)
https://x.com/rerundotio/status/1968806896959402144
This is a use of Rerun that I haven't seen before!
This is pretty fascinating!!!
Typically people use Rerun to visualize robotics data - if I'm following along correctly... what's fascinating here is that Adam for his master's thesis is using Rerun to visualize Agent (like ... software / LLM Agent) state.
Interesting use of Rerun!
https://github.com/gustofied/P2Engine
For sure, for instance Google has ADK Eval framework. You write tests, and you can easily run them against given input. I'd say its a bit unpolished, as is the rest of the rapidly developing ADK framework, but it does exist.
heya, building this. been used in prod for a month now, has saved my customerâs ass while building general workflow automation agents. happy to chat if ur interested.
darin@mcptesting.com
(gist: evals as a service)
There is no way to prove the correctness of non-deterministic (a.k.a. probabilistic) results for any interesting generative algorithm. All one can do is validate against a known set of tests, with the understanding that the set is unbounded over time.
It really reads to me like, "you should build a running water circuit", then presenting you how easy it is to phone a plumber and let them free ride on the matter, but beware to not use a project manager as real people implement project management of plumbery themselves."
There's something(s) about @tptacek's writing style that has always made me want to root for fly.io.
There is a lot of stuff I should do. From making my own CPU from a breadboard of nand gates to building a CDN in Rust. But aint got time for all the things.
That said I built an LLM following Karpathy's tutorial. So I think it aims good to dabble a bit.
Yeah, itâs a never-ending curve.
I built an 8-bit computer on breadboards once, then went down the rabbit hole of flight training for a PPL. Every time I think Iâm "done," the finish line moves a few miles further.
Guess we nerds are never happy.
One should be melting sand to get silicon, anything else it's too abstract to my taste.
Glad youâve got all that time on your hands. I am still working on the fusion reactor portion of my supernova simulator, so that I can generate the silicon you so blithely refer to.
Given the premise, one could also say we nerds are forever happy.
Non sequitur.
If you are a software engineer, you are going to be expected to use AI in some form in the near future. A lot of AI in its current form is not intuitive. Ergo, spending a small effort on building an AI agent is a good way to develop the skills and intuition needed to be successful in some way.
Nobody is going to use a CPU you build, nor are you ever going to be expected to build one in the course of your work if you donât seek out specific positions, nor is there much non-intuitive about commonly used CPU functionality, and in fact you donât even use the CPU directly, you use translation software whit itself is fairly non-intuitive. But thatâs ok too, you are unlikely to be asked to build a compiler unless you seek out those sorts of jobs.
EVERYONE involved in writing applications and services is going to use AI in the near future and in case you missed the last year, everyone IS building stuff with AI, mostly chat assistants that mostly suck because, much about building with AI is not intuitive.
Very early in TFA it explains how easy it is to do. That's the whole point of the post.
It's good to go through the exercise, but agents are easy until you build a whole application using an API endpoint that OpenAI or LangChain decides to yank, and you spend the next week on a mini migration project. I don't disagree with the claim that MCP is reinventing the wheel but sometimes I'm happy plugging my tools and data into someone else's platform because they are spending orders of magnitudes more time than me doing the janitor work to keep up with whatever's trendy.
I have been playing with OpenAI, Anthropic, and Groqâs APIs in my spare time and if someone reading this doesnât know it, they are doing the same thing and they are so close in implementation that itâs just dumb that they are in any way different.
You pass listing of messages generated by the user or the LLM or the developer to the API, it generates a part of the next message. That part may contain thinking blocks or tool calls (local function calling requested by the LLM). If so, you execute the tool calls and re-send the request. After the LLM has gathered all the info it returns the full message and says I am done. Sometimes the messages may contain content blocks that are not text but things like images, audio, etc.
Thatâs the API. Thatâs it. Now there are two improvements that are currently in the works:
1. Automatic local tool calling. This is seriously some sort of afterthought and not how they did it originally but ok, I guess this isnât obvious to everyone.
2. Not having to send the entire message history back. OpenAI released a new feature where they store the history and you just send the ID of your last message. I canât find how long they keep the message history. But they still fully support you managing the message history.
So we have an interface that does relatively few things, and that has basically a single sensible way to do it with some variations for flavor. And both OpenAI and Anthropic are engaged in a turf war over whose content block types are better. Just do the right thing and make your stuff compatible already.
I feel like one small piece is missing to call it an agent? The ability to iterate in multiple steps until it feels like it's "done". What is the canonical way to do that? I suspect that implementing that in the wrong way could make it spiral.
When a tool call completes the result is sent back to the LLM to decide what to do next, that's where it can decide to go do other stuff before returning a final answer. Sometimes people use structured outputs or tool calls to explicitly have the LLM decide when it's done, or allow it to send intermediate messages for logging to the user. But the simple loop there lets the LLM do plenty of it has good tools.
So it returns a tool call for "continue" every time it wants to continue working? Do people implement this in different ways? It would be nice what method it has been trained on if any.
Actually tool "ping 8.8.8.8" never quits unless running on windows. This can spawn many processes that kill the server.
This is one of the first production grade errors I've made when I started my programming. I had a widget that would ping the network, but every time someone went on the page, a new ping process would spawn
If you look at the actual code, it runs ping -c 5. I agree ping without options doesn't terminate.
THEY SEND THE WHOLE CONTEXT EVERY TIME? Man that seems... not great. sometimes it will go off and spin on something.. seems like it would be a LOT better to roll back than to send a corrective message. hmmm...... this article is nerd-sniping on a massive scale ;D
Sending the whole context on each user message is essentially what the model remembers of this conversation. ie: it is entirely stateless.
I've written some agents that have their context altered by another llm to get it back on track. Let's say the agent is going off rails, then a supervisor agent will spot this and remove messages from the context where it went off rails, or alter those with correct information. Really fun stuff but yeah, we're essentially still inventing this as we go along.
There is context caching in many models. It is less expensive if you enable that.
In the Responses API, you can implicitly chain messages with `previous_response_id` (I'm not sure how old a conversation you can resurrect that way). But I think Codex CLI actually sends the full context every time? And keep in mind, sending the whole context gives you fine-grained control over what does and doesn't appear in your context window.
Anyways, if it nerd sniped you, I succeeded. :)
Just the other day we tried to explain inner workings of cursor etc to a bunch of colleagues who had a very complicated view how these agents achieve what they do. Awesome post. Makes it easier for me the next time. The options are so big. But one should say that an agent with file access etc, is easy to write but hard to control. If you want to build yourself a general coding agent a bit more thought needs to be put into the whole thing. Otherwise you might end up with a âdd -if=/dev/random -of=/â or something ^^ and happily execute it.
Its easy to create a toy, but much harder to make something right! Like anything, so much weird polish stuff creeps in at the 90% mark.
> so much weird polish stuff creeps in at the 90% mark.
That is where the human in the loop needs to focus on for now :)
Does anyone have an understanding - or intuition - of what the agentic loop looks like in the popular coding agents? Is it purely a âwhile 1: call_llm(system, assistant)â, or is there complex orchestration?
Iâm trying to understand if the value for Claude Code (for example) is purely in Sonnet/Haiku + the tool system prompt, or if thereâs more secret sauce - beyond the âsugarâ of instruction file inclusion via commands, tools, skills etc.
Claude Code is an obfuscated javascript app. You can point Claude Code at it's own package and it will pretty reliably tell you how it works.
I think Claude Code's magic is that Anthropic is happy to burn tokens. The loop itself is not all that interesting.
What is interesting is how they manage the context window over a long chat. And I think a fair amount of that is serverside.
> Claude Code is an obfuscated javascript app. You can point Claude Code at it's own package and it will pretty reliably tell you how it works.
This is why I keep coming back to Hacker News. If the above is not a quintessential "hack", then I've never seen one.
Bravo!
I've been running the obfuscated code through Prettier first, which I think makes it a bit easier for Claude Code to run grep against.
No need to take guesses - the VS Code GitHub Copilot extension is open source amnd has an agent mode with tool calling:
https://github.com/microsoft/vscode-copilot-chat/blob/4f7ffd...
You can reverse engineer Claude Code by intercepting its HTTP traffic. It's pretty fascinating - there are a bunch of ways to do this, I use this one: https://simonwillison.net/2025/Jun/2/claude-trace/
I thought this was informative: https://minusx.ai/blog/decoding-claude-code/
Generally, that's pretty much it. More advanced tools like Claude Code will also have context compaction (which sometimes isn't very good), or possibly RAG on code (unsure about this, I haven't used any tools that did this). Context compaction, to my understanding, is just passing all the previous context into a call which summarizes it, then that becomes to new context starting point.
https://github.com/sst/opencode opencode is open source. Here's a session I started but haven't had time to get back to which is using opencode to ask it about how the loop works https://opencode.ai/s/4P4ancv4
The summary is
The beauty is in the simplicity: 1. One loop - while (true) 2. One step at a time - stopWhen: stepCountIs(1) 3. One decision - "Did LLM make tool calls? â continue : exit" 4. Message history accumulates tool results automatically 5. LLM sees everything from previous iterations This creates emergent behavior where the LLM can: - Try something - See if it worked - Try again if it failed - Keep iterating until success - All without explicit retry logic!
Have a look at https://github.com/anthropics/claude-code/tree/main/plugins/... to see how a fairly complex workflow is implemented
> nobody knows anything yet
that sums up my experience in AI over the past three years. so many projects reinvent the same thing, so much spaghetti thrown at the wall to see what sticks, so much excitement followed by disappointment when a new model drops, so many people grifting, and so many hacks and workarounds like RAG with no evidence of them actually working other than "trust me bro" and trial and error.
That is because for the people for whom AI is actually working/making money they would prefer to keep it a secret on what and how they are doing it, why attract competition?
I think we'd get better results if we thought of it as a conscious agent. If we recognized that it was going to mirror back or unconscious biases and try to complete the task as we define it, instead of how we think it should behave. Then we'd at least get our own ignorance out of the way when writing prompts.
Being able to recognize that 'make this code better' provides no direction, it should make sense that the output is directionless.
But on more subtle levels, whatever subtle goals that we have and hold in the workplace will be reflected back by the agents.
If you're trying to optimise costs, and increase profits as your north star. Having layoffs and unsustainable practices is a logical result, when you haven't balanced this with any incentives to abide by human values.
> Imagine what itâll do if you give it bash. You could find out in less than 10 minutes. Spoiler: youâd be surprisingly close to having a working coding agent.
Okay, but what if I'd prefer not to have to trust a remote service not to send me
Obviously if you're concerned about that, which is very reasonable, don't run it in an environment where `rm -rf` can cause you a real problem.
Also if you're doing function calls you can just have the command as one response param, and arguments array as another response param. Then just black/white list commands you either don't want to run or which should require a human to say ok.
blacklist is going to be a bad idea since so many commands can be made to run other commands with their arguments.
Yeah I agree. Ultimately I would suggest not having any kind of function call which returns an arbitrary command.
Instead, think of it as if you were enabling capabilities for AppArmor, by making a function call definition for just 1 command. Then over time suss out what commands you need your agent do to and nothing more.
There are MCP configured virtualization solutions that is supposed to be safe for letting LLM go wild. Like this one:
https://github.com/zerocore-ai/microsandbox
I haven't tried it.
You can build your agent into a docker image then easily limit both networking and file system scope.
Probably could be a bit cleaner, but it worked for me.Putting it inside docker is probably fine for most use cases but it's generally not considered to be a safe sandbox AFAIK. A docker container shares kernel with the host OS which widens the attack surface.
If you want your agent to pull untrusted code from the internet and go wild while you're doing other stuff it might not be a good choice.
Could you point to some resources which talk about how docker isn't considered a safe sandbox given the network and file system restrictions I mentioned?
I understand the sharing of kernel, while I might not be aware of all of the implications. I.e. if you have some local access or other sophisticated knowledge of the network/box docker is running on, then sure you could do some damage.
But I think the chances of a whitelisted llm endpoint returning some nefarious code which could compromise the system is actually zero. We're not talking about untrusted code from the internet. These models are pretty constrained.
> âYou only think you understand how a bicycle works, until you learn to ride one.â
This resonates deeply with me. That's why I built one myself [0], I really really love to truly understand how coding agents work. The learning has been immense for me, I now have working knowledge of ANSI escape codes, grapheme clusters, terminal emulators, Unicode normalization, VT protocols, PTY sessions, and filesystem operations - all the low-level details I would have never think about until I were implementing them.
[0] https://github.com/vinhnx/vtcode
>> âYou only think you understand how a bicycle works, until you learn to ride one.â
> This resonates deeply with me. That's why I built one myself [0]
I was hoping to see a home-made bike at that link.. Came away disappointed
Good one! Sorry to disappoint you. But personally, that line strike deeply with me, honestly.
It's conflating two issues though. Most people who can ride a bike can't explain the physics. They really don't know how it works. The bicycle lesson is about training the brain on a new task that cannot be taught in any other way.
This case is more like a journeyman blacksmith who has to make his own tools before he can continue. In doing so, he gets tools of his own, but the real reward was learning what is required to handle the metal such that it makes a strong hammer. And like the blacksmith, you learn more if you use an existing agent to write your agent.
Agree, to me, the wheel is the greatest invention of all. Everyone could have rode a bike, but the underlying physic and motion that came to `riding` is a whole another story.
I agree. I find LLMs a bit overblown. I don't think most people want to use chat as their primary interface. But writing a few agents was incredibly informative.
I did that, burned 2.6B tokens in the process and learned a lot: https://transitions.substack.com/p/what-burning-26-billion-p...
> A subtler thing to notice: we just had a multi-turn conversation with an LLM. To do that, we remembered everything we said, and everything the LLM said back, and played it back with every LLM call. The LLM itself is a stateless black box. The conversation weâre having is an illusion we cast, on ourselves.
the illusion was broken for me by Cline context overflows/summaries, but i think its very easy to miss if you never push the LLM hard or build you own agent. I really like this wording, amd the simple description is missing from how science communicators tend to talk about agents and LLMs imo
So I wrote an MCP using your code: https://gurddy-mcp.fly.dev. You can get the source code from https://github.com/novvoo/gurddy-mcp.
Question, how hard is it for someone new to agents to dip their toes into writing a simple agent to get data? (e.g., getting reviews from sites for sentiment analysis?)
Forgive if I get someting wrong: From what I see, it seems fundamentally it is a LLM being ran each loop with information about tools provided to it. On each loop the LLM evaluates inputs/context (from tool calls, inputs, etc.) and decided which tool to call / text output.
You can prototype this without writing any code at all.
Fire up "claude --dangerously-skip-permissions" in a fresh directory (ideally in a Docker container if you want to limit the chance of it breaking anything else) and prompt this:
> Use Playwright to fetch ten reviews from http://www.example.com/ then run sentiment analysis on them and write the results out as JSON files. Install any missing dependencies.
Watch what it does. Be careful not to let it spider the site in a way that would justifiably upset the site owners.
Dont you need to setup Playwright MCP first?
No. I don't use Playwright MCP at all - if the coding agent can run Python code it can use the Playwright Python library directly, if Node.js it can use the Playwright Node library.
Interesting, thanks for the info.
I wanted to run claude headlessly (-p) and playwright headlessly to get some content. I was using Playwright MCP and for some reason claude in headless mode could not open playwright MCP in headless mode.
I never realized i can just use playwright directly without the playwright MCP before your comment. Thanks once again.
Spoiler: it's not actually that easy. Compaction, security, sandboxing, planning, custom tools--all this is really hard to get right.
We're about to launch an SDK that gives devs all these building blocks, specifically oriented around software agents. Would love feedback if anyone wants to look: https://github.com/OpenHands/software-agent-sdk
Only on HN is there a âwell, actuallyâ with little substance followed by a comment about a launch.
The article isnât about writing production ready agents, so it does appear to be that easy
How autonomous/controllable are the agents with this SDK?
When I build an agent my standard is Cursor, which updates the UI at every reportable step of the way, and gives you a ton of control opportunities, which I find creates a lot of confidence.
Is this level of detail and control possible with the OpenHands SDK? Iâm asking because the last SDK that was simple to get into lacked that kind of control.
This work predates agents as we know them now and was intended for building chat bots (as in irc chat bots) but when auto-gpt I realized I could formalize it super nicely with this library:
https://blog.cofree.coffee/2025-03-05-chat-bots-revisited/
I did some light integration experiments with the OpenAI API but I never got around to building a full agent. Alas..
I agree with the sentiment but I also recommend you build a local only agent. Something that runs on llama.cpp or vllm, whatever... This way you can better grasp the more fundamental nature of what LLM's really are and how they work under the hood. That experience will also make you realize how much control you are giving up when using cloud based api providers like OpenAI and why so mane engineers feel that LLM's are a "black box". Well duh buddy you been working with apis this whole time, of course you wont understand much working just with that.
ive been trying this for a few week, but i dont at all currently own hardware good enough to be useful for local inference.
ill be trying again once i have written my own agent, but i dont expect to get any useful results compared to using some claude or gemini tokens
My man, we now have llms that are anywhere between 130 million to 1 trillion parameters available for us to run locally, I can guarantee there is a model for you there that even your toaster can run. I have a RTX 4090 but for most of my fiddling i use small models like Qwen 3 4b and they work amazing so there's no excuse :P.
well, i got some gemini models running on my phone, but if i switch apps, android kills it, so the call to the server always hangs... and then the screen goes black
the new laptop only has 16GB of memory total, with another 7 dedicated to the NPU.
i tried pulling up Qwen 3 4B on it, but the max context i can get loaded is about 12k before the laptop crashes.
my next attempt is gonna be a 0.5B one, but i think ill still end up having to compress the context every call, which is my real challenge
I recommend use low quantized models first. for example anywhere between q4 and q8 gguf models. Also dont need high context to fiddle around and learn the ins and outs. for example 4k context is more then enough to figure out what you need in agentic solutions. In fact thats a good limit to impose on yourself and start developing decent automatic context management systems internally as that will be very important when making robus agentic solutions. with all that you should be able to load an llm no issues on many devices.
I've found it much more useful to create an MCP server, and this is where Claude really shines. You would just say to Claude on web, mobile or CLI that it should "describe our connectivity to google" either via one of the three interfaces, or via `claude -p "describe our connectivity to google"`, and it will just use your tool without you needing to do anything special. It's like custom-added intelligence to Claude.
You can do this. Claude Code can do everything the toy agent this post shows, and much more. But you shouldn't, because doing that (1) doesn't teach you as much as the toy agent does, (2) isn't saving you that much time, and (3) locks you into Claude Code's context structure, which is just one of a zillion different structures you can use. That's what the post is about, not automating ping.
Honest question, as your comment confuses me.
Did you get to the part where he said MCP is pointless and are saying he's wrong?
Or did you just read the start of the article and not get to that bit?
I'd second the article on this, but also add to it that the biggest reason MCP servers don't really matter much any more is that the models are so capable of working with APIs, that most of the time you can just point them at an API and give them a spec instead. And the times that doesn't work, just give them a CLI tool with a good --help option.
Now you have a CLI tool you can use yourself, and the agent has a tool to use.
Anthropic itself have made MCP server increasingly pointless: With agents + skills you have a more composeable model that can use the model capabilities to do all an MCP server can with or without CLI tools to augment them.
I feel the CLI vs MCP debate is an apples to oranges framing. When you're using claude you can watch it using CLI's, running brew, mise, lots of jq but what about when you've built an agent that needs to work through a complicated API? You don't want to make 5 CRUD calls to get the right answer. A curated MCP tool ensures it can determinism where it matters most.. when interacting with customer data
> Another thing to notice: we didnât need MCP at all. Thatâs because MCP isnât a fundamental enabling technology. The amount of coverage it gets is frustrating. Itâs barely a technology at all. MCP is just a plugin interface for Claude Code and Cursor, a way of getting your own tools into code you donât control. Write your own agent. Be a programmer. Deal in APIs, not plugins.
Hold up. These are all the right concerns but with the wrong conclusion.
You don't need MCP if you're making one agent, in one language, in one framework. But the open coding and research assistants that we really want will be composed of several. MCP is the only thing out there that's moving in a good direction in terms of enabling us to "just be programmers" and "use APIs", and maybe even test things in fairly isolated and reproducible contexts. Compare this to skills.md, which is actually defacto proprietary as of now, does not compose, has opaque run-times and dispatch, is pushing us towards certain models, languages and certain SDKs, etc.
MCP isn't a plugin interface for Claude, it's just JSON-RPC.
I think my thing about MCP, besides the outsized press coverage it gets, is the implicit presumption it smuggles in that agents will be built around the context architecture of Claude Code --- that is to say, a single context window (maybe with sub-agents) with a single set of tools. That straitjacket is really most of the subtext of this post.
I get that you can use MCP with any agent architecture. I debated whether I wanted to hedge and point out that, even if you build your own agent, you might want to do an MCP tool-call feature just so you can use tool definitions other people have built (though: if you build your own, you'd probably be better off just implementing Claude Code's "skill" pattern).
But I decided to keep the thrust of that section clearer. My argument is: MCP is a sideshow.
I still don't really get it, but would like to hear more. Just to get it out of the way, there's obvious bad aspects. Re: press coverage, everything in AI is bound to be frustrating this way. The MCP ecosystem is currently still a lot of garbage. It feels like a very shitty app-store, lots of abandonware, things that are shipped without testing, the usual band-wagoning. For example instead of a single obvious RAG tool there's 200 different specific tools for ${language} docs
The core MCP tech though is not only directionally correct, but even the implementation seems to have made lots of good and forward-looking choices, even if those are still under-utilized. For example besides tools, it allows for sharing prompts/resources between agents. In time, I'm also expecting the idea of "many agents, one generic model in the background" is going to die off. For both costs and performance, agents will use special-purpose models but they still need a place and a way to collaborate. If some agents coordinate other agents, how do they talk? AFAIK without MCP the answer for this would be.. do all your work in the same framework and language, or to give all agents access to the same database or the same filesystem, reinventing ad-hoc protocols and comms for every system.
i treat MCP as a shorthand for "schema + documentation, passed to the LLM as context"
you dont need the MCP implementation, but the idea is useful and you can consider the tradeoffs to your context window, vs passing in the manual as fine tuning or something.
Yeah I was inspired after https://news.ycombinator.com/item?id=43998472 which is also very concrete
I love everything they've written and also Sketch is really good.
The formatting of the code is messed up on my phone. I was looking at the first bit thinking `call` was a function returning `None`. I thought initially it was doing some clever functional programming stuff but, no, just a linebreak that shouldn't be there.
I lost all respect for fly.io last time they published an article swearing about people are insane to not believe in vibe coding.
Looks like they keep up with the swearing in the companyâs blog. Just not my thing I guess.
I don't think "insane to not believe in vibe coding" is a fair summary of https://fly.io/blog/youre-all-nuts/ - that post wasn't about vibe coding (at least by its I-think-correct definition of prompt-driven coding where you don't pay any attention to the code that's being written), it was about AI-assisted engineering by professional software developers.
It did have some swear words in - as did many of the previous posts on the Fly.io corporate blog.
Worth highlighting that both OP article and the one Simon linked are by @tptacek, who is also one of the top commenters here on HN.
His fly.io posts are very much in his style. I figure they let him post there, without corp-washing, because any publicity is good publicity.
I would like an LLM to be integrated in the shell so I don't have to learn all the Unix tools arguments and write Bash scripts.
I am thinking of building agents that can partly replace manual testing using a headless browser.
You should write agents if you want to learn how agents work, if the problem you are trying to solve is not solved yet or if you are convinced that you will do much better job solving the problem again. Otherwise is just reinventing the wheel.
Maybe we should write an agent that writes an agent that writes an agent...
I realize now what I need in Cursor: A button for "fork context".
I believe that would be a powerful tool solving many things there are now separate techniques for.
crush-cli has this. I think the google gemini chat app also has this now.
The author shoulda written a REPL
No, because I know that "agents" are token burning machines - for me they're less efficient than the chat interface, slower and burning much more tokens.
I'm not surprised that AI companies would want me to use them though.. I know what you're doing there :)
.text-gray-600 { color: black; }
What's wrong with the OWASP Top Ten?
Author on Twitter a few years ago: https://x.com/tqbf/status/851466178535055362
The bravado posturing in this article is nauseating. Sure, there are a few serious points buried in there, but damn...dial it down, please.
they kinda feel like the cgi perl scripts of the mid 2020s.
You mean late 1990âs? :)
the point around MCPs is spot on
The Codex agent has an official TypeScript SDK now.
Why would Fly.io advocate using the vanilla GPT API to write an agent, instead of the official agent?
Because you won't learn as much using an agent framework, and, as you can see from the post, you absolutely don't need one.
It is also very simple to be a programmer.. see,
print "Hello world!"
so easy...
But that didn't use the H100 I just bought to put me out of my own job!
> Itâs Incredibly Easy
And this will work great until next week's update when Ralph responses will consist of "I'm sorry, it would be unethical for me to respond with lies, unless you pay for the Premium-Super-Deluxe subscription, only available to state actors and firms with a six-figure contract."You're building on quicksand.
You're delegating everything important to someone who has no responsibility to you.
Its easy to switch to an open source model
I love that the thing you singled out as not safe to run long term, because (apparently) of woke, was my weird deep-cut Labyrinth joke.
How.. please don't say use langxxx library
I am looking for a language or library agnostic pattern like we have MVC etc. for web applications. Or Gang of Four patterns but for building agents.
I'm not going to link my blog again but I have a reply on this post where I link to my blog post where I talk about how I built mine. Most agents fit nicely into a finite state machine or a directed acyclic graph that responds to an event loop. I do use provider SDKs to interact with models but mostly because it saves me a lot of boilerplate. MCP clients and servers are also widely available as SDKs. The biggest thing to remember, imo, is to keep the relationship between prompts, resources, and tools in mind. They make up a sort of dynamic workflow engine.
The whole post is about not using frameworks; all you need is the LLM API. You could do it with plain HTTP without much trouble.
When I ask for Patterns, I am seeking help for recurring problems that I have encountered. Context management .. small llms ( ones with small context size) break and get confused and forget work they have done or the original goal.
Start by thinking about how big the context window is, and what the rules should be for purging old context.
Design patterns can't help you here. The hard part is figuring out what to do; the "how" is trivial.
That's why you want to use sub-agents which handle smaller tasks and return results to a delegating agent. So all agents have their own very specialized context window.
That's one legit answer. But if you're not stuck in Claude's context model, you can do other things. One extremely stupid simple thing you can do, which is very handy when you're doing large-scale data processing (like log analysis): just don't save the bulky tool responses in your context window once the LLM has generated a real response to them.
My own dumb TUI agent, I gave a built in `lobotomize` tool, which dumps a text list of everything in the context window (short summary text plus token count), and then lets it Eternal Sunshine of the Spotless Agent things out of the window. It works! The models know how to drive that tool. It'll do a series of giant ass log queries, filling up the context window, and then you can watch as it zaps things out of the window to make space for more queries.
This is like 20 lines of code.
Did something similar - added `summarize` and `restore` tools to maximize/minimize messages. Haven't gotten it to behave like I want. Hoping that some fiddling with the prompt will do it.
FYI -- I vouched for you to undead this comment. It felt like a fine comment? I don't think you are shadowbanned but consider emailing the mods if you think you might me.
> Give each call different tools. Make sub-agents talk to each other, summarize each other, collate and aggregate. Build tree structures out of them. Feed them back through the LLM to summarize them as a form of on-the-fly compression, whatever you like.
You propose increasing the complexity of interactions of these tools, and giving them access to external tools that have real-world impact? As a security researcher, I'm not sure how you can suggest that with a straight face, unless your goal is to have more vulnerable systems.
Most people can't manage to build robust and secure software using SOTA hosted "agents". Building their own may be a fun learning experience, but relying on a Rube Goldberg assembly of disparate "agents" communicating with each other and external tools is a recipe for disaster. Any token could trigger a cascade of hallucinations, wild tangents, ignored prompts, poisoned contexts, and similar issues that have plagued this tech since the beginning. Except that now you've wired them up to external tools, so maybe the system chooses to wipe your home directory for whatever reason.
People nonchalantly trusting nondeterministic tech with increasingly more real-world tasks should concern everyone. Today it's executing `ping` and `rm`; tomorrow it's managing nuclear launch systems.
Write an agent, it's easy! You will learn so much!
... let's see ...
client = OpenAI()
Um right. That's like saying you should implement a web server, you will learn so much, and then you go and import http (in golang). Yeah well, sure, but that brings you like 98% of the way there, doesn't it? What am I missing?
That OpenAI() is a wrapper around a POST to a single HTTP endpoint:
Plus a few other endpoints, but it is pretty exclusively an HTTP/REST wrapper.
OpenAI does have an agents library, but it is separate in https://github.com/openai/openai-agents-python
I think you might be conflating an agent with an LLM.
The term "agent" isn't really defined, but its generally a wrapper around an LLM designed to do some task better than the LLM would on its own.
Think Claude vs Claude Code. The latter wraps the former, but with extra prompts and tooling specific to software engineering.
That's not an agent, it's an LLM. An agent is an LLM that takes real-world actions
No, it's saying "let's build a web service" and starting with a framework that just lets you write your endpoints. This is about something higher level than the nuts and bolts. Both are worth learning.
The fact you find this trivial is kind of the point that's being made. Some people think having an agent is some kind of voodoo, but it's really not.
An agent is more like a web service in your metaphor. Yes, building a web server is instructive, but almost nobody has a reason to do it instead of using an out of the box implementation once itâs time to build a production web service.
maybe more like âletâs write a web server but letâs use a library for the low level networking stackâ. That can still teach you a lot.
Didn't see such a bad piece of writing for a long time. Serously guys, is it just me? It's hard to read for some reason.
A very good blog article that I have read in a while. Maybe MCP could have been involved as well?