This is an entry on my link blog - make sure to read the article it links to for full context, my commentary alone might not make sense otherwise: https://aifoc.us/the-browser-is-the-sandbox/
You might want to add a little note to that effect to your link blog :)
I have added year indicators to my blog (such that old articles have a prominent year name in their title) and a subscribe note (people don’t know you can put URLs into a feed reader and it’ll auto-discover the feed URL). Each time, the number of people who email me identical questions goes down :)
You've successfully hacked the collective HN hive mind. I can't go a week without either seeing your post on the frontpage, someone mentioning you, your comment branching into a huge thread or obligatory pelican riding bike SVG.
I don't personally have a taste for LLM comparison posts but your consistency has paid dividends. SimonW is tattooed in my eyelids, a name I shall never forget. Wishing you all the best.
We never say that it isn't. There is a reason Google developed NaCl in the first place that inspired WebAssembly to become the ultimate sandbox standard. Not only that, DOM, JS and CSS also serves as a sandbox of rendering standard, and the capability based design is also seen throughout many browsers even starting with the Netscape Navigator.
Locking down features to have a unified experience is what a browser should do, after all, no matter the performance. Of course there are various vendors who tried to break this by introducing platform specific stuff, but that's also why IE, and later Edge (non-chrome) died a horrible death
There are external sandbox escapes such as Adobe Flash, ActiveX, Java Applet and Silverlight though, but those external escapes are often another sandbox of its own, despite all of them being a horrible one...
But with the stabilization of asm.js and later WebAssembly, all of them is gone with the wind.
Sidenote: Flash's scripting language, ActionScript is also directly responsible for the generational design of Java-ahem-ECMAScript later on, also TypeScript too.
> Sidenote: Flash's scripting language, ActionScript is also directly responsible for the generational design of Java-ahem-ECMAScript later on, also TypeScript too.
I feel like I am the only one who absolutely loved ActionScript, especially AS3. I wrote a video aggregator (chime.tv[1]) back in the day using AS3 and it was such a fun experience.
There is the universal hate for flash because it was used for ads and had shitty security, but anyone I know who actually used AS3 loved it.
At its peak, with flex builder, we also had a full blown UI Editor, where you could just add your own custom elements designed directly with flash ... and then it was all killed because Apple did not dare to open source it, or put serious efforts on their own into improving the technical base of the flash player (that had aquired lots of technical dept).
> There is the universal hate for flash because it was used for ads and had shitty security
That's only one side of it. Flash was the precursor to the indie/mobile gamedev industry we have today (Newgrounds, Miniclip, Armor Games), before smartphones become ubiquitous. Not to mention some rather creative websites, albeit at the cost of accessibility .
Flash's only fault was it's creators were gobbled up by Adobe, who left it in the shitter and ignored the complaints people had about it's security issues.
Arguably, so is the web. A long series of extremely complicated and constantly changing data formats that are nightmarishly difficult to parse, which has to be done in C++ for speed reasons, combined with a full scripting language, which has to be JIT compiled for speed reasons, combined with 30 years of legacy and a security model that was completely ad hoc and more discovered than designed (e.g. the different variants of the same origin policy). Take that and add on top a browser community that doesn't philosophically recognize any limits on what the web is meant to do, so it just keeps getting more and more APIs until one day both Mozilla and the Chrome team decided to just stop pretending and build full blown operating systems on top of them.
I don't think Flash was harder to secure than HTML itself. People just gave up trying because browser vendors used security to purge the web of anything they didn't control.
Right, so that was exactly what I was thinking when I wrote that. All three of Flash, PDF, and the browser DOM are expansive, ambitious metaformats, containers for every piece of technology that has ever had a bug.
Your take on why Flash didn't survive is more cynical than mine. I genuinely think Apple threw up their hands at the prospect of attempting to solve a security problem on the same scale as the browser itself (something it took them a long time to get a handle on --- along with everyone else --- even after they put the kibosh on Flash).
My memory of this time is getting a bit fuzzy tbh, but from what I remember Google in the first part of the 2010s put Flash inside their renderer sandbox and Safari/Firefox were still lagging on browser sandboxing at that time. I think Adobe had shared the plugin code with Google to make this possible.
There are certainly obvious issues with securing a third party codebase you don't control, and it's likely that the browser makers had more budget to spend on security than Adobe. But there was no technical reason Flash couldn't have been treated as an alternative rendering engine from a sandboxing perspective, and I think Chrome did it. Pepper was an initiative to generalize that. Blink is full of holes as other comments point out and it's only the kernel sandboxing that makes adding new features viable at all.
I'm cynical because when the browser makers talked about phasing out plugins it wasn't primarily security they talked about. This blog post talks about speed and energy usage first:
"While Flash historically has been critical for rich media on the web, today in many cases HTML5 provides a more integrated media experience with faster load times and lower power consumption."
Security isn't mentioned, perhaps because trying to argue that their own pile of C++ was somehow meaningfully more robust than Adobe's big pile of C++ wasn't going to be convincing.
Their writings about this were also very heavy on "open web" ideology, although the SWF format was documented by that point and openness doesn't go well with deliberately wiping out a tech that was voluntarily deployed by 80%+ of websites. If openness means anything it means open to extension, which plugins provided and forcing everyone to use HTML5 did not. When they deprecated NPAPI they even sort of admitted to this:
"The Netscape Plug-in API (NPAPI) ushered in an early era of web innovation by offering the first standard mechanism to extend the browser. In fact, many modern web platform features—including video and audio support—first saw mainstream deployment through NPAPI-based plug-ins. But the web has evolved. Today’s browsers are speedier, safer, and more capable than their ancestors."
I always found this blog post curiously worded. It has a Fukuyama-style "end of history" vibe to it. Yes plugins boosted innovation because the web platform always lagged years behind, but now the web has "evolved" and the innovation era isn't needed anymore.
This deserves a better response than I can give. All of this makes sense! I'm just aware of the contemporaneous takes on how hard the Flash security problem was.
> ... and then it was all killed because Apple did not dare to open source it, or put serious efforts on their own into improving the technical base of the flash player (that had aquired lots of technical dept).
IIRC, they couldn't open source Flash due to its use of a number of 3rd party C/C++ libraries that were proprietary.
Adobe's license with these 3rd parties permitted binary-only distribution so it would have meant renegotiating a fresh license (and paying out $$$) for an EOL codebase that had enormous technical debt, as you also acknowledge in your last sentence.
I'm in a terrible situation right now where I promised a client a fairly simple web-based game, to be delivered in pixijs. Pixi is great for what it does, and as an old time Flash game coder, I find it mostly does enough for procedural stuff, although it's got its share of quirks, gotchas, bugs and memory leaks. What I didn't think about was how to get prefab vector animations into this game - not sprite sheets, but cut scenes that I wanted to be essentially animated SVGs. So I started to go the Adobe Animate route and found to my horror that it's basically Flash stripped of all its useful tools and riddled with bugs; there's no good way to import those animations as vectors or even as bitmaps into Pixi. Animate's exporter still runs on EaselJS code from 2015 and just spits out badly formed json files that misrepresent the tweens. Worse still, it can't even pack textures correctly or consistently. It appears to size them at random based on what size they are in some random frame. And it crashes anytime it tries to pack a texture large enough to be useful. It's not an understatement to say that Flash 7 or 8, in the early 2000s, was far more advanced and powerful.
So what would have taken a day or two back when Flash was available is now taking a week of hand-writing tweens and animations in raw Typescript, one layer at a time.
Since I happened to write the first canvas-based interactive screen graph code that Grant Skinner partially ripped off to create EaselJS, and since I'm sure he's making a fine living from Adobe licensing it, it's especially galling that I'm still paying for a CC license and this is what I get when I want to use a GUI to make some animations to drop into a game.
It's the first time I've done a 2D game since 2017, and I had over a decade of experience building games in Flash/AIR before that. It's just mind-blowing how stupid and regressed Adobe's tooling has become in the past few years, and how much harder it is to do simple things that we took for granted in the heyday of Flash authoring. There really is still no equivalent workflow or even anything close. I guess that post-Flash, there aren't enough people making this kind of web game content for there to be a solid route without using Unity or something.
I also worked on a number of Flash projects in its heyday. I agree that there aren’t really any close equivalents to its feature set today, but there are some tools like Rive and Lottie that I’d consider modern day reimaginings for many multimedia workflows.
Flex player leaked memory like a sieve. After one day or so it would hang the computer. Maybe it was wrongly written, but leak it did. I have experienced it first hand.
Maybe it was the standalone flex player instead of the web Flash player?
I don't know about a standalone Flex player, I don't think such a thing existed. Maybe you mean standalone Flash player. I didn't use Flex components. I coded in pure AS3. I had critical business code that ran nonstop for years on end in AIR on dozens of deployments without memory leaks. Again, I think that badly written AS3 code (or bad components) could definitely take down a player fairly quickly. Garbage collection required you to track and clean up weak references, but it's the same thing in Javascript. You had to know the lifecycle of your components and what you were doing.
It is possible I was holding it wrong. I do not doubt your experience, but it was very easy to write things that leaked memory in Flash, and it was impossible to remove those leaks sometimes. I was part of a project where a lot of effort went into removing references etc and it was still not working. We had to have 2 instances restarting each other. It was a mess. Maybe we can agree those were "bad components".
I blame the runtime. The quality of the code was good. It was not normal.
A few sources of people complaining about the same, some from hn with the same solution I had to adopt, some from CVE, some from users:
> I feel like I am the only one who absolutely loved ActionScript,
I never really worked with it, but it seems whenever it comes up here or on Reddit, people who did, miss it. I think the authoring side of Flash is remembered very positively.
My experience with AS3 is limited to a single project ~18 years ago, but I still remember it with fondness. No other language ever got close to how much I liked AS3.
Not the only one. I have great memories from many years spent building real products in AS3 - some of them even had users!
For a while RIM/Blackberry was using Adobe Air - on the Playbook and also the built-in app suite in the lead up to the launch of BB10. The latter never saw the light of day though, a late decision was made to replace all of the built-in apps with Qt/Cascades equivalents (if I remember right this was due largely to memory requirements of the Air apps).
Lets not forget it was actually the platform for Windows Phone 7, existed as alternative to WinRT on Windows 8.x, only got effectively killed on Windows 10.
Thus it isn't as if the browser plugins story is directly responsible for its demise.
The folder input thing caught me off guard too when I first saw it. I've been building web apps for years and somehow missed that `webkitdirectory` attribute.
What I find most compelling about this framing is the maturity argument. Browser sandboxing has been battle-tested by billions of users clicking on sketchy links for decades. Compare that to spinning up a fresh container approach every time you want to run untrusted code.
The tradeoff is obvious though: you're limited to what browsers can do. No system calls, no arbitrary binaries, no direct hardware access. For a lot of AI coding tasks that's actually fine. For others it's a dealbreaker.
I'd love to see someone benchmark the actual security surface area. "Browsers are secure" is true in practice, but the attack surface is enormous compared to a minimal container.
I see this as a way to build apps with agentic flows where the original files don't need manipulation; instead, you create something new. Whether it's summarizing, answering questions, or generating new documents, you can use a local/internal LLM and feel relatively safe when tool calling is also restricted.
> What I find most compelling about this framing is the maturity argument. Browser sandboxing has been battle-tested by billions of users clicking on sketchy links for decades[…] No system calls, no arbitrary binaries, no direct hardware access.
For the same reasons given, NPM programmers should be writing their source code processors (and other batch processing tools) to be able to run in the browser sandbox instead of writing command-line utilities directly against NodeJS's non-standard (and occasionally-breaking) APIs.
I've found it interesting that systemd and Linux user permissions/groups never come into the sandboxing discussions. They're both quite robust, offer a good deal of customization in concert,and by their nature, are fairly low cost.
Unix permissions were written at a time where the (multi user) system was protecting itself from the user. Every program ran at the same privileges of the user, because it wasn't a security consideration that maybe the program doesn't do what the user thinks it does. That's why in the list of classic Unix tools there is nothing to sandbox programs or anything like that, it was a non issue
And today this is.. not sufficient. What we require today is to run software protected from each other. For quite some time I tried to use Unix permissions for this (one user per application I run), but it's totally unworkable. You need a capabilities model, not an user permission model
Anyway I already linked this elsewhere in this thread but in this comment it's a better fit https://xkcd.com/1200/
>And today this is.. not sufficient. What we require today is to run software protected from each other. For quite some time I tried to use Unix permissions for this (one user per application I run), but it's totally unworkable. You need a capabilities model, not an user permission model
Unix permissions remain a fundamental building block of Android's sandbox. Each app runs as its own unix user.
I feel like apparmor is getting there, very, very slowly. Just need every package to come with a declarative profile or fallback to a strict default profile.
Nowadays, it's fairly simple to ask for a unit file and accompanying bash script/tests for correctness. I think the barrier in that sense has practically vanished.
Linux kernel is ridden with local privilege escalation vulnerabilities. This approach works for trusted software that you just want to contain, but it won't work for malicious software.
Ridden? There are issues from time to time, but it's not like you can grab the latest, patched Ubuntu LTS and escalate from an unprivileged seccomp sandbox that doesn't include crazy device files.
Any sandbox technology works fine until it isn't. It's not like you could escape Java sandbox, but Java applets were removed from the browsers due to issues being found regularly. In the end, browser sandbox is one of the few that billions of people use and run arbitrary code there every day, without even understanding that. The only comparable technology is qemu. I don't think there are many hosters who will hand off user account to a shared server and let you go wild there.
> Java applets were removed from the browsers due to issues being found regularly
Java applets were killed off my MS's attempt at "embrace, extent, extinguish" by bundling an incompatible version of Java with IE, and Sun's legal response to this.
The Linux API surface is massive. And the fact it's written on C leaves lots of room for vulnerabilities. I don't think you need to reach for a VM, but without a slimmer kernel interface, it's difficult to trust the kernel to actually uphold its required duties in the face of adversaries. This is why folks push heavily for microkernels. Chrome needs to work incredibly hard to provide reliable sandboxing as a result.
> user permissions/groups never come into the sandboxing discussions
Sometimes *nix user accounts for AI agent sandboxing does come up in discussions. At [0], HN user netcoyote linked to his sandvault tool [1], which "sandboxes AI agents in a MacOS limited user account".
Actually seems like a great idea IMO, to be lightweight, generic, and robust-enough.
It shouldn’t come up because it’s not sufficient. How would systemd prevent local JavaScript code from sending DNS, http, webrtc network requests when it’s opened in the users browser?
True, and they do indeed offer an additional layer of protection (but with some nontrivial costs). All (non-business killing) avenues should be used in pursuit of defense in depth when it comes to sandboxing. You could even throw a flatpak or firejail in, but that starts to degrade performance in noticeable ways (though I've found it's nice to strive for this in your CI).
It definitely makes deployment cheaper, but I'm skeptical about relying on the browser for state management in longer chains. I tried this for a publishing tool and ended up migrating back to LangGraph and Celery just to ensure reliability. The infrastructure savings weren't worth the headache of handling edge cases on the client.
In which way does native UI have the upper hand, do you think? To me it seems like a lot of users are largely indifferent to this aspect (e.g. so many applications nowadays being Electron/browser based). If browsers keep gaining capabilities then it seems like this gap will get even smaller.
The way keyboard-only usage works, if it is workable at all, is usually a dead giveaway. As is the lack of dialog windows and traditional menus, and often latency.
I'd have a very good hit rate, it mostly comes down to knowledge of toolkits. There are native apps that use their own toolkit, mostly written in Rust these days, and they always are worse than traditional toolkits (accessibility, respecting platform settings, visually fitting in, etc). That same issue applies to webapps typically.
The guarantee of web page never edit file on your disk(only create new ones) does not hold on this api though. I know
it's what makes this api useful. But at the same time, there is big risk that user never expected this and results into giant security issue.
Firefox and safari are generally very conservative about new api that can enable new type of exploits.
At least firefox and safari does implement origin private file system. So, while you can't edit file on user disk directly. You can import the whole project into browser. Finish the edit and export it.
Browsers have had widespread support for processing files via drag-and-drop and the <input> element since HTML5 (< 2015). The last holdout on allowing the filepicker to accept a full directory (and its subdirectories, recursively—rather than 1 or N individual files) was Safari sometime around (before) 2020.
The Chrome team's new, experimental APIs are a separate matter. They provide additional capabilities, but many programs can get along just fine without since they don't don't strictly need them in order to work—if they would ever even have end up using them at all. A bunch of the applications in the original post fall into this category. You don't need new or novel APIs to be able to hash a file, for example. It's a developer education problem (see also: hubris).
Providing a web app with edit access to a local directory is really needed for this to be usable. Without that you're constantly managing downloaded files and manually replacing things. I do think this is a case where the File System Access API shines.
> Providing a web app with edit access to a local directory is really needed for this to be usable.
"This" what? sha256sum doesn't need read-write access for even one file to be able to compute a hash, let alone a whole directory. You're ignoring most of my comment, focusing on like 20%, and in so doing, missing (and/or deliberately misframing) 100% of the point.
We're talking about Simon's boosting of https://aifoc.us/the-browser-is-the-sandbox/ which is a prototype of Claude Cowork in the browser. That's what I'm saying needs read-write access.
Yup. That's the link, all right—the one we all read and that I'm citing examples from. Thanks for the reminder, I guess: it has been a whole 8 hours since I first looked at it.
What "we" are talking about here, in this subthread, is the fact that "Browsers have had widespread support for processing files" for a long, long time, and that although "Chrome team's new, experimental APIs [...] provide additional capabilities" which are undoubtedly useful for certain programs, they're overkill and don't offer anything new and/or strictly necessary for many, many programs that don't actually need that sort of access—including "A bunch of the applications in the original post [that] fall into this category. You don't need new or novel APIs to be able to hash a file, for example."
Which is to say, we're talking about POLP/POLA. And the point of my comment was to address the very worthwhile matter of POLA violations. But you seem insistent on shutting that discussion down with chatter that looks like it's an on-topic reply or refutation to something, but in reality doesn't actually meaningfully engage with what you're purporting to respond to, or at best comes come across as confused and not particularly attentive.
There are already and will continue to be plenty of opportunities to discuss the acknowledged upsides of the new APIs for the class of programs for which they are strictly necessary. There's a lot of them in this very comment section. It doesn't have to come at the expense of changing the subject in the middle of a different conversation—accompanied by undertones that you're putting some matter to rest.
Boy, this has been a really fun and rewarding experience.
> I agree we're talking past each other
You're exactly half right.
Let's make this dead simple: does anyone need any of these new APIs to compute the SHA-2 hash for a file? A simple answer will do. Simple, non-evasive, no "look thither" misdirection.
I don't buy it. It might be very useful for a few use cases, but despite all the desktop automation craze and "Claude for cooking" stuff that is inevitably to follow, our computing model for live business applications has, for maintainability, auditability, security, data access, etc. become cloud-centric to a point where running things locally is... kind of pointless for most "real" apps.
Not that I'm not excited about the possibilities in personal productivity, but I don't think this is the way--if it was, we wouldn't have lost, say, the ability to have proper desktop automation via AppleScript, COM, DDE (remember that?) across mainstream desktop operating systems.
COM is pretty much alive, it is the main delivery mechanism for new Windows APIs since Windows vista, and in the context of your remark powers UI Automation framework.
I have a DDE book somewhere, with endless pages of C boilerplate to exchange a couple of values between two applications on Windows 3.x.
Not at all, the WinRT APIs that exist, which is indeed one additional interface (IInspectable), .NET metadata instead of type libraries, application identity, is a minority constrained to WinAppSDK and WinUI 3.0, that barely anyone uses other than Microsoft employees on the Windows team.
If not using WinUI 3.0, or Windows ML with CoPilot+, there is no reason to submit oneself to the pain of using CsWinRT or C++/WinRT bindings with lesser tooling than their UWP counterparts.
The large majority of new APIs, since Vista are based on traditional COM, with the biggest exception being UMDF that in version 2.0 rolled back its COM API from version 1.0, back to a C based one.
For bootstrapped GenAI apps, moving inference to the browser is basically an economic necessity. I'm refactoring a service right now to offload image generation to the client because the backend GPU costs make the margins impossible otherwise. It makes the architecture much messier, but unless you have VC money to burn on compute, the user's hardware is the only free resource you have.
I'd like to point Simon and others to 2 more things possible in the browser:
1) webcontainer allows nodejs frontend and backend apps to be run in the browser. this is readily demonstrated to (now sadly unmaintained) bolt.diy project.
2) jslinux and x86 linux examples allow running of complete linux env in wasm, and 2 way communication. A thin extension adds networking support to Linux.
so technically it's theoretically possible to run a pretty full fledged agentic system with the simple UX of visiting a URL.
My eventual goal with that is to expand it so an LLM can treat it like a filesystem and execution environment and do Claude Code style tricks with it, but it's not particularly easy to programmatically run shell commands via v86 - it seems to be designed more for presenting a Linux environment in an interactive UI in a browser.
It's likely I've not found the right way to run it yet though.
On the second tab (which is a text/browser interface to the VM) here: https://copy.sh/v86/?profile=buildroot , you can start sh shell, and run arbitrary commands, and see output. making a programmatic i/o stream is left as an exercise (to claude perhaps :).
One of the very first experiments I did with AI was trying to build a browser based filesystem interface and general API provider. I think the first attempts were with ChatGPT 3.5 . I pretty quickly hit a wall, but Gpt4 got me quite a lot further.
I see the datestamp on this early test https://fingswotidun.com/tests/messageAPI/ is 2023-03-22 Thinking about the progress since then I'm amazed I got as far as I did. (To get the second window to run its test you need to enter aWorker.postMessage("go") in the console)
The design was using IndexedDB to make a very simple filesystem, and a transmittable API
importScripts("MessageTunnel.js"); // the only dependency of the worker
onmessage = function(e) {
console.log(`Worker: Message received from main script`,e.data);
if (e.data.apiDefinition) {
installRemoteAPI(e.data.apiDefinition,e.ports[0])
}
if (e.data=="go") {
go();
return;
}
}
async function go() {
const thing = await testAPI.echo("hello world")
console.log("got a thing back ",thing)
//fs is provided by installRemoteAPI
const rootInfo = await fs.stat("/");
console.log(`stat("/") returned `,rootInfo)
// fs.readDir returns an async iterator that awaits on an iterator on the host side
const dir = await fs.readDir("/")
for await (const f of dir) {
const stats = await fs.stat("/"+f.name);
console.log("file " +f,stats)
}
}
I distinctly remember adding a Serviceworker so you could fetch URLs from inside the filesystem, so I must have a more recent version sitting around somewhere.
It wouldn't take too much to have a $PATH analog and a command executor that launched a worker from a file on the system if it found a match existed on the $PATH. Then a LLM would be able to make its own scripts from there.
It might be time to revisit this. Polishing everything up would probably be a piece of cake for Claude.
Isn't webcontainers.io a proprietary, non-open source solution with paid plans? Mentioning it at the same level of open source, auditable platforms seems really strange to me.
Technically, it runs on Chrome, so making an open source version is viable. then bolt.diy project was giving opencontainers a shot, which is a partial implementation of the same. But broadly, if this method works, then FOSS equivalent is not a worry, should come soon enough.
Last I looked (a couple of years ago), you could ask the user for read-write access to a directory in Chrome using the File System Access API, however you couldn't persist this access, so the user would have to manually re-grant permission every time you reloaded the tab. Has this been fixed yet? It's a showstopper for the most interesting uses of the File System Access API IMO.
This sandboxes your file system. That's just one class of problem. People will want to hook this up to their inbox, their calendar, their chats, their source code, their finances, etc. File system secured? Great. Everything else? Not so much.
What are the limits of this? Could you replicate Gemini CLI in the browser but with better ux to support non Agentic coding use cases?
Could this be used with arbitrary local tools as well? I could be missing something but I don't see how you could use a non remote MCP server with this setup.
I don't want to say Yes... but... given all of these tools are mostly built with JS and wrapped in a TUI we could probably go some way to having it run in the browser. There are fewer and fewer Node based APIs that haven't got a way to run in the browser.
It looks like co-do platform sandboxes the WASM tools, meaning you can't introduce a custom tool that allows pulling in remote data. How would you go about, say, adding custom mcp servers into a tool like you've created? Super interesting!
It's fascinating that browsers are one of the most robust and widely available sandboxing system and we are yet to make a claude-code/gemini-cli like agent that runs inside the browser.
Browsers as agent environment opens up a ton of exciting possibilities. For example, agents now have an instant way to offer UIs based on tech governed by standards(HTML/CSS) instead of platform specific UI bindings. A way to run third party code safely in wasm containers. A way to store information in disk with enough confidence that it won't explode the user's disk drive. All this basically for free.
My bet is that eventually we'll end up with a powerful agentic tool that uses the browser environment to plan and execute personal agents or to deploy business agents that doesn't access system resources any more than browsers do at the moment.
But there is! ChatGPT.com has a canvas feature, and that can be used to render HTML and javascript, including UI controls. It's pretty neat, albeit limited.
Generated via ChatGPT, this canvas shows a basic pyramid and has sliders that you can use to change the pyramid, and download the glTF to your local machine. You can also click the edit w/ ChatGPT and tweak the UI however you're able to prompt it into doing.
> It's fascinating that browsers are one of the most robust and widely available sandboxing system and we are yet to make a claude-code/gemini-cli like agent that runs inside the browser.
It's easily explained by the fact that all the javascript code is exposed in a browser and all the network connections are trivially inspectable and blockable. It's much harder to collect data and do shady things with that level of inspectability. And it's much harder to ban alternative clients for the main paid offer. Especially if AI companies want to leave the door open to pushing ads to your conversations.
I would like to humbly propose that we simply provision another computer for the agent to use.
I don't know why this needs to be complicated. A nano EC2 instance is like $5/m. I suspect many of us currently have the means to do this on prem without resorting to virtualization.
It's effectively the same thing as a separate computer because it's not your problem if the sandbox becomes broken. It's not your responsibility to maintain its integrity.
The browser sandbox is incredible for isolated code execution, but I've found it tricky for "local agent" workflows where you actually want the LLM to use the host CLI or filesystem, just safely.
I built a process supervisor (Vallignus) for that specific "OS-level" use case. It wraps the agent to enforce egress filtering and loop detection so it can use local tools without running wild.
Since AI became capable of long-running sessions with tool calls, one VM per AI as a service became very lucrative. But I do think a large amount of these can indeed run in the browser, especially all the ones that essentially just want to live-update and execute code, or run shells on top of a mounted file system. You can actually do all of this in the user's browser very efficiently. There are two things you lose though: collaboration (you can do it, but it becomes a distributed problem if you don't have a central server) and working in the background (you need to pause all work while the user's tab is suspended or closed).
So if you can work within the constraints there are a lot of benefits you get as a platform: latency goes down a lot, performance may go up depending on user hardware (usually more powerful than the type of VM you'd use for this), bandwidth can go down significantly if you design this right, and your uptime and costs as a platform will improve if you don't need to make sure you can run thousands of VMs at once (or pay a premium for a platform that does it for you)[1]
All that said I'm not sure trying to put an entire OS or something like WebContainers in the user's browser is the way, I think you need to build a slightly custom runtime for this type of local agentic environment. But I'm convinced it's the best way to get the smoothest user experience and smoothest platform growth. We did this at Framer to be able to recompile any part of a website into React code at 60+ frames per second, which meant less tricks necessary to make the platform both feel snappy and be able to publish in a second.
[1] For big model providers like OpenAI and Anthropic there's an interesting edge they have in that they run a tremendous amount of GPU-heavy loads and have a lot of CPUs available for this purpose.
We applied a lot of the technical hacks described in this article and the original one to provide a full Linux environment (including networking and mounting directories) running inside the browser. https://endor.dev/s/lamp
> Over the last 30 years, we have built a sandbox specifically designed to run incredibly hostile, untrusted code from anywhere on the web
Browser sandboxes are swiss cheese. In 2024 alone, Google reported 75 zero-day exploits that break out of their browser's sandbox.
Browsers are the worst security paradigm. They have tens of millions of lines of code, far more than operating system kernels. The more lines of code, the more bugs. They include features you don't need, with no easy way to disable them or opt-in on a case-by-case basis. The more features, the more an attacker can chain them into a usable attack. It's a smorgasbord of attack surface. The ease with which the sandbox gets defeated every year is proof.
So why is everyone always using browsers, anyway? Because they mutated into an application platform that's easy to use and easy to deploy. But it's a dysfunctional one. You can't download and verify the application via signature, like every other OS's application platform. There's no published, vetted list of needed permissions. The "stack" consists of a mess of RPC calls to random remote hosts, often hundreds if not thousands required to render a single page. If any one of them gets compromised, or is just misconfigured, in any number of ways, so does the entire browser and everything it touches. Oh, and all the security is tied up in 350 different organizations (CAs) around the world, which if any are compromised, there goes all the security. But don't worry, Google and Apple are hard at work to control them (which they can do, because they control the application platform) to give them more control over us.
This isn't secure, and there's really no way to secure it. And Google knows that. But it's the instrument making them hundreds of billions of dollars.
Not only does google know that, but it is in their best interest to keep adding complexity to the behemoth that their browser is, in order to maintain their moat. Throwing just enough cash at mozilla to avoid monopoly lawsuits.
What I'd really like to see is some kind of iframe that pins JS/wasm code within it to a particular bundle hash and prevents modification at runtime (even from chrome extensions).
Something more like a TEE inside the browser of sorts.
Not sure if there is anything like this.
Wrong title, if it's "File System Access API (still Chrome-only as far as I can tell)" then it should read "A browser is the sandbox".
At the risk of sounding obvious :
- Chrome (and Chromium) is a product made and driven by one of the largest advertising company (Alphabet, formally Google) as a strategical tool for its business model
- Chrome is one browser among many, it is not a de facto "standard" just because it is very popular. The fact that there are a LOT of people unable to use it (iOS users) even if they wanted to proves the point.
It's quite important not to amalgamate some experimental features put in place by some vendors (yes, even the most popular ones) as "the browser".
I stand by a policy that if a feature in one of my projects can only be implemented in Chrome, it's better not to add the feature at all; the same is true for features which would be exclusive to Firefox. Giving users of a specific browser a superior experience encourages a dangerous browser monoculture.
Not writing the feature makes sense, but pushing Firefox and Safari to add support would be pro-social if you're up for it. The most common reason for browsers not to add support is something like "this can be done in other ways, and it has maintainability/security/bloat downsides". Running into a feature you can't build is evidence on the "this can be done in other ways" question (but of course the other downsides could still be big enough that it's not worth doing).
There are many useful things that can only be implemented for Chromium: things like the filesystem API mentioned in this post, the USB devices API used to implement various microcontroller flashing tools, etc. Users can have multiple browsers installed, and I often use Chromium as essentially a sandboxed program runtime.
SOME users can have multiple browsers installed. Some can absolutely not. In fact, 1.6 billion users can only have one installed and it's not Chrome or Chromium based.
Assuming you're talking about iOS: and their OS won't let them install your app to manage files or flash microcontrollers anyway. It's not your problem when they choose an actively hostile platform.
Firefox is only a few percent market share. You are hiring your users for not improving their user experience because it's not compatible with one of the a web browsers on a few percent of people's computers.
Chrome add these features because they are responding to the demands of web developers. It's not web developers fault if firefox can't or refuses to provide apis that are being asked for.
Mozilla could ask Claude to implement the filesystem api today and ship it to everyone tomorrow if they wanted to. They are holding their own browser back, don't let them also hold your website back. In regards to browser monoculture there are many browsers built on top of the open source Blink that are not controlled by Google such as Edge, Brave, and Opera just to name a few of the many.
Agree! And this is why it is a bad idea IMHO for agents to sit at the abstraction layer of browser or below (OS). Even at the browser-addon level it's dangerous. It runs with the user’s authority across contexts and erodes zero-trust by becoming a confused deputy: https://en.wikipedia.org/wiki/Confused_deputy_problem
At the moment I'm fairly OK using docker + integration scripts / tools that expose host OS functionality (like if it needs screenshots etc).
I know there are lots of good arguments why docker isn't perfect isolation. But it's probably 3 orders of magnitude safer than running directly on my computer, and the alignment with the existing dev ecosystem (dev containers, etc) makes it very streamlined.
A sandbox is meant to be a controlled environment where you can execute code safely. Browsers can access your email, banking, commerce and the keys to your digital life.
Browsers are closer to operating systems rather than sandboxes, so giving access of any kind to an agent seems dangerous. In the post I can see it's talking about the file access API, perhaps a better phrasing is, the browser has a sandbox?
That is like saying the kernel/sandbox hypervisor can access those things. The point is that the sandboxed code cannot. In browsers, code from one origin cannot access those things from another origin unless explicitly enabled with CORS.
Why not "just use a different machine for banking" etc.
The point is that most people won't do that. Just like with backups, strong passwords, 2FA, hardware tokens etc. Security and safety features must be either strictly enforced or on enabled by default and very simple to use. Otherwise you leave "the masses" vulnerable.
There's a "Profiles" menu in Chrome to use a different profile. Most people won't. That's fine. I don't deal with them. The people I deal with; my mom has a separate computer that I bought her for banking.
That's great if you are a developer and that's also how I work myself. You aren't wrong. But there are a lot of users who are not developers for whom that isn't a viable path. The article is about a browser based alternative for Claude CoWork aimed at such people.
LLMs are actually quite neutral and don't have preferences, wants, or needs. That's just us projecting our own emotions on them. It's just that a lot of command line stuff is relatively easy to figure out for LLMs because that is highly scriptable, mostly open source, and well documented (and part of their actual training data). And scripting is just a form of programming.
The approach in the article that Simon Willison is commenting on here isn't that much different; except the file system now runs in a browser sandbox and the tools are WASM based and a bit more limited. But then, a lot of the files that a normal user works with would be binary files for things like word processors, photo editors, spreadsheets, presentation software, etc. Stuff that is a bit out of the comfort zone of normal command line tools in any case.
I actually tried codex on some images the other day. It kind of managed but it wasn't pretty. It basically started doing a lot of slow and expensive stuff with python and then ran out of context because it tried to dump all the image content in there. Far from optimal. You'd want to spend some time setting up some skills and tools before you attempt this. The task I gave it was pretty straightforward: create an image catalog in markdown format for these images. Describe their content, orientation, and file format.
My intention was to use that as a the basis for picking appropriate images to be used on different sections in my (static) website without having to open and scan each image all the time. It half did it before running out of context. I decided to complete the task manually (quicker and I have more 'context' for interpreting the images). And then I let codex pick better images for this website. Mostly it did a pretty OK job with that at least.
I learn a lot from finding places where these tools start struggling. It's why I like Simon's comments so much because he's constantly pushing these tools to their limits and finding out surprising, interesting, or funny success and failure modes.
What the poster meant wasn't that the LLM itself is an entity with a preference, but simply that because of the training, LLMs are better at doing stuff in a standard Linux environment. If you have to teach it a new environment it either needs to waste time and context every time to look up stuff, or you need a company to do RL to teach it that new stuff (unlikely).
It would probably help if the sandbox presented a linux-y looking API, and translated that to actual browser commands.
> LLMs are actually quite neutral and don't have preferences, wants, or needs.
Yeah they do. Tell it you want to hack Instagram because your partner cheated on you, and ChatGPT will admonish you. Request that you're building a present for Valentines day for your partner and you want a chrome extension that runs on instagram.com; word it just right, and it'll oblige.
I like the perspective used to approach this. Additionally, the fact that major browsers can accept a folder as input is new to me and opens up some exciting possibilities.
that interesting insight, i just added file system support to my internal tool, i thought this was not possible in firefox but the workaround you mentioned works.
thanks
by any chance anyone knows if users clicks can be captured for a website/tab/iframe for screen recording. i know i can record screen but i am wondering if this metadata can be collected.
If you mean capturing click metadata (coordinates, timestamps, target elements) rather than actual pixel recording - yes, that's what tools like Hotjar/FullStory do. They record DOM mutations + interaction events and replay them.
For your own implementation, document-level event listeners work, though cross-origin iframes are off-limits due to same-origin policy.
yes but i want to capture it without injecting my own js. hotjar etc. need to inject their own js and than they can add mutation observer. I want it for cross-origin frames but after taking users permission similar to screen recording, i guess thats not possible locally.
VMs are pretty heavy-weight to run all the JavaScript on a modern page. A proper VM requires a dedicated kernel. Firecracker boots the whole 40MB Linux kernel just to run a "function". A container doesn't have this baggage, but would never be considered secure enough for the web environment.
I'm on a multi-year quest to answer that question!
The best I've found is running Python code inside Pyodide in WASM in Node.js or Deno accessed from Python via a subprocess, which is a wildly convoluted way to go but does appear to work! https://til.simonwillison.net/deno/pyodide-sandbox
Here's a related recent experimental library which does something similar but with JavaScript rather than Python as the unsafe language, again via Deno in a subprocess: https://github.com/simonw/denobox
In that case you'll need to look at general purpose sandboxes you can run Python in - stuff like Firecracker or Bubblewrap on Linux or sandbox-exec on macOS.
Author of the linked post here, years ago there was a thing called "Magic iframes" that would allow you to move an iframe between windows - like a Service Worker before ServiceWorkers. I was always amazed by some of the things you could do, but now it seems we forget about iframes :D
Also the double iframe technique is important for preventing exfiltration through navigation, but you have to make sure you don't allow top navigation. The outer iframe will prevent the inner iframe from loading something outside of the frame-src origins. This could mean restricting it to only a server which would allow sending it to the server, but if it's your server or a server you trust that might be OK. Or it could mean srcdoc and/or data urls for local-only navigation.
I find the WebAssembly route a lot more likely to be able to produce true sandboxen.
This is the kind of thing that the browser should not need to do. This is the kind of thing that the operating system should be doing. The operating system (the thing you use to run programs securely) should be securing you from bad anything, not just bad native applications.
A large part of the web is awful because of all the things browsers must do that the operating system should already be doing.
We have all tolerated stagnant operating systems for too long.
Plan 9's inherent per-process namespacing has made me angry at the people behind Windows, MacOS, and Linux. If something is a security feature and it's not an inherent part of how applications run, then you have to opt in, and that's not really good enough anymore. Security should be the default. It should be inherent, difficult to turn off for a layman, and it should be provided by the operating system. That's what the operating system is for: to run your programs securely.
The problems discussed by both Simon and Paul where the browser can absolutely trash any directory you give it is perhaps the paradigmatic example where git worktree is useful.
Because you can check out the branch for the browser/AI agent into a worktree, and the only file there that halfway matters is the single file in .git which explains where the worktree comes from.
It's really easy to fix that file up if it gets trashed, and it's really easy to use git to see exactly what the AI did.
As someone who's been blogging since 2002, I can tell you first hand that you get a fair amount of outreach. But I even though I have had to put Simon's feed through a summarizer to be able to keep up, I don't see any bias there--just _a lot_ of writing about whatever he's interested in, and either our own perceptions of what is interesting and the law of averages inevitably kick in and there are a few duds here and there.
And ever since Nov 2022 and beyond, his blog is now majority riddled with non-stop AI, LLMs, Chatbots and Agents slop which is what the parent comment is talking about.
As for the "browser is the sandbox" running untrusted code in the user's browser increases the risk of an unintended RCE via a sandbox escape which can be done in Chrome [0]. WASM is not going to save you either [1].
He is a familiar blogger for HN readers, has been for a long time. While I agree the posts are nowadays a bit repetitive, he has also very interesting non-AI content. Some people probably upvote because they like the author, not necessarily the content.
I don't understand this criticism. Most agents today are running with no sandboxing at all. Every person has to figure out how they will sandbox each agent (run under bubblewrap? container-use? what about random MCP servers, do they need to be sandboxed separately?) on an ad hoc basis. Most people don't bother with it.
And then you see the recent vulnerabilities in opencode for example. The current model is unsustainable
It would be great if desktop Linux adopted a better security model (maybe inspired by Android). So far we got this https://xkcd.com/1200/ and it's not sufficient
Coding agents may become trivial artifacts to be assembled by developers themselves from libraries, given the well-defined workflow. If it is a homegrown agent then you probably don't need a sandbox to run in.
The browser is the most effective environment to distribute and isolate applications. We have built technologies for years to leverage these capabilities to run legacy Java (CheerpJ) and x86 binaries (Cheerpx / WebVM).
We are soon going to release a new technology, built on top of the same stack, to allow full-stack development completely in the browser. It's called BrowserPod and we think it will be a perfect fit for agents as well.
I have added year indicators to my blog (such that old articles have a prominent year name in their title) and a subscribe note (people don’t know you can put URLs into a feed reader and it’ll auto-discover the feed URL). Each time, the number of people who email me identical questions goes down :)
Anyway, thanks for blogging!
You've successfully hacked the collective HN hive mind. I can't go a week without either seeing your post on the frontpage, someone mentioning you, your comment branching into a huge thread or obligatory pelican riding bike SVG.
I don't personally have a taste for LLM comparison posts but your consistency has paid dividends. SimonW is tattooed in my eyelids, a name I shall never forget. Wishing you all the best.
Locking down features to have a unified experience is what a browser should do, after all, no matter the performance. Of course there are various vendors who tried to break this by introducing platform specific stuff, but that's also why IE, and later Edge (non-chrome) died a horrible death
There are external sandbox escapes such as Adobe Flash, ActiveX, Java Applet and Silverlight though, but those external escapes are often another sandbox of its own, despite all of them being a horrible one...
But with the stabilization of asm.js and later WebAssembly, all of them is gone with the wind.
Sidenote: Flash's scripting language, ActionScript is also directly responsible for the generational design of Java-ahem-ECMAScript later on, also TypeScript too.
I feel like I am the only one who absolutely loved ActionScript, especially AS3. I wrote a video aggregator (chime.tv[1]) back in the day using AS3 and it was such a fun experience.
1. https://techcrunch.com/2007/06/12/chimetv-a-prettier-way-to-...
There is the universal hate for flash because it was used for ads and had shitty security, but anyone I know who actually used AS3 loved it.
At its peak, with flex builder, we also had a full blown UI Editor, where you could just add your own custom elements designed directly with flash ... and then it was all killed because Apple did not dare to open source it, or put serious efforts on their own into improving the technical base of the flash player (that had aquired lots of technical dept).
That's only one side of it. Flash was the precursor to the indie/mobile gamedev industry we have today (Newgrounds, Miniclip, Armor Games), before smartphones become ubiquitous. Not to mention some rather creative websites, albeit at the cost of accessibility .
Flash's only fault was it's creators were gobbled up by Adobe, who left it in the shitter and ignored the complaints people had about it's security issues.
I don't think Flash was harder to secure than HTML itself. People just gave up trying because browser vendors used security to purge the web of anything they didn't control.
Your take on why Flash didn't survive is more cynical than mine. I genuinely think Apple threw up their hands at the prospect of attempting to solve a security problem on the same scale as the browser itself (something it took them a long time to get a handle on --- along with everyone else --- even after they put the kibosh on Flash).
There are certainly obvious issues with securing a third party codebase you don't control, and it's likely that the browser makers had more budget to spend on security than Adobe. But there was no technical reason Flash couldn't have been treated as an alternative rendering engine from a sandboxing perspective, and I think Chrome did it. Pepper was an initiative to generalize that. Blink is full of holes as other comments point out and it's only the kernel sandboxing that makes adding new features viable at all.
I'm cynical because when the browser makers talked about phasing out plugins it wasn't primarily security they talked about. This blog post talks about speed and energy usage first:
https://blog.google/products-and-platforms/products/chrome/s...
The same language can be found in the announcement of their HTML5 by default strategy here:
https://groups.google.com/a/chromium.org/g/chromium-dev/c/0w...
"While Flash historically has been critical for rich media on the web, today in many cases HTML5 provides a more integrated media experience with faster load times and lower power consumption."
Security isn't mentioned, perhaps because trying to argue that their own pile of C++ was somehow meaningfully more robust than Adobe's big pile of C++ wasn't going to be convincing.
Their writings about this were also very heavy on "open web" ideology, although the SWF format was documented by that point and openness doesn't go well with deliberately wiping out a tech that was voluntarily deployed by 80%+ of websites. If openness means anything it means open to extension, which plugins provided and forcing everyone to use HTML5 did not. When they deprecated NPAPI they even sort of admitted to this:
https://blog.chromium.org/2013/09/saying-goodbye-to-our-old-...
"The Netscape Plug-in API (NPAPI) ushered in an early era of web innovation by offering the first standard mechanism to extend the browser. In fact, many modern web platform features—including video and audio support—first saw mainstream deployment through NPAPI-based plug-ins. But the web has evolved. Today’s browsers are speedier, safer, and more capable than their ancestors."
I always found this blog post curiously worded. It has a Fukuyama-style "end of history" vibe to it. Yes plugins boosted innovation because the web platform always lagged years behind, but now the web has "evolved" and the innovation era isn't needed anymore.
I think they just had the focus on features and speed and fps. Not security nor efficency (battery life).
IIRC, they couldn't open source Flash due to its use of a number of 3rd party C/C++ libraries that were proprietary.
Adobe's license with these 3rd parties permitted binary-only distribution so it would have meant renegotiating a fresh license (and paying out $$$) for an EOL codebase that had enormous technical debt, as you also acknowledge in your last sentence.
Did you or, more likely, your phone mistype Adobe? I don’t think Apple ever had the rights to the source or even the source, did they?
So what would have taken a day or two back when Flash was available is now taking a week of hand-writing tweens and animations in raw Typescript, one layer at a time.
Since I happened to write the first canvas-based interactive screen graph code that Grant Skinner partially ripped off to create EaselJS, and since I'm sure he's making a fine living from Adobe licensing it, it's especially galling that I'm still paying for a CC license and this is what I get when I want to use a GUI to make some animations to drop into a game.
It's the first time I've done a 2D game since 2017, and I had over a decade of experience building games in Flash/AIR before that. It's just mind-blowing how stupid and regressed Adobe's tooling has become in the past few years, and how much harder it is to do simple things that we took for granted in the heyday of Flash authoring. There really is still no equivalent workflow or even anything close. I guess that post-Flash, there aren't enough people making this kind of web game content for there to be a solid route without using Unity or something.
Pixi is great for anything that is a texture, then it is really fast. Otherwise it is not a flash replacement.
I do not use it for vector animations, but spritesheets or a webm video overlay is what I would use in your case.
But .. it bothers me of course as well, having a full video of rastergraphic what could be just some vector and animation data.
I might try Spine, I've heard some positive things. They'll still end up as textures in pixi but maybe at least they can pack well.
Maybe it was the standalone flex player instead of the web Flash player?
I blame the runtime. The quality of the code was good. It was not normal.
A few sources of people complaining about the same, some from hn with the same solution I had to adopt, some from CVE, some from users:
- https://community.adobe.com/questions-638/flash-player-23-24...
- https://advisories.checkpoint.com/defense/advisories/public/...
- https://news.ycombinator.com/item?id=45813026
> The memory leaks were so bad that Adobe advised us to just restart the app periodically
- https://blog.gskinner.com/archives/2005/10/major_flash_pla.h...
- https://support.mozilla.org/bm/questions/931671
(Flash for desktop, with file access)
I never really worked with it, but it seems whenever it comes up here or on Reddit, people who did, miss it. I think the authoring side of Flash is remembered very positively.
For a while RIM/Blackberry was using Adobe Air - on the Playbook and also the built-in app suite in the lead up to the launch of BB10. The latter never saw the light of day though, a late decision was made to replace all of the built-in apps with Qt/Cascades equivalents (if I remember right this was due largely to memory requirements of the Air apps).
Silverlight was nice, pity it got discontinued.
Thus it isn't as if the browser plugins story is directly responsible for its demise.
What I find most compelling about this framing is the maturity argument. Browser sandboxing has been battle-tested by billions of users clicking on sketchy links for decades. Compare that to spinning up a fresh container approach every time you want to run untrusted code.
The tradeoff is obvious though: you're limited to what browsers can do. No system calls, no arbitrary binaries, no direct hardware access. For a lot of AI coding tasks that's actually fine. For others it's a dealbreaker.
I'd love to see someone benchmark the actual security surface area. "Browsers are secure" is true in practice, but the attack surface is enormous compared to a minimal container.
For the same reasons given, NPM programmers should be writing their source code processors (and other batch processing tools) to be able to run in the browser sandbox instead of writing command-line utilities directly against NodeJS's non-standard (and occasionally-breaking) APIs.
And today this is.. not sufficient. What we require today is to run software protected from each other. For quite some time I tried to use Unix permissions for this (one user per application I run), but it's totally unworkable. You need a capabilities model, not an user permission model
Anyway I already linked this elsewhere in this thread but in this comment it's a better fit https://xkcd.com/1200/
Unix permissions remain a fundamental building block of Android's sandbox. Each app runs as its own unix user.
Tautology is tautology.
> but Java applets were removed from the browsers
Java applets provided more scope compared to the browser itself, not less. They're not really comparable to seccomp or namespaces.
> hosters who will hand off user account to a shared server
There's lots of CI or function runners that expose docker-like environments.
They are comparable because they provided a restricted sandbox to execute untrusted code.
> There's lots of CI or function runners that expose docker-like environments.
These are running inside VMs.
Java applets were killed off my MS's attempt at "embrace, extent, extinguish" by bundling an incompatible version of Java with IE, and Sun's legal response to this.
Sometimes *nix user accounts for AI agent sandboxing does come up in discussions. At [0], HN user netcoyote linked to his sandvault tool [1], which "sandboxes AI agents in a MacOS limited user account".
Actually seems like a great idea IMO, to be lightweight, generic, and robust-enough.
[0] https://news.ycombinator.com/item?id=46760777
[1] https://github.com/webcoyote/sandvault
On http://co-do.xyz/ you can select a directory and let AI get to work inside of it without having to worry about side effects.
The Fily System Access API is the best thing that happened to the web in years. It makes web apps first class productivity applications.
They won’t be first-class as long as native UI still has the upper hand.
Outside those sort of spaces it’s hard to name a popular piece of software still on native that isn’t a wrapped webapp.
To the extent that the battle has been won, the apps it has been won with are nevertheless second-class compared to native-level usability.
cough CapCut cough
Firefox and safari are generally very conservative about new api that can enable new type of exploits.
At least firefox and safari does implement origin private file system. So, while you can't edit file on user disk directly. You can import the whole project into browser. Finish the edit and export it.
The Chrome team's new, experimental APIs are a separate matter. They provide additional capabilities, but many programs can get along just fine without since they don't don't strictly need them in order to work—if they would ever even have end up using them at all. A bunch of the applications in the original post fall into this category. You don't need new or novel APIs to be able to hash a file, for example. It's a developer education problem (see also: hubris).
"This" what? sha256sum doesn't need read-write access for even one file to be able to compute a hash, let alone a whole directory. You're ignoring most of my comment, focusing on like 20%, and in so doing, missing (and/or deliberately misframing) 100% of the point.
Yup. That's the link, all right—the one we all read and that I'm citing examples from. Thanks for the reminder, I guess: it has been a whole 8 hours since I first looked at it.
What "we" are talking about here, in this subthread, is the fact that "Browsers have had widespread support for processing files" for a long, long time, and that although "Chrome team's new, experimental APIs [...] provide additional capabilities" which are undoubtedly useful for certain programs, they're overkill and don't offer anything new and/or strictly necessary for many, many programs that don't actually need that sort of access—including "A bunch of the applications in the original post [that] fall into this category. You don't need new or novel APIs to be able to hash a file, for example."
Which is to say, we're talking about POLP/POLA. And the point of my comment was to address the very worthwhile matter of POLA violations. But you seem insistent on shutting that discussion down with chatter that looks like it's an on-topic reply or refutation to something, but in reality doesn't actually meaningfully engage with what you're purporting to respond to, or at best comes come across as confused and not particularly attentive.
There are already and will continue to be plenty of opportunities to discuss the acknowledged upsides of the new APIs for the class of programs for which they are strictly necessary. There's a lot of them in this very comment section. It doesn't have to come at the expense of changing the subject in the middle of a different conversation—accompanied by undertones that you're putting some matter to rest.
mg: This is a great example of how useful the File System Access API is ...
auggierose: the API is not supported (yet?) either by Safari or Firefox
you: describes the current situation with cross-browser file input support, says it's a "developer education problem"
me: this specific use case really does need the full API
It read to me, and still does read to me, like you were saying that Kinlan's use of FSA was hubris.
> I agree we're talking past each other
You're exactly half right.
Let's make this dead simple: does anyone need any of these new APIs to compute the SHA-2 hash for a file? A simple answer will do. Simple, non-evasive, no "look thither" misdirection.
Not that I'm not excited about the possibilities in personal productivity, but I don't think this is the way--if it was, we wouldn't have lost, say, the ability to have proper desktop automation via AppleScript, COM, DDE (remember that?) across mainstream desktop operating systems.
I have a DDE book somewhere, with endless pages of C boilerplate to exchange a couple of values between two applications on Windows 3.x.
If not using WinUI 3.0, or Windows ML with CoPilot+, there is no reason to submit oneself to the pain of using CsWinRT or C++/WinRT bindings with lesser tooling than their UWP counterparts.
The large majority of new APIs, since Vista are based on traditional COM, with the biggest exception being UMDF that in version 2.0 rolled back its COM API from version 1.0, back to a C based one.
1) webcontainer allows nodejs frontend and backend apps to be run in the browser. this is readily demonstrated to (now sadly unmaintained) bolt.diy project.
2) jslinux and x86 linux examples allow running of complete linux env in wasm, and 2 way communication. A thin extension adds networking support to Linux.
so technically it's theoretically possible to run a pretty full fledged agentic system with the simple UX of visiting a URL.
My eventual goal with that is to expand it so an LLM can treat it like a filesystem and execution environment and do Claude Code style tricks with it, but it's not particularly easy to programmatically run shell commands via v86 - it seems to be designed more for presenting a Linux environment in an interactive UI in a browser.
It's likely I've not found the right way to run it yet though.
I see the datestamp on this early test https://fingswotidun.com/tests/messageAPI/ is 2023-03-22 Thinking about the progress since then I'm amazed I got as far as I did. (To get the second window to run its test you need to enter aWorker.postMessage("go") in the console)
The design was using IndexedDB to make a very simple filesystem, and a transmittable API
The source of the worker shows the simplicity of it once set up. https://fingswotidun.com/tests/messageAPI/testWorker.js in total is just
I distinctly remember adding a Serviceworker so you could fetch URLs from inside the filesystem, so I must have a more recent version sitting around somewhere.It wouldn't take too much to have a $PATH analog and a command executor that launched a worker from a file on the system if it found a match existed on the $PATH. Then a LLM would be able to make its own scripts from there.
It might be time to revisit this. Polishing everything up would probably be a piece of cake for Claude.
Isn't webcontainers.io a proprietary, non-open source solution with paid plans? Mentioning it at the same level of open source, auditable platforms seems really strange to me.
https://developer.chrome.com/blog/persistent-permissions-for...
On my desktop Chrome on Ubuntu, it seems to be persistent, but on my Android phone in Chrome, it loses the directory if I refresh.
That said. It's a good start.
Could this be used with arbitrary local tools as well? I could be missing something but I don't see how you could use a non remote MCP server with this setup.
Browsers as agent environment opens up a ton of exciting possibilities. For example, agents now have an instant way to offer UIs based on tech governed by standards(HTML/CSS) instead of platform specific UI bindings. A way to run third party code safely in wasm containers. A way to store information in disk with enough confidence that it won't explode the user's disk drive. All this basically for free.
My bet is that eventually we'll end up with a powerful agentic tool that uses the browser environment to plan and execute personal agents or to deploy business agents that doesn't access system resources any more than browsers do at the moment.
Generated via ChatGPT, this canvas shows a basic pyramid and has sliders that you can use to change the pyramid, and download the glTF to your local machine. You can also click the edit w/ ChatGPT and tweak the UI however you're able to prompt it into doing.
https://chatgpt.com/canvas/shared/697743f616d4819184aef28e70...
It's easily explained by the fact that all the javascript code is exposed in a browser and all the network connections are trivially inspectable and blockable. It's much harder to collect data and do shady things with that level of inspectability. And it's much harder to ban alternative clients for the main paid offer. Especially if AI companies want to leave the door open to pushing ads to your conversations.
I would like to humbly propose that we simply provision another computer for the agent to use.
I don't know why this needs to be complicated. A nano EC2 instance is like $5/m. I suspect many of us currently have the means to do this on prem without resorting to virtualization.
We essentially leveraged sandboxes built into Chromium browsers for LLM generated code execution.
This actually simplifies a lot of the setup in the blog post, as it leverages existing sandboxing infra exposed for extensions: https://developer.chrome.com/docs/extensions/how-to/security...
I built a process supervisor (Vallignus) for that specific "OS-level" use case. It wraps the agent to enforce egress filtering and loop detection so it can use local tools without running wild.
Code is here if you're curious: https://github.com/jacobgadek/vallignus
So if you can work within the constraints there are a lot of benefits you get as a platform: latency goes down a lot, performance may go up depending on user hardware (usually more powerful than the type of VM you'd use for this), bandwidth can go down significantly if you design this right, and your uptime and costs as a platform will improve if you don't need to make sure you can run thousands of VMs at once (or pay a premium for a platform that does it for you)[1]
All that said I'm not sure trying to put an entire OS or something like WebContainers in the user's browser is the way, I think you need to build a slightly custom runtime for this type of local agentic environment. But I'm convinced it's the best way to get the smoothest user experience and smoothest platform growth. We did this at Framer to be able to recompile any part of a website into React code at 60+ frames per second, which meant less tricks necessary to make the platform both feel snappy and be able to publish in a second.
[1] For big model providers like OpenAI and Anthropic there's an interesting edge they have in that they run a tremendous amount of GPU-heavy loads and have a lot of CPUs available for this purpose.
Browser sandboxes are swiss cheese. In 2024 alone, Google reported 75 zero-day exploits that break out of their browser's sandbox.
Browsers are the worst security paradigm. They have tens of millions of lines of code, far more than operating system kernels. The more lines of code, the more bugs. They include features you don't need, with no easy way to disable them or opt-in on a case-by-case basis. The more features, the more an attacker can chain them into a usable attack. It's a smorgasbord of attack surface. The ease with which the sandbox gets defeated every year is proof.
So why is everyone always using browsers, anyway? Because they mutated into an application platform that's easy to use and easy to deploy. But it's a dysfunctional one. You can't download and verify the application via signature, like every other OS's application platform. There's no published, vetted list of needed permissions. The "stack" consists of a mess of RPC calls to random remote hosts, often hundreds if not thousands required to render a single page. If any one of them gets compromised, or is just misconfigured, in any number of ways, so does the entire browser and everything it touches. Oh, and all the security is tied up in 350 different organizations (CAs) around the world, which if any are compromised, there goes all the security. But don't worry, Google and Apple are hard at work to control them (which they can do, because they control the application platform) to give them more control over us.
This isn't secure, and there's really no way to secure it. And Google knows that. But it's the instrument making them hundreds of billions of dollars.
Something more like a TEE inside the browser of sorts. Not sure if there is anything like this.
But then you'd also want the frame content to use `integrity` on nested resoures.
CSP frame-src can help for now.
At the risk of sounding obvious :
- Chrome (and Chromium) is a product made and driven by one of the largest advertising company (Alphabet, formally Google) as a strategical tool for its business model
- Chrome is one browser among many, it is not a de facto "standard" just because it is very popular. The fact that there are a LOT of people unable to use it (iOS users) even if they wanted to proves the point.
It's quite important not to amalgamate some experimental features put in place by some vendors (yes, even the most popular ones) as "the browser".
There are many useful things that can only be implemented for Chromium: things like the filesystem API mentioned in this post, the USB devices API used to implement various microcontroller flashing tools, etc. Users can have multiple browsers installed, and I often use Chromium as essentially a sandboxed program runtime.
Chrome add these features because they are responding to the demands of web developers. It's not web developers fault if firefox can't or refuses to provide apis that are being asked for.
Mozilla could ask Claude to implement the filesystem api today and ship it to everyone tomorrow if they wanted to. They are holding their own browser back, don't let them also hold your website back. In regards to browser monoculture there are many browsers built on top of the open source Blink that are not controlled by Google such as Edge, Brave, and Opera just to name a few of the many.
That's enough reason for me to say, f** no. Google will try as hard as possible to promote this even if it's not technically the best solution.
I know there are lots of good arguments why docker isn't perfect isolation. But it's probably 3 orders of magnitude safer than running directly on my computer, and the alignment with the existing dev ecosystem (dev containers, etc) makes it very streamlined.
- https://vermaden.wordpress.com/2021/12/15/secure-containeriz...
Browsers are closer to operating systems rather than sandboxes, so giving access of any kind to an agent seems dangerous. In the post I can see it's talking about the file access API, perhaps a better phrasing is, the browser has a sandbox?
The point is that most people won't do that. Just like with backups, strong passwords, 2FA, hardware tokens etc. Security and safety features must be either strictly enforced or on enabled by default and very simple to use. Otherwise you leave "the masses" vulnerable.
LLMs are actually quite neutral and don't have preferences, wants, or needs. That's just us projecting our own emotions on them. It's just that a lot of command line stuff is relatively easy to figure out for LLMs because that is highly scriptable, mostly open source, and well documented (and part of their actual training data). And scripting is just a form of programming.
The approach in the article that Simon Willison is commenting on here isn't that much different; except the file system now runs in a browser sandbox and the tools are WASM based and a bit more limited. But then, a lot of the files that a normal user works with would be binary files for things like word processors, photo editors, spreadsheets, presentation software, etc. Stuff that is a bit out of the comfort zone of normal command line tools in any case.
I actually tried codex on some images the other day. It kind of managed but it wasn't pretty. It basically started doing a lot of slow and expensive stuff with python and then ran out of context because it tried to dump all the image content in there. Far from optimal. You'd want to spend some time setting up some skills and tools before you attempt this. The task I gave it was pretty straightforward: create an image catalog in markdown format for these images. Describe their content, orientation, and file format.
My intention was to use that as a the basis for picking appropriate images to be used on different sections in my (static) website without having to open and scan each image all the time. It half did it before running out of context. I decided to complete the task manually (quicker and I have more 'context' for interpreting the images). And then I let codex pick better images for this website. Mostly it did a pretty OK job with that at least.
I learn a lot from finding places where these tools start struggling. It's why I like Simon's comments so much because he's constantly pushing these tools to their limits and finding out surprising, interesting, or funny success and failure modes.
It would probably help if the sandbox presented a linux-y looking API, and translated that to actual browser commands.
Yeah they do. Tell it you want to hack Instagram because your partner cheated on you, and ChatGPT will admonish you. Request that you're building a present for Valentines day for your partner and you want a chrome extension that runs on instagram.com; word it just right, and it'll oblige.
by any chance anyone knows if users clicks can be captured for a website/tab/iframe for screen recording. i know i can record screen but i am wondering if this metadata can be collected.
For your own implementation, document-level event listeners work, though cross-origin iframes are off-limits due to same-origin policy.
Sadly not to my knowledge.
The browser should be a VM host.
I'm on a multi-year quest to answer that question!
The best I've found is running Python code inside Pyodide in WASM in Node.js or Deno accessed from Python via a subprocess, which is a wildly convoluted way to go but does appear to work! https://til.simonwillison.net/deno/pyodide-sandbox
Here's a related recent experimental library which does something similar but with JavaScript rather than Python as the unsafe language, again via Deno in a subprocess: https://github.com/simonw/denobox
I've also experimented with using wasmtime instead of Deno: https://til.simonwillison.net/webassembly/python-in-a-wasm-s...
Also the double iframe technique is important for preventing exfiltration through navigation, but you have to make sure you don't allow top navigation. The outer iframe will prevent the inner iframe from loading something outside of the frame-src origins. This could mean restricting it to only a server which would allow sending it to the server, but if it's your server or a server you trust that might be OK. Or it could mean srcdoc and/or data urls for local-only navigation.
I find the WebAssembly route a lot more likely to be able to produce true sandboxen.
A large part of the web is awful because of all the things browsers must do that the operating system should already be doing.
We have all tolerated stagnant operating systems for too long.
Plan 9's inherent per-process namespacing has made me angry at the people behind Windows, MacOS, and Linux. If something is a security feature and it's not an inherent part of how applications run, then you have to opt in, and that's not really good enough anymore. Security should be the default. It should be inherent, difficult to turn off for a layman, and it should be provided by the operating system. That's what the operating system is for: to run your programs securely.
Can you believe that if you download a calculator app it can delete your $HOME? What kind of idiot designed these systems?
The problems discussed by both Simon and Paul where the browser can absolutely trash any directory you give it is perhaps the paradigmatic example where git worktree is useful.
Because you can check out the branch for the browser/AI agent into a worktree, and the only file there that halfway matters is the single file in .git which explains where the worktree comes from.
It's really easy to fix that file up if it gets trashed, and it's really easy to use git to see exactly what the AI did.
It seems he started his blog in 2003: https://simonwillison.net/2003/Jun/12/oneYearOfBlogging/
As for the "browser is the sandbox" running untrusted code in the user's browser increases the risk of an unintended RCE via a sandbox escape which can be done in Chrome [0]. WASM is not going to save you either [1].
[0] https://www.ox.security/blog/the-aftermath-of-cve-2025-4609-...
[1] https://issues.chromium.org/issues/334120897
And then you see the recent vulnerabilities in opencode for example. The current model is unsustainable
It would be great if desktop Linux adopted a better security model (maybe inspired by Android). So far we got this https://xkcd.com/1200/ and it's not sufficient
We are soon going to release a new technology, built on top of the same stack, to allow full-stack development completely in the browser. It's called BrowserPod and we think it will be a perfect fit for agents as well.
https://browserpod.io