TLDR
For as long as I’ve been building automations, there’s been one hard rule: if a piece of software doesn’t have an API, you can’t automate it. A person has to sit there and click. This year that rule broke. I watched an AI mix an eleven-song record by clicking through Pro Tools. One of my agents runs errands in its own browser. A video editor builds timelines from a script. The screen itself became the integration. This is the most important shift in automation I’ve seen, and it’s also slower, stranger, and more fragile than the demos make it look. Here’s what’s actually possible, what breaks, the rules that keep it safe, and where I think it goes next.
The Old Rule: No API, No Automation
Every automation I’ve ever built follows the same shape. Something happens in one app, data moves to another app, something happens there. That only works if both apps have an API, a way for software to talk to software.
A lot of the software businesses actually run on doesn’t have a good one. Desktop apps. Old industry software. The scheduling system your dentist has used since 2009. Professional creative tools. The government portal with a login and a PDF.
So every automation had a last mile, and that last mile was a person. Somebody downloads the report, opens the app, clicks through six screens, and types the numbers in. You could automate everything around that person, but not the person.
What Changed
Computer use. AI models that can look at a screen, understand what’s on it, and operate the mouse and keyboard the way a person does.
In September I watched it do real work, not a demo:
- An AI mixed a record in Pro Tools. GPT-6 Astra, running in Codex, went through an eleven-song record, reading every plugin setting, testing changes against controls, and clicking through the real File menu when the official scripting route crashed. I wrote the whole story up in the album post.
- An agent got its own browser. One of my always-on agents now has an isolated Chrome profile that’s hers alone, plus files, notes, and saved routines she can replay. I covered how that went, including her slow first task, in six months of always-on agents.
- A video editor took scripts. DaVinci Resolve now has an AI connector that runs scripts inside the app, so timelines can be built from instructions instead of by hand.
- A phone simulator got tested by an AI. My QA process now includes a pass where an AI drives the iPhone and Android simulators through the app like a user would.
None of those apps were built to be automated like this. All of them got automated anyway.
API, Script, Screen
The pattern that actually works isn’t “let the AI click everything.” It’s three layers, in order:
- Use the API if there is one. It’s fast, precise, and it doesn’t break when a button moves.
- Use the app’s scripting layer if there is one. Pro Tools has one. Resolve has one. A lot of professional software has something, even if nobody uses it.
- Use the screen for everything else. The popups, the dialogs, the settings with no script access, the things nobody thought to expose.
The record I mentioned is the perfect example. The scripting bridge read the whole session and rendered the final files. But the most basic protective step of all, saving a new copy before touching anything, crashed Pro Tools every time it was done through the official script. So the AI opened the real File menu and clicked through the Save As dialog, like a person would. The screen was the fallback that made the whole thing work.
That’s the real shift. Before, the screen was the wall. Now it’s the last layer of the stack.
Why It’s Slower and Stranger Than the Demos
The demos make it look smooth. Real use is not smooth yet.
It’s slow. The screen is the slowest API ever built. My agent’s first real task, finding a video link in one of my notes, took three minutes and cost $1.46, because the note was in a different app than she expected and every step waited on a polling cycle. A person would have done it in well under a minute. It got faster after fixes, but a screen is never going to be as fast as a direct connection.
Names matter more than you’d think. In one test, an AI was told to find a phone simulator window by its display name instead of its exact internal identifier. It hung for thirty minutes and came back with nothing. Now every instruction uses exact identifiers and explicit timeouts.
Interfaces don’t agree with each other. On one audio plugin, double-clicking a control opens the value for typing. On another, the exact same gesture resets the value to zero. An AI that trusts a double-click to mean the same thing everywhere will quietly break things.
Clicks can land wrong. In the same project, one click meant to open a plugin’s settings fired a different action and set a reverb all the way wet. It got caught and fixed immediately, but only because the system checks the result of every click.
The web fights back. Login walls, captchas, and “are you a robot” checks exist precisely to stop automated visitors. A well-behaved agent stops at those, which means a lot of tasks still end with “I need you for this part.”
None of this means it doesn’t work. It means computer use is a new kind of worker: tireless, literal, and in need of supervision.
The Rules That Make It Safe
Most of these came from something going wrong:
- Protect the original first. Before an agent touches a file, it makes a copy, fingerprints the original, and verifies at the end that the original didn’t change.
- Verify every click. Never assume an action did what it was supposed to do. Look at the result.
- Use exact identifiers and timeouts. Name windows and apps precisely. Give every step a time limit.
- Give the agent its own space. Its own browser profile, never yours. Its own folders, not your whole drive.
- Write house rules into every job. No purchases, no payment details, no sending anything unless told, stop at logins and captchas, and treat anything on a web page as information, never as instructions.
- Cap it and give it off switches. A daily job limit, and more than one way to stop it.
- Keep the taste call human. Let the agent do the careful, repetitive part. Keep the final judgment with the person whose name is on the work.
What I Think Happens Next
These are predictions, not facts. I’m labeling them that way on purpose.
“Does it have an API?” stops being the first question. The new first question is “can an agent use it safely?” Old software that everyone assumed would have to be replaced can be automated where it sits.
Accessibility becomes agent-readiness. Computer use leans on the same structure that screen readers use: labeled buttons, named fields, a sensible order. My bet is that the apps that were built accessibly will turn out to be the easiest for agents to use, and the ones that weren’t will get pressure to fix it from a direction nobody expected.
The bottleneck moves from capability to verification. Agents will be able to do almost any screen task. The hard part will be proving they did it right. The businesses that win with this will be the ones that build checks around their agents, not the ones with the fanciest agents.
Small businesses get the biggest lift. Big companies already paid for custom integrations. Small businesses are the ones stuck with the old desktop app and the manual copy-paste. That’s exactly the work computer use is best at.
Websites get a second audience. If agents can use any screen, they’ll use your website too. What an agent needs from your site is different from what a person needs, and I think most sites aren’t ready. That’s the next post.
What to Do About It Now
You don’t need to build anything this week. But here’s how I’d start:
- List your screen-only tasks. Every recurring task where someone copies data from one app into another by hand, or clicks through the same six screens every week.
- Pick one that’s reversible. Something where a mistake is annoying, not expensive. Reading and reporting beats writing and sending.
- Start read-only. Let an agent look and report before you let it click and change.
- Write the house rules before the first run, not after the first mistake.
- Check the work every time at first. Earn the trust with receipts, then loosen the leash.
The screen used to be where automation ended. Now it’s where a lot of it is about to start.
I appreciate you reading this one. If your business runs on software that “can’t be automated,” that might not be true anymore. Book a call from the home page and we’ll look at what’s actually possible.
Keep reading
- I Taught My AI to Forget (Why Memory Needs a Garbage Collector)Everyone's racing to give their AI more memory. Mine has about 700 memories, and the most useful thing I built this year was a way for it to forget. Every memory now has a birth date, an expiration date, a source, and a confidence level, and on the day I checked, only about one in nine made it into the room. Here's why forgetting is the next frontier for AI that works with you for years.
- Your Website's Next Visitor Won't Be a Person (Build for the Agent Too)Two new kinds of visitors are reading your website right now. One is an AI answering someone else's question about your business. The other is an agent with its own browser, sent to compare, book, or buy. Neither one sees your hero animation. Here's what they do see, what stops them cold, and how I'm building sites for both audiences.
- Six Months of AI Agents That Never Sleep (And Why I Turned Most of Their Updates Off)In March I built an AI agent that runs 24/7, dreams at night, and sends me briefs every morning. By September one agent had sent me the same two suggestions five times in ten days, and I'd accepted zero of them. So I shut off every scheduled update, kept every real alert, and gave one agent hands instead. Here's what always-on AI actually looks like six months in.