For a while, Sense could show you where something was. Ask how to approve a claim and it would spotlight the button, or play a short guided journey through the screens. Useful, and safe, and slightly frustrating: the assistant knew exactly which control you needed and could do nothing but point at it.
This post is about the step from pointing to acting — the agent opening a screen, reading what is on it, pressing the control, filling the field — and about the decisions that made that acceptable to ship.
Four tools, in ascending order of nerve
Navigate: open an app-relative route. Read the screen: return the route and the controls on it, with their labels and whether they are disabled. Invoke: press a control. Fill: set a form field. The first two change nothing. The last two act, and each one is confirmed with the user before it runs.
Fill is deliberately the weakest tool on the platform. It never saves. If the agent needs to change stored data, the right instrument is the record-update tool it already has: transactional, auditable, testable, and independent of whether a browser is open. Filling a field exists for a different job — staging something on screen so the person can look at it, adjust it, and press save themselves. An agent that could type into a form and submit it would have bypassed every approval gate we have written about; an agent that can only type is a colleague leaning over your shoulder.
Addressing the screen without describing it
The obvious way to let an agent press a button is to give it the screen's markup and let it find a selector. It works in a demo and breaks the first time a tenant moves the button, which in a platform where every organisation designs its own layouts is roughly immediately.
The platform already had a better answer, built for the guided tours: every meaningful control carries a stable semantic anchor — an identifier for what the control is, not where it sits. The agent addresses controls by anchor. Read-the-screen returns anchors with their labels; invoke and fill take an anchor. A redesign moves the target; it does not break the agent. It also means the agent never sees a selector or a pixel position, which keeps the tool surface small and the failure modes legible.
Consent lives in the browser
Where to put the "are you sure?" was the design decision that took longest. The platform has a server-side approval system for agent writes, and the reflex was to add these tools to it. We did not, and the reason is what each side knows.
The server sees an instruction like invoke the submit action on a sales order. The browser sees a button labelled Submit Order on a screen the user is looking at. The browser's confirmation is the meaningful one, because it can show the person exactly what is about to be pressed in the words they read on screen. So consent is a browser-side setting — ask every time by default, changeable from the profile menu — and every acting tool shows a confirmation naming the control before it runs. Organisations that want a second, server-side gate can add the tools to their guardrail policy; the two gates compose rather than fight, because the browser round-trip happens before the approval step.
Round trips, not fire-and-forget
Pointing could be fire-and-forget: the model emitted "spotlight this", the client did it, and nobody needed to know whether it worked. Acting cannot. An agent that does not know whether the click landed will confidently tell the user it did.
So each acting tool round-trips. The server pauses the turn, the browser performs the action, and the result — success or failure, the new route, the controls now on screen — comes back to the model before it continues. The rendezvous reuses the same mechanism the platform uses for live approvals, which means the streaming response and the answer to it can land on different servers without anyone noticing. Surfaces that cannot act — the mobile app, an external assistant over the MCP endpoint — declare that up front, and the tools are simply not offered to them; a turn never parks waiting for a browser that is not there.
When the screen is too big to describe
A dense screen has more controls than fit in the space reserved for describing them. The tempting failure is to truncate silently; the model then concludes the control does not exist and tells the user so, with conviction. Both truncation points now report what they dropped, so the model can say "the screen has more controls than I could read; tell me which area" rather than inventing an absence.
What it is for
Not for doing your work for you. For the gap between knowing and doing that every complex application has: "take me to the pricing rules for this account and put the new rate in the field so I can check it before I save". "Open the dispatch board for tomorrow and show me the unassigned jobs." "Fill the return reason and leave the rest to me." The agent handles the navigation and the typing; the person handles the decision and the save. That is the same division of labour as everywhere else in Sense — it reads freely, it drafts with evidence, and a human is on the trigger — applied to the screen itself.
Related: Sense Assist · How Sense earns trust · Reads everything, writes almost nothing