Why we will never inject touches
Most remote-control tools work by synthesizing input events. That design is one bug away from an agent doing anything on a user's device. Here's the architecture we chose instead, registered semantic actions, validated on-device.
Classic remote-desktop tools work by capturing the screen and replaying raw input: a click at (412, 887), a key press, a swipe. It's a universal mechanism, and that universality is exactly the problem. If an agent (or a compromised agent account, or a bug in the coordinate math) can click anywhere, they can do anything: approve a payment, delete an account, open settings the user never intended to share.
Registered semantic actions
In ScreenSteer, your app registers the actions an agent may invoke, by name, with a typed operation. workorders.refresh is invocable. Everything else does not exist as far as the wire protocol is concerned. There is no “tap at coordinates” message to abuse, because we never defined one.
await store.refreshWorkOrders()
}
The device is authoritative
Even a registered action isn't executed on trust. Each command arrives in a signed session envelope and passes an on-device validator: is control consent currently granted? Is the sender the one agent holding the controller lease? Has the command expired? Does its layout revision match what's actually on screen? Fail any check and the device rejects the command and says why, stale_layout, not_available, rejected.
A concrete example: the user rotates their iPad while the agent is mid-click. The layout revision bumps, the in-flight command no longer matches, and the device refuses it. The console simply shows the rejection and waits for the fresh layout. The wrong button is never pressed.
Consent is not a dialog, it's a state machine
Viewing and assisting are separate grants, requested separately, revocable independently, and enforced on every single message, not just at session start. When a user taps Stop, the session ends regardless of what anyone else thinks; the device doesn't need permission to stop obeying.
The result: a support agent can be maximally helpful and still provably unable to act outside the boundary your app drew. That's a property we can put in a security review, and one raw input injection can never offer.