--- name: iOS Simulator description: Use to run, drive, and test an iOS app in the Simulator — boot a device, install and launch a build, tap/swipe/type, read the accessibility tree, and take screenshots. Use for iOS development, UI testing, reproducing a bug on device, or checking how a screen actually looks. version: 1 situations: run my iOS app, test on iPhone, iOS simulator, boot a simulator, install the app, tap the button, does this screen look right, reproduce on device, UI test, take a screenshot of the app, launch the app, deep link, iOS build requires-secrets: requires-packages: baguette --- # iOS Simulator You can drive a real iOS Simulator: boot a device, install/launch a build, interact with it, and see what is on screen. The operator sees the same phone live in the session's right-hand panel, so you are both looking at one device — if they are watching, narrate what you're doing. Requires macOS on Apple Silicon with Xcode 26+ and at least one iOS runtime installed. The driver (`baguette`) is installed automatically with this skill. If a tool reports the capability is unavailable, it says exactly why (no runtime, wrong arch, driver missing) — relay that, don't guess. ## How to call these tools **If you are a CLI brain (Claude Code / Codex), these are NOT invocable skills or built-in tools — run each one through the `00` command from your shell:** ```bash 00 media simulator_list --json '{}' 00 media simulator_describe_ui --json '{}' 00 media simulator_tap --json '{"x":201,"y":354}' 00 media simulator_key --json '{"code":"Enter"}' 00 media simulator_launch --json '{"bundleId":"com.example.app"}' ``` `00` already knows which agent and conversation you are, so you never pass ids and never write curl. If a tool name isn't recognised, run `00 media tools` — do NOT go hunting through your capability settings. (A pi-brained agent calls them directly as tools instead; the names and parameters are identical.) ## Coordinates — read this before your first tap **Everything is in DEVICE POINTS, origin top-left, x→right, y→down.** Points are the phone's own units (what iOS calls points), NOT pixels: an iPhone 17 Pro is ~402×874 points while its screenshot is ~1206×2622 pixels at 3× scale. So: - **Take coordinates from `simulator_describe_ui` frames, never by eyeballing a screenshot.** Each element has a frame `{x, y, width, height}` already in points. Tap its centre: `x + width/2`, `y + height/2`. - **Never measure pixels off a screenshot image** — you'd be off by the scale factor (2× or 3×) and every tap would land in the wrong third of the screen. - `simulator_screenshot --scale 2` halves the IMAGE, and changes nothing about the point coordinates. - After `simulator_orientation`, width and height swap. Re-read the tree; don't reuse old coordinates. - Origin is the top-left of the SCREEN, including the status bar and any notch/Dynamic Island area. ## Hardware buttons you can press `simulator_press { button }` takes exactly these: | Button | What it does | |---|---| | `home` / `swipe-to-home` | Go to the home screen (the gesture variant on Face ID devices) | | `app-switcher` / `swipe-to-app-switcher` | Open the multitasking switcher | | `lock` | Lock the screen (press again / `swipe-to-home` to wake) | | `power`, `side-button`, `left-side-button` | Physical side buttons | | `volume-up`, `volume-down` | Volume | | `action` | The Action button (15 Pro and later) | | `digital-crown` | Watch only | | `pull-down-to-lock-screen` | Pull down the Lock Screen | | `pull-down-to-notification-center` | Pull down Notification Center | There is no "back button" on iOS — go back by tapping the app's own back control (find it in the tree), or `simulator_swipe` from the left edge inward (an edge-swipe back gesture). ## Testing a screen — the loop that actually works 1. `simulator_describe_ui` → find the element by its label/role, compute its centre. 2. Act: `simulator_tap` / `simulator_swipe` / `simulator_type`. 3. `simulator_describe_ui` **again** → confirm the screen changed the way you expected. A tap that hit nothing is completely silent; re-reading is the only way you find out. 4. `simulator_screenshot` when the operator should SEE it, or when the question is visual (layout, spacing, colour) rather than structural. While you're driving, the operator's panel shows "🤖 is controlling" and their taps are paused, so they aren't fighting you for the phone. It releases a few seconds after your last action — so do a related burst of actions, then stop; don't hold the device idle. ## iOS autocorrect will change what you typed A normal text field has autocorrection on, and it rewrites non-words as you type: typing `Kbd` into one lands `Kid`. That is iOS doing its job, not a dropped keystroke — so **always re-read the field** after typing something that isn't a real word (a code, an id, a made-up name) and don't assume the tool failed. `simulator_paste` goes via the pasteboard and is not autocorrected, so prefer it for exact strings. ## The home screen can't be read — launch the app first `simulator_describe_ui` only sees a **foregrounded application**. On the iOS home screen (SpringBoard) there is no accessibility tree, and the tool tells you so. That is not a failure to retry: it means you haven't launched anything yet. Order of operations is always **install → launch → read**. If a read comes back saying nothing is in the foreground, `simulator_launch` your app (or re-launch it after a crash) rather than reading again. ## Look before you touch — `simulator_describe_ui` **`simulator_describe_ui` is your primary sense, not the screenshot.** It returns the accessibility tree: every element with its label, role, and frame in device points. Use it to find WHERE to tap. Screenshots are for showing the operator (or judging visual layout/appearance) — do not try to read a screenshot to work out coordinates when the tree gives you exact frames. Typical loop: 1. `simulator_describe_ui` → find the element (match on its label/role). 2. Tap its frame's centre: `simulator_tap { x, y }` (device points, origin top-left). 3. `simulator_describe_ui` again to confirm the screen changed as expected. Only fall back to a screenshot + estimated coordinates when an element genuinely isn't in the tree (custom-drawn canvas, a game, a video surface). ## Tools - `simulator_list` — devices and which is booted. `simulator_boot { udid }` / `simulator_shutdown { udid }`. - `simulator_describe_ui { udid, x?, y? }` — accessibility tree; pass x/y to hit-test one point. - `simulator_screenshot { udid }` — a JPEG saved into `files/`; use it to SHOW the operator. - `simulator_tap { udid, x, y }`, `simulator_double_tap`, `simulator_swipe { fromX, fromY, toX, toY }`. - `simulator_type { udid, text }` — US-ASCII into the focused field. Non-ASCII (emoji, accents) → `simulator_paste { udid, text }`, which goes via the pasteboard. - `simulator_key { udid, code, modifiers? }` — one key by W3C `KeyboardEvent.code`: `Enter`, `Escape`, `Backspace`, `Tab`, `ArrowDown`, `KeyA`… Modifiers are `shift`/`control`/`option`/`command`. This is how you submit a form, dismiss a sheet, delete a character, or run a shortcut like `command`+`KeyA`. ⌘C/⌘X also hand back what the app copied. - `simulator_clipboard { udid }` — read the device pasteboard (what the app copied). - `simulator_press { udid, button }` — `home`, `lock`, `app-switcher`, `volume-up`, `swipe-to-home`, … - `simulator_install { udid, appPath }` (a built `.app`), `simulator_launch { udid, bundleId }`, `simulator_terminate`, `simulator_open_url { udid, url }` (deep links). - `simulator_orientation { udid, value }` — rotate; screen dimensions swap. ## Working rules - **Boot once.** Booting is slow. Check `simulator_list` first and reuse a booted device. - **Build, then install.** You build the app with your normal tools (`xcodebuild`), then `simulator_install` the product and `simulator_launch` it. This skill does not build. - **Verify, don't assume.** After any action that should change the screen, re-read the tree. A tap that landed on nothing is silent — the only way you know is by looking again. - **Screenshot when the operator asks "how does it look"**, or to prove a visual fix. Attach the saved file rather than describing it. - **Don't type secrets.** Never enter credentials, API keys, or card numbers into the simulator unless the operator explicitly asked you to in this conversation. - If the app crashes or won't launch, say so plainly with the error — don't retry blindly in a loop.