ZSeven-W/dsh-android

A live Android device inside a DeepSeek Harness conversation — emulator or USB phone, driven entirely through adb.
_{20 agent tools • in-process live stream, no external helper • three-button navigation panel • Gradle build & run • Vision OCR}

_{npm: @zseven-w/dsh-android · Current plugin release: 0.1.0-rc.12 · Tested with DSH 0.1.5-rc.1}

English · 简体中文 · 繁體中文 · 日本語 · 한국어 · Français · Español · Deutsch · Português · Русский · हिन्दी · Türkçe · ไทย · Tiếng Việt · Bahasa Indonesia

_{An Android device streamed and controlled from inside a DSH conversation — the agent's tool call in the center, the live device panel on the right}

DSH Android gives the agent a real Android device inside the conversation — and gives you the pixels. The agent can start a stream on an emulator or a USB-connected phone, build and install a Gradle project, drive the UI by resource-id/text or by OCR, read logcat, and inspect processes and memory, while a live stream of the device renders in a persistent sidebar panel where you can tap, drag, rotate, and press Back / Home / Recents directly on the video. No image blocks and no screen-recording files: visual bytes reach the UI only through signed, expiring URLs served by the DSH webserver.

There is exactly one code path. adb devices -l reports a serial, and that serial is a device's only identity — emulator-5554, a USB serial, or an ip:port target all behave identically. The plugin is bound to no emulator product (AVD, Genymotion, WSA, a cloud device farm), and there is no simulator/real-device split to reason about.

All 20 tools are registered on every host and return plain JSON — visual bytes reach the UI only through presentationMeta + signed routes, never as image blocks. When adb cannot be resolved the tools stay registered and every call fails with an explanatory error naming the fix.

Coordinates are normalized 0..1 of the streamed frame everywhere. The frame follows the display rotation (a landscape app streams 2400×1080 on a 1080×2400 device) and input tap shares that same space, so no client-side rotation math exists anywhere in this plugin.

• Sidebar panel. The live view lives in a persistent right-hand panel (a fixed dock that pushes the conversation aside, or a centered overlay on narrow viewports). It renders the live PNG stream and accepts click-to-tap and drag-to-gesture directly on the video, with a toolbar carrying ◁ Back, ○ Home, □ Recents, rotate, screenshot, and refresh. A device menu runs the five device-level actions (notification shade, quick settings, lock, wake, assistant). The device picker lists every adb device in ONE list, grouped by kind, with offline AVDs shown as a hint pointing at android_boot rather than a boot-on-click. Size modes and frame styles (frameless / bezel / phone shell) work as in the iOS twin; the panel adapts its aspect ratio from the frame's own natural size, so a rotation needs no configuration.
• Compact conversation cards. Tool results render as one-line cards with no inline imagery: the device name, an action sub-label, a status badge, and an "open in sidebar" cue. Clicking the row opens the panel.
• Status capsule above the input. While the panel is closed and a stream is online, a small pill appears above the composer and opens the panel when clicked.
• Standard mode and Code Mode. Standard sessions use the host-projected presentationMeta; nested Code Mode dispatches carry no meta, so the client reconstructs the identical meta from the durable result JSON — the panel, the cards, and the capsule work in both.

• The browser never talks to adb, and there is no inner port to talk to. The stream is produced in this process and served from memory; every byte crosses the DSH webserver origin through plugin-owned /_dsh/dsh-android/* routes: /stream/<token> (live multipart PNG), /screenshot/<token> (cached PNG), plus /grant, /switch-device, /devices, /capture, /status, /control, and /device-action. This is a strictly smaller attack surface than a proxied loopback stream server.

• A triple loopback fence by default, applied before any capability is read. The transport peer must be a loopback address, the Host header must name a loopback authority (so a DNS-rebinding Host is rejected), and Fetch-Metadata/Origin must be same-origin. Host and Origin are caller-controlled data and are never trusted on their own.

• trustedAuthorities for deployments reached through a reverse proxy. A loopback peer proves the last hop came from this machine, which is always true behind a proxy — while the Host the proxy forwards is the deployment's public name, so every route above answers 403 and the plugin only works through an SSH tunnel. The optional config accepts those authorities explicitly, the way DSH's own --trusted-host does for its webserver:

  - id: dsh-android
    config:
      trustedAuthorities:
        # Any Host port; browser Origin must use the scheme's default port.
        - dsh.example.com
        # Port 8443 on either scheme — the reverse-proxy spelling.
        - dsh.example.com:8443
        # Pinned to one scheme, which is what you want when only one is served.
        - https://dsh.example.com:8443

  An entry may be a bare host, a host:port, or a pasted origin, and what it covers is decided by what it says:

  Entry	Matches	Origin accepted
  host	that hostname on any Host port	that host on HTTP 80 or HTTPS 443
  host:port	Host: host:port	that port on http or https
  https://host	that hostname on any Host port	HTTPS on that host, port 443
  https://host:port	Host: host:port	https on that port only

  The port is always compared, because an origin is scheme + host + port: an Origin from another application on the same hostname but a different port is refused. Browsers classify that as same-site rather than cross-site, so nothing else in the fence would stop it, and the request would still execute even though its response is unreadable to that origin. Listing an authority never narrows anything either — a loopback Host is judged on its own, so adding localhost for an SSH tunnel cannot break the local panel.

  Ports are normalized numerically; browsers omit scheme-default ports, so https://host:443 accepts the canonical Origin: https://host. Explicit default Host ports may likewise be omitted. Distinct configured schemes and ports coexist rather than hiding each other. trustedAuthorities must be an array of strings; a YAML scalar is rejected. Credential-bearing URLs, paths, queries and fragments are not authorities.

  Configure public authorities only behind a proxy that authenticates every plugin route. Origin is a browser protection, not authentication: an API client can forge both Host and Origin and send taps, text or rotations. The plugin does not establish that caller's identity. All POST routes, including /grant and /status, require Origin; token-gated stream/screenshot GETs do not. The peer-address fence remains mandatory, so direct non-loopback clients are refused regardless of headers. An empty list keeps public authorities refused.

• HMAC-SHA256 capabilities expiring within 10 minutes, formatted base64url(payload).base64url(mac) and signed with a 32-byte per-DSH-home key (<DSH_HOME>/cache/dsh-android/stream-access.key, mode 0600, created atomically). A capability minted for one device stops working the moment another device takes the stream slot, and a screenshot capability cannot be replayed against the stream route.

• The screenshot route serves exactly one directory. Paths are walked with lstat (any symbolic link is refused), finished with a realpath containment check, opened with O_NOFOLLOW, size-bounded, and re-validated after the read — so a file swapped for a symlink between minting and fetching is never served.

• /grant never boots anything. It only starts the frame loop for a device that is already online, and it refuses (409 device_busy) to yank the stream away from another device. Switching devices requires the explicit /switch-device gesture; booting an AVD stays with the android_boot tool.

• Keep-alive and idle stop. A crashed frame loop restarts in the background (~5 s delay); with zero consumers the stream stops itself after 5 minutes. Intentional stops are never fought.

• Node ≥ 24.11.0.
• adb, from the Android SDK platform-tools, resolved in this order: the ADB environment variable → adb on PATH → <ANDROID_HOME>/<ANDROID_SDK_ROOT>/the per-OS default SDK root + /platform-tools/adb. Install it with sdkmanager "platform-tools", with Android Studio, or with brew install --cask android-platform-tools. Without adb the plugin still loads and all 20 tools register; every call then explains what is missing.
• A device: an emulator of any product, or a phone with USB debugging enabled. The emulator launcher is optional and only android_boot-by-AVD-name needs it — everything else works with whatever adb can see.
• DSH ≥ 0.1.0-rc.6 with the web bundle for the panel. Headless profiles work too: all 20 tools function normally, just without the live view.
• macOS host for OCR (only android_find_text / android_tap_text / android_wait_for need it): the plugin compiles its bundled assets/ocr.swift with swiftc on first use into ~/Library/Caches/dsh-android/bin/ocr. On Linux and Windows hosts those three tools report that OCR needs the macOS Vision framework; the other 17 are unaffected. Overrides: DSHPLUGIN_ANDROID_OCR_DIR, DSHPLUGIN_ANDROID_OCR_SWIFT, DSHPLUGIN_ANDROID_SWIFTC.
• ADBKeyboard (optional, for CJK and emoji input): adb shell input text is ASCII-only. Install ADBKeyboard on the device and select it as the active IME, and non-ASCII text is delivered through its broadcast interface. Without it, non-ASCII typing is REFUSED with the install hint — never silently mistyped.

There is no WebDriverAgent equivalent to build, sign, trust, or re-sign every seven days. Enable USB debugging, plug the phone in, accept the authorization prompt on the device, and it appears in android_devices with every tool working against it. An unauthorized device is reported as such with the prompt hint, not as a mysterious failure.

Three honest caveats:

• Frame rate is lower over USB — roughly 2–5 fps against a phone versus 5–10 fps on an emulator, because every frame crosses the USB link as a full PNG.
• CJK typing needs ADBKeyboard (see above); this affects emulators and phones alike.
• android_shutdown cannot power off a phone. adb has no such verb; the tool says so instead of pretending.

Measured on an emulator (Android 14, 1080×2400):

The single persistent child is what buys this: spawning one adb per frame costs ~50–100 ms before any pixels move. Expect ~5–10 fps on an emulator and ~2–5 fps on a USB phone, depending on the machine and the screen density.

dsh plugin --profile web add @zseven-w/dsh-android@latest
dsh web

Or add it as a dependency of an existing profile package:

pnpm add @zseven-w/dsh-android

1. Discover devices — "List the Android devices." → android_devices.
2. Start the stream — "Stream emulator-5554." → android_boot. The panel opens with the device live. (An AVD name boots that emulator first.)
3. Tap on the video — tap or drag directly on the panel, or let the agent drive: "Open Settings, then tap Display." → android_interact, or android_ui_tree + android_tap_element for identity-based taps, or android_find_text + android_tap_text when the tree is blind.
4. Build and run your app — "Build and run /path/to/MyApp." → android_build_run. A full Gradle build takes minutes; when it lands, the app launches and you watch it live in the panel.
5. Read the logs — "Show the last two minutes of logcat for com.example.app." → android_logs.

• Every tool says adb is unavailable — the error names the three resolution tiers. Set ADB=/path/to/adb, put adb on PATH, or install the SDK platform-tools (sdkmanager "platform-tools").
• The device is unauthorized — accept the USB debugging prompt on the device screen. android_devices reports the state honestly rather than hiding the device.
• android_boot cannot find an AVD — the emulator launcher was not discoverable. Start the emulator by any means; it appears in android_devices as soon as adb sees it, and android_boot then takes its serial.
• Non-ASCII text is refused — install ADBKeyboard and select it as the input method (see Requirements). The refusal is deliberate: input text would silently drop or mangle the characters.
• android_find_text says OCR is unavailable — OCR needs a macOS host (Apple's Vision framework). The 17 non-OCR tools work everywhere.
• The stream stops by itself — that is the idle policy, not a crash: with zero consumers (panel closed, no cards mounted, no route active) the stream stops after 5 minutes and restarts on the next tool call or panel open. A crashed loop restarts on its own within ~5 seconds.
• Rotation looks wrong on the launcher — launchers and Settings pin themselves to portrait and ignore user_rotation. That is normal Android behaviour, not a plugin bug; rotate inside an app that allows it.

pnpm install
pnpm run build      # host tsc + client bundle → lib/
pnpm run typecheck
pnpm test           # every static suite; no device required

The scripts/ smoke suites exercise the built lib/. All of them are static except dev-emulator-smoke.mjs, which needs a device and reports SKIP (exit 0) when there is none.

If the panel streams a solid white (or black) image while android_ui_tree still sees real UI elements, the emulator's host-GPU framebuffer readback is broken on your machine (a known gfxstream issue on some macOS hosts — screencap itself returns blank frames, so every screen tool is affected). Relaunch the emulator with software rendering:

emulator -avd <name> -gpu swiftshader_indirect

or set hw.gpu.mode=swiftshader_indirect in the AVD's config.ini. Physical devices are never affected.

• A higher-frame-rate source. The StreamSource seam is deliberately pluggable: an scrcpy-server + WebCodecs H.264 path would replace the per-frame PNG stream without touching the routes, the tools, or the panel.
• Compose preview hot reload. The iOS twin hot-swaps SwiftUI previews as a dylib; Compose has no equivalent hot-swap primitive today, so this stays a future item rather than a shipped-and-flaky one.

• DSH iOS Simulator — the same architecture for the iOS Simulator and USB-connected iPhones
• DSH Crew — dispatch work to DSH agents from Claude Code / Codex
• DSH Noema — long-term memory for DSH
• DSH OpenPencil — inspect and edit .op design documents inside a conversation

• Android SDK platform-tools (adb) — resolved at runtime, never redistributed: Google's SDK licence does not permit bundling it.
• ADBKeyboard — Senzhk — the optional on-device IME behind non-ASCII typing (Apache-2.0; not bundled).
• Architecture and route posture shared with dsh-ios, from which this plugin is ported.
• See THIRD_PARTY_NOTICES.md for the full notices.

License: MIT

The default auto uses Vision on macOS and optional Tesseract on Windows/Linux. Tesseract and its language data must be installed by the operator; the plugin does not install them. Chinese character words are joined within each TSV line; Latin word separators are preserved.

• DSHPLUGIN_ANDROID_OCR_BACKEND: auto, vision, tesseract.
• DSHPLUGIN_ANDROID_TESSERACT_BINARY: Tesseract executable path.
• DSHPLUGIN_ANDROID_TESSERACT_LANGUAGES: eng+chi_sim.
• DSHPLUGIN_ANDROID_TESSERACT_DATA_DIR: Optional language-model directory.

Tesseract CLI