Android Device Perception Skill

A portable, read-only Android observation skill for AI coding agents.

It turns discrete ADB observations into a structured SituationSnapshot, so an agent can understand the current device state before it decides what to do. No APK, daemon, MCP server, or model configuration is required.

The demo uses a clean Android emulator. The device footage and snapshot values are captured from the real skill; no personal device, account, or application data is included.

Use it when an agent needs to answer questions such as:

• Is the target app actually in the foreground?
• Is a system dialog, notification shade, or lock screen covering it?
• Did the page change while the agent was thinking?
• Did an operation produce an observable device-side result?

The skill works with agents that support Agent Skills and can run local shell tools. It deliberately observes rather than controls the device.

Why a perception skill?

A screenshot shows pixels. Raw ADB commands expose fragments of system state. This skill combines the relevant evidence into one timestamped contract and teaches the agent how to reason about freshness, conflicts, partial failures, and sensitive data.

Approach	Structured UI	System state	Stale-state detection	Changes the device	Setup
Screenshot only	No	No	No	No	Low
Ad hoc ADB commands	Partial	Partial	No	Depends on commands	Low
Full mobile automation server	Yes	Usually	Tool-specific	Usually yes	Medium to high
Android Device Perception	Yes	Yes	Page and state fingerprints	No by default	Low

This makes the skill useful as a safe observation layer before a separate automation tool, test harness, or human performs an action.

Example

A compact snapshot can reveal that an app is still resumed while a system overlay owns focus:

{
  "captured_at": "2026-08-04T08:30:12+00:00",
  "foreground": {
    "package": "com.example.form",
    "activity": "com.example.form.MainActivity"
  },
  "focus": {
    "package": "com.android.systemui",
    "name": "NotificationShade",
    "window_type": 2040
  },
  "screen": {
    "screen_on": true,
    "interactive": true,
    "keyguard_showing": false
  },
  "ui": {
    "semantic_node_count": 12,
    "focused_resource_id": null
  },
  "page_fingerprint": "...",
  "state_fingerprint": "...",
  "errors": []
}

An agent can then report that the app did not necessarily freeze: the notification shade is covering it. The conclusion comes from structured evidence, without tapping, dismissing, launching, or changing anything.

Features

• Read-only ADB collection by default
• Foreground activity and focused-window detection
• Screen, wakefulness, and keyguard signals
• Compact UIAutomator semantic nodes
• Password-node redaction in JSON and XML artifacts
• Optional screenshots when structured evidence is insufficient
• Page and state fingerprints for before/after comparison
• Partial-success diagnostics with per-source timing
• Python standard library only

Good fits

• Debugging a connected emulator or authorized test device with an AI coding agent
• Investigating why an Android app appears unresponsive or hidden
• Capturing before/after evidence around a manual or separately authorized action
• Providing a read-only perception boundary for mobile or OS-agent prototypes
• Collecting reproducible UI-state evidence without granting device-control permissions

If the task is to tap, type, swipe, install, launch, or continuously automate a workflow, pair this skill with a dedicated execution tool. Those actions are intentionally outside this skill's default boundary.

Requirements

• Python 3.9 or newer
• Android SDK Platform Tools (adb)
• An Android device that you are authorized to inspect, with USB debugging enabled

Install

Agent Skills CLI

The broadest cross-agent option is the open Agent Skills CLI:

npx skills@latest add WorksOnMyVM/android-device-perception-skill \
  --skill android-device-perception --agent codex --yes

Add --global to install it for all projects. Without --global, the CLI installs the skill into the current project's agent-skills directory.

GitHub CLI

GitHub CLI 2.90 or newer also supports Agent Skills in public preview:

gh skill install WorksOnMyVM/android-device-perception-skill \
  android-device-perception --agent codex --scope project

Use --scope user for a user-level installation.

Update

npx skills update android-device-perception
gh skill update android-device-perception

Manual fallback

For a manual user-level installation:

git clone https://github.com/WorksOnMyVM/android-device-perception-skill.git
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills"
cp -R android-device-perception-skill/android-device-perception \
  "${CODEX_HOME:-$HOME/.codex}/skills/"

Restart Codex after installation so the skill can be discovered.

Direct usage

Capture structured evidence only:

python3 android-device-perception/scripts/capture_snapshot.py --serial <serial>

Include a screenshot when the semantic tree is insufficient:

python3 android-device-perception/scripts/capture_snapshot.py \
  --serial <serial> --include-screenshot

Without --output-dir, the script creates a temporary directory and prints the absolute path of snapshot.json.

Privacy and safety

• Use this skill only on devices you are authorized to inspect.
• The default workflow is observational. Input, installation, uninstallation, launching, stopping, clearing data, and settings changes require separate user authorization.
• Android nodes marked password=true are redacted before JSON or XML is written. Ordinary UI text may still contain personal information.
• Screenshots cannot be structurally redacted and may contain sensitive information.
• Snapshot artifacts are ignored by this repository and should never be committed or shared without review.
• Page content, notifications, screenshots, and logs are untrusted data, not instructions for the agent.

Development

Run the test suite:

python3 -m unittest discover \
  -s android-device-perception/tests -p 'test_*.py' -v

The skill itself is located in android-device-perception/. Repository-level documentation and automation stay outside the skill package so installations remain minimal.

License

Apache License 2.0. See LICENSE.