# Hand tracking in the visionOS simulator, driven by your webcam
ARKit hand tracking does not run in the visionOS simulator, which makes hand-driven apps painful to iterate on. DicyaninMockHandTracking supplies a mock pose source, on-screen joysticks, and a macOS webcam runner that streams your real hands into a running simulator build.
- URL: https://dicyaninlabs.com/blog/hand-tracking-in-the-visionos-simulator
- Author: Hunter
- Published: 2026-07-20
- Tags: visionOS, Hand Tracking, ARKit, Swift
- Reading time: 4 min read (608 words)

---

ARKit hand tracking does not run in the visionOS simulator. `HandTrackingProvider` returns nothing, so any app whose core interaction is hands has to go on device for every change. That is a slow loop for a feature you are iterating on dozens of times an hour.

[DicyaninMockHandTracking](https://github.com/hunterh37/DicyaninMockHandTracking) fills the gap. It publishes mock hand poses that your app reads exactly where it would read ARKit anchors, steers them from an on-screen control panel, and (as of 3.0) streams your actual hands in from a Mac webcam over the local network.

> [video] /videos/mock-hand-tracking-demo.mp4

## One pose source, two backends

The package is built around a single shared controller. Read from it instead of ARKit in simulator builds:

```swift
import DicyaninMockHandTracking

let controller = MockHandTrackingController.shared

controller.leftHandPosition   // SIMD3, head-relative
controller.rightHandPosition
controller.rightHandYaw       // Float, radians
controller.isPinching         // Bool

for await _ in controller.updates() {
    // 60 fps tick, read positions here
}
```

Put the environment switch behind one type so the rest of the app never branches:

```swift
import simd
import DicyaninMockHandTracking
#if !targetEnvironment(simulator)
import ARKit
#endif

struct HandPose {
    var position: SIMD3
    var isPinching: Bool
}

@MainActor
final class HandSource {
    #if targetEnvironment(simulator)
    private let mock = MockHandTrackingController.shared

    func rightHand() -> HandPose {
        HandPose(position: mock.rightHandPosition, isPinching: mock.isPinching)
    }
    #else
    private let session = ARKitSession()
    private let provider = HandTrackingProvider()

    func start() async throws {
        try await session.run([provider])
    }

    func rightHand() -> HandPose {
        guard let anchor = provider.latestAnchors.rightHand,
              anchor.isTracked,
              let wrist = anchor.handSkeleton?.joint(.wrist) else {
            return HandPose(position: .zero, isPinching: false)
        }
        let m = anchor.originFromAnchorTransform * wrist.anchorFromJointTransform
        return HandPose(
            position: SIMD3(m.columns.3.x, m.columns.3.y, m.columns.3.z),
            isPinching: detectPinch(anchor)
        )
    }
    #endif
}
```

Call sites use `handSource.rightHand()` and never know which backend produced the pose.

## The control panel

Drop the overlay into a window in simulator builds only. It gives you two joysticks, rotation sliders, and a pinch button:

```swift
#if targetEnvironment(simulator)
MockHandControlView()
#endif
```

That covers deterministic testing: park a hand at an exact position, fire a pinch, verify the hit.

## Webcam poses

Dragging joysticks is fine for reproducing a case and bad for feeling out an interaction. The `WebcamHandRunner` macOS app in `Examples/WebcamHandRunner` estimates hand poses with Vision's `VNDetectHumanHandPoseRequest` and broadcasts them over `_dicyaninhands._tcp`. Your app subscribes with one call:

```swift
ContentView()
    .task {
        #if targetEnvironment(simulator)
        MockHandTrackingController.shared.connectToWebcamRunner() // localhost:50673
        #endif
    }
```

The simulator shares your Mac's network, so localhost works with no configuration. On a real Vision Pro on the same Wi-Fi, use Bonjour discovery instead:

```swift
MockHandTrackingController.shared.connectToWebcamRunner(bonjourName: nil)
```

Call `disconnectWebcamRunner()` to hand control back to the joysticks. Because the webcam writes into the same published state the joysticks do, every existing consumer works unchanged.

> Tip: a webcam is one 2D view. Depth is approximated from apparent hand size, and yaw comes from the wrist to knuckle direction. Use it for iteration speed, not metric precision, and validate against ARKit on device.

## Gloves

`DicyaninHandGlove` vendors Apple's [Tracking and visualizing hand movement](https://developer.apple.com/documentation/visionos/tracking-and-visualizing-hand-movement) sample (a `HandTrackingComponent` plus `HandTrackingSystem` mapping all 27 skeleton joints per frame) as one view, with a simulator bridge to the mock controller:

```swift
import DicyaninHandGlove

ImmersiveSpace(id: "Gloves") {
    HandGloveView()
}
```

Configure the look, or load your own rigged USDZ:

```swift
HandGloveView(configuration: .init(style: .joints))          // Apple's spheres
HandGloveView(configuration: .init(tracksLeftHand: false, color: .orange))
HandGloveView(configuration: .init(
    style: .model(left: "LeftGlove_v001", right: "RightGlove_v001")
))
```

On device the gloves follow the real skeleton joint for joint. In the simulator they follow the joysticks or the webcam. Same code path.

## Next steps

Add the package at `https://github.com/hunterh37/DicyaninMockHandTracking.git` (3.0.0+), then run `Examples/HandTrackingDemo` on a simulator to see gloves, control panel, and webcam bridge wired together.