Microsoft HoloLens Hand Tracking

Default user input for Microsoft HoloLens is with gestures, gaze and voice; supplemented by Bluetooth devices such as the Clicker that is shipped with the device, Bluetooth keyboard and mice and the latest Xbox One S Bluetooth enabled controller.

Microsoft HoloToolkit for Unity provides prefabs and scripts to quickly develop in Unity and handling hand gesture events such as the air tap, drag, resize and other interesting object interactions.

One of the events available is detecting if the user’s hand is within the detection cone in front of the HoloLens known as the “Gesture Frame” but covers only the Hold, Manipulation and Navigation gestures.

We can infer that the HoloLens is able to track the hand — how do we retrieve the hand’s position in the holographic space?

The solution is similar to the approach done with the HandsManager.cs in the HoloToolkit through the InteractionManager and using the SourceUpdated Event.

void Awake()
{
    InteractionManager.SourceDetected += InteractionManager_SourceDetected;
    InteractionManager.SourceLost += InteractionManager_SourceLost;
    InteractionManager.SourceUpdated += InteractionManager_SourceUpdated;
}

To access the detected hand position in the Holographic space, the data is available through the state.properties.location attribute from the InteractionSourceState.   In the following snippet,  an object is instantiated when a hand is first detected and tracked in a Dictionary.

private void InteractionManager_SourceDetected(InteractionSourceState state)
{
    if (state.source.kind != InteractionSourceKind.Hand)
    {
        return;
    }
    trackedHands.Add(state.source.id);

    var obj = Instantiate(TrackingObject) as GameObject;
    Vector3 pos;
    if (state.properties.location.TryGetPosition(out pos))
    {
        obj.transform.position = pos;
    }
    trackingObject.Add(state.source.id, obj);
}

When the user’s hand moves within the gesture frame, it will generate InteractionManager.SourceUpdated events so we can move the tracking object to the updated hand position.

private void InteractionManager_SourceUpdated(InteractionSourceState state)
{
    uint id = state.source.id;
    Vector3 pos;

    if (state.source.kind == InteractionSourceKind.Hand)
    {
        if (trackingObject.ContainsKey(state.source.id))
        {
            if (state.properties.location.TryGetPosition(out pos))
            {
                trackingObject[state.source.id].transform.position = pos;
            }
        }
    }
}

Most applications will only need to use the gesture events in the HoloToolkit but in case there’s a desire to create a hologram of a fireball around the magic user’s casting hand — this is a possible solution, just remember to have the forefinger prominent so the hand is detected in the “ready” position.

The rest of the clean-up code needed is posted in the Unity Project: https://github.com/ritchielozada/HoloLensHandTracking