Enhancing Mixed Reality Cloud Apps with Vuforia

Introduction

Early mobile phones equipped with a camera module provided the convenience of in the moment capture without having to carry an actual camera; with portable processing power and network speeds improved, video chat became a common application. Further improvements in computational capabilities placed mobile cameras in machine vision and image processing apps.

Visual Markers

One innovation is the use of a picture to hold information that can be processed readily with an ordinary camera snapshot. Early solutions included QR Codes and other visual tags to serve as an equivalent to the bar code that can hold more information. The primary use case is to hold information that can be passed to an application from unique identification, data payload or even a website URL and open a browser.

Current solutions available from open source projects and commercial providers support marker image processing not just for data payloads but the ability to compute the position of the marker in world space for use in augmented and mixed reality solutions.

Augmented and Mixed Reality Cloud Apps

The Vuforia SDK supports marker and 3D object processing along with several other features listed on the Vuforia developer resources. Markers also do not have to be machine-centric such as bar codes we see on product packaging or QR codes which are obviously not for human consumption. Vuforia works with custom pictures provided it has enough distinguishing features and payload data on VuMarks which can embed data strings on a human friendly image or logo.

In a hack project, we created a Mixed Reality client app for a popular music streaming music service on HoloLens with Vuforia image markers. The core demo application connects to a cloud API that allows the user to retrieve playlists, album art which are then posted on the user’s walls.

Using image target markers placed on cloud connected speakers, we display the current playing song along with basic commands such as skipping to the next song and transferring the playback to another speaker. The markers identify the specific speaker unit and position to place the song name, album art and animated note model.

In cases where multiple devices need to be differentiated, VuMarks can be used to hold a unique identifier that is passed to a cloud data provider. In other projects, we used VuMarks to identify IoT (Internet of Things) units to provide telemetry, notification summaries and command and control through cloud IoT API’s. Since the markers also provide 3D spatial position information, they are used to identify rooms and serve as a reference anchor point to superimpose a room layout model and identify equipment inside. In the following sample image, the “ACME” logo is for the human viewer and the surrounding glyphs pass information to the application.

Summary

Technology and tools continue to improve, today we extend mobile computing capabilities through cloud services and simplify user to world interaction and minimizing user manual input by using alternative systems such as the camera and voice command. We’ll eventually have machine vision and device integration that may make markers seem primitive, in the meantime it is an innovative solution for identification, data payloads and spatial positioning.

Resources

For questions, comments or contact – follow/message me on Twitter @rlozada