Recently I’ve been working on a proof of concept  project for a client that uses Microsoft’s Mixed Reality Hololens device.

The Hololens is an intriguing device, giving the user – (or should I say “wearer” ?) – of the device the ability to place holograms – 3D “imaginary” shapes –  into the space they see in front of them. This is the “mixed” in Mixed Reality. You see through the clear lenses, and in addition, you see holographic objects in your view as if they were placed in the 3D space around you.

Relativity

Einstein showed us that there is no fixed point in space. Everything is relative. So, when you want to put a hologram in space, where is it exactly?

Similarly to DICOM images, where everything is relative to the patient, we can say with the hololens that everything is relative to the user. More precisely, relative to the sensors on the headset, but with calibration and reasonable care the origin point is roughly at the xz plane of the user’s eyes, with the y axis pointing up and the z axis pointing towards the user’s back.

Above, we see the wearer “gazing” at the fish hologram.

So, where exactly is the fish hologram?

Example Code

Microsoft provides a fair amount of Direct X example code on GitHub.
Let’s take a look at a simple example.

In the BasicHologram example, we see in BasicHologramMain.cpp at line 579

// The simplest way to render world-locked holograms is to create a stationary reference frame
// based on a SpatialLocator. This is roughly analogous to creating a "world" coordinate system
// with the origin placed at the device's position as the app is launched.
m_stationaryReferenceFrame = m_spatialLocator->CreateStationaryFrameOfReferenceAtCurrentLocation();


OK this makes sense. When you start up your application, when you want to grab a fixed, i.e. “world locked” frame, you use CreateStationaryFrameOfReferenceAtCurrentLocation().
If you position a hologram in this frame of reference, it will stay there, and if you move around it will stay in that position:

So above we see our intrepid goggle wearer moving, and the fish hologram stays positioned relative to the original frame of reference.

After working with this sample code for a bit, I then moved on to the next feature I needed; namely, access to the front facing color video camera.

You can’t get there from here

Alas, there just seemed to be no way to access the video stream from within this example code. Something to do with it being in the WinRT architecture vs the UWP architecture I think. But I gave up trying to figure it out, and moved to a code example that had access to the video camera, the HolographicFaceTracking example.

It didn’t take long to move my existing code over to this new example, and although it was in Microsoft’s proprietary Managed C++, I decided to bite the bullet and stick with this as a basis for my experimentation.

Everything seemed to be working correctly, and I was able to get the video stream and draw it to a texture so I could see it within the Hololens view. But after a while, I realized there was one problem. My Holographic shape was not behaving naturally;  as I moved towards it or away from it, it stayed the same size. Normally, if you viewed our friendly fish, positioned in the “world” coordinate system, it would get bigger as you walked towards it, as below:

Fish at one distance

Fish some steps closer.

 

However this wasn’t happening. It was puzzling. Until I finally realized that the FaceTracking Example just couldn’t be doing what it was saying it does.

The code in the BasicHologram example has the following comment when it places the hologram:

 // The simplest way to render world-locked holograms is to create a stationary reference frame
  // based on a SpatialLocator. This is roughly analogous to creating a "world" coordinate system
  // with the origin placed at the device's position as the app is launched.

Ok, that looks correct.

And the FaceTracking example has the same comment in the same area of code:

 // The simplest way to render world-locked holograms is to create a stationary reference frame
 // based on a SpatialLocator. This is roughly analogous to creating a "world" coordinate system
 // with the origin placed at the device's position as the app is launched.

I’m guessing that many of you have already guessed the problem; the code doesn’t match the comment.

In the BasicHologram example, we have the code that matches the comment.

m_stationaryReferenceFrame = m_spatialLocator->CreateStationaryFrameOfReferenceAtCurrentLocation();

And of course in the FaceTracking example, we have the code that doesn’t match the comment.

 m_referenceFrame = m_locator->CreateAttachedFrameOfReferenceAtCurrentHeading();

So, of course in the latter case, the hologram stays “fixed” but always relative to the origin of the user. It stays fixed only with respect to rotation, though. It moves with the user with respect to distance.

Above we see our fish gazer looking normally at the hologram.

If he takes a few steps backwards, with the “Attached Frame of Reference”, the fish moves with him, and so appears to not change size. But now the part that confused me.

If the user moves backwards, and then rotates their position, the hologram stays in place. So it’s invariant to rotation.

This all makes good sense to me now. And I’m moving forward on my experiment. But the moral of the story is: Trust comments, but verify.