{"id":569,"date":"2019-06-14T20:59:41","date_gmt":"2019-06-15T00:59:41","guid":{"rendered":"http:\/\/dominionsw.com\/wordpress\/?p=569"},"modified":"2019-11-25T07:53:23","modified_gmt":"2019-11-25T11:53:23","slug":"hololens-and-directx-3d-the-ecstasy-and-the-agony-of-example-code-part-1","status":"publish","type":"post","link":"https:\/\/www.dominionsw.com\/?p=569","title":{"rendered":"Hololens and DirectX 3D- the Ecstasy &#8211;  and the Agony of example code, Part 1."},"content":{"rendered":"<p>Recently I&#8217;ve been working on a proof of concept\u00a0 project for a client that uses Microsoft&#8217;s Mixed Reality Hololens device.<\/p>\n<p>The Hololens is an intriguing device, giving the user &#8211; (or should I say &#8220;wearer&#8221; ?) &#8211; of the device the ability to place holograms &#8211; 3D &#8220;imaginary&#8221; shapes &#8211;\u00a0 into the space they see in front of them. This is the &#8220;mixed&#8221; in Mixed Reality. You see through the clear lenses, and in addition, you see holographic objects in your view as if they were placed in the 3D space around you.<\/p>\n<h2>Relativity<\/h2>\n<p>Einstein showed us that there is no fixed point in space. Everything is relative. So, when you want to put a hologram in space, where is it exactly?<\/p>\n<p>Similarly to DICOM images, where everything is relative to the patient, we can say with the hololens that everything is relative to the user. More precisely, relative to the sensors on the headset, but with calibration and reasonable care the origin point is roughly at the xz plane of the user&#8217;s eyes, with the y axis pointing up and the z axis pointing towards the user&#8217;s back.<\/p>\n<p><a href=\"http:\/\/dominionsw.com\/wordpress\/wp-content\/uploads\/2019\/06\/Capture.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-573 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture.png\" alt=\"\" width=\"1269\" height=\"1084\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture.png 1269w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-300x256.png 300w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-768x656.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1024x875.png 1024w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-624x533.png 624w\" sizes=\"auto, (max-width: 1269px) 100vw, 1269px\" \/><\/a><\/p>\n<p>Above, we see the wearer &#8220;gazing&#8221; at the fish hologram.<\/p>\n<p>So, where exactly is the fish hologram?<\/p>\n<h2>Example Code<\/h2>\n<p>Microsoft provides a fair amount of Direct X <a href=\"https:\/\/github.com\/microsoft\/Windows-universal-samples\">example code on GitHub.<\/a><br \/>\nLet&#8217;s take a look at a simple example.<\/p>\n<p>In the BasicHologram example, we see in BasicHologramMain.cpp at line 579<br \/>\n<code><br \/>\n\/\/ The simplest way to render world-locked holograms is to create a stationary reference frame<br \/>\n\/\/ based on a SpatialLocator. This is roughly analogous to creating a \"world\" coordinate system<br \/>\n\/\/ with the origin placed at the device's position as the app is launched.<br \/>\nm_stationaryReferenceFrame = m_spatialLocator-&gt;CreateStationaryFrameOfReferenceAtCurrentLocation();<br \/>\n<\/code><\/p>\n<p>OK this makes sense. When you start up your application, when you want to grab a fixed, i.e. &#8220;world locked&#8221; frame, you use\u00a0<code>CreateStationaryFrameOfReferenceAtCurrentLocation().<\/code><br \/>\nIf you position a hologram in this frame of reference, it will stay there, and if you move around it will stay in that position:<a href=\"http:\/\/dominionsw.com\/wordpress\/wp-content\/uploads\/2019\/06\/Capture-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-576 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1.png\" alt=\"\" width=\"1413\" height=\"1290\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1.png 1413w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1-300x274.png 300w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1-768x701.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1-1024x935.png 1024w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/Capture-1-624x570.png 624w\" sizes=\"auto, (max-width: 1413px) 100vw, 1413px\" \/><\/a><\/p>\n<p>So above we see our intrepid goggle wearer moving, and the fish hologram stays positioned relative to the <em>original<\/em> frame of reference.<\/p>\n<p>After working with this sample code for a bit, I then moved on to the next feature I needed; namely, access to the front facing color video camera.<\/p>\n<h2>You can&#8217;t get there from here<\/h2>\n<p>Alas, there just seemed to be no way to access the video stream from within this example code. Something to do with it being in the WinRT architecture vs the UWP architecture I think. But I gave up trying to figure it out, and moved to a code example that had access to the video camera, the HolographicFaceTracking example.<\/p>\n<p>It didn&#8217;t take long to move my existing code over to this new example, and although it was in Microsoft&#8217;s proprietary Managed C++, I decided to bite the bullet and stick with this as a basis for my experimentation.<\/p>\n<p>Everything seemed to be working correctly, and I was able to get the video stream and draw it to a texture so I could see it within the Hololens view. But after a while, I realized there was one problem. My Holographic shape was not behaving naturally;\u00a0 as I moved towards it or away from it, it stayed the same size. Normally, if you viewed our friendly fish, positioned in the &#8220;world&#8221; coordinate system, it would get bigger as you walked towards it, as below:<\/p>\n<p>Fish at one distance<\/p>\n<p><a href=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishSmall.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-580 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishSmall.png\" alt=\"\" width=\"682\" height=\"708\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishSmall.png 682w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishSmall-289x300.png 289w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishSmall-624x648.png 624w\" sizes=\"auto, (max-width: 682px) 100vw, 682px\" \/><\/a><\/p>\n<p>Fish some steps closer.<\/p>\n<p><a href=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-581 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig.png\" alt=\"\" width=\"1028\" height=\"1002\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig.png 1028w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig-300x292.png 300w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig-768x749.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig-1024x998.png 1024w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/fishBig-624x608.png 624w\" sizes=\"auto, (max-width: 1028px) 100vw, 1028px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>However this wasn&#8217;t happening. It was puzzling. Until I finally realized that the FaceTracking Example just couldn&#8217;t be doing what it was saying it does.<\/p>\n<p>The code in the BasicHologram example has the following comment when it places the hologram:<\/p>\n<pre><code> \/\/ The simplest way to render world-locked holograms is to create a stationary reference frame\r\n  \/\/ based on a SpatialLocator. This is roughly analogous to creating a \"world\" coordinate system\r\n  \/\/ with the origin placed at the device's position as the app is launched.<\/code><\/pre>\n<p>Ok, that looks correct.<\/p>\n<p>And the FaceTracking example has the same comment in the same area of code:<\/p>\n<pre><code> \/\/ The simplest way to render world-locked holograms is to create a stationary reference frame\r\n \/\/ based on a SpatialLocator. This is roughly analogous to creating a \"world\" coordinate system\r\n \/\/ with the origin placed at the device's position as the app is launched.<\/code><\/pre>\n<p>I&#8217;m guessing that many of you have already guessed the problem; the code doesn&#8217;t match the comment.<\/p>\n<p>In the BasicHologram example, we have the code that matches the comment.<\/p>\n<pre><code>m_stationaryReferenceFrame = m_spatialLocator-&gt;CreateStationaryFrameOfReferenceAtCurrentLocation();\r\n<\/code><\/pre>\n<p>And of course in the FaceTracking example, we have the code that doesn&#8217;t match the comment.<\/p>\n<pre><code> m_referenceFrame = m_locator-&gt;CreateAttachedFrameOfReferenceAtCurrentHeading();<\/code><\/pre>\n<p>So, of course in the latter case, the hologram stays &#8220;fixed&#8221; but always relative to the origin of the user. It stays fixed only with respect to rotation, though. It moves with the user with respect to distance.<a href=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-582 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1.png\" alt=\"\" width=\"1125\" height=\"962\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1.png 1125w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1-300x257.png 300w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1-768x657.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1-1024x876.png 1024w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/TopView1-624x534.png 624w\" sizes=\"auto, (max-width: 1125px) 100vw, 1125px\" \/><\/a><\/p>\n<p>Above we see our fish gazer looking normally at the hologram.<\/p>\n<p><a href=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-583 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2.png\" alt=\"\" width=\"1202\" height=\"982\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2.png 1202w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2-300x245.png 300w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2-768x627.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2-1024x837.png 1024w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/topView2-624x510.png 624w\" sizes=\"auto, (max-width: 1202px) 100vw, 1202px\" \/><\/a><\/p>\n<p>If he takes a few steps backwards, with the &#8220;Attached Frame of Reference&#8221;, the fish moves with him, and so appears to <em>not<\/em> change size. But now the part that confused me.<\/p>\n<p><a href=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-584 size-full\" src=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3.png\" alt=\"\" width=\"1291\" height=\"1324\" srcset=\"https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3.png 1291w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3-293x300.png 293w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3-768x788.png 768w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3-998x1024.png 998w, https:\/\/www.dominionsw.com\/wp-content\/uploads\/2019\/06\/vertical3-624x640.png 624w\" sizes=\"auto, (max-width: 1291px) 100vw, 1291px\" \/><\/a><\/p>\n<p>If the user moves backwards, and then <em>rotates<\/em> their position, the hologram stays in place. So it&#8217;s invariant to rotation.<\/p>\n<p>This all makes good sense to me now. And I&#8217;m moving forward on my experiment. But the moral of the story is: Trust comments, but verify.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently I&#8217;ve been working on a proof of concept\u00a0 project for a client that uses Microsoft&#8217;s Mixed Reality Hololens device. The Hololens is an intriguing device, giving the user &#8211; (or should I say &#8220;wearer&#8221; ?) &#8211; of the device the ability to place holograms &#8211; 3D &#8220;imaginary&#8221; shapes &#8211;\u00a0 into the space they see [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-569","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/posts\/569","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=569"}],"version-history":[{"count":12,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/posts\/569\/revisions"}],"predecessor-version":[{"id":610,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=\/wp\/v2\/posts\/569\/revisions\/610"}],"wp:attachment":[{"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=569"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=569"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dominionsw.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=569"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}