My #HoloLens 101 notes

HoloLens, Technical stuff, Work
See comments

Yes finally I have a HoloLens device (thank you IdentityMine for footing the bill and facilitating the purchase). Even though IdentityMine has a long history of 3D development, and we have been working with HoloLens devices for the past few months in collaboration with Microsoft, so far my inputs have been limited to helping the team brainstorm concepts, test the apps, give feedback, and help with some presales activities. In the contrary of some of my colleagues, I am not a 3D developer yet, and I am eager to learn.

Here are a few notes I took during the learning stages. I am sure that many of you are in the same boat as I am, and these may come handy if you want to progress with HoloLens and Unity3D development.

A few preliminaries

As I mentioned, IdentityMine has been developing for HoloLens for a while now. You can see a session that my esteemed colleague Rene Schulte and I gave at the recent Build 2016 conference about an application that we built.

Even though you can code HoloLens applications with any 3D framework that supports Direct3D, the consensus these days (and the solution that Microsoft recommends) is to use Unity. This is a middleware (meaning that this is a set of APIs that will create code that then runs on a number of platforms, including HoloLens).

Rene Schulte has a very good blog post with recommendations. You should probably read it too!

Charging the device

You would think that charging the device is trivial! Well in fact… it charges with a micro-USB cable similar to what you use for your mobile phone (except if you have the latest generation which uses USB-C). Like for mobile phones, there are two kinds of cables: Charging only, and Charging+Data. Whichever you use, make sure that you test your cables and use the ones which charge the fastest. When testing cables to charge my mobile phones, I found huge discrepancies (some cables charge up too 3-4 times faster than others). A good choice is to use the cable and charger which come with the HoloLens device itself!

With a full charge you will be able to use the device for a few hours. I didn’t time it really, but when learning to code, I leave the device on and use it on and off for a whole evening without needing to charge. You can then charge it overnight, which is very convenient (it takes a while to get a full charge, so plan accordingly!).

Calibrating

Before you start using the device, you need to calibrate it. This is important so that your eyes can focus properly on the holograms. The Calibration app takes a couple of minutes to execute. The very first time you start the device, you will also get a tutorial about gestures. Later on, the Learn Gestures app can be started separately. Probably a good idea to run through this basic information! Note that you can modify and retrieve the Interpupillary distance (IPD) from the Device Portal (see below). This is an important value for a good experience!

Using apps

What better way to get started than to use apps! For a first try, the Holograms app (preinstalled) is cool: It allows you to place holograms anywhere in the room, and observe them from diverse angles. There are even some video holograms. It’s a good idea to use this app to learn to interact with the menus. Note that you will need an internet connection for this app to work!

In addition to the preinstalled apps, you have a choice of apps developed specifically for HoloLens in the store. I like the HoloTour, which gives you a guided tour of Rome and a high plateau in Peru. I hope we will see more content coming soon! Also available are games like Young Conker and Roboraid. Another app named Holo Anatomy will teach you about the human body. While HoloTour, Young Conker and Roboraid are what I would call full blown apps, Holo Anatomy is more of a proof of concept. Still cool to try!

Make sure to try the Fragments game too! It is probably the most compelling holographic experience at the moment, and integrates very nicely with the room you are in! One of the character was seen sitting on my office chair for example!!

There is also a great app developed on suggestions from the users’ community, called Galaxy Explorer. This app is available as source code on Github and you need to build it. See below in the “Learning to code” section.

You can also install 2D universal apps for Windows 10. Any UWP app should run fine though some (like Skype) have been customized for the HoloLens device. I installed OneDrive (which allows me to watch videos stored in the cloud), Groove Music (which lets me stream music from my OneDrive), Cast (a great universal podcast app which synchronizes content between all Windows 10 devices) and a few more.

Unfortunately I cannot find a way to download and save a file on the device, so for now the only media you can consume is online streamed media. This is OK but I hope we’ll be able to get some media on the drive for offline consumption at some point.

Here are a few screenshots. Note that some are a screen grab of the portal videostream so the quality is not great, but they are just illustrating the point. The actual quality is much better!

13173339_10154069664749651_6183228024374949655_o
The game “Fragments” is one of the most impressive HoloLens experiences at the moment.

20160512_133624_HoloLens
Groove Music and Skype pinned to my home office wall

2016-05-12_14-04-31
On a HoloSkype call with my colleague Rene. We both can draw on the scene (him in green and me in blue).

Finding your device’s IP address

In many occasions you will need your device’s IP address. To do this, wear the device, and then say “Hey Cortana”. After Cortana shows up and you hear the sound meaning that she is listening, ask “What is my IP address”. It will show up. Alternatively, you can open the Settings app, and then navigate to Network & Internet, Wi-Fi, Advanced Options and write down the IPv4 address. Note that you can also find it using your Router’s configuration menu, if you have access to it. It might be a good idea to configure your router to always use the same IP address for the device.

Using the device portal or the app

The device portal is a must-try. To open it is quite easy: First you need to find your device’s IP address (see above). Then you can enter this IP address in a web browser (ignore warnings about invalid security certificate), and it will connect to your device which acts as a web server. Note that on some networks, this will be prevented to work so make sure that the network you use is open to this.

Alternatively, you can try the HoloLens app available here. The cool thing is that it is a universal app, and so you can also let it run on a Windows 10 mobile phone. UWP FTW!!

The web portal lets you do the following operations:

  • Checking the device status and other information such as device name and Windows version, online status, temperature, battery level etc
  • Check your interpupillary distance (IPD) and modify this value
  • Change sleep settings. I recommend setting a high enough value so the device doesn’t go to sleep while you are coding, which can cause deployment to fail!
  • See in real time (almost) what the user is seeing. This is called Mixed Reality Capture (MRC). It even lets you make videos and take pictures (which include the holograms!). Note however that there is a lag of a couple of seconds between what the user sees and what is shown on the screen. Also, the screen refresh rate will drop to 30 FPS while the MRC is active (instead of ideally 60FPS). If you have fast animations, this can be a little disconcerting.
  • See the list of processes running on the device and getting performance information.
  • Seeing a list of installed apps and providing some management tools. You can even side load an app from this page.
  • Getting crash information.
  • Forcing the device to run in kiosk mode (i.e. disabling the start menu, disabling Cortana and hiding pinned applications). Great for demos!
  • Logging
  • Simulating rooms: You can take a capture of a given room and save it. Then you can pass this recording to another developer, who can use this page to load it to his device. Great when you need to test an app in some specific room conditions.
  • Networking settings and information such as device IP, MAC address etc
  • Virtual input, allowing the developer to simulate keyboard input. Note however that you can simply use a bluetooth keyboard if you want to easily enter text.

The universal app seems to show the live stream with less lag. Definitely worth a try!

Taking screenshots and videos

You can take screenshots and videos of what you see in two different ways. Note however that the resolution of the screenshots and the videos is not going to be as good as the real thing, because they downplay the device resolution when screen capture is on.

Using the device portal

If you are an operator for someone else trying the headset, launch the device portal (see above) and navigate to the Mixed Reality Capture section. You will see buttons allowing you to record, take photo or even see the live stream (with a few seconds delay).

Using Cortana

More spontaneously, if you are in a great experience on HoloLens and want to share with the world, you can take a screenshot by saying “Hey Cortana” and then “Take a picture”. Similarly, for videos, say “Hey Cortana” and then “Take a video”. When you are done filming, say “Hey Cortana” and then “Stop video”. This takes a bit of exercise to get right, so make sure to train before you do it for real. The pictures and videos will then show up in the Photos application on the device, as well as in the Device Portal, where they can be downloaded from.

Letting people try the HoloLens

For a better experience, you should always get a new user’s Interpupillary distance IPD! Use the Calibration app to do that. Alternatively, there are some devices one can purchase, but they require some know how so make sure to learn how to use them. Once a user’s IPD has been determined, you can always modify it or retrieve it from the Device Portal (see above).

One thing I noticed after letting a few “newbies” try it out: The “tap” gesture is not as easy as it sounds. A good friend never got it right. So in addition to the calibration, it is interesting to get some time to do the Learn Gestures applications too. At first it is a bit hard to guide them to start the Calibration application. What I did to help is this:

  • Connect the UWP app to the device and observe what they are seeing. Sure it drops the frame rate but it is really helpful to know what they are currently seeing, in the beginning.
  • You can do some gestures for them. Simply put your own hand in front of the visor and bloom or air tap. For example to teach my friend how to air tap, at first I told her to simply look at a tile and then I air tapped myself. This helped her to understand the gaze gesture better.
  • There is a clicker that comes with the HoloLens device, that you can use instead of an “air tap” gesture.

It is pretty overwhelming at first, and for us it is easy to forget that it is pretty overwhelming at first I was happy to have the occasion to observe a few people trying it out for the first time and learn from this experience.

About Unity

It’s easy to confuse Unity (the 3D middleware) with Unity (the Inversion-of-control framework). The fact that both of these are used by Microsoft makes it even more confusing. In case of doubts, make sure to use Unity3D instead of just Unity. In this post, I will simply talk about Unity and this is NOT the IOC one ;)

When you install a new version of Unity3D, it might install side by side with older version. DO NOT GET CONFUSED! If you open a HoloLens Unity project with a version of Unity that is not suitable, you will get VERY confusing error messages. Make absolutely certain that you have the correct version of Unity open. Currently, the version is

Learning to code

The HoloLens academy is a Microsoft offering and has a growing number of tutorials. You can find all the information on their webpage.

Note that you don’t strictly need a device to get started. There is a free emulator which works with Visual Studio and lets you try your code out (see below). It’s a good way to wait for a device, as getting one can take a while due to overwhelming demand.

Learning to deploy

Before you even get to coding, it would be a good idea to learn to build and deploy a project. For instance you can follow these steps to download and build an existing project, the open source Galaxy Explorer:

  • Go to the Galaxy Explorer GitHub repo
  • Fork or download the code as Zip file, and extract it.
  • Start Unity3D
  • In the start dialog, press Open
  • Navigate to the folder GalaxyExplorer and press Select
  • Wait until Unity loads the project
  • Open the menu File / Build Settings
  • Make sure that the Windows Store platform is selected
  • Set SDK to Universal 10
  • Set UWP Build Type to D3D
  • Make sure that Build and Run is set to Local Machine.
  • Press the Build button
  • In the Build Windows Store folder selection menu, create a New Folder and name it App.
  • Make sure that the App folder is selected.
  • Press Select Folder.

This will start the build process. Note that this only creates the Visual Studio project with all necessary files. You will still need to open the created Solution file. Follow the steps:

  • Navigate to the App folder that you created earlier.
  • Open the GalaxyExplorer.sln file in Visual Studio 2015.
  • Make sure to select the following configuration: Release / x86 / Remote Machine.
  • If the Remote Connections dialog shows up, enter your device’s IP address in the Address field and make sure that “Universal (Unencrypted Protocol)” is the Authentication Mode selected. Then press Select to establish a connection to your device.
  • If this is the first time you deploy, you will need to enter a pin. The pin should be shown on your device, but if it is not, go to Settings, Update, For Developers, Pair. Then enter the pin into the Visual Studio dialog.
  • Finally, select Debug / Start without Debugging.

If everything is configured correctly, the application should start, and you can feel confident that deploying works for your future studies.

Don’t let errors like “System.Object doesn’t exist” startle you

If you run the examples of the HoloLens academy, you might encounter some weird errors when you generate the Visual Studio code from Unity, and then open the resulting SLN file in Visual Studio. You will see a LOT of errors because the Nuget packages have not been restored yet. Do not fret, you can just build, which will force the Nuget packages to get restored, and all the ugly errors should go away.

Match the file name and the class name

In C#, it is highly recommended that the file name and the class name match, and that you have only one class by file. But these are recommendations only. When coding in Unity however, I had some compilation errors because the file name didn’t match the class name. If you follow the early tutorials (for example the 101), they will ask you to create a new C# script file with a certain name (for example GazeGestureManager). Make sure to enter that name as you create the script. If you don’t do this, but instead you create a new script file with the default name, and then rename the script file to GazeGestureManager, Unity will refuse to do anything with this file. This is because the class name inside the file and the file name don’t match. So be careful to follow the steps exactly.

Sometimes Unity starts MonoDevelop instead of Visual Studio

In the course of the tutorials, you will sometimes have to edit some C# script files. Unity comes preinstalled with the MonoDevelop code editor, but using Visual Studio is more comfortable. To ensure that Visual Studio is launched, check the following settings: Edit / Preferences / External Tools / External Script Editor and make sure it is set to Visual Studio 2015.

Even with this setting properly configured, it can happen that Unity gets confused and starts MonoDevelop anyway. In my experience the file opens in Visual Studio anyway and you can just close MonoDevelop and move on.S

Starting the emulator

The emulator is a good way to get started when you don’t have a device with you. Strictly speaking, it’s not really faster to deploy to the emulator than to a real device, and of course the experience is not comparable, so you will probably prefer an actual device if you have a chance. You can also simulate gazing at objects, tapping them, walking around etc (see below).

The emulator can be started from Visual Studio directly (it is called HoloLens Emulator in the list of all the emulators and devices). If you don’t see it, you might have to install it first :)

I had some cases where the emulator refused to start, because my machine didn’t have enough memory (for example on a Surface Pro 3 with 8GB RAM, the emulator requires 2GB RAM and Windows decided there wasn’t enough left… annoying!). In that case, you should do the following:

  • Create a BAT file.
  • Enter the following command:
    “C:\Program Files (x86)\Microsoft XDE\10.0.11082.0\XDE.exe” /sku HDE /video 1268×720 /vhd “C:\Program Files (x86)\Windows Kits\10\Emulation\HoloLens\10.0.11082.1039\flash.vhd”
  • Save the BAT file and restart your machine
  • Before you do anything else, run the BAT file to start the emulator.
  • Then only start Visual Studio and deploy to the emulator, which is already running.

Issues when deploying to the device

In theory you can deploy to the device using a USB cable attached to your laptop, or using an internet connection and “Remote Machine” (not Remote Device like stated in some locations in the official documentation).

To deploy using your internet connection, you must make sure that the device is connected to the same network as your PC. I had a few issues when trying to deploy at the hotel, and I suspect that there was a firewall or something which was preventing me from successfully deploying. I am still investigating this. At home it seems to work fine.

Make sure the device is awake…

This may sound silly but it happened to me a few times… I remove the device, place it on the table upside down (which seems the safest way), and then after I make changes to the code I hit Ctrl-F5… and it fails. What happens is that after a few minutes, the device goes to sleep. Even if you wake it up quickly after you start deploying, it doesn’t seem to work (I think there is a delay for the device to reconnect to the network). So really, make sure that it is up and running before you hit Ctrl-F5!

FAT vs NTFS

I seem to have issues deploying to the device when the source code is on a FAT filesystem (SD card). If I copy the same source code to my main SDD (which is NTFS), then it works. I will investigate more. It’s not completely surprising because I had the same issue before with Windows 10 UWP applications, and the HoloLens is a Windows 10 device after all. It would be nice to be able to do this from an SD card though, because a typical HoloLens project is pretty large. I’ll update this if I find a solution.

Deploying over USB

Here too I am having issues when I am trying to deploy over USB. I am using the original HoloLens USB cable, and I am getting a cryptic error (Unexpected Error: -2145648626). Here too I will try to find more information and update this. So far I didn’t manage to solve it, but here are a few indications from other people in the community that might help you:

  • If your HoloLens device goes to sleep, and then you wake it up and try to deploy to it, it might fail. In this case, restart Visual Studio and try again.
  • Sometimes the USB connection keeps connecting and disconnecting continuously. In this case, shut the device off, then plug the USB cable in. This will wake the device up. Then you can try deploying again.

Overall, it seems that deploying over Wifi is more reliable than over USB. That’s a shame because USB works faster. My guess is that it’s a firmware issue and I hope it will get fixed some time in the future.

Conclusion

Well that’s a pretty long post… so I will stop here and publish. I am not stopping my investigations and learnings however, so you can probably expect more such notes in the future. I hope you enjoyed reading and that it is useful!

Happy holocoding!
Laurent

Previous entry | Next blog entry

Responses to “My #HoloLens 101 notes”

  1. serge Says:

    Do you really see hollolens hitting the business introduction instead of gaming or university research or will it be an other failure like the Kinect was outside game industry ?

    When you see how long it takes and strong knowledge you need to build an professionnal app, I have the feeling that comapnies willing to integrate it in their own business will be refrain by the app development cost and price of the device itself, which will be big, unless MS is targeting only big company which have money..

    what is your feeling on this ?

  2. lbugnion Says:

    Hi,

    Keeping in mind that I don’t have a crystal ball, what I can say is this: (1) the technology is a lot more compelling than Kinect was for the wider public. As much as I love Kinect, people often didn’t quite get what it did. With HoloLens, we put people in the center of the magic, and they see it happen with their own eyes. That’s a pretty cool feeling. (2) AR/VR is not just a Microsoft initiative. Google is at it, maybe also Apple, Samsung is doing it and Facebook with Oculus. (granted, these are VR and not AR but it’s the same domain). There is a lot more public interest for AR than we ever saw for Kinect. (3) Based on what I saw, even before the devices are available for wide public consumption, we get a lot of requests. That’s also not something we observed with Kinect. We actually have people coming to us asking for AR/VR, while in the past we were the ones proposing Kinect based solutions to clients to achieve what they wanted to do.

    So yeah I am cautiously optimistic. Whatever happens, HoloLens is definitely showing what’s possible and pushing the envelope and that is positive.

Comments for My #HoloLens 101 notes