HoloLens Blueprints

Experience the virtual and real worlds coming together with HoloLens

Abhijit Jana

Manish Sharma

Mallikarjuna Rao

BIRMINGHAM - MUMBAI

HoloLens Blueprints

Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: June 2017

Production reference: 1160617

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78728-194-3

www.packtpub.com

Credits

About the Authors

Abhijit Jana works with Microsoft as a development consultant as part of Microsoft Services. As a consultant, his job is to help customers design, develop, and deploy enterprise-level secure solutions using Microsoft technologies. Apart from being a former Microsoft MVP (Most Valuable Professional), he is a speaker and author as well as an avid technology evangelist. He has delivered sessions at prestigious Microsoft events, such as TechED, Web Camps, Azure Camps, Community TechDays, Virtual TechDays, DevDays, and developer conferences. He loves to work with different .NET communities and help them with different opportunities.

He is a well-known author and has published many articles on various .NET community sites. You can follow him on Twitter at @abhijitjana. He has authored the book Kinect for Windows SDK Programming Guide (ISBN: 1849692386 ISBN 13: 9781849692380).

Abhijit lives in Hyderabad, India, with his wife, Ananya, and a beautiful little angel, Nilova.

Manish Sharma works with Microsoft as a solution architect as part of Microsoft Services. As, a solution architect, he is responsible for leading large enterprise-transformational engagements defining technology/solution roadmaps, conducting architecture and technical evaluations of enterprise engagements, and architecting multimillion-dollar solution developments, and maintenance of outsourcing management. He is also a technology evangelist and speaker at prestigious events, such as Microsoft TechEd on the latest and cutting edge technologies, such as HoloLens, Internet of Things, connected cars, and cloud technologies such as Azure.

Mallikarjuna Rao works with Microsoft as UX Designer/3D mixed reality artist as part of Microsoft Services. He has 15 years of experience in the animation industry, with around 12 years' international exposure.

He has worked as a senior 3D asset supervisor for feature films, television series, commercials, and game projects. Mallikarjuna has worked on pivotal projects for big names in the industry, such as Walt Disney Pictures, Disney Junior, Warner Bros Pictures, Yash Raj Films, and UTV Motion Pictures, which put him under the limelight at the international level. He has an in-depth knowledge of the 3D Asset production pipeline and UX design principles.

Acknowledgments

Writing this book would not have been possible without help of many people. We had a wonderful time while writing, which was mainly due to the skills support, dedication and motivation of people around us.

First of all, we are extremely thankful to Kundan Prakash, Ankur Agarwal and Shanmugadas Chembakassery Sivadasan, Suresh Viswanath for their continuous support and motivation from the time we started writing this book.

We are deeply thankful to the entire team at Packt Publishing, including Tushar, Parul, Narendra, Ritika, Smeet, Denim, Safis, Dhanya, Tejal, Jason and Arvind. We would like to extend our thanks to Content Development Editor, Yusuf and our Technical Editor Akansha. They have been very supportive and helpful from the beginning. Thank you all for your effort and dedication.

A sincere thanks to Michael Washington for their insightful and deep technical review. He helped us to identify and fill the gaps and improve the quality of the book.

We would like to thank Charul Pant and Akshay Dixit for their offline review of the book and sharing their feedback. It was extremely helpful to get some feedback from avid reader and technology geek like you people. Thank you very much for your time and help us improving the book over the period.

We spent time in writing when we should have sleeping, spending time with family or playing with our kids. We wouldn't have never been able to write this book without support from our family. Be it late night working or meeting over weekends, they supported us every bit and pieces.

About the Reviewer

Michael Washington is an ASP.NET, C#, and Visual Basic programmer. He has extensive knowledge of process improvement, billing systems, and student information systems. He is a Microsoft MVP. He has a son, Zachary, and resides in Los Angeles with his wife, Valerie.

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Disclaimer

The opinions in this book are purely our personal opinions and do not reflect in any way the opinion of our employers.

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at http://www.amazon.in/dp/1787281949.

If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

I dedicate this book to my parents, my lovely wife Ananya and my little angle Nilova - Abhijit Jana

I dedicated this book to my parents, my wife Joohi and lovely son Atharv - Manish Sharma

I would like to dedicate this book to my parents, my wife Sunitha and kids Sai Thanish, Rudhvi sriman -Mallikarjuna Rao

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Digital Reality - Under the Hood

Virtual Reality - what it is?

Realizing Virtual Reality

Virtual Reality in the field

Augmented Reality - what it is?

Visualization techniques for Augmented Reality

Screen-based

Augmented Reality head-mounted globally displays

Projection-based

Augmented Reality in the field

Mixed Reality - what it is?

Mixed Reality in the field

Digital Reality and the current market place

Summary

HoloLens – The Most Natural Way to Interact

A closer look at HoloLens hardware

Inside the HoloLens device

Cameras and sensors

The Lenses

Speakers

Microphones

The processing units

Inertial Measurement Unit

Vent

Connectivity

Control buttons

LED indicator

Putting things together

Turning your device on/off

Cortana on HoloLens

The Mixed Reality world - HoloLens shell

Holograms in reality

HoloLens, holograms and your real world

Hologram and the Windows Holographic platform

Recording holograms - Mixed Reality capture

HoloLens interaction and application model

Interacting with holograms

Gaze

Gesture

Voice

Spatial Audio

Spatial Mapping

HoloLens app life cycles

HoloLens Clicker

Controlling your HoloLens remotely

Key differentiators

Immersive with surroundings

Independent standalone device

No screen staring

Mixed not virtual

Broad scope

Exploring the possibilities with HoloLens

Virtual collaboration

Remote expert guidance

People-centric applications

Development process and team building

Building up the team

Designers

3D Artists

Developers

Audio designers

Leads

Development process overview

Requirement Envisioning

Storyboarding

Planning, Designing and Prototyping

Planning and Design

Prototyping

Develop

Deploy and Test

Summary

Explore HoloLens as Hologram - Scenario Identification and Sketching

Envisioning

Training scenario for HoloLens

Features

Collaboration

Scenario prioritization

Scenario elaboration

Primary View

Lenses

Speakers

Cameras

Volume control buttons

Brightness control buttons

Sketching the scenarios

Sketching - interaction with speakers

Assets design and development

3D Modeling

3D Modeling tools

3D Modeling workflow

Build your 3D Model

Model sheet - base reference images

Scene creation

Importing model sheet

Import side image

Import front image

Import top image

3D Model construction with cube

Edge loop

Vertex adjustment

Mirroring

Texturing workflow

UV Unwrapping of model

Create texture in Photoshop

Export as Filmbox (FBX)

Assembling assets in Unity3D

Things to consider while using the 3D Asset Pipeline

Maintaining quads

Checking Surface Normals

Maintaining grouping

Optimizing the scene file

Setting up a development environment

Prerequisites

Installation checklist

Using HoloLens emulator

Summary

Explore HoloLens as Hologram - Developing Application and Deploying on Device

Getting started - creating a new project

The Unity3D basics

The Core interface

Scene View

Game View

Object Hierarchy

Project Assets

Inspector

The Toolbar

The Core concepts

Game Object

Components

Prefabs

Tags

Coordinate system

Setting up a project for holographic application

Setting up the HoloToolkit - Unity

Get The HoloToolkit - Unity

Preparing the package

Import the package

Applying project settings

Behind the scenes

Updating the initial scene

Importing 3D Assets

Exploring the 3D Object

Saving the scene

Build, run, and test the initial setup

Giving life to object - scripting

Highlighting the objects - enabling Gaze

The Gaze manager

The Cursor

Adding the Box Collider

See Gaze in action

Behind the scene

Adding actions to objects - applying gesture

Adding Air Tap on Lenses

Air Tap on Lenses - see it in action

Adding Air Tap on speaker

Air Tap in speaker - see it in action

Behind the scenes

Interaction using Voice Command

Enabling the Microphone capabilities

The Keyword Manager

Defining Keywords

Implementing actions for Keywords

Mapping with Keyword Manager, Keyword and actions

Voice Command - see it in action

Behind the scenes

Interaction using text to speech

Setting up text to speech Manager

Integration of text to speech

Text to speech in application launch

Text to speech in Air Tap

Text to speech - see it in action

Behind the scenes

Deploying your app

Deploy using Visual Studio

Deploy using Build Automation Window

Deploy using Device Portal

Load the package

Start the app

Deploy using Unity remoting

Configure HoloLens device for remoting

Holographic emulation - remoting from Unity

Running the app on a device

Testing your application

Lighting conditions

Space

Angle of views

View distance

Audio or visual hints

Ambient sounds

Summary

Remote Monitoring of Smart Building(s) Using HoloLens - Scenario Identification and Sketching

Envisioning

Solution scope overview

Devices and Sensors

Event Processing

Visualization

Real-time monitoring

Analyse and Act

Scenario prioritization

Scenario elaboration

Internet of Things - Remote Monitoring

Devices

Cloud Gateway (Azure IoT Hub)

Steam Processing (Azure Stream Analytics)

Data Store (Azure Storage)

Machine Learning

Real-time visualization through HoloLens

APIs

Holographic application

Sketching the scenarios

Sketching - Interaction with Rooms and Sensors

3D Model - structure and components

3D Model of building

3D Model of sensors

Summary

Remote Monitoring of Smart Building(s) Using HoloLens - Developing Application and Deploying on Device

Solution development

Logical design diagram

Setting up the backend Infrastructure

Create the Azure Event Hub

Create the Azure Cosmos DB

Create the Stream Analytics Job

Building up the Azure IoT solution

Connecting Device with Azure Event Hub - simulator

Developing the Web API

Setting up a project for a holographic application

Getting started - creating a new project

Import the package

Applying Project Settings

Updating the initial scene

Importing 3D Assets - building model

Exploring the building's 3D Objects

Save the scene

Build, run and test the initial setup

Tracking building floors from 3D Model

Adding Gaze

Adding the cursor

Adding the Box Collider

Add tag to each floor for mapping

Tracking Gaze for each floor

See Gaze in action

Verify the Gazed floor

Connecting a holographic app with Azure

Defining data models

Enabling the InternetClient capabilities

Building Azure Bridge

Getting the data

Attaching the Azure Bridge

Binding data with sensor indicators

See the live data in action

Enabling sensor binding for all floors

Adding actions to floors - applying gesture

Adding Air Tap on floors

Displaying detailed information

Designing the Panel layouts

Grouping UI elements - adding Canvas Group

Binding Data with Panel

Invoking the Panel

Closing the Panel

Air Tap on Lenses - see it in action

Deploying your app onto a device

Extending the current solution

Adding Voice Command

3D Model placement as table top mode

Additional interaction with services

Summary

Build End-to-End Retail Solution - Scenario Identification and Sketching

Envisioning

Scenario elaboration

Catalog and merchandising

Visualization

Order placement

Sketching the scenarios

Sketching - interaction for catalog and items

Sketching - interaction with 3D Model

3D Model

Summary

Build End-to-End Retail Scenario - Developing Application and Deploying on Device

Getting the 3D Assets ready

Unity Project and Asset Bundle setup

Importing 3D retail assets

Assets Bundles generation

Solution development

Logical design diagram

Building the backend Azure Solution

Creating the Azure Blob Storage

Creating the Azure Cosmos DB

Developing the Web API

Setting up the holographic project

Importing the main menu 3D Model

Exploring the main menu

Building , running and testing the initial solution

Making the main menu Intractable

Enabling Gaze with cursor

Attaching the script with menu items

Enabling Air Tap with menu items

API connector - connecting with Web API

Defining the application data model

Enabling the IntenetClient capabilities

API Connector

Getting the data

Attaching the API connector

Getting holograms on the fly

Handling the Air Tap on menu item

Downloading assets and rendering holograms on the fly

Associating toolbars with holograms

The toolbar controls

Instantiating toolbars on the fly

Showing controls based on types

Dynamic holograms in action

Adding action to the toolbar controls

Remove product

Add to Cart

Changing pictures for Wall Paints

Placing the holograms - spatial mapping

Deploying your app on a device

Extending the current solution

Dynamic menu for products

Adding Voice Command

Application and storage access security

Rotating the individual product

Summary

Possibilities

HoloLens and Microsoft Bot Framework

Solution

Client authentication

Start conversation

Send message

Receive message

HoloLens in healthcare

Solution

Search data from HealthVault

HoloLens and live streaming

Solution

Setting up media services for live streaming

Consume live streaming within holographic application

Handling secure keys with Azure Key Vault

Summary

Microsoft HoloLens in Enterprise

Microsoft HoloLens Commercial Suite

HoloLens device management using Microsoft Intune

Manual registration

Automatic registration

Installing applications on HoloLens

Device Portal for application deployment

Business Store for application deployment

Summary

Preface

Digital reality brings immersive experiences, such as transporting you to different world or places, making you interact within those immersive, mixing digital experiences with reality, and ultimately, opening new horizons to make you more productive.

The spectrum and scenarios of digital reality are huge; to