Table of Contents

Kinect for Windows SDK Programming Guide

Credits

About the Author

Acknowledgement

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Instant Updates on New Packt Books

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Understanding the Kinect Device

Components of Kinect for Windows

Inside the Kinect sensor

The color camera

IR emitter and IR depth sensor

How depth data processing works

Tilt motor

Microphone array

LED

Kinect for Windows versus Kinect for Xbox

Where can you use Kinect

Summary

2. Getting Started

System requirements for the Kinect for Windows SDK

Supported operating systems

System configuration

The Kinect sensor

The Kinect for Windows sensor

The Kinect for Xbox sensor

Development tools and software

Evaluation of the Kinect for Windows SDK

Downloading the SDK and the Developer Toolkit

Installing Kinect for Windows SDK

Installing the Developer Toolkit

Components installed by the SDK and the Developer Toolkit

Kinect management service

Connecting the sensor with the system

Verifying the installed drivers

Not able to view all the device components

Detecting the loaded drivers in Device Manager

Testing your device

Testing Kinect sensors

Testing the Kinect microphone array

Looking inside the Kinect SDK

Features of the Kinect for Windows SDK

Capturing the color image data stream

Processing the depth image data stream

Near Mode

Capturing the infrared stream

Tracking human skeleton and joint movements

Capturing the audio stream

Speech recognition

Human gesture recognition

Tilting the Kinect sensor

Getting data from the accelerometer of the sensor

Controlling the infrared emitter

The Kinect for Windows Developer Toolkit

The Face Tracking SDK

Kinect Studio

Making your development setup ready

The Coding4Fun Kinect Toolkit

Summary

3. Starting to Build Kinect Applications

How applications interact with the Kinect sensor

Understanding the classification of SDK APIs

Kinect Info Box – your first Kinect application

Creating a new Visual Studio project

Adding the Kinect libraries

Getting the Kinect sensor

The Kinect sensor

Defining the Kinect sensor

The collection of sensors

Starting up Kinect

Inside the sensor.Start() method

Enabling the data streams

Identifying the Kinect sensor

Initializing the sensor using device connection ID

Stopping the Kinect sensor

The Stop() method does the clean-up operation

Displaying information in the Kinect Info Box

Designing the Info Box UI

Binding the data

A quick look at INotifyPropertyChanged

Using INotifyPropertyChanged for data binding

Setting the DataContext

Setting up the information

That's all!

Dealing with the Kinect status

Monitoring the change in sensor status

Properties of the StatusChangedEventArgs class

Resuming your application automatically

Building KinectStatusNotifier

Setting up an application

How it works

Using KinectStatusNotifier

Test it out

Summary

4. Getting the Most out of Kinect Camera

Understanding the Kinect image stream

Types of color images

Different ways of retrieving the color stream from Kinect

Event model

Polling model

KinectCam – a Kinect camera application

Setting up the project

Designing the application – XAML and data binding

Capturing color image from the Kinect camera

Enabling the color stream channel

Enabling a channel with the image format

Choosing the image format

Disabling the color stream channel

Attaching the event handler

Processing the incoming image frames

Rendering image frames on the UI

Running the KinectCam

Looking inside color image stream helpers

The ColorImageStream class

The ColorImageFrame class

Capturing frames on demand

Extending the KinectCam

Getting the frame number

Changing image format on the fly

Bind available image formats

Changing the color image format

Calculating frame rate

How to calculate frame rate

Capturing and saving images

Saving images periodically

Trying to save image frames directly

Changing the sensor elevation angle

Maximum and minimum elevation angles

Adjusting the Kinect sensor angle

Playing around the color pixels

Applying RGB effects

Making grayscale effects

Inverting the color

Applying more effects to the camera

Applying the backlight compensation mode

Applying slow motion effects

Kinect Camera Effects – application

Seeing in low light

Making your application perform better

Using the Coding4Fun toolkit

Installing the Coding4Fun Kinect toolkit

Using assembly

Using the NuGet package

Using Coding4Fun Kinect libraries in your application

Summary

5. The Depth Data – Making Things Happen

Understanding the depth data stream

Depth data – behind the scenes

Stereo triangulation

Capturing and processing depth data

Enabling the depth stream channel

Attaching the event handler

Processing the depth frames

Depth data at first look

Looking inside depth image stream helpers

Depth data and distance

How the distance is calculated

Getting the distance from a particular pixel

Accessing the range of distance

Colorize depth data processing

Working with depth range

Special depth range values

Depth data distribution

Player index with depth data

How player index works

Identifying players

Getting the depth and player index automatically

A 3D view of depth data

The basics of the coordinate system

Basic elements of 3D graphics

Setting up the project

Give it a 3D effect

Creating the ViewPort

Using the camera

Controlling the camera position

Creating the 3D Model

Building the mesh object

Setting up the initial data points

Getting the depth data from Kinect

Have a look at 3D depth

Summary

6. Human Skeleton Tracking

How skeleton tracking works

Steps to remember

Skeleton tracking with the Kinect SDK

Start tracking skeleton joints

Tracking the right hand

Setting up the project

Creating a joint placeholder

Get Kinect running and instantiate skeleton tracking

Enabling and disabling the skeleton stream

Processing the skeleton frames

Mapping the skeleton joints with UI elements

Running the application

Adding more fun

Flow – capturing skeleton data

An intrusion detector camera application

Adding night vision

Looking inside skeleton stream helpers

The skeleton frame

The skeleton stream

Skeleton-tracking mode

Default skeleton tracking

Seated skeleton tracking

Using seated-skeleton tracking

Points to be considered with seated-skeleton tracking

Skeleton tracking in near mode

The Skeleton

Skeleton-tracking state

Counting the number of tracked skeletons

Choosing which skeleton to track

Skeleton-tracking ID

Monitoring changes in the skeleton

Limiting tracking for the intrusion-detector camera

The building blocks – Joints and JointCollection

Joint-tracking state

Steps to be followed for joint tracking

Create your own joints data point

Bones – connecting joints

Bone sequence

Bone sequence for a default skeleton

Bone sequence for a seated skeleton

Drawing bones between joints

Adjusting the Kinect sensor automatically and giving live feedback to users

Skeleton smoothing – soften the skeleton's movement

What causes skeleton jitters

Making skeleton movement softer

Smoothing parameters

How to check if skeleton smoothing is enabled

Exponential smoothing

Skeleton space transformation

The Advanced Skeleton Viewer application

Debugging the applications

Using conditional breakpoints

Using Kinect Studio

Getting data frames together

Summary

7. Using Kinect's Microphone Array

Verifying the Kinect audio configuration

Troubleshooting: Kinect USB Audio not recognizing

Using the Kinect microphone array with your computer

The Kinect SDK architecture for Audio

Kinect microphone array

The major focus area of Kinect audio

Why microphone array

Audio signal processing in Kinect

Taking control over the microphone array

Kinect audio stream

Starting and stopping the Kinect audio stream

Starting audio streaming after a time interval

Kinect sound recorder – capturing Kinect audio data

Setting up the project

Designing the application – XAML and data binding

Recording the Kinect audio

Starting the recording

Playing the recorded audio

Running the Kinect Sound Recorder

Processing the audio data

Echo cancellation

Noise suppression

Automatic gain control

Audio data processing with the Kinect sound recorder

Sound source localization

Sound source angle

Confidence level

Beamforming

Beam angle mode

Extending the Kinect Sound Recorder with sound source localization

Summary

8. Speech Recognition

How speech recognition works

Using Kinect with your Windows PC speech recognition

Beginning with Microsoft Speech API (SAPI)

Steps for building speech-enabled applications

Basic speech-recognition approach

Building grammar

Using Choice and GrammarBuilder

Appending new grammars

Building grammar using XML

Creating grammar from GrammarBuilder

Loading grammar into a recognizer

Unloading grammars

Draw What I Want – a speech-enabled application

Setting up the project

Designing the application – XAML and data binding

Data binding

Instantiating speech recognizer

Working with the speech recognition engine

Configuring Kinect audio

Creating grammar

Start the speech recognizer

Drawing an object when speech is recognized

Testing your application

Summary

9. Building Gesture-controlled Applications

What is a gesture

Approaches for gesture recognition

Basic gesture recognition

Gesture-detection technique

Representing skeleton joints

Calculating the distance between two joints

Building a clapping-hands application

Setting up the project

Implementing the gesture recognizer

Defining the types of gestures

Defining the types of recognition results

Creating the event argument for the gesture

Wrapping up everything with the gesture recognition engine

Plugging gestures into the application

Testing your application

A virtual rope workout application

Hands-raised-above-head gesture recognition

Steps to recognize basic gestures

Algorithmic gesture recognition

Which gestures can be considered as algorithmic

Understanding the algorithmic gesture detection approach

Implementing an algorithmic gesture

Adding gesture types

Extending the Event argument

Adding a GestureHelper class

Defining the GestureBase class

Implementing the SwipeToLeftGesture class

Adding the ZoomIn, ZoomOut, and SwipeToRight gesture classes

Implementing the GestureRecognitionEngine class

Using the GestureRecognitionEngine class

A demo application

Making it more flexible

Weighted network gesture recognition

What is a neural network

Gesture recognition with neural networks

Jump tracking with a neural network – an example

Template-based gesture recognition

Building gesture-enabled controls

Making a hand cursor

Getting the hand-cursor point

Identifying the objects

Enabling action for the objects

The Basic Interaction – a WPF application

Key things to remember

Summary

10. Developing Applications Using Multiple Kinects

Setting up the environment for multiple Kinects

Plugging the first Kinect sensor

Plugging the second Kinect sensor

Kinect sensors require an individual USB Controller

Multiple Kinects – how to reduce interference

Detecting multiple Kinects

Getting access to the individual sensor

Different ways to get a Kinect sensor's reference

Developing an application with multiple Kinects

Setting up the project

Designing the UI

Creating the KinectInfoCollection

Getting information from Kinects

Running the application

Controlling multiple sensor status changes

Extending Multiple Kinect Viewer with status change

Registering and handling the status change

Running the application

Identifying the devices automatically

Integrating with KinectStatusNotifier

Capturing data using multiple Kinects

Handling a failover scenario using Kinects

Challenges faced in developing applications using multiple Kinects

Applications where multiple Kinects can be used

Summary

11. Putting Things Together

Taking Kinect to the Cloud

Required components

Windows Azure

The Windows Azure SDK

The Kinect for Windows SDK

Designing the solution

Real-time implementations

Remotely using the Kinect with Windows Phone

Required components

The Windows Azure Service Bus

The Windows Phone SDK

Designing the solution

Real-time implementations

Using Kinect with a Netduino microcontroller

Required components

Microsoft .Net Micro Framework

Netduino

The Netduino SDK

Blinking of the on-board LED

Changing the Deployment Transport

Running the application

Connecting Kinect to a Netduino

Using an Internet connection

Listening to the request

Sending a request from a Kinect application

Taking it further

Augmented reality applications

Working with face tracking

Working with XNA and a 3D avatar

Summary

Index

Kinect for Windows SDK Programming Guide

Kinect for Windows SDK Programming Guide

Copyright © 2012 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2012

Production Reference: 1191212

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-84969-238-0

www.packtpub.com

Cover Image by Sandeep Babu (<sandyjb@gmail.com>)

Credits

Author

Abhijit Jana

Reviewers

Atul Gupta

Anoop Madhusudhanan

Atul Verma

Acquisition Editor

James Keane

Lead Technical Editor

Susmita Panda

Technical Editors

Prasanna Joglekar

Dipesh Panchal

Farhaan Shaikh

Nitee Shetty

Copy Editors

Brandt D'Mello

Insiya Morbiwala

Aditya Nair

Alfida Paiva

Project Coordinator

Yashodhan Dere

Proofreaders

Ting Baker

Matthew Humphries

Indexer

Rekha Nair

Graphics

Valentina D'silva

Aditi Gajjar

Production Coordinator

Nitesh Thakur

Cover Work

Nitesh Thakur

About the Author

Abhijit Jana works at Microsoft as a .NET Consultant, as part of Microsoft Services Global Delivery, India. As a consultant, his job is to help customers design, develop, and deploy enterprise-level secure solutions using Microsoft Technology. Apart from being a former Microsoft MVP, he is a speaker and author as well as an avid technology evangelist. He has delivered sessions at prestigious Microsoft events such as TechEd, Web Camps, Azure Camps, Community Tech Days, Virtual Tech Days, DevDays, and developer conferences. He loves to work with different .NET communities and help them with different opportunities. He is a well-known author and has published many articles on different .NET community sites.

He shares his thoughts on his personal blog at http://abhijitjana.net. You can follow him on Twitter at @abhijitjana. Abhijit lives in Hyderabad, India, with his wife, Ananya and a beautiful little angel Nilova.

Note

Disclaimer

The opinions in this book are purely my personal opinions and do not reflect in any way the opinions of my employers.

Acknowledgement

Writing this book would not have been possible without the help of many people. I had a wonderful time while writing, which was mainly due to the skills, support, dedication, and motivation of the people around me.

First of all I am extremely thankful to Sachin Joshi, Pinal Dave, and Prasant Kraleti for the continuous support and motivation they gave me from the time I started writing this book. They have been awesome with their support at every stage of writing.

I am deeply thankful to the entire team at Packt Publishing, including Prasad, Susmita, Mayur, Prasanna, Dipesh, Farhaan, and Nitee. I would like to extend my thanks to the Project Coordinator, Yashodhan, for his support from the beginning. Thank you all for your effort and dedication.

A sincere thanks to Atul Gupta and Anoop Madhusudanan for their insightful and excellent technical review. They helped me to identify and fill the gaps and improve the overall quality of this book.

I would like to acknowledge the efforts of Atul Verma for his extended support for in-depth technical review, and also for his time in discussing, peer coding, and providing feedback on many topics.

I would like to thank Jebarson Jebamony for his excellent peer review for this book, and also for spending his time and effort in sharing his thoughts and feedback for improving the content. He also helped me to organize content and design many demo applications.

I would like to thank Arka Bhattacharya and Atul Sharma for their offline review of the book and for sharing their feedback. A big thank you to Rajesh R. Nair for helping me on designing sketches and icons, and also Rishabh Verma for capturing and sharing the dismantled sensor images with me.

My sincere thanks to Jag Dua and Sanjoyan Mitra, two true leaders I have worked with. I would like to extend my thanks to Jag for giving me his Kinect sensor when I was overseas and was urgently looking for a Kinect sensor for some experimentation.

I was fortunate enough to be present at many seminars and conferences over the past year, on Kinect. This helped me to interact with many developers and students who are really passionate about programming with Kinect. Thanks to each one of them for spending their time with me and discussing about their problems and questions.

A big thanks to the Kinect for Windows team, the Kinect for Windows Community, and my Community friends, and MSPs who helped me in writing this book. I would like to thank my friends Kunal Chowdhury, Abhishek Sur, Dhananjay Kumar, Suresh Bemagani, Sheo Narayan, and Sharavan Kasagoni for their continuous support and help while writing this book. I am also thankful to the bloggers on the various Kinect topics, and also the researchers who have been working and experimenting day in and day out with Kinect. On many occasions I have been reading their posts and referring to them.

I spent time in writing when I should have been sleeping, spending time with family, or playing with my newborn child. I'd never have been able to write this book without the support of my wife, Ananya. I cannot even express her love and support while I was writing this book. Thank you Ananya.

Being a Community lover and an active blogger, I have been writing blogs for the last couple of years; but this is the first time I am putting something in the form of a book. The credit goes to each one of you who has been connected with me and have been my blog reader and supporter.

I would really appreciate it if you would contact me at <abhijitjana@outlook.com> for any kind of clarification.

About the Reviewers

Atul Gupta is currently a Principal Technology Architect at Infosys' Microsoft Technology Center, Infosys Labs. With more than 16 years of experience working on Microsoft technologies, his expertise spans User Interface technologies, and he currently works on touch and gestural interfaces with technologies such as Windows 8 and Kinect. He has prior experience on Windows Presentation Foundation (WPF), Silverlight, Windows 7, Deepzoom, Pivot, PixelSense, and Windows Phone 7.

He has co-authored the book ASP.NET 4 Social Networking, Packt Publishing. Earlier in his career, he has also worked on technologies such as COM, DCOM, C, VC++, ADO.NET, ASP.NET, AJAX, and ASP.NET MVC. He is a regular reviewer for Packt Publishing and has reviewed books on topics such as Silverlight and Generics.

He has authored papers for industry publications and websites, some of which are available on Infosys' Technology Showcase (http://www.infosys.com/microsoft/resource-center/pages/technology-showcase.aspx). Along with his colleagues from Infosys, he is also an active blogger (http://www.infosysblogs.com/microsoft). Being actively involved in professional Microsoft online communities and developer forums, he has received Microsoft's Most Valuable Professional award for multiple years in a row.

Anoop Madhusudanan has been a Microsoft MVP in C# for the last 3 years and has more than 10 years of experience with Microsoft technologies. Presently, he is working as a Solution Architect with the Cloud & Mobile Center of Excellence, Marlabs Inc. He works across multiple Microsoft technologies and platforms including Windows 8, ASP.NET, Windows Azure, and so on, across domains including education, healthcare, and telecom.

He blogs at http://amazedsaint.com and is the developer of various open source frameworks such as BrainNet Neural Network Library, ElasticObject, SilverDraw, MetaCoder, and so on. He is also an active contributor to CodeProject. His Twitter handle is @amazedsaint.

Atul Verma is a Technical Consultant at Microsoft Services Global Delivery and is a graduate from NIT, Hamirpur. He has been developing enterprise-level secure and scalable solutions using agile software methodologies for the past seven years. His technical expertise includes WPF, ASP.NET, WCF, SharePoint, Dynamics CRM, and Kinect for Windows. Apart from this, he also contributes to technical communities, technical seminars, open source projects, and blogs. He is currently studying the essence of Indian culture and loves to spend quality time with his family. You can follow him on Twitter at @verma_atul.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Instant Updates on New Packt Books

Get notified! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page.

I dedicate this book to my parents, my lovely wife Ananya and my little angel Nilova.

Preface

Ever since its inception, Kinect has brought about a revolution in the field of NUI and hands-free gaming. There is no wonder that Kinect went on to shatter all records and become the fastest selling electronic device on earth. Although touted as a controller for Xbox console, Kinect applicability is beyond gaming domain and you can think of building applications for diverse domains such as health care, robotics, imaging, education, security, and so on. Thus we have the Kinect for Windows sensor, that enables applications to interacts with users via gestures and voice, and this opens up avenues that developers couldn't even have imagined before.

This book is mainly focussed on the Kinect for Windows SDK with which you can build applications that can leverage the power of the Kinect sensor. This book doesn't require any prior knowledge about the platform from the reader and its strength is the simplicity in which the concepts have been presented using code snippets, a step-by-step process, and detailed descriptions. This book covers:

A practical step-by-step tutorial to make learning easy for a beginner

A detailed discussion of all the APIs involved and the explanations of their usage in detail

Procedures for developing motion-sensing applications and also methods used to enable speech recognition

What this book covers

Chapter 1, Understanding the Kinect Device, introduces Kinect as a hardware device. You will get an insight into the different components that make up Kinect and the technology behind this device, which makes the components work together. This chapter will also give an overview of the difference between Kinect for Xbox and Kinect for Windows sensor. You will also become familiar with different possibilities of domain specific applications that can be developed using the Kinect sensor.

Chapter 2, Getting Started, introduces the Kinect for Windows SDK, its features, and how to start working with the Kinect sensor. In this chapter, you will get to know about the requirements for preparing your development environment. This will also walk you through a step-by-step guide for downloading and installing the SDK. You will delve into the installed components to verify that everything is set up properly. This chapter will also provide you with a quick lap around the different features of the Kinect for Windows SDK as well as introduce the Kinect for Windows Developer Toolkit.

Chapter 3, Starting to Build Kinect Applications, explains the step-by-step process of building your first Kinect-based application. You will understand how applications interact with the Kinect sensor using the SDK libraries. This chapter will give you an in-depth guide on how to start building Kinect applications using the Kinect for Windows SDK and Visual Studio. You will also learn how to deal with applications when there is any change in the device status.

Chapter 4, Getting the Most Out of Kinect Camera, covers the in-depth discussion of the Kinect color camera and how to use it. In this chapter, you will learn about the different types of image streams and different approaches to retrieve them from the Kinect sensor. You will get an understanding of Color camera stream pipeline and its events. You will also explore the different features of the Kinect for Windows SDK that control the color camera and process the color data. This chapter will give you an understanding of processing color images and applying different effects to the captured images and how to save the image frames. You will also learn how you can use the Kinect camera to capture images in low light.

Chapter 5, The Depth Data – Making Things Happen, explores the fundamentals of the Kinect depth sensors and how they produce depth information. This chapter describes how to work with object distances and player indices from the captured depth data. You will also learn about the capturing of data using the near mode and also get a quick view of generating 3D depth data.

Chapter 6, Human Skeleton Tracking, describes how a Kinect sensor tracks the human skeleton and how you can leverage the features of the Kinect for Windows SDK to play around with tracked skeletons and joints. You will also learn how to change the sensor elevation angle based on the player position. This chapter also explores how skeletons can be tracked in a seated mode. You also learn about details of the skeleton joints and bone hierarchy. The sample application in this chapter will help you to understand the APIs for skeleton tracking in better ways such as using Kinect as an intrusion detector. At the end of this chapter, you will be familiar with a few debugging tips and tricks to boost your development speed.

Chapter 7, Using Kinect's Microphone Array, introduces the microphone array that captures and processes the audio signal. You will learn why Kinect uses an array of microphones rather than a single microphone. In this chapter you will get an insight into the Kinect audio processing pipeline that helps Kinect to capture good-quality audio signals and makes Kinect a highly directional audio device. This chapter provides you with information on how to capture and record audio signals using the Kinect microphone array and process the audio data for better quality. You will also learn about different concepts such as Noise Suppression, Automatic Gain Control, Echo Cancellation, and Beam forming.

Chapter 8, Speech Recognition, introduces the building of speech-enabled applications using Kinect. You will explore how speech recognition works and how Kinect's microphone array helps Kinect to recognize human speech. This chapter also shows how you can use Kinect as the default speech recognition device for your PC. You will also learn about the Microsoft Speech API and how it is integrated with Kinect for Windows SDK, which helps us to build speech-enabled applications.

Chapter 9, Building Gesture-controlled Applications, describes how to build applications that can be controlled by human gestures. You will learn different approaches for recognizing gestures and how to apply these approaches in the form of programs to build motion-sensing applications using the Kinect sensor. This chapter will also help you understand how to build some gestured-enabled controls.

Chapter 10, Developing Applications Using Multiple Kinects, explains how multiple