Its been a while since I wrote about Kinect, I have to say I really miss it because Kinect is really cool device and there is a lot can be done with it.

Because I’ve talked a lot on the Kinect Nui its time to talk about Kinect Speech Recognition

Kinect .NET SDK–Getting Started , Kinect – Getting Started – Control Camera Angle , Kinect–How to Apply Smooths Frame Display Using TransformSmoothParameters , Kinect – Create Buttons ,  Kinect – Calculator – Adjust Skeleton Movements To Mouse , Kinect Reception–Introduction , Kinect Reception – Tech Stuff

Download Demo Project

Prerequisites

Before you can start with Kinect Speech Recognition you need to install the following:
* Even if you’re working with 64bit OS you need to download the 86x versions because Kinect Runtime works only with 86x. *

• Microsoft Speech Platform - Software Development Kit (SDK) (Version 10.2)
• Microsoft Speech Platform - Server Runtime (Version 10.2)
• Kinect for Windows Runtime Language Pack

Step 1: Add References and Using

Add the following references:

1. Microsoft.Speech
   (C:\Program Files (x86)\Microsoft Speech Platform SDK\Assembly\Microsoft.Speech.dll)
2. Microsoft.Research.Kinect
   (C:\Program Files (x86)\Microsoft Research KinectSDK\Microsoft.Research.Kinect.dll)

Using:

using Microsoft.Research.Kinect.Audio;
using Microsoft.Speech.AudioFormat;
using Microsoft.Speech.Recognition;

And I’ve created 3 global variables:

KinectAudioSource _kinectSource;
SpeechRecognitionEngine _speechEngine;
Stream _stream;

Step 2: Get Installed Recognizers

Kinect Speech Recognition works with Microsoft Speech SDK and you need to assign a Recognizer to the Kinect before you can start.

SpeechRecognitionEngine.InstalledRecognizers();

The above method will bring a list of all installed recognizers on your machine.

Step 3: Register SpeechRecognitionEngine & Build Words Grammar

Once you choose the Speech Recognizer and click the start button, I’m talking the RecognizerInfo value from the selected item and pass it over to the BuildSpeechEngine method.

private void BtnStartClick(object sender, RoutedEventArgs e)
{
    if (listInstalledRecognizers.SelectedItem == null) return;
    var rec = (RecognizerInfo)listInstalledRecognizers.SelectedItem;
 
    DisableUI();
 
    BuildSpeechEngine(rec);
}

Now, using the RecognizerInfo and create a new instance of SpeechRecognitionEngine.

And start building your Word Grammar using Choices, the word grammar is very important for the speech recognition, without those words he can’t recognize any word you say.

After you define the words you need to add them inside a GrammarBuilder Initialized with the RecognizerInfo Culture. The GrammarBuilder is a simple way to build the grammar before adding it to the Grammar object itself.

Now load the grammar to the SpeechRecognitionEngine object and register to those 3 events:

• SpeechHypothesized
• SpeechRecognized
• SpeechRecognitionRejected

void BuildSpeechEngine(RecognizerInfo rec)
{
    _speechEngine = new SpeechRecognitionEngine(rec.Id);
 
    var choices = new Choices();
    choices.Add("word");
    choices.Add("calculator");
    choices.Add("my computer");
    choices.Add("mspaint");
    choices.Add("close");
 
    var gb = new GrammarBuilder { Culture = rec.Culture };
    gb.Append(choices);
 
    var g = new Grammar(gb);
 
    _speechEngine.LoadGrammar(g);
    //recognized a word or words that may be a component of multiple 
    //complete phrases in a grammar.
    _speechEngine.SpeechHypothesized += new EventHandler
     <SpeechHypothesizedEventArgs>(SpeechEngineSpeechHypothesized);
    //receives input that matches any of its loaded and enabled Grammar 
    //objects.
    _speechEngine.SpeechRecognized += new EventHandler
     <SpeechRecognizedEventArgs>(_speechEngineSpeechRecognized);
    //receives input that does not match any of its loaded and enabled 
    //Grammar objects.
    _speechEngine.SpeechRecognitionRejected += new EventHandler
                             <SpeechRecognitionRejectedEventArgs>
                             (_speechEngineSpeechRecognitionRejected);
 
 
    //C# threads are MTA by default and calling RecognizeAsync in the same
    //thread will cause an COM exception.
    var t = new Thread(StartAudioStream);
    t.Start();
}

Step 4: Connect Kinect Audio Input To Speech Recognition Engine

After setting SpeechRecognitionEngine with our list of words and events we just need to connect Kinect Audio Source to the SpeechRecognitionEngine.

First you need to create new instance KinectAudioSource and define the audio settings for the Kinect Audio Source.

Now, starting the KinectAudioSource and assign it to our stream (our global variable)

Back to our SpeechRecognitionEngine and call the SetInputToAudioStream method with the stream object from KinectAudioSource.

Now the Kinect Audio is connected to SpeechRecognitionEngine and you can start the speech recognition using RecognizeAsync.

void StartAudioStream()
{
    _kinectSource = new KinectAudioSource
    {
        FeatureMode = true,
        AutomaticGainControl = true,
        SystemMode = SystemMode.OptibeamArrayOnly,
        MicArrayMode = MicArrayMode.MicArrayAdaptiveBeam
    };
 
    _stream = _kinectSource.Start();
 
    _speechEngine.SetInputToAudioStream(_stream,
                    new SpeechAudioFormatInfo(
                        EncodingFormat.Pcm, 16000, 16, 1,
                        32000, 2, null));
 
    _speechEngine.RecognizeAsync(RecognizeMode.Multiple);
}

Download Demo Project

Enjoy