I'm interested in performing a thorough evaluation of various speech recognition engines, however I currently don't have the means to do so.

Could the VoxForge corpus be used to evaluate various systems, such as the Google Chrome Voice API, Bing, Nuance NDEV, etc.?

I understand that the open source toolkits have evaluation functions, that are used for evaluating the acoustic & language models of their own systems.

Could HTK/Julian/Sphynx etc. be used to evaluate another system?

This would require some code to compare the transcriptions in the VoxForge corpus to the output transcriptions of the various ASR systems.

Any advice or assistance that you could offer would be greatly appreciated.

--- (Edited on 10/22/2013 7:59 pm [GMT-0500] by ) ---

> I'm interested in performing a thorough evaluation of various speech recognition engines, however I currently don't have the means to do so.

--- (Edited on 10/23/2013 21:30 [GMT+0400] by nsh) ---

Thank you for your recommendation of the NIST toolsets. (previously this website was offline.)

SCTK & SCLITE seem best suited to perform the evaluation:
http://www.nist.gov/itl/iad/mig/tools.cfm

What format are the VOXFORGE transcription files in?

SCLite accepts 'trn', 'txt','stm' & 'ctm'.
http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/infmts.htm

The following example output, doesn't appear to be in either 'trn', 'stm' or 'ctm':

jaiger-12032006-6/mfc/vf6-01 HE CRIED AND SWUNG THE CLUB WILDLY

What tasks am I interested in evaluating?

Real-time continuous speech, in telephone calls, with channels segmented between speakers (no diarization required.)

This will be spontaneous speech (not read,) with a large vocabulary.

--- (Edited on 10/23/2013 7:43 pm [GMT-0500] by ) ---

>  The following example output, doesn't appear to be in either 'trn', 'stm' or 'ctm':

Yes, it has to be converted. It can be done with a simple script.

> Real-time continuous speech, in telephone calls, with channels segmented between speakers (no diarization required.) This will be spontaneous speech (not read,) with a large vocabulary.

So you can automatically sort out Google/Bing which are unable to recognize narrowband telephone calls and Nuance default engine too. You can compare only specialized Nuance engine and CMUSphinx, both should be equal in accuracy except the price/license. That's it, I saved you a month to run a comparison.

--- (Edited on 10/25/2013 01:37 [GMT+0400] by nsh) ---

> Yes, it has to be converted. It can be done with a simple script.

I've been searching for a script, however I haven't been able to locate anything on SPHINX / FESTIVAL / etc.

> So you can automatically sort out Google/Bing which are unable to recognize narrowband telephone calls and Nuance default engine too. You can compare only specialized Nuance engine and CMUSphinx, both should be equal in accuracy except the price/license. That's it, I saved you a month to run a comparison.

I appreciate your expert advice, it is helping me head into the right direction!

I was interested in training Bing/Nuance with narrowband telephone calls (rather than the wideband microphone recordings.)

I understand that the LM (and pronunciation dictionary) will need to be adjusted to fit my domain. Is the MITLM toolkit best suited to interpolate the LMs?

--- (Edited on 10/27/2013 7:38 pm [GMT-0500] by julian) ---

> I've been searching for a script, however I haven't been able to locate anything on SPHINX / FESTIVAL / etc.

--- (Edited on 10/29/2013 01:09 [GMT+0300] by nsh) ---