So if the user just sits there and says nothing, the recognizer will keep putting out results as if it does hear something? Sounds like your system is set to be too sensitive.

Generally there will be a setting that you can vary to tell the recognizer what "silence" really is, which might not be completely silent at all. You don't say which recognizer you are using, but with Julius for example you could play with the -lv (level threshold). Raising this level would tell Julius to stop listening even if there is noise still happening.

The problem of course is that out in the wild your application will hear a wide range of different noise levels which vary during a long session. You might want to build in a step in the setup that listens for the level of silence and then takes this into account for the real detection.

Alternatively there has been some work done on the convolution of existing training samples with various impulse responses. See http://voxforge.org/home/forums/message-boards/audio-discussions/-what-are-best-practices-for-collecting-speech-for-a-free-gpl-speech-corpus/3