[SOLVED] Sending Speech Input to Sofa

[Jeremy]

Hi all,

I would like to improve the ability of the user to move the camera during a simulation, currently they must put a haptic device down in order to move the mouse, which is not realistic. In a real surgery, the surgeon would tell the camera operator what to do, so we downloaded CMUSphinx and have it listening for voice commands from a restricted dictionary.

The issue that we are currently having is figuring out how to send the camera motions to Sofa. We have two ideas, but cannot figure out how to correctly implement either of them:

1) Create a plugin that listens for speech and then calls the event handler. We know how to create a plugin and have it listen for speech, however we need it to be running in a loop so it continuously listens for speech. At the moment our plugin runs only on initPlugin and we do not know how to have it run continuously.

2) Have our initPlugin method spawn the speech to text code as a separate program which then pipes the commands back into Sofa. However, we do not know how to pipe the commands from our speech to text program into Sofa.

Does anyone have any advice for either of these approaches or know where to go from here?

Thank you!
Jeremy

[Guillaume]

Hi Jeremy,

That’s a nice and very interesting project! I think your are in the right direction with (1) but you are missing the component-oriented aspect of SOFA.

Here is my suggestion:
– Create a simple and empty plugin. Refer to plugins/Geomagic or plugins/LeapMotion (these are driver plugins that link to external libraries).
– In your plugin, create a SOFA Component. It could inherit from sofa::component::controller::Controller like GeomagicDriver or LeapMotionDriver. This component will create (and discuss with) a thread dedicated to speech listening and will use the event handler of SOFA.
– In your scene, add your Component to activate your awesome feature.
– In runSofa, load your plugin and run your scene 🙂

Some topics about events in SOFA:
https://www.sofa-framework.org/community/forum/topic/keyboard-events-in-sofa/
https://www.sofa-framework.org/community/forum/topic/how-to-send-data-to-sofa-through-socket/
https://www.sofa-framework.org/community/forum/topic/how-to-use-external-data-in-sofa/

Hope this helps,
Guillaume

[Jeremy]

Thank you! We created a component and were able to spawn our own thread in the bwdInit() method following the example of SensableEmulation/OmniDriverEmu.cpp. We haven’t implemented the speechEvent yet, but this is a big step towards that.

Jeremy

[Hugo]

Thank you for your feedback Jeremie.
Keep us updated about this nice work! At my best knowledge, SOFA has never been used with speech control. Such a module would be extremely interesting!

[jjcasmar]

Looks really nice. Would it be possible to create a QThread (for example) during the component init method and keep that thread listening and sending signals to a slot in the component which handle the action? This way you shouldn’t have to create a dedicated library which links to sofa. Just an idea, I don’t know if it’s possible

[Jeremy]

That’s actually similar to what we did, except we used Boost threads instead of QThreads. Basically in bwdinit() we spawn a boost thread which runs an infinite loop which listens for speech input. After it hears an accepted command, it sets a flag in our component telling it what it heard. Then in our handleEvent() method, we handle animateBeginEvents by checking the flags to see if it heard anything and, if so, we move the camera.

We thought about creating our own speechEvent class to handle events, but that turned out to be more difficult than moving the camera on animateBeginEvents, although it would have been much more elegant, IMO. We also had difficulty sending events from the separate thread, but we could modify data elements without any issues, which is one of the reasons why we chose to do it this way.

Jeremy

[Author]

Posts