Fish eye cameras and anatomical model in SOFA

[Anonymous]

Hello,

I was wondering it is possible to add fish-eye cameras in SOFA. I am trying to add small fish-eye cameras around a surgical instruments and use the images to create a bird-eye view, however I haven’t find any information about the use of cameras in the simulator.
Moreover, if this is not possible I would like to create my simulation environment in SOFA and export it in Blender. Is there a repository with models of anatomical surgical setup ( instruments and human anatomy)?

Thank you in advance for your support with this matter.

[Bruno Marques]

Hi @Angela_Fifi,

As-is in SOFA, there’s no way of changing the camera’s intrinsic parameters using exisiting components.
There is a plugin, called SofaCV (computer vision) that contains a component called CalibratedCamera, and that can take OpenCV-style camera calibration data in the form of:
– A 3×4 matrix encoding both extrinsic and intrinsic camera parameters
– the K, R, and t parameters as expressed in OpenCV
– A glModelView / glProjection matrix if you’re more comfortable with OpenGL
– A position, up/lookat/ vector, & projection matrix
– …

This plugin is hosted at this address: https://gitlab.inria.fr/mimesis/ComputerVision/SofaCV, and while the plugin is Open source, Access is only given upon request I believe.

Otherwise, If you prefer going for your own implementation, I’d suggest to look into the VisualManager class (sofa::core::visual::VisualManager) That let’s you override the OpenGL camera parameters of your scene through the preDrawScene / postDrawScene methods.

If you want to go for the SofaCV plugin, If you want to go for the SofaCV plugin, I believe you would have to create an account on gitlab.inria.fr, and I should then be able to add you to the project’s members 🙂

[Anonymous]

Hi Bruno,

Thank you for your answer. As for the moment I just need to have a static scene and get the images I may will just create the model in SOFA and use it in another software. But I will consider to use the plugin in the future.

I am actually working in minimally invasive surgery applications, do you know if there are model already available?

Thanks,
Angela

[Hugo]

Hi @faragassorobot-t-u-tokyo-ac-jp

And thank you for your question on the forum!
I am glad to see Bruno’s help clarified your question about the fish-eye effect.

I can answer you about the question concerning the availability of models/anatomy in SOFA. The examples of medical simulations that you can see in SOFA online videos often use patient data. These data (even if anonymized) can not be provided in our open-source repository for legal reasons.

You can however find some geometry- and tool-meshes in SOFA/share/mesh. An example is provided in the simulation examples/Demos/liver.scn

I hope this helps, do not hesitate if you have any further question.
Best wishes,

Hugo

[nhnhan]

Hi @bmarques,
I’m so happy to hear that there is a plugin helping us to be able to add a fish-eye camera to SOFA. I have been doing the exact things described above, therefore, It would be nice if I can play with this plugin. However, apparently, I need to acquire an account to access gitlab.inria.fr from an Inria member. Could you please provide me one? Thank you and looking forward to your response soon.
Nhan

[Bruno Marques]

Hi @nhnhan, this plug-in is now available on GitHub!
The plug-in has been split into 2 repositories also, SofaCV, which contains mainly interface code, and. ImageProcessing, which contains the code you’re interested in, and depends on SofaCV.
You can check out both plugins at the following link:
https://github.com/mimesis-inria/SofaCV
https://github.com/mimesis-inria/ImageProcessing

[nhnhan]

Hi @bmarques,
Thank you very much for sharing your wonderful works. I have been struggling to compile these plugins to SOFA, however, they seem not to be updated for a while. Specifically, they require OpenCV 3.2 for Imageprocessing, and OpenCV 3.0 for Sofacv. I have attempted to install these packages on my computer (ubuntu 20.04 which already provides OpenCV 4.2) but not successful. I just wonder that do you have intent to upgrade these plugins to be more compatible? I personally think these plugins will be very crucial. Thank you very much.
Nhan

[btt]

Dear @bmarques,
I have successfully compiled SofaCV with Sofa, but things seem even rougher with Imageprocessing plugin. When I activated ninja, many fatal errors popped up and it seems this plugin is no longer compatible with the current versions of SOFA (I tried v19, v20.06 and master branch). I just wonder is there any update in the coding of this plugin to make it more compatible with the current version of SOFA? It would be great if you could update it. Anyway, thank you for your wonderful works and I can wait to play with it.
Best regards,
btt

[Hugo]

Hey @nhnhan @newbie

I see you are curious about plugins developed by @bmarques and his team!

I do not know it personnally, but we could try to make them both compatible with OpenCV 3.2 or even OpenCV 4.2, but I am completely ignorant about the inside of these plugins.

@bmarques do you think it would be easily doable? do you have some spare time for us (since you’re off do not hesitate to say no!)

@nhnhan @newbie I could try to help me with some errors you get.

Best wishes,

Hugo

[nhnhan]

Hi @hugo,
Thank you for your effort in connecting with the authors of these plugins. These ones are of my interest and I really want to exploit them. I have faced some errors as shown below:
#error "OpenGL headers have been moved to Sofa.GL. Therefore you will need to link against Sofa.GL if you need OpenGL (PR1649), and include <sofa/gl/DrawToolGL.h> instead of this one."

Then when I tried to include <sofa/gl/DrawToolGL.h> as recommended, another error said “there is no such file”
Hope some one can have solution or any suggestiong will also be appreciated
Nhan

[btt]

Hi @hugo,
It is glad to hear your effort to help us. I do managed to install SofaCV by include the DataFilename.h as you suggested in here. I will try to create a PR regarding this issue later on.
In case of Imageprocessing plugin, you can find the most common types of error messages below:
`/home/btt/Desktop/sofa/src-master/SofaKernel/modules/SofaHelper/compat/sofa/core/visual/DrawToolGL.h:32:2: error: #error “OpenGL headers have been moved to Sofa.GL. Therefore you will need to link against Sofa.GL if you need OpenGL (PR1649), and include <sofa/gl/DrawToolGL.h> instead of this one.”
#error “OpenGL headers have been moved to Sofa.GL. Therefore you will need to link against Sofa.GL if you need OpenGL (PR1649), and include <sofa/gl/DrawToolGL.h> instead of this one.”
/home/btt/Desktop/ExtPlugins/ImageProcessing/src/ImageProcessing/camera/common/CameraSettings.h:14:10: fatal error: sofa/gl/template.h: No such file or directory
#include <sofa/gl/template.h>
/home/btt/Desktop/sofa/src-master/SofaKernel/modules/SofaHelper/compat/sofa/helper/gl/Transformation.h:28:13: warning: “Transformation has been stripped of its OpenGL code, use sofa::gl::TransformationGL if you need its OpenGL implementation.”
SOFA_PRAGMA_WARNING(“Transformation has been stripped of its OpenGL code, use sofa::gl::TransformationGL if you need its OpenGL implementation.”)

These errors occurs to many files along with many warning regarding header. I have tried to link library of Open.GL as recommended in PR#1649 , but it did not work. Hope to see your attempt’s results. Thank you again for your great support
best,

[Hugo]

Hey @newbie

The Open.GL is a CMake package recently created (after v20.12 release). This means the branch of the plugin you are using is compatible with the master branch of SOFA.

Regarding a PR, it would be most welcome and appreciated by the community indeed! Do not hesitate to notify your PR in this post as well.

Best wishes,

Hugo

[Bruno Marques]

Hi @newbie, @Hugo

I’ve updated the codebase to match Ubuntu 20.04’s default version, in SofaCV and ImageProcessing.

I only did the essentials (no extra options enabled such as SofaQtQuickGUI widgets, Python3 bindings, OpenCV_Contrib features, etc…) so that the plugins would compile, but hopefully that should put you on track, if you’re willing to go the extra mile.

https://github.com/mimesis-inria/SofaCV/pull/4
https://github.com/mimesis-inria/ImageProcessing/pull/2

If you have questions, I’ll try to answer the best I can, although my answers might arrive quite late, as I’m not working on this project anymore.

Best of luck!
– Bruno

[Author]

Posts