-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add interactive Qt panel to VR scene #43
Comments
💯 this will be a great feature! |
This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43
To follow up on this, @martinken recently added classes to VTK for adding Qt widget into the VR scene: |
@jcfr Perfect! @mohammadrashid0917 just started working on this. We'll re-adjust the plan to use these classes. Mohammad, please hang on. In the meantime you can look at these classes. |
@jcfr We were lucky this time that minimal/no work was duplicated, but we need to coordinate a bit more closely to make sure we avoid parallel efforts. How could we do that? Where can we find planned VTK virtual reality developments? Do you have a category in the VTK bugtracker, labs page, …? |
The plan described at the top of this issue is outstanding. I should probably have done a better job communicating about it with folks working on OpenVR at Kitware. Moving forward, I think sharing the plan like the one at the top of this issue on VTK discourse would help. Internally, the folks working on OpenVR features know that I am coordinating effort on the Slicer side and I think it worked pretty well as I was able to follow up here. That said, I agreed that the feedback loop could be shorter. May be having a list of contact associated with key VTK area would help ?
Considering that this type of feature are driven by customer needs and often planned and executed in a short time frame, i think it is challenging for us to communicate about this features with a high level of granularity and far ahead. |
To follow up, here is a Slicer PR backporting the specific VTK changes to Slicer/VTK fork. See Slicer/Slicer#1141 I have checked that it compiles on Linux, after someone check that Slicer/VTK compiles on Windows and macOS, it will be integrated. |
These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43
Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43
These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43
Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
This commit is intended to implement features described in KitwareMedical#43 by integrating changes originally associated with the following branches: * dgmato/gui-widgets-interactions - initially developed between 2021-12-15 and 2022-05-23 by @dgmato with contributions from @cpinter. - based on the "gui-widgets-module" branch * dgmato/gui-widgets-module - initially developed between 2018-10-18 and 2021-10-0523 by @cpinter with contributions from @dgmato, @lassoan, @SalehChoueib and @mohammadrashid0917 - with first commit titled "ENH: Initial VR home widget implementation" based of "dgmato/virtual-widget" branch. * dgmato/virtual-widget - initial commit contributed by @SalehChoueib on 2018-10-18 (ENH: Add virtual widget from arbitrary Qt Widget !WIP!) - follow-up development by @cpinter and @mohammadrashid0917 between 2019-05-13 and 2019-06-13 ------------ Commits originally associated with branch "dgmato/gui-widgets-interactions": * ENH: Remove unused UI widget in GUI Widgets module * ENH: Do not load avatar models when setting up VR pointer * ENH: Add new button to set up interaction Set up will include getting the VR controller transform node and applying that transform to the pointer model. * ENH: Add GUIWidgets module to Virtual Reality category for easy access * ENH: Load avatar models and transform pointer by right controller transform * ENH: Reorganize code and remove unused code * COMP: Fix build error due to Slicer API changes * ENH: Add new module target library vtkSlicerMarkupsModuleMRMLDisplayableManager * ENH: Send mouse release event to complete simulated mouse click after press event * ENH: Modify coordinate system for GUI widget Origin is located on the top left corner in Qt widgets. * ENH: Add button "Start interaction" to simulate interaction with VR pointer * ENH: Add module to simulate VR pointer display and movement ------------ Commits originally associated with branch "dgmato/gui-widgets-module": * ENH: Add buttons in GUIWidgets to show VR widgets; Fix include dirs variable in CMake * ENH: Add GUIWidgets module that provides in-VR Qt widget This is still a prototype, a few crucial features are needed: - Handle mouse interactions on the widget: forward events to Qt - Support moving the widget to arbitrary position - Implement outline tube and handle to grab with the controller - Option to keep widget floating in certain position relative to head motion (~HUD) - Implement laser pointer and use the intersection with these widgets as mouse interaction See more history: https://github.com/cpinter/VtkQWidgetTest * ENH: Creating and registering VR segment editor widget Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) * BUG: Fix issues in VR home widget and the data widget * ENH: Creating and registering VR data module widget Creates and registers the VR data module widget (which can be switched to from the VR home widget) * ENH: Convert the interactor style to use the new vtkMRMLViewInteractorStyle Fix build error with Slicer-4.10 * STYLE: Add function documentation * ENH: Initial VR home widget implementation Squashed commits (originally associated with branch "dgmato/virtual-widget") * ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget * ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa * ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. * ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed * ENH: Add virtual widget from arbitrary Qt Widget !WIP! Co-authored-by: Andras Lasso <[email protected]> Co-authored-by: Csaba Pinter <[email protected]> Co-authored-by: David Garcia Mato <[email protected]> Co-authored-by: Mohammad Rashid <[email protected]> Co-authored-by: Sal Choueib <[email protected]>
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Some widgets were missing from the UI Re KitwareMedical#43
Formerly we saw the widget "from behind" when it appeared. Re KitwareMedical#43
A workaround was implemented, a better solution will be needed in the long run. Re KitwareMedical#43
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Some widgets were missing from the UI Re KitwareMedical#43
Formerly we saw the widget "from behind" when it appeared. Re KitwareMedical#43
A workaround was implemented, a better solution will be needed in the long run. Re KitwareMedical#43
At the start of the Pick event the interactor style invokes vtkCommand::PickEvent. With this Python script it is possible to do the clicking with the right trigger. def onClick(caller, event): gw.onStartInteractionButtonClicked() vris = slicer.modules.virtualreality.viewWidget().interactor().GetInteractorStyle() vris.MapInputToAction(vtk.vtkEventDataDevice.RightController, vtk.vtkEventDataDeviceInput.Trigger, vtk.VTKIS_PICK) vris.AddObserver(vtk.vtkCommand.PickEvent, onClick) Re KitwareMedical#43
Without this fix, although the first button press worked on any child widget of the set GUI widget, after this first click, no other button reacted to the interaction. Now all the buttons in the GUI widget work as expected. Re KitwareMedical#43
Using the branch https://github.com/cpinter/SlicerVirtualReality/tree/gui-widget-20230612 the interactions can be set up to work like shown in this video: 2023-06-15.VRWidgetDemo.mp4Note that I had to apply a few workwrounds in the code and use a bunch of scripting to achieve this. Next steps can be:
|
Squashed commits: ENH: Create framework for registering VR modules Adds registerModule function to qMRMLVRView which can be called to create a new button for that module on the Modules bar in the VR home widget KitwareMedical#43 ENH: Make VR home widget controls functional These changes make it so that changes in the module widget (viewable in Slicer) are reflected in the home widget (viewable in VR) and vice versa KitwareMedical#43 ENH: Add VR optimized Qt Style Sheet This style sheet makes the text in the VR menu widget bigger, and gives the buttons and slider handles different colors for unpressed, hover, and pressed statuses. KitwareMedical#43 ENH: Add VR home widget WIP Its UI is more or less complete but events are not handled and style sheet is needed ENH: Add virtual widget from arbitrary Qt Widget !WIP!
Creates and registers the VR data module widget (which can be switched to from the VR home widget) KitwareMedical#43
Creates and registers the VR segment editor widget (which can be switched to from the VR home widget) KitwareMedical#43
Some widgets were missing from the UI Re KitwareMedical#43
Formerly we saw the widget "from behind" when it appeared. Re KitwareMedical#43
A workaround was implemented, a better solution will be needed in the long run. Re KitwareMedical#43
At the start of the Pick event the interactor style invokes vtkCommand::PickEvent. With this Python script it is possible to do the clicking with the right trigger. def onClick(caller, event): gw.onStartInteractionButtonClicked() vris = slicer.modules.virtualreality.viewWidget().interactor().GetInteractorStyle() vris.MapInputToAction(vtk.vtkEventDataDevice.RightController, vtk.vtkEventDataDeviceInput.Trigger, vtk.VTKIS_PICK) vris.AddObserver(vtk.vtkCommand.PickEvent, onClick) Re KitwareMedical#43
Without this fix, although the first button press worked on any child widget of the set GUI widget, after this first click, no other button reacted to the interaction. Now all the buttons in the GUI widget work as expected. Re KitwareMedical#43
A very flexible way for showing menus in VR would be to implement off-line rendering of any Qt widget in a texture that is shown on a VTK plane actor, which can respond to virtual mouse events generated by the laser pointer (currently called probe mode in VTK) in VR.
Tasks that need to be done, start with first and only move to next one once it is done.
Useful UI panels (extend as needs arise):
Related issues
Related pull requests
The text was updated successfully, but these errors were encountered: