Hololens2 performance issues

[iamx1212]

Hello, I used UnityVolumeRendering to load a CT (Dicom) image and deployed it to the Hololens2 real machine. After opening it, it was very stuck. The model can be displayed but when I look at the model, it becomes very slow and stuck. What should I do?

[iamx1212]

[mlavik1]

Hi @iamx1212 !

I don't have a Hololens device myself, but I know there have been other users using this plugin with Hololens, so I believe it should be possible to get acceptable performance.

First of all: Is this a publically available dataset that you can share with me?

If not, could you check how large the dimension of the dataset is? If it's very large, you could consider:
A: Open the volume rendering settings window, enable "Show downscale prompt", re-import your DICOM and choose "yes" when asked if you want to downscale the dataset
or B (better): Use some software (I think 3D Slicer can do that) to downsample the dataset.

That should reduce the graphics memory usage (and memory bandwidth usage), and probably improve performance.

Also, you might want to check the appearance settings on the VolumeRenderedObject:

• Enable ray termination: ON
• Disable cubic interpolation: OFF
• Disable Lighting: OFF (unless you can afford it - it's expensive though)

[iamx1212]

I have been trying to solve the lag problem recently, but it hasn't had a particularly good effect.
Uploading CT.zip…

[mlavik1]

Just a thought: Is the performance problem bad in all render modes? (Direct volume rendering vs isosurface rendering)? Maybe not so much in Maximum Intensity Projection mode?

I saw that Hololens 2 uses a resolution of 2880x1406, which might make volume rendering a bit tricky - considering its GPU. You could try change the resolution, or in Unity: UnityEngine.XR.XRSettings.renderScale = 0.5f or something. Might look really bad, but would be interesting to see how much it affects the performance.

You could also try to change the MAX_NUM_STEPS values (one for each render mode) in the DirectVolumeRenderingShader.shader

[iamx1212]

Yes, there will be lag in volume rendering, maximum density, and isosurface drawing at MAX-NUM-STEPS 512. Lowering the MAX-NUM-STEPS value can slightly alleviate this.
I used UnityEngine XR The XRSettings.renderScale=0.5f method can also alleviate lag, but my method may have been incorrect. When using it with Holo on my head, the entire model shook noticeably as I walked around.
script.zip

[mlavik1]

@iamx1212 hmm, if even maximum intensity projection mode is slow, then I'm afraid it's not looking very promising.. Since I don't own a Hololens device it's hard for me to investigate if this is just due to the hardware, or if there are any performance issues, but I think I'm running out of suggestions now...
Do you need to run the app natively on the Hololens, or would it be an option to run it through a PC's GPU?
I don't have experience with Hololens, but on Oculus you have Link/Airlink for that. I saw there's something called "Holographic Remoting Player" for Hololens.

[DiangF]

Do you have any measures to address the issue of loading rendering on HoloLens? thanks

[mlavik1]

@DiangF Hmm, what do you mean with "loading rendering"? Are you running into any issues loading datasets?

@iamx1212 I've recently added a new "sampling rate multiplier" setting. If you check out latest from "master" branch you can try it out.
Try reducing the value and see how much the FPS improve. Feel free to share here! :)

I now also do opacity correction, so the only visual different should now be more/less artifacts (the colours don't change like they did before)

[shiauyang]

Hi @mlavik1,
Firstly, be pretty appreciating at your EasyVolumeRenderer project, it helps me a lot.

I use HoloLens2 to display only slice, so changes MAX_NUM_STEPS, disable the DEPTHWRITE_ON and disable cubic interpolation is not helpful for me to improve FPS on HoloLens2, it all for volume rendering.

And i have downscale my .dcm from 100 MB to 80MB, it improves FPS a little, but when rendering slice, the FPS will be the same low.

You said that use URP will help boost FPS, i change my unity project to URP and change SliceRenderingShader.shader to URP, it successufully run on HoloLens2 but the performance remains the same.

Is there any suggestion for improve FPS for only slice rendering?

Finally, thank you very much

[mlavik1]

Hi @shiauyang ,

I still don't have a Hololens device available, so I'm unfortunately not able to look into that myself, but I'll try to answer what I can:

• You no longer need to change the "MAX_NUM_STEPS" value. Instead, you can now adjust the "Sampling rate multiplier" in the VolumeRenderedObject inspector (see screenshot in comment above)
• I don't think URP should improve performance for you on Hololens. URP enables you to use FidelityFX upscaling, but I don't think that's available on Hololens standalone..
• Regarding downscaling: Did you downscale the size of the whole dataset, in a tool like Slicer?

Also, what do you mean when you say "rendering slice"? Are you talking about the "Slice renderer" window? I would suspect that one to be really fast, but you may need to disable the VolumeRenderedObject, or at least its' MeshRenderer.

[shiauyang]

Thank you for your constructive replied @mlavik1,

There are some methods I tried to improve FPS for HoloLens2 APP natively:

In my project, after add slice in Slice renderer, there will be new object 'slice plane' under VolumeContainer. I just want to render it without 3D volume, so, yes, I disable MeshRenderer in MeshRenderer.

From my experiment, adjusting Sampling rate multiplier didn't change FPS in my HoloLens2, I think is about volume rendering, not slice rendering.

Both URP and HDRP work well for HoloLens2, I see your post about performance to do it, thinking about self-define shader at SRP can improve FPS, and the results are same as you think. It can't improve FPS. https://github.com/mlavik1/UnityVolumeRendering/blob/master/Documentation/General/VolumeRendering/ShadowVolumes.md

About render scale, I have tried XRSettings.renderScale = 0.5f and 0.1f for both built-in render pipeline and URP, however, the FPS remained unchanged and because the lower resolution, HoloLens cannot render object stably and precisely.

Regarding downscaling, both python scripts and 3D slicer can downsample my CT dataset, nonetheless, there will some data miss which is not I want. Hence, I will not use this method despite it can boost FPS.

Here's my thought: Batches (drawcall) is pretty slow <10, so I don't think is CPU-bound and vertices is <600 which is acceptable for HoloLens2 GPU, but FPS is about 15, really don't have any ideas.

Do you have any suggestion for changing SliceRenderingShader.shader or any idea?

Pretty appreciate for your work, time and replied.