Surgical VR

The video above is the client-facing overview of the product. It gives a sense of what the experience is; the rest of this page covers how it was built.

This project involved building out a comprehensive surgical learning experience in VR, covering the patient journey through a mix of 2D media, 3D scenes, standard video, stereoscopic 3D surgical footage and 360 content.

• Integrated varied media types into one chapter-based VR flow that stayed clear and usable in headset.
• Built and supported interactive reinforcement moments between chapters using 3D content, text and guided interaction.
• Worked on efficient playback and coordination of multiple concurrent video streams inside a real product.
• Contributed across content integration, interaction logic, product polish and release-quality problem solving.

Stereoscopic 3D robotic surgery footage

The robotic surgery footage in this product is genuine stereoscopic 3D, not flat video shown in a headset. A surgical robot's endoscope is already a stereo camera: it captures two channels, one per eye, because that is how the surgeon perceives depth at the console. We take that native stereo pair straight from the robot and keep it stereo all the way through the pipeline, rather than flattening it to a single mono image the way conventional surgical video does.

In headset each eye is fed its own channel, so the viewer gets the same depth cue the operating surgeon had. That changes what the footage can teach. Instrument depth, how far a tool actually is from a vessel, the layering of tissue planes, whether a structure sits in front of or behind another one - all of that is spatial information that a flat recording simply throws away. For dissection content, where the entire skill is judging planes and depth correctly, this is the difference between watching a procedure and reading one.

Presentation matters as much as the source. The footage plays on a large screen anchored in front of the viewer, but the screen is not a flat quad. A shader curves it around the user so the surface stays at a consistent viewing distance from edge to edge, which keeps the far sides of a wide image comfortable to look at instead of forcing the eyes to work harder the further out they track. The screen sits inside the 360 capture of the operating theatre, so the surgical view is framed by the real room it came from.

• Preserved the robot's native per-eye stereo pair end to end instead of collapsing it to mono.
• Drove correct per-eye playback in headset so the depth separation reads accurately rather than as a gimmick.
• Curved the video surface in-shader for comfortable wide-angle viewing at a consistent distance.
• Chaptered the procedure so a specific phase of the operation can be jumped to directly and rewatched in 3D.

Volumetric DICOM rendering

The most technically involved part of the product. CT and MRI scans arrive as DICOM: a stack of greyscale slices whose intensity values map to tissue density. Clinicians are used to reading those slices flat. The question worth asking in VR is whether you can render the volume itself, in real time, and let someone move around it.

• Load the DICOM slice stack into a 3D texture, so the scan becomes a sampleable volume rather than a set of images.
• Render by raymarching through that volume per pixel, accumulating colour and opacity along each ray.
• Drive appearance from a transfer function mapping scan intensity to colour and opacity, so bone, soft tissue and air can be isolated or blended without touching the source data.
• Support clipping so the volume can be cut away interactively to expose internal structure.

Raymarching a volume is expensive, and VR makes it worse: you are rendering twice, at high resolution, and a dropped frame in a headset is not a cosmetic problem. The largest saving came from a two-stage approach. A broad phase marches a low-resolution copy of the scan first, cheaply establishing where the ray will and will not hit anything worth sampling. Only then does the full-resolution march run, and only across the intervals that survived. A CT volume is mostly air and empty space, so a very large proportion of the expensive samples are discarded before they are ever taken.

The rest was the usual balance: step count against quality, early ray termination, and deciding where fidelity could be traded without misleading the viewer. That last constraint matters more with medical data than with games art, because the image is supposed to represent something real.

The main value here was delivery: making a lot of different content types feel like one coherent training product, while learning more about VR UX, media-heavy implementation work and performance tradeoffs in production.