VR Explorations

Throughout 2016 and the beginning of 2017, when I was still living in Oslo and working for Postmenn-Stripe (now PXLR), we did several internal R&D projects to explore the possibilities and limitations of Virtual Reality - and how to best produce content for the platform.

Coming from a visual effects background, focusing on commercials, movies and TV, the visual quality of an image or video clip has always been paramount. This we wanted to take with us when producing content for this new medium.

The part of the projects described here was handled by me together with my colleague Mariusz Kolodziejczak.

Photogrammetry

Photogrammetry is the process of creating digital models, of real world objects and locations, based on photographs. If done right - it can create imagery virtually indistinguishable from the original. Although labor intensive and not suited for all kind of objects (ie glass and transparent objects will give a poor result) it has great potential when you want to share an experience in VR.

For a test case we chose the Stave Church (Stavkirke) at the Norwegian Museum of Cultural History at Bygdøy in Oslo, Norway. This old, wooden building has a lot of character and many beautiful details to experience.

Inside the Stave Church

It took around 800 pictures, clocking in at 21 megapixels each, to reconstruct the inside of the church. We shot on two Canon 5D Mk II cameras which were kitted with a 24 and 50mm lens. The exposure times were quite long (around 60 seconds) since we only had the natural light available to us which passed through the two open doors in the building. We also took great care to not disturb the other visitors - since the shoot was done during a regular visiting hours. With two photographers we spent around 3 hours shooting the interior this way.

The resulting images were processed in Agisoft Photoscan where we had to take care to mask out tourists (and each other). We also processed all the RAW images to floating point EXR files to preserve the dynamic of the natural light to the best possible degree.

Post processing and cleanup were done in a mix of Autodesk Maya and 3DCoat. Since there was quite a bit of overlapping geometry further up towards the roof we had to spend a significant amount of time doing this.

The final model had to be split up before bringing it over to Unreal Engine - since we wanted to preserve as much detail as possible (although we did reduce the polygon density of our original model quite a bit). Several 8K texture maps were generated as well.

As a final touch we experimented with adding the sound of munks, church bells and creaking wood - to enhance the experience. Using the HTC Vive VR Headset you can walk around and look at whatever peak your interest.

The video below shows the experience from outside and inside.

We also did stereoscopic stripmaps suitable for the Samsung GearVR or other platforms, like Google Cardboard, with the proper viewer. These will give you full depth perception and have excellent image quality, but will not allow for movement and sound naturally.

Stereoscopic stripmap

You can download the full size image here (you'll need a proper viewer for this to make sense).

Exteriors

Since we were there shooting the interior of the Stave Church we also did a quick test outside. And by quick I mean not more than 15 minutes by one photographer to walk around and snap images from the ground level. The captures was a mix of 24mm and 100mm lens shots.

As with the interior model we processed these with Agisoft Photoscan, but since there was less occlusion and overlap we did not spend much time doing cleanup before bringing the model into Unreal Engine.

We also imported a model of a house from Røros - which I photographed some years ago when I went there for a visit. The main challenge with this one was the grass on the roof - which we had to remove since the photogrammetry software could not create a good enough model there. This was mainly because of the wind made the grass move between the frames.

Old Artifacts

Preservation of cultural heritage and old artefacts is another venue where photogrammetry is well suited.  Capturing organic forms with intricate details is no extra work this way. You can then choose to bring the captured object into VR, 3D print it or save for later reference in case the object in question is lost, stolen or destroyed.

To evaluate small scale objects with a lot of details we were lucky enough to get access to an image-set of a "carved scull". The original images were provided by the photographer Steffen Aaland at Glitch Studios and were shot using a Phase One IQ250, a 50 megapixel medium format camera. Focus stacking was utilised to get required depth of field (i.e. get the whole scull in focus).

We brought the scull into VR as well and it was pretty incredible to be able to pick it up from the table and inspect it up close.

Bringing reality into VR

As part of our SkatteFUNN project we also looked into a plethora of other methods to "acquire reality" with the highest possible fidelity.

Areas of extra interest were HDRI (High Dynamic Range Imaging), to ensure the experience in VR would be as close to "being there" as possible, as well as making sure the source material held as high a resolution as possible. Even though the screens of todays VR headsets are pretty low resolution this will improve - and when this happens we can re-export the images for the new formats.

To optimise the process we photographed the scenes in a number of ways - from more traditional pano-stitch (although with offset per eye) - to a full stitch with Cara VR for Nuke. The latter giving the overall best result, but was way more demanding in artist-time and processing. As an example stitching 30-something 25 megapixel stills would consume over 125GB or system RAM. For one frame! This scene would crash in Windows, every time, but would process fine on Linux.

Some examples of images we did were the following three. The one from Vøyen gård were processed in CaraVR while the rest were stitched more traditionally per eye, but converted to stripmap in Nuke.

The main challenge with the latter were that we had to shoot one set of images per eye (and using 5 image bracketing for the HDRI), which led to session length of around 40+ minutes per location. In that timespan the sun and sky (in case of clouds) managed to move quite a bit - forcing us to replace the sky in several of the locations.

Vøyen Gård, Oslo

Vøyen Gård download (you'll need a proper viewer for this to make sense).

Blå, Oslo

Blå download (you'll need a proper viewer for this to make sense).

Vulcan, Oslo

Vulcan download (you'll need a proper viewer for this to make sense).

Lunar Fligt vs The Rift

I hereby declare Lunar Flight to be my favorite Rift experience. It has by far been the best and most pleasant (read: least nauseating) experience. That might also have to do with me finally getting around to calibrating the correct IPD (inter pupillary distance) for my Rift.

Most likely the slow movement of the lunar lander and the static cockpit around the player is the main contributor in my case.

I really like that you can look around the cockpit as well - and when you look directly at a button you can activate it by pressin Y on the controller which is nice.

At first I tried to hook up the SpaceNavigator, which would be perfect for this, but it seems a bug in Unity only let's me register one axes (Z-) so until that's fixed I have to rely on the Logitech F710 gamepad.

However; it's not an easy game/simulator - so far no success, only crashes.

Now if I could only get to try out the famed Eve Valkyrie demo...

Rift: Day two

Yesterday I got UDK up and running and took Epic Citadel for a spin. Enabling the Rift was as easy as running the level from the editor, open the console (by pressing "~") and typing the following "stereo on" and "hmdwarp on".

Console in the UDK

Tracking, it seems, was enabled out of the box so nothing else was needed.

The verdict? It kicks ass! The graphics, even though they are made for phones, are very impressive on the Rift. My motion sickness took longer to set in as well. I don't know if it is because I'm starting to get used to the experience, but what helps, for me at least, is to close my eyes when I move around with the mouse - since that is what confuses the senses. Moving back and forth along the direction I look in is not a big issue.

Having two monitors when working with the Rift isn't necessarily the optimal solution since there are issues with vsync apparently. What happens is that you'll get tearing on one of the displays - and that might as well be the headset. Which is something you would like to avoid.

Since I have an iMac on my desk already I figured I'd just install a VNC server on the Windows machine and only have the Rift attached to the graphics card. That way I can control the menus/desktop with a VNC client and then close the connection before I put the headset on. Works like a charm! :)

VNC to the rescue.

I don't think the flavor of VNC server matters much, but I opted for RealVNC myself.

Oculus Rift: First Impressions

This Monday UPS delivered my Oculus Rift - which were a pleasant surprise since it's still, as of today, which is Sunday, listed as "Processing for delivery" on my account page on the Oculus web site.

Shipping box.

Nice and sturdy packaging.

High product value going on here. Even if it's "just" a developers kit. Me like :)

As I'll probably will be bringing the Rift with me the case will be handy to have.

As the OS X devkit weren't available when I got the kit didn't have anything to test it on as I'm mostly on Mac and Linux these days. Still - the new PC arrived after a couple of days so no big deal. Had to fiddle a bit around to get it up and running though; It's been quite a while since I put together a "gaming rig" and a dated BIOS prevented me from booting with the Nvidia card.

Getting the system up and running with Windows 7 and the rest of the software needed took most of yesterday so it weren't until today I could hook up the Rift and do a test run. Very excited!

Current setup with a small 1280x720 display to extend/mirror the Rift. Oh, and a big iMac 27" in the background. Kind of the reason I don't want any more big displays at the moment ;)

My first impression, when I put on the goggles, were: "Wow! Those pixels are big!". Which is nothing new, and something I were prepared to experience, but still. On the other hand I did fire up the goggles while still on the desktop and a pure, blue, background doesn't give you much to look at. Having since learned that you can start the Unity Tuscany demo on your other display and switch it over to the Rift using the F9 key makes things a bit easier. As others have noted; you can't navigate your desktop with the Rift!

It's a must to have a second monitor to navigate the desktop and in-game menus.

I know there's work being done in this area, but I haven't had time to test it out myself yet. There's an active thread on the MTBS3D forum about this.

That said; the Tuscany demo is very nice and you kind of forget those pixels when you move around - which is the best part. The Rift tracks your head movement very precisely and you really get the feeling you're "there". Until, after a couple of minutes, you get pretty nauseous. At least I did...

As I'm prone to motion sickness this didn't surprise me at all, but I think the main reason in this case were from moving around using the keyboard and mouse. As long as I'm in one place, just looking around I didn't feel a thing. My hope is that having positional tracking as well will alleviate this since the cues picked up by the middle ear will correlate to what the eyes see. I might try to implement some means to test this later on. Later when I tried out some other demos the effect weren't nearly as strong so I hope it will pass entirely in the future.

Firing up Hawken were a nice surprise. As it's not released with Rift support yet, at first I didn't understand why my POW was looking straight down, until I saw the headset laying on the desk next to me with the front down. Seems like the tracking works as expected, but the camera-rendering isn't enabled yet. There might be a way of enabling this in an ini file, but I haven't gotten that far yet. The same goes for the Citadel demo in the Unreal Developers Kit. Works fine with the head tracking, but I need to tweak the same (?) ini file to turn on the 3D view. Something to try out tomorrow.

The Rift being a developers kit I didn't expect too much, but so far I'm very happy with it. If the guys at Oculus VR manage to implement positional tracking as well as a more pixel-dense display I think the consumer version will be great product. And I hope this time around VR won't flop like it did back in the 90s.

Merging multiple DVI signals using an FPGA

A small followup to my previous post. Inspired by an article on Hackaday I did some more research on the use of FPGA to manipulate DVI streams.

It seems this has already been done. I came over this project at the Institute of Visual Computing in Germany. Direct link to pdf here. They talk about several different methods to parallelize rendering of a scene. "Sort-first" is a way of splitting the image into multiple parts (quadrants) and have different computers render each one - then combining it with a custom FPGA.

The downside is that their solution seem to require genlocked cards. This would require professional GPU's like the Quadro line from NVidia which are not exactly the best for gaming and also carries a high cost. On the other hand, there is enough RAM on the FPGA that they can handle +- 2 lines difference in sync using a small buffer/FIFO. The main difference to what we need is that we have both GPUs mounted in the same PC. That way, when we tell the 3D engine to start sending the two frames rendered in each GPU they will start at the same time - more or less. Hopefully this will let us get away without using genlock, but that's still just a theory. Running high-rez at 60Hz the timing will be tight anyway.

Update:
It seems the 618 series DVI comparator from Colorado Video has the capability to do what we require. Unfortunately they do not mention what the max resolution is, if you need genlocked signals or what kind of latancy (if any) the unit has. At $1390 it's not exactly cheap either. Good to know someone has done it though.

Maximizing Performance for VR

Like everybody else that took part in the Kickstarter I'm eagerly waiting for my Oculus Rift. Latest news is that they will start shipping kits later this month so hopefully the wait will be over soon!

While we wait there's a lot of time for speculation. For example; how will it perform with modern games? Doom 3 BFG is after all not the most taxing game any more. Most modern graphic engines require a pretty beefy hardware setup - both on the CPU and GPU side. Even then you aren't guaranteed a minimum of 60 fps all the time. When wearing a HMD like the Rift anything below 60 fps could ruin the immersive experience as a result of stuttering.

As the screen of the Rift developer kit only has a resolution of 1280x800 pixels it's not bad as for pixel fill, but it can still tax the computer when rendering two eyes/cameras. Nobody yet know how pixel-heavy the consumer version will be, but 1920x1080 is not totally unlikely.

So how do we ensure a high and consistent frame rate? First of all; turning off vsync would be a bad idea since tearing will destroy the immersion just as much as lower frame rates. Turning off high detail, anti-aliasing, shadows etc would for sure make the games run smoother, but where's the fun in that? Also; as the Rift has relatively low horizontal resolution per eye (1280 / 2 = 640 pixels) you'll probably need some anti aliasing to take care of hard edges. This is something that has already been confirmed by people that has tried out early versions of the Rift.

Obviously adding one or more graphic cards would help, but it's not that easy. SLI or CrossFire are not good solutions when generating VR related imagery since they both add 1+ frame of latency depending on the mode used. That's 16ms at 60Hz which automatically puts us above 20ms latency since rendering one frame takes 1 frame already. They can also in some cases suffer from other artefacts.

Latency, in combination with frame rate, is a big factor since you'll easily develop motion sickness if the computer generated world you perceive through your eyes does not match up with the actual movement your head performs and your middle ear detects. Both Michael Abrash of Valve Software and John Carmack at ID Software have written at length about latency and suggests ways to at least partly mitigate it. Both articles are well worth a read. The main goal would be to get latency below 20ms as anything below that would not be noticeable for most of us.

What we need is a way to render the left and right eye on separate graphic cards. Having the game engine support this, in comparison to rendering both views through one card shouldn't be too hard. Back in the "old days" we would do this when working with stereoscopic content - since we had to use two projectors with polarized glass to show the content so it's not that different. The difficult part is to merge them back together into one signal - as that is what the Rift expects.

A custom FPGA could be the solution. I'm no expert, but since we won't manipulate the signal in any way - the computational requirements shoudn't be too bad - especially for the resolutions we are talking about for the developers kit.

Since DVI, as I understand it, don't send the whole frame as a packet, but streams each pixel as if it were an analog video signal, we should be able to mix, or rather switch, the signals using the FPGA in real time. This is important since we don't want to introduce any more latency to the pipe. Since the left and right image will be located at each halve of the frame respectively we would simply tell the FPGA to switch between the two signals when passing the middle of the screen and back again at the right edge of the screen.

There are, however, a couple of challenges with this solution, the most prominent being sync (or genlock/framelock in TV-lingo). We would need to ensure that each pixel on the screen, be it for the left or right eye, scans out at the precise same time or else we would get artifacts in the form of slipping scan lines and skewing of the image.


Taking it one step further; to reduce the latency on the computer/GPU side if things one might even be able to have the computer render frames at 120Hz, but only let half of them through the FPGA - giving the display 60Hz - which it can handle. Although that would probably require some kind of buffering since the clock signal of the input would be half of what the display requires.

I'm making quite a few assumptions here, but it would be very interesting to test this out properly. I wonder if theres's time to learn VHDL before my Rift arrives?

Virtual Reality Redux

It seems Virtual Reality is gaining popularity again. I remember back in the 90' when VR was the big hype, but never took off because og the lacking hardware - both to produce the images fast enough as well as capturing user input accurately enough. There was also the problem with lag since the hardware available needed time to crunch the numbers.

Nowadays there's no question of the hardware being powerful enough and as we've seen the last couple of years the sensor technology and finding innovative ways of deploying it certainly has improved.

A while ago I came over the VR game gun which shows how "easy" it is to use off the shelf components and software and make something quite cool - as long as you have an idea.



This system, although functioning, would require a very larg venue to be practical so keeping yourselfe stationary, but still moving around in the virtual environment has been one of the other big issues needing solving. Creating a big sphere, ie a gigantic hamster ball, where you can round around inside is not a new idea, but I haven't seen anyone actually creating such a device until now.



My biggest concern about this sollution is momentum. No matter what material you decide to use to construct the sphere it will have considerable mass and coming to a complete halt after running will test your balance. I would love to try it out though :)

More information at the VirtuSphere homepage.