Year 4, Semester 2 - Perspective Detective: an augmented reality game on the PlayStation Vita

January 2016 - April 2016

This semester we had a few lectures on augmented reality (AR) and a few on anaglyph 3D and were tasked with making a game that takes full advantage of one of these technologies. I chose AR and made a game called Perspective Detective.

It's all about cameras

Both the lectures on anaglyph 3D and especially the lectures on AR involved lots of camera vector/matrix calculations and it got me thinking about how to use cameras as part of gameplay rather than just a way to view gameplay. I'd also been playing The Witness (which is excellent) so I had puzzles on the brain and eventually this led to the concept for Perspective Detective: virtual cameras attached to AR markers which see different versions of the level than the physical camera. As shown in the footage above, I designed 4 puzzles as a proof-of-concept for this mechanic:

Level 1 teaches the player the basic mechanics of the game: move from orange to green, don't trust the physical camera, virtual cameras show the true path. Level 2 reinforces these ideas and also forces the player to count out their steps because of a larger level and the fact that the physical camera shows nothing but the start and end points (and the player of course).

Level 3 introduces a new perspective trick by making the player look at the level from the opposite direction. The controls are relative to the player not the camera, so by turning things around like this it reverses the controls - to move 'right' you now press left. The arrow atop the player shows the direction of up on the d-pad. The final level uses multiple virtual cameras with alternating halves of the full path. The player has to flick between looking at the physical camera feed to see where they are and the two virtual camera feeds to see where to move next.

Visuals

I used this camera model by 3dregenerator and built the UI using these sheets by Buch. Both are free for non-commercial use. I also made some simple effects to make everything a bit nicer:

• When the physical camera loses track of one of the virtual camera markers, the lost virtual camera feed is rendered as old-school static with a message saying "CAMERA FEED LOST". There's another message for when the level marker is lost.
• Screen fades hide the transition between levels. When you complete a level the screen quickly fades to white, then loads the next level, then fades back in. It works quite well.
• When the player falls off the level they get shrunk as they go so it looks like they've fallen a long way.
• There are small angular brackets in the corners of the virtual camera feeds similar to the ones you see on camera viewfinders. There's also a flashing red circle and 'REC' message in the top left.

How it works

In AR the only fixed point is the camera, so that acts as the origin for your coordinate system. The Sony marker tracking library gives you a transformation matrix to go from the camera to the centre of the marker - this transformation includes position, orientation and scale. If we want to position something relative to a marker we just build a local transformation matrix to do whatever we want, then post-multiply that by the marker matrix to get into world coordinates. The axes relative to each marker are shown on the left of my lovingly hand-drawn diagram; the marker lies in the xy-plane and z increases up from the marker.

So we have the position of the centre of the marker, a. If we do a local transformation along z we'll get position b and if we do a further transformation along y we'll get position c. The eye of the camera is at b, the look-at vector is (c - b) and the up vector is (b - a). With these we can build a view matrix and then render the level geometry from the perspective of a camera floating above the marker. As the marker moves, so does the view.

Thoughts

I had plans for another mechanic but decided not to do it because I really didn't need to and had plenty of other work to be getting on with. Later levels would have had colored glass panes attached to markers and viewing the level through these panes would reveal a different set of information, both for the physical and virtual cameras. This was directly inspired by a similar mechanic in The Witness.

​Surprisingly this is the first time I've made a puzzle game and I thoroughly enjoyed designing the levels for it. I might continue playing with this multi-camera dynamic in a purely virtual format as AR games aren't exactly easy to play, though I do think the tactile nature of the player physically moving the cameras (albeit via their markers) adds a lot to the game. Overall I'm really happy with it as a proof-of-concept and those who have played it seemed to really enjoy it!