Case Study: Joycestick UX
UX Researcher & Lead Designer
Above: a short clip of one of the chapter experiences in Joycestick, which is an HTC Vive interactive virtual reality game based on James Joyce's Ulysses. Read an overview.
The process of designing Joycestick's user experience was a monthslong discussion that involved everyone on the team. This was due to a number of factors:
Everything was new. There were no best practices on how to turn a novel (a notoriously dense one at that) into a virtual reality game, or why that would even be a good idea in the first place. It's also not like we were experienced game developers – we were a bunch of college kids, and was only starting to dawn on us what kind of a gargantuan undertaking Joycestick would be. Most of our engineers, while accomplished in areas like AI and web dev, hadn't touched Unity or VR before. Our writing team had more experience with satire and one-act plays than they did gaming storyboards. So the early production process was vague at best, and there was a massive learning curve across every team. 

But those first few months were thrilling: everything was ideas, everyone had them, and many of them were good. We just had to figure out which ones were achievable given our time and skill constraints. To that end, we aimed to have an MVP of Joycestick for an upcoming literary conference. That gave us six weeks of runway to write, storyboard, sketch, model, design, and develop a minimum interactive product. It was rough, and there were a lot of all-nighters. But we learned a ton: what works, what really does not work, who is suited to what job, and that everything takes about four times as long as you think it will take.
There was a divide on what the product should be. The big early internal debate was whether Joycestick should be a passive experience or an interactive game. On one side you had the writers, who pushed for a product that was mostly a guided experience (i.e., little to no action on the part of the wearer). The argument was that messing around with Ulysses too much would invite all sorts of problems: a lack of authenticity, fan backlash, and that converting a challenging masterpiece of writing into a video game was impossible. “Ulysses is not Super Mario 64,” was the general vibe.

On the other side you had the engineering team, who cared less about preserving Joyce’s precise sentiment in Telemachus, or Circe. They saw a passive, please-sit-down-and-do-not-move product as a waste of VR’s potential. They argued that any VR experience could not truly be immersive without first being interactive, and that to rob the wearer of the freedom to move and touch was to rob them of both their agency and their curiosity. OK, they didn’t say that. It was a team of all male engineers. They said something like “screw you guys, we want to build Nazi Zombies, but with this Joyce guy and VR and stuff.” But deep down I like to think that’s what they meant.
And then you had a handful of people, myself included, who thought the two strands weren’t so disparate. When we presented Joycestick at a conference in Singapore, one of our advisors, Andrew Kuhn, gave a talk on the history of books. He described the novel as being the original virtual reality machine, one that has “a trillion dollars worth of special effects that can be conjured up by the mind’s eye.” It has stuck with me ever since. If you think about it, the similarities are striking. Both novels and VR let you slip into vivid, multi-sensory worlds, ones with characters and plot and intrigue. You can take on a character, you can be a fly on the wall — it all depends on how it’s written. Both allow you to wander and imagine in ways that cinema and television could never do. They are not for sharing: they are both deeply personal and private in nature. Indeed, you could make the case, as Andrew is wont to do after a few drinks, that VR is just an inferior form of the book.
In any matter, it became clear that passive VR was a closer experience to cinema than it was to a novel, and the case for an interactive game won out.
It wasn’t clear who our audience was. Who exactly Joycestick was for was another question. There was comically little overlap between the two most obvious audiences:  a) fans of VR and b) fans of Joyce. There were huge gaps in age, technological ability, and in what each group wanted to get out of the experience. We decided to opt for a triangulation method, whereby we’d try to give both groups some of what they really cared about. 
For Joyceans, authenticity is paramount. We didn’t mess around his Joyce’s words: they were narrated as written, and incorporated into our game’s soundscape at all points. Realistic environments were another huge draw for Ulysses fans: we were offering them a chance to step into a long-gone Dublin of the 1920s. And these people are experts: they know every detail, every description. We couldn’t afford screw it up. The team spent months researching how to go about designing the environments. We combed through every clue in the text. We asked historians stuff like what kind of wallpaper a middle-class salesman like Leopold Bloom would have in his 1920s garden-level Georgian abode. Our project manager Professor Joe Nugent, engineering lead Ryan Reede, and I went to Dublin for four days on a mid-semester research trip. We built 3D models off of architectural drawings, and for the Martello Tower environment, we had a team give us a 3D point cloud of the building based off of a scan they had previously done. 
We put in the work, and it paid off. Joyceans generally gave us top marks for how the environments felt and looked. In fact, they usually said it was the thing they enjoyed the most — after all, they know the novel inside-out, so Joycestick’s storyline in some ways didn’t matter to them. That wasn’t the case for VR fans, who tended to be young college students. They didn’t really care that the absinthe glass in Proteus was period-accurate, and they’d never touched the book before. We had to make sure that the game stood on its own as a narrative. The story obviously couldn’t capture everything in the novel, but we did our best to communicate the narrative through interaction, as you can see in the video up top. It conveys the essence of the story in Calypso, while still incorporating actionable video game behavior. This allowed college students to play a full-on game while still learning crucial tenets of a novel they’d never pick up, even if it was assigned reading.​​​​​​​

An early rendering of the Martello Tower (top) versus what the final tower looked like in the headset (bottom)

We didn't know how Joycestick was going to be used. It was obvious from the start that VR was not going to be a mainstream platform, at least not in its present form. As Joycestick was a fully interactive project, we knew platforms like Cardboard and Daydream weren’t an option (we developed Joycestick for the HTC Vive system). VR marketplaces like Steam only catered to people who had VR setups in their homes. These people were largely hardcore gamers, and were not likely looking to check out an experimental, media-boundary-pushing literary project. Which begged the question: where did something like Joycestick belong? 
We decided to treat Joycestick as a traveling art installation. This reflected the inherent space and equipment needs of VR, while also acknowledging that its roots are grounded in the arts rather than tech. We ended up touring Joycestick in this capacity in Boston, Rome, Dublin, Toronto, Singapore, Amazon HQ in Seattle, and at the Smithsonian National Museum of American History.

PM Joe Nugent (left) and student Ryan Reede (right) showcasing the game in Singapore.                Credit: The Straits Times 

Joycestick as an installation presented experience limitations. As an exhibit at conferences, we had to get through large numbers of people in a short space of time. We had to figure out how break up Joycestick into small, accessible pieces so that we could give people a satisfying but brief spin. We ended up switching the experience from being a linear, chapter-by-chapter game to a radial, fragmented model. We turned the Martello Tower, the setting for the opening chapter, into a home base of sorts. It functioned as a museum, with several dozen interactive objects that were important to the story. Playing with some of the 3D models acted as a portal to different chapter experiences, such as the location in the video above (7 Eccles St for all the Joyce fans reading this), and you were returned to the tower upon completion of each chapter.
As Ulysses is hardly a typical narrative, this didn’t present many authenticity issues and allowed users to dip in and out of different chapters as they pleased. Indeed, the tower itself acted as a perfectly satisfactory experience itself for many users, and was sometimes used as such when the queue for Joycestick extended into the dozens. 
Getting from A to B. This is a debate that is far from settled in VR—how does the user get around? Some VR games, like the popular The Lab, use teleportation. The user identifies a point in the environment that they’d like to go to via the controller, and then they are transported to that spot. Some games allow the user to physically walk around that point, which is usually around a 9 ft x 9 ft square (though this varies by each room setup). Physical walking is great when it works, but it limits environment size and bumping into real-world objects in your living room is a real nuisance. 
There was another, less popular walking method: what we called “the running man,” due to its similarity to the popular Vine dance (it was a new thing at the time, trust me). This basically called for users to alternate moving the controllers up and down as if on a jog, with each movement swing propelling the user forward in whichever direction they were looking.
After some A/B testing, we opted for the Running Man. It’s better suited to larger environments, and I think it’s the more natural of the two. For the few who found it disconcerting, we also enabled physical walking. As we were offering the experience as an exhibit, we were able to control the environment so that they wouldn’t walk into anything.

An in-game clip with the Running Man method shown.

Trigger fingers. After figuring out the narrative, the overall flow, and how to get around, all that was left to do was figure out how exactly the user interacted with the world objects. Each HTC Vive controller was equipped with a wide array of buttons, triggers, and touch pads. Given that a lot of our users would be older academics, we tried to make the controls as intuitive as possible. Hitting the menu button brought up the menu, you scrolled through the menu via the touchpad. Squeezing the trigger let you pick up objects; releasing it dropped them. If you held an object, swung the controller, and released the trigger, the object would be thrown across the environment thanks to our physics engine (school groups visiting our exhibit in the Smithsonian basically spent their entire time mastering this aspect of the experience).
For almost all of our users, Joycestick was their first time using VR. The learning curve in using it is no joke, so we always had a student help out a user at the start. The student would direct them around the environment, telling them how the controls work, and answered questions. Over time we developed a tutorial, which helped reduce the learning curve and allowed student helpers to be a little more hands off.
Takeaways. The most exciting lesson we learned from Joycestick is that it can be replicated. The same reasons why Ulysses was a good fit for VR — the similarities, the pedagogical implications — are the same reasons most novels can work in this setting. That’s why, at the end of the project’s main development, the engineering team began working on an agile framework to make novel-VR adaptation more accessible to other schools. Not only that, but the team decided to take Joycestick open-source so that other universities could make use of our extensive code libraries and repositories. I’m being as transparent as I can about our UX and design research, in the hopes of making another team’s job a little bit easier. It’s a brave new world. Get out there and explore it!
Back to Top