Situating 3D Interaction in Desktop Environment 2011-2012
Academic Support: Alex Olwal, Cati Boulanger. Assistance in Software Development: Akimitsu Hogge
With the rise of augmented reality, direct 3D interaction is gaining a momentum, but it is not relevant for everyday users as traditional 2D GUIs outperform 3D with better precision and comforts in many daily tasks. I believe however, there are situations where users can clearly benefit by direct 3D interaction - tasks managements, creative work, collaboration - and we should switch our focus from replacing conventional UIs with 3D, to how to ‘fuse’ 3D into traditional desktop interface.
SpaceTop is a design and technology solution to fuse 2D and direct 3D interactions in a single desktop workspace. Users can reach inside the screen, and type, click, draw in 2D, and directly manipulate interfaces that float in the 3D space above the keyboard. With SpaceTop’s unique interaction visualization techniques, users can transition between 2D and 3D seamlessly. SpaceTop’s applications showcase the power of such integrated workflows with fast switching between interactions of multiple fidelities.
Transitioning between direct and indirect, and 2D and 3D are challenging and the first guiding principle for the UI design was to create a modeless workflow. Sliding door metaphor helps users smoothly shift focus from the “main” 2D document to “background” contents floating behind. Touchpad interaction with multiple 2D windows floating in 3D space introduces interesting challenges. To address this issue, Shadow Touchpad emulates a 2D touchpad below each of the tilted 2D documents floating in 3D space.
In SpaceTop, the background tasks occupy a fixed position in the 3D space behind the main task, allowing users to rely on their spatial memory to retrieve them. This spatial persistence mitigates some of the cognitive load associated with conventional task management systems. Sliding door or stack interaction can be directly applied to categorize, remember, and retrieve tasks
The user can use his dominant hand to scroll through a main document, while simultaneously using the other hand to flip through a pile of other documents, visualized in 3D space, to find a relevant piece of text. The user can then drag that piece of text into the main document through the more precise touchpad interaction. In this way, users can quickly switch back-and-forth between low-bandwidth , high-precision interactions (copying lines) and high- bandwidth, low-precision interactions (rifling through documents), or use them simultaneously.
The objective for the hardware design was to create something that looks as close to a conventional desktop computer as possible, so that users can feel familiar. Through multiple iterations, I ended up creating a structure with adjustable hinge that can move in the range of tilting angles within which most of the users can find a comfortable spot.
Through the transparent screen, users can view floating UIs, both on the screen plane and in the 3D space behind it or on the bottom surface through a dynamic perspective correction. One depth camera (Kinect) faces the user and tracks the head to enable motion parallax. Another depth camera points down towards the interaction space and detects the position and pinch-gestures of the user’s hands. The setup can also detect if and where the user’s fingers are touching on the 2D input plane. To achieve this, I re-engineered light-weight algorithms for detecting touch and pinch previously explored by Andy Wilson to make it suitable for seamless transition between touch and pinch mode.