Feature development, shader programming, UX, pipeline, modeling, prototyping, all art direction.
Since I've joined the Tilt Brush team, I've had the opportunity to work on over 10 features through prototyping and iteration across 7 releases. I've prototyped and developed new features, designed interactions, programmed and art directed new brushes, and developed practices for how our small team builds and manages assets.
Genre: Creation tool | Platforms: Oculus Rift, Vive, Windows MR | Languages: C#, HLSL, Cg, GLSL, Python, MEL | Tools: Unity, Maya, Adobe CC
Team Size: 10 | Duration: 15+ releases over 2 years
Tilt Brush on Oculus Rift
This was my first big project on Tilt Brush, which was also our first port to another platform. Much of the groundwork laid here gave us insight how to bring Tilt Brush to new hardware.
Much of the UX challenge involved navigating input styles. Our visuals for the Vive relied heavily on the large touch pad, which the Oculus lacked. We explored ways to extend the Touch controller's form factor beyond its small face buttons and joystick.
I created prototypes beginning with sketches and ideas, creating and editing assets, iterating on concepts in-engine, and working with engineers to bring the final prototype to ship.
The primary problem we were aiming to solve with the Touch controllers was translating not only visuals to a smaller form factor, but also to mapping swipes and presses of a single touch pad to multiple face buttons and joystick. Making use of capacitive touch was a key part of the final design.
I authored the Touch controller geometry so we could reuse all existing controller textures for both platforms, regardless of the very different shape of the two controller geometries. This allowed us to use the same textures and materials across both platforms.
Lights & backdrop Panels
The Lights panel allows users to create custom lighting setups for a given sketch by changing the orientation and color of three virtual lights. This panel went through many permutations as I looked for ways to condense lighting concepts into a single panel, integrate color pickers, and work through interaction models for lights themselves.
I ended up pulling from traditional drawing exercises when looking for a proxy for the scene users were lighting, and writing custom shaders to flatten widgets while the panel wasn't in use.
The Backdrop panel allows users to adjust fog density and create a custom environment. Our final design landed on a large sphere with trackball style interaction (for the skybox) and a custom button for fog.
The most requested feature from users has been the ability to select and manipulate strokes. I designed and implemented audio, icons, widgets and our outline effect for Selection. Through the process, I parsed feedback from the team and grabbed people as they passed in the hall. The final implementation used a prototype I created for helping users better select/deselect in 3D as a key portion of the design.
There was no documentation to reference when I began creating new brushes in Tilt Brush. I had to reverse engineer how brushes worked in order to create new ones, having to sort out how geometry was generated on the CPU, augmented with shaders on the GPU and brought to a finish using textures and material parameters.
This brush combines 3 separate channels scrolling at different rates generated from one texture, multiplied by a gradient lookup to create its stylized effect.
Made from scans of acrylic paint + specular PBR rendering
vertex displacement, custom tube brush geometry, and darkened backfaces
This brush uses intentionally stuttery flipbook animation to recreate the feeling of hand-drawn animation and classic cartoons.
As the project has grown, so has the scope of what's possible when creating art with Tilt Brush.
To help illustrate this, I created a new intro sketch. I tried to create an environment that was larger than what most beginners set out to paint, but tried to balance it with clearly visible strokes and other pieces, to help give folks an idea of where to start and alleviate the anxiety of starting on a blank page. I also created the sound design for the sketch.
Our work launched with new improvements to the sketchbook panel, which I art directed while also working on this new transition state.
shader based TRANSITION
Along with designing the sketch itself, I worked with my creative director on an animation transition for when the sketch loads in. He and I wrote shaders for the effect, animating either the stroke geometry or stroke textures in.
There were no existing guidelines for creating new assets when I joined Tilt Brush. Most elements were tuned by eye and no quick reference to what Tilt Brush's style was. I took 6 weeks to overhaul our UX design to reduce the user's mental load and help build a foundation for new work in future milestones.
Touching every piece of Tilt Brush's interface for consistency, redesigning icons, writing documentation for all new and existing patterns, and creating master files with batch scripts for quick export.
Vector to Unity Pipeline
This is a workflow I developed to generate meshes from vector paths in Illustrator, translated to Maya, then exported to Unity. We've used this process for generating new panels, making models conforming to existing geometry, and translating icon designs to 3D elements.
Key art & experimental features
When I'm not working on Tilt Brush directly, I'll provide feedback to the rest of the team by creating art with experimental features. I also help spot issues with parity across the Tilt Brush toolkit and Tilt Brush's online gallery in an effort to make sure all content authored with Tilt Brush looks its best no matter where it's seen.
A fun side effect of this is I have also had the opportunity to create a few sketches for Google marketing to use with their social media accounts.