NUI for LS3

The Defense Advanced Research Projects Agency (DARPA) is working with Boston Dynamics to create a new kind of military robot. The Legged Squad Support System (LS3) program is developing a large quadruped robot with autonomous navigation capabilities, meaning it can find its own way through the world. This pack mule robot, designed to carry heavy gear for overburdened soldiers, would be embedded into military fire squads.

In late 2011, DARPA awarded CMU’s National Robotics Engineering Center (NREC), DEZUDIO, and Kicker Studio a contract to launch an R&D study called “A Human-Centered Research and Design Process for the LS3 Natural User Interface”. The study goal was to create an ideal experience for soldiers that would allow them to communicate with the robot in a head’s up manner, utilizing an interface that would also inform the robot control architecture, or the technology development behind the robot. Our team’s goal was to create a natural and intuitive user-centered interface for the soldiers to further facilitate human-robot interactions.


An interdisciplinary approach.

Kicker was particularly excited about being part of an interdisciplinary collaboration. This team of specialists approached and dissected the primary challenge–to create an interface that gives the soldier confident control of the robot–from multiple angles. Together we worked toward a solution from the design and robotics side that took into consideration the human and the technological factors that would impact the LS3 program.

Our team included principal investigator Carl Wellington, senior roboticist Matt Powers from NREC who worked on the LS3 robot’s sensing capabilities, and Tzu-Wei Powers and Raelynn O’Leary, interaction designers from DEZUDIO, who have extensive user research experience with military personnel. Kicker brought to the project our natural user interface expertise and our experience integrating emerging technologies, such as voice, touch, gesture, and haptics, into intuitive interfaces.

To start, we studied current methods of military communication and then spent a week with soldiers at Ft. Benning, observing team interactions in operational scenarios with and without a robot stand-in. We had them run through a variety of combat scenarios, and we had them test multiple modes of interaction with the robot, including voice, audio, gesture, and haptic cues and feedback. The soldiers provided invaluable insights to what would work and would not work for them.



Based on the results of our field study, we established the following design principles:

  • Robot interaction must be doctrinally aware but contextually flexible.
  • The mundane has to be easy.
  • Efficiency trumps specificity.
  • The robot is there to serve the squad.

These principles guided our team as we set about creating natural cues and feedback alongside the robot control architecture. We then tested and validated our design with eight soldiers in a one-day workshop to make sure what seemed natural to us felt natural to them. Based on this feedback, we refined and aligned the interface design with the robot control architecture system.



A robot that serves the soldier.

Our project’s ultimate goal was to explore ways to leverage the potential technologies in LS3 for heads-up interaction rather than the heads-down joystick interaction commonly used for military robots. We were asked: How might heads-up interaction with LS3 work for the soldier? Our team asked: What is the soldier already doing and how can we add interacting with the LS3 to those tasks with minimal interference? Because, as one soldier stated, “The robot is supposed to attach to us, so we’re not going to manipulate what we do for it. This is coming for us, not the other way around.”

The team wanted to ensure that the interaction design informed the control architecture design, ultimately impacting the technology behind the LS3 robot, to make it something that the soldiers can and will use.

We made recommendations for the interface from two levels: the user experience and technological specifications. We designed both sides with these key considerations in mind:

  • Leverage military vernacular
  • Build redundancy to allow flexibility
  • Utilize context to reduce cognitive load
  • Understand heads-up can still be dangerous
  • Create efficiency
  • Account for emotional stress


Our conclusion: In this context, intuitive, natural interactions are even more critical for safety and productivity and the robot interactions have to consider the stress of high-risk environments. In order to leverage the existing military communication, the interface needs to include a shared, adapted and unique vocabulary between the robot and the soldier. Natural, redundant interaction based on context and personal preference means soldiers have multiple ways to lead the robot. In order to reduce the cognitive load, controlling the robot should be contextual and not require extra steps. Eyes-free operation allows for minimal distraction, because heads-up can still be dangerous. The ideal interface is efficient with its one-command actions and it accounts for emotional stress by allowing special urgent commands and actionable feedback.

It was our team’s recommendation that communications be streamlined through the interface design by leveraging existing knowledge, existing mental models and the soldier’s environment to reduce the soldiers’ cognitive load. This would, in turn, narrow the robot’s problem space (the system by which the robot “thinks” and solves problems) and enable better reasoning algorithms, resulting in a more intelligent robot that is less prone to errors. These recommendations together would allow for a more natural, efficient and successful user experience.

Our team’s efforts may potentially influence a new generation of military robotics designed to be safer, more natural, and therefore, more useful to soldiers in the field.


New blog post: Eyes on the Road! Or why my car should NOT be a giant smartphone on wheels: hours ago