You are here

Robovie II helps elderly customers in a supermarket

dylan-glas's picture

Researchers at the ATR Intelligent Robotics and Communication
Laboratories in Kyoto, Japan have recently developed a proof-of-concept
system showcasing the capabilities of ubiquitous networked robot
systems for social robotics applications targeted towards assisting the

In a televised demonstration at a Japanese shopping mall, a Robovie
II robot accompanied an elderly customer through a supermarket,
carrying her groceries and reminding her to buy items on her shopping
list when they entered appropriate areas of the supermarket.  The
demonstration encompassed all stages of the interaction, including
entry of the shopping list from home and detection of the customer when
she entered the shopping mall.

This video shows all stages of the interaction:

  1. The customer enters her shopping list from home using an iPhone.
    Using a speech interface, she tells a virtual robot what she wants to
    buy.  This audio data is processed by a remote operator and entered
    into a database.
  2. The customer goes to the shopping mall. As she approaches the
    entrance, her iPhone automatically connects to the WiFi access point
    provided by the networked robot system. The system recognizes her MAC
    address and loads her shopping information.
  3. Inside the shopping mall, a network of laser range finders
    anonymously tracks the trajectories of customers.  By associating the
    signal strength of the WiFi connection with these observed
    trajectories, the location of the customer carrying the iPhone is
  4. The robot approaches the customer and offers to go shopping with
    her.  As they enter the supermarket, the robot carries a shopping
    basket to hold her groceries.
  5. The robot follows the customer through the supermarket. Whenever an
    area is entered which corresponds with an item on the shopping list,
    the robot reminds the customer to buy that item, which the customer
    places in the robot's shopping basket.

Real-World Challenges
There are many difficult recognition and control problems in
this scenario, including tracking of the customer, localization of the
robot, path planning through crowded spaces, localization in a changing
environment, and speech recognition in a noisy shopping mall. Although
proof-of-concept solutions exist for some of these problems, they do
not provide the high levels of safety required for operation alongside
real customers in a supermarket environment. For reasons of safety and
reliability, a remote operator oversees and assists the robot's

Semi-Autonomous Control
The robot control in this demonstration is semi-autonomous.
Initially, the robot autonomously patrols around the entrance to the
shopping mall, performing its own path planning with the help of the
external human position tracking system.  The robot is able to approach
the target customer and initiate conversation autonomously.

Once the robot has successfully approached the customer and
begun speaking, the human operator takes over control.  The operator
drives the robot, assists its speech recognition, and triggers its
utterances.  The user interface prompts the operator to perform these
actions, but the operator is given the final decision as to whether to
execute an action or not.

Future Vision
The scenario in this video demonstrates a vision for the future
of ubiquitous networked robot systems, where networked social robots
will provide support services for the elderly in everyday life.