In this post we are mainly concentrating on the core concept and the benefits of an approach called RoboEarth which will be highly beneficial for the future robotic applications in science and industry. RoboEarth is a world-wide platform which robots can use to exchange position and map information as well as task-related, hardware independent action recipes. This will enable manufacturers worldwide to break down their costs and efforts for reproducing software algorithms for robot behavior over and over again. The RoboEarth framework can store all relevant data from algorithms to complex behavior descriptions that allows robots to act autonomously in an unknown, unspecified environment. Especially in the field of interaction with humans. RoboEarth can bring forward the behavior of robots and simplify the software design for developers in that field.
Today’s robots are not capable of understanding unstructured environments. The systems available rely on the specification of every eventuality a system will have to cope with in executing its tasks. Each response to a contingency has to be programmed in advance. During performing such a pre-programmed task, present robots almost completely rely on feedback. Looking at the result of its action, the robot will try to make the necessary adjustments. The worst thing about this is that the next time the robot has to perform the same action in the same environment again, it has to start all over again, build a world model from sensor data and close the feedback loop to adjust the actions needed to accomplish the task at hand. The lack of memorization and learning prevents the robot from improving its sensing and action capabilities over time. This approach faces serious limitations because the real world is generally too nuanced, too complicated and too unpredictable to be summarized within a limited set of specifications. There will inevitably be novel situations and the robotic system will always have gaps, conflicts or ambiguities in its knowledge and action instructions.
Furthermore the growing number of cheap sensors, especially networked sensors and increased resolution of sensors result in an exponential growth in sensor data. This raises the problem of extracting meaning and purpose from these bursts of sensor data. A solution to the above problems is to store the robots knowledge of the environment and the actions needed to perform its task in a global world-wide accessible database. If that particular robot or any other robot is to perform a similar action in a more or less similar environment and it has access to the stored knowledge, the robot can even if it did not perform the task before- improve on the earlier obtained sensing and action result. This is exactly what the Robo Earth approach is about: sharing knowledge between robots all over the world by building up a huge knowledge database on the world with its objects, their affordances and high level action description data (called recipes), that describe actions of the robots in general way applicable to different hardware platforms that are not completely identical in construction. This reusable knowledge of the world will provide a powerful feed forward to any robots 3D sensing, acting and learning capabilities.
The main objective of Robo Earth is to develop a system design capable of carrying out useful tasks autonomously in circumstances that were not planned for explicitly at design time. Robo Earth exploits a new approach towards endowing robots with advanced perception and action capabilities. In the core of the innovation a world-wide web-style database will be developed and the self-awareness of robot platforms will be enhanced significantly. Apart from the database an interface between the database and the systems referring to the database will be developed. By incorporating the use of the database, robotic systems are expected to perform better in the field of 3Dsensing (object recognition and localization), control strategies (linking perception and action) and learning. As a consequence applications in the field of healthcare robotics, security robotics etc. can profoundly benefit from that. The proliferation of the publicly available services on the Web is a boon for the community at large Some examples: for an example consider figure 1 of a demonstrator for human-machine interaction. As a consequence users get robots with much more abilities without teaching anything.
The Robo Earth platform can have special benefit on the industrial sector. As all interfaces are standardized and algorithms and action descriptions are saved in a general abstracted high-level language, manufacturers only have to build their robot platform with a tailored hardware-abstraction layer to the Robo Earth platform (see Figure 2).
The complete knowledge that has to be usually provided to any single robot type by the producing company can be in the best case left out as all abilities and environmental knowledge will be already accessible through the RoboEarth system. Furthermore the RoboEarth database can use all the data obtained from robots about objects, geographical positions, maps or behavior to build up a global world mode and to use this data to learn and abstract in order to create new abilities and information automatically. As an example, the object data from RoboEarth will improve the processing of 3D sensing data because the robot will know the sensor characteristics and the likely location of most objects. The actions taken by the robot will be interpreted and translated in action recipes (action descriptions), which can be linked to corresponding objects or describe general actions or algorithms. A universal language will be designed to define action recipes that are stored and exchanged via Robo Earth. Moreover, the knowledge of the world from Robo Earth will be used for the design of learning controllers, which improve the action recipes on the basis of the observed result for a particular task. The improved action recipes will be uploaded to RoboEarth again for future reuse.
RoboEarth will demonstrate rigorously, that sharing information across multiple entities and providing a repository for storing this information and extracting commonalities greatly speeds up the learning and adaptation process and allows systems to perform complex tasks and enhances the interaction capabilities between humans and robots profoundly. It is proposed to make the system self learning, through the evaluation of the system performance in executing tasks. Currently, most autonomous systems that employ adaptation and learning keep this information to themselves. What is discovered through many trials and iterations is typically discarded when the system is decommissioned. RoboEarth will serve as a repository for the learned information that can be useful to other robots/autonomous systems when performing similar tasks. This can range from control architectures to detailed algorithms for motion control to learned parameters across similar robotic platforms. So, many different types of data will be stored in the database. One abstract way of viewing what RoboEarth is trying to accomplish is to have a better balance between computation and memorization. Currently, the emphasis is mainly on computation; most research focuses on developing algorithms that solve progressively more difficult problems. There is very little emphasis on memorization and sharing information. Why should thousands of systems essentially solve the same problems over and over again? The RoboEarth platform will be a standard core around which to build dedicated applications in a lot of different fields including human-machine interaction.
III. Basic Concept
In the future, Robo Earth will contain all reusable secure knowledge of the world gathered by robots all over the world. The knowledge from Robo Earth will greatly improve each individual robots 3D-sensing ability as well as its action recipe learning ability as all available knowledge is being fed back to the robot.It registers both static and dynamic objects, information on how to recognize movable objects, situations and corresponding possible actions. The use of Robo Earth as a source for local world models to improve 3D-sensing and learning actions is a natural and highly necessary combination to deal with the real-world environment. Each robot performing a successful task in a specific part of the world will upload both its local world model and the corresponding task-related action recipes to Robo Earth. There is no principle limit to the number of robots that can be given access to this web-based register covering complete buildings, cities etc. This generates an immensely strong learning environment. An essential ability for a robot in order to create abstract action descriptions named action recipes is to recognize what kind of action it is performing or what kind of situation it is actually in. This is a rather complex task and can be described as self awareness. Normally a robot only has knowledge of a certain sequence of actions and movements to solve a task. That does not include the knowledge of what is actually executed on the robot platform. But this higher level knowledge that we want to call labeling of situations/actions is essential for the aspired vision of a worldwide shared Robo Earth world model. A robot has to know what kind of action it is performing in order to share this knowledge with others. As a consequence it certainly must have cognitive abilities. The robot has to recognize certain situations and actions it is executing and has to classify those activities by using language terms as classifiers. Natural language terms are only used for identifying recipes instead of a General Unique Identifier (GUID). This makes it easier for a human user to label certain actions when a robot is directly taught to perform certain actions. In order to implement such a system and make a robot work in an unstructured environment it will need a local world model of the environment including both static and dynamic objects, which is based on the combination of 3D-sensing data and already existing knowledge of the environment. For intermediate update rates it will be possible to write dynamic objects to the database. As Robo Earth is implemented as a database system, dynamic objects cannot be accessed from there in real time. Nevertheless, the aim is to guarantee database system update rates between 1 and 5 seconds. The object has a GUID which allows for dynamic updates. Furthermore, the fusion component can filter dynamic objects and create for example tracks out of position data which allows for predicting positions of dynamic objects. As an example, RoboEarth could feature a hypothesis-based dynamic object tracker that stores data-associated objects by using model based observers as a function of the object state, which typically contains pose and velocity information. On request, Robo Earth will return a predicted state estimate for the specified objects, which will be the input for the robots local tracker with a much higher update rate in the order of tens of Hz.
Furthermore a plan of actions for handling and manipulating these objects which are learned from 3D-sensing data and from the local world model is essential. The combination of 3D-sensing with action recipes is crucial for any useful application of robotics in unstructured environments. Each individual robot collects data through its sensors. From this data objects are detected and labeled. The measured object data is uncertain due to measurement noise, such that the data needs to be associated with real world objects, which in turn need to be tracked as a function of time. The set of both static and tracked dynamic objects constitutes the local world model that is mostly deployed on the robot itself. This local world model is published to Robo Earth for later use by other robots entering a similar environment. Robo Earth itself also deploys part of the world model, named the global world model, to manage the consistency between incoming local world model data from multiple robots and to build up probabilistic models of the initiation, continuation and finalization of objects in the environment at hand, which models in turn are used by individual robots for data association and object tracking in the local world model. Also, the existing object data from Robo Earth will improve the robots 3D sensing capabilities, because the robot will already know where to expect which objects in the sensor data. Any robot that enters a particular environment for which Robo Earth provides global world model data will exploit all available knowledge of the environment to improve its performance. A system will need to interpret the task to be executed and combine this with the information of the environment, its objects and their action affordances it receives through its 3D sensing and other sources. The task has to be broken down into executable steps: the action recipes such as grabbing objects, pushing objects etc. This process of recognition and planning of actions can also be improved through the sharing of information through Robo Earth. Robo Earth will provide methods for recognizing situations and actions based on uncertain data and afterwards translate recipes back to specific execution plans.
IV. RoboEarth Platform
Within the RoboEarth project a basic system will be developed, that is able to manage the above mentioned data types and execute basic fusion algorithms on the data.
4.1 Basic architecture
The basic architecture (see Figure 3) includes the central RoboEarth system that holds different containers for the described types of data. Especially between position data and map data there is a high level of synergy as maps can be partly constructed out of position information. A fusion component responsible for merging comparable information is highly relevant. Position information is always afflicted with a degree of uncertainty depending on the sensors and algorithms. Identical objects can have for example slightly different positions. The task of the fusion component is to eliminate such inconsistencies. Moreover there is a container for the previously defined recipes. As a lot of different robot platforms should contribute to Robo Earth there is a high plausibility that different platforms are sending recipes for the same actions, e.g. lifting a cup of tea.
The recipes are saved in a platform independent XML-based language. Consequently all kind of robots can interpret that language and transfer the information saved in the recipes to their specific hardware platform. In order to merge these recipes together or to make an appointed distinction between them a Learning & Reasoning Component‚ is used that analyzes the recipes and uses a rule base for handling with this problem. Furthermore it must be able to learn from previous results. The approach will basically rest upon logical reasoning and learning algorithms. This means that rules are defined on how to put together recipes with the same labels but with different content. Based on those rules, the system tries to merge both recipes into one common recipe. As those rules are defined in the knowledge base there is certain plausibility that they are not correct. As a result merged recipes can lead to wrong behavior. This wrong behavior is detected for example by a human who gives a feedback or by automatic observance. As a consequence the rule for merging that kind of recipe is evaluated with negative points. If a certain rule for merging recipes gets too much negative feedback it is deleted or automatically adapted. The learning approach is based on methods from the field of reinforcement learning.
The last container in the architecture can save environmental dependent knowledge. This can be dependencies between objects, time constraints, etc. As an example we can think of the process of making coffee. The coffee powder and the water have to be filled into the coffee machine before it is started. This kind of information will be saved in the Environment Dependent Knowledge Base. In order to merge these data or to make an appointed distinction a conceptual similar Learning &Reasoning Component is used like for the recipes. It also uses a rule base and must be able to learn from previous results. Since RoboEarth should be a web-based platform where robot platforms all over the world can contribute to and get information from, certain interfaces are needed. The output interface can provide information concerning positions, maps, recipes and environmental knowledge upon request. For recipes and knowledge results a language developed in Robo Earth is used. For position information and map data a comparison between already established standard interfaces like Open Street Map  or the Google Maps protocol will be made. The most convenient solution will be chosen. Although the chosen demonstrator concerns indoor use only for the sake of reducing complexity, Robo Earth is not limited to that. Robo Earth can therefore benefit from existing map protocols that can be used indoors as well as outdoors. Furthermore, Robo Earth will rather use the available data of existing databases then copying them. Implementing an external interface is used as a proof of concept for the open platform concept of Robo Earth. The same fact holds for the input interface. Robot platforms can provide recipes, position data and environment knowledge to Robo Earth using the same protocols similar to those in the output interface. See Figure for an abstract illustration of the interface concept.
V. Advancements that RoboEarth will bring about
Robots already can take on the simple logistic chores in hospitals, factories and homes. The industrial and scientific state-of-the-art in nano electronics, embedded systems and software and mechatronics has evolved so rapidly in the past years that we can actually make a next step in robot development. By increasing sensory inputs, Intelligent processing and mechatronics enabled high reactiveness, robots can start to deliver a type of support that is pro-active, sensible and human centered instead of reactive and object driven.
5.1 : 3D sensing
Current sensing systems have focused on building feature based maps from scratch and without controlling the robot motion. The map proposed in Robo Earth is composed of recognizable objects. The robot motion is based on the online acquired map, and would be guided in order to identify and locate objects in the database. The available database information is going to be exploited as appearance models prior information for producing maps composed of objects instead of features.
5.2 World modeling
There will be various advances in world modeling. First of all a fast, real-time capable multi-sensor-multi-target data association and object tracking approach that works in unstructured and dynamically changing environments with typically a large number of objects is developed. The key will be to combine reduced data association approaches such as state partitioning and pruning of the association hypotheses tree with the a prior knowledge from Robo Earth. Moreover Robo Earth will allow for better models. From the history of object data in Robo Earth, validated probabilistic models for initiation, continuation and termination of objects will be created for the purpose of data association and validated probabilistic models for dynamic model uncertainty and measurement uncertainty will be created for the purpose of tracking. Moreover, RoboEarth will provide input for the dynamic object prediction models and their initial state estimates. As a result, RoboEarth data association and tracking approach will show superior performance (quality of tracking) and robustness (insensitive to uncertainties such as temporary object occlusion and measurement or communication latency) as compared to existing approaches. As a further point a standardized object definition and description for exchange between robots and RoboEarth will be established.
5.3 Labeling and recognition of situations and actions
There will be several advances on the field of labeling and recognition of situations/actions which will profoundly improve the self-awareness capabilities of robots and their ability to interact with humans. First of all a unique interface between different kinds of robot platforms and RoboEarth is established. First of all a defined, general to use component which includes different specific approaches from the field of situation/action labeling into one common architecture usable not only on one specific robot platform but on all types of robots is developed. The usage of this interface allows creating and labeling situations/actions that are provided to RoboEarth. As a consequence not every robot has to learn every single action. It can search RoboEarth for adequate solutions for its problem and use the information found there for enhancing its abilities and cognitive skills. Furthermore the interface component will work vice versa and translates the labeled situations/actions defined in a higher-level language back to a platform specific representation. Moreover actions and situations on different robot platform can be identified as similar (e.g. grabbing a cup of tea, grabbing a cup of coffee). Consequently RoboEarth will follow a component driven approach. Basic skills do not have to be implemented over and over again, but can be easily obtained from RoboEarth; because of a unique labeling approach. Once a situation or action is labeled correctly it can be directly used by all robot platforms which are connected to RoboEarth. With a robust basic architecture and a large set of basic algorithms on the field of transferring low level to high-level data, especially the cognitive skills of a robot can be enhanced what allows developing more complex systems that are able to communicate and understand their environment. One of the most important facts is that every robot platform does not have to implement basic skills from scratch, but can get the information from RoboEarth in a representation the described component can translate back into a robot platform specific description. The result will be robots with much better cognitive skills in order to understand their environment and human beings. As a consequence they can provide much more help on different fields of applications like health care, home environment etc.
5.4 Learning controller
The learning controllers are responsible for learning new abilities on the robot system derived from already gained knowledge. Furthermore learning controllers are directly implemented in the RoboEarth system to use the data saved there to gather new information by learning with methods like data mining etc. As a consequence prior information obtained by similar robots when performing a new task is used. This will greatly improve the performance of robots when they encounter new situations, since they can leverage past information. Moreover there will be extensions to robots of varying levels of dissimilarity. Even though different types of robots will perform the same task in different ways, there will inevitably be commonalities. Methods will be developed to automatically extract the commonalities from the RoboEarth repository and thus greatly expand the impact of prior information. Also standardization is an important topic: one of the major hurdles facing the ubiquitous deployment of robots is the lack of standards. The success of the internet is in large part due to the standardized way in which one interfaces to it. While we do not assume that the methods that we will develop will lead to a single standard, they will inevitably start the standardization process, which will be iterative by nature. RoboEarth will explore more than just feed forward corrections to robot motion. There will be also no restriction to pre parameterized commonalities across different systems. These commonalities should be learned and extracted automatically from the database/repository.
In order to unify the various approaches for representing and communication knowledge about situations and actions, a language will be defined which is small enough to be not overloaded and therefore usable in an easy way. On the other hand, the language has to be extensible to be able to provide special information when needed. An important advance of this language will be the systematic and clear representation and may be a basis for further applications. Proven concepts from existing languages will be used and integrated in the concept. One promising approach is to extend the framework of XABSL. For RoboEarth, an intermediate layer will be introduced, which is able to specify not only high-level behaviors (e.g. open door), but also a fine-granular representation of action segments (e.g. approach handle, close gripper, move it down ). To integrate these action descriptions into a generalized view, probabilistic ontologies will be considered. Their advantage is the combination of the terminological clearness with the fuzziness of real-world data.
VI. CONCLUSION AND OUTLOOK
RoboEarth stores and shares world information and object action recipes that can be used by robots all over the world. It enables the development of model-based 3D-sensing and model-based learning controllers for action recipes. For successful operation in unstructured real-world environments it is absolutely necessary that robots exploit all already available knowledge of the environment. Instead of inventing the wheel from scratch all the time, resources can be used to further innovate. This will enable the step towards more advanced robot applications in unstructured environments. The simple vacuum cleaning robot will become a general purpose house cleaning robot, the static train station ticket machines will become station helper robots.
Robots will reuse and expand on each others knowledge and as a consequence also enhance the abilities to interact with humans. In the future RoboEarth can become a standard platform for robotic applications. In the project RoboEarth the basic components and demonstrators will be developed. However, a necessary condition for the success of RoboEarth will be the adoption of and contribution to RoboEarth by 3rd parties from both industry and academia.