An Introduction to Mobile Manipulators

Robotics engineering refers to the science and process of design, development, operation, and management of robots. It combines various disciplines, including electrical, mechanical, artificial intelligence, to create machines that mimic human actions with minimum intervention. 

A typical robot has three functionalities. These components are the foundation of robotics and enable diverse applications. 

• Perception: Robots can interpret their surroundings based on the data received through sensors like cameras, LiDAR, or IMU sensors. 

• Decision-Making: Robots rely on algorithms and control systems to process the data they receive and decide what actions to take. 

• Actuation: This aspect is responsible for movement of the robot and its various parts to perform different tasks with help of different motors and mechanisms. 

Robots can be classified based on their application, motion, or functionality. Based on their end-use they can be classified as: 

• Domestic Robots: Help in various household chores and tasks  

• Industrial Robots: AMRs, robotic arm that are used in manufacturing & logistics  

• Medical Robots: Robots that are used in surgery, rehabilitation, and patient monitoring. 

• Autonomous Vehicles: Self-driving cars; drones / AMRs used for surveying and cleaning.  

Some of the key technologies that have made AMRs possible are:  

• High-end Sensors: AMRs use tools like LiDAR, image sensors, and depth sensors to interpolate their environment. 

• Artificial Intelligence and Machine Learning: AI enabled decision-making and adaptability, use of machine learning to learn and improve their performance allows AMRs to handle dynamic situations like avoiding obstacles or rerouting when the paths are blocked. 

• Simultaneous Location and Mapping: Leveraging data from an advance image sensor, this technology helps the AMR to build a map of its surrounding and move around the space.  

Mobile manipulators are complex systems that combine mobility with a robotic arm for advanced manipulation tasks. In this blog, we will discuss some key functions and the development flow of mobile manipulators. 

Typical Sequence:

• Listen and Decode Voice Command
  When a voice command is given, the robot uses advanced NLP to interpret the command and decode it into actionable tasks. 

• Object Detection Leveraging Computer Vision
  With help of camera and depth sensors AMRs can identify the object and precise location within the environment.  

• Autonomous Navigation
  With help of a map created by SLAM, the AMR navigates around obstacles and dynamically adapts to the environment, to reach the detected object. 

• Grasp and Place Object at a Particular Spot Using Precise Manipulation
  When the AMR reaches the object, its robotic arm—equipped with end-effectors for precise grasping—picks up the object. Then, with careful coordination, it places the object at the desired location. 

The above-mentioned sequence emphasizes the need of synergy between image sensors, voice recognition, machine vision, mobility, and manipulation.  

Typical Development Workflow:

Robot Design and Modelling:
There are many 3D modelling software tools that can be leveraged for designing. Fusion 360 is one of the tools that can leverage customized robot design to suit the specific requirements of a use case. 

Planning and Flow development

A typical flow includes:  

• Speech-to-text: As a first step, the AMR needs to hear the voice commands and convert it into text. This feature enables interaction between AMRs and humans. This is possible with the use of advanced speech recognition algorithms. These algorithms are capable of supporting different languages and accents. There are also additional algorithms for audio processing, and they also support noisy environments. 

• LSTM (Long Short-Term Memory): Long Short-Term Memory (LSTM) is an enhanced version of the Recurrent Neural Network (RNN), it is an enhanced version of an (ANN) artificial neural network in the field of artificial intelligence (AI) and deep learning. It governs how the system processes sequential data for better context understanding. Once the voice command is converted to text, it is processed by an LSTM model. It ensures that the robot interprets commands accurately, even when there are dependencies between words. 

• YOLO (You Only Look Once): Once the command is understood, the robot leverages YOLO to find the object in the environment. It is an innovative algorithm that identifies and locates the object in real-time. These three components work together to ensure that the system understands the user’s intent, processes it efficiently, and locates the desired objects in real time. 

• Object Tracking and Positioning Algorithm: This is the method for tracking objects in dynamic environments leveraging different algorithms. Megapose6d / 6D Pose Algorithm helps in determining the object’s position and orientation in 3D space. 

Simulation and Demo:

Based on modelling / simulation environment and practical demo, the algorithms can be refined further.  

One of the critical tasks is to control and manipulate the effector (robotics arm) to pick and drop the object. Forward kinematics is a process of calculating where the end effector of a robot is, based on the known angles or positions of the robot’s joints. Forward kinematics is based on the advanced mathematical algorithms that calculate the end position of an effector.  

Forward kinematics is important for:  

• Position identification: Ensures that the robot’s end effector reaches the correct location. 

• Path Planning and Simulation: Helps predict & visualize the robot’s movements in software before executing them physically. 

Forward kinematics calculates the end position whereas inverse kinematics helps in determining the position of different angles and joints to ensure that the end effector reaches a desired point, and a specific target position and orientation in space.  

Inverse Kinematics is important for:  

• Precise Manipulation: Instruction and steps for AMRs to move joints and angles in specific ways to make sure end effector reaches the desired target.  

• Motion Planning: The motion planning ensures that the arm moves smoothly and accurately to the desired position.  

Together, forward, and inverse kinematics is the base for motion and manipulation of AMRs ensuring smooth motion. 

To summarize, while mobile manipulators highlight enormous potential for various applications, it requires advanced software and substantial computational resources for perception, planning, and control.  

At the eInfochips Robotics center of excellence, we have developed 3 version of the autonomous mobile robots equipped with latest Slam and Navigation techniques. We have also developed digital twins for simulation and training of the AI/ML algorithms. In the next blog, we shall discuss the development challenges and key technologies for the development of mobile manipulators in detail.