C++, ROS2, Open Robotics Turtlebot3 Burger, 2D LiDar, OpenAI Gymnasium, Deep Reinforcement Learning

View This Project on GitHub

Description

In this project, I designed and implemented an OpenAI Gymnasium environment for training a Deep Reinforcement Learning model to enable autonomous navigation of a Turtlebot3 Burger through complex hallway environments. The system leverages 2D LiDAR sensors for perception and employs state-of-the-art Deep RL algorithms to learn collision-free navigation policies.

Project Overview

graph TB
    subgraph Environment["Simulation Environment"]
        TB3[Turtlebot3 Burger]
        LIDAR[2D LiDAR Sensor]
        WORLD[Hallway Environment]
    end

    subgraph RL["Reinforcement Learning"]
        GYM[OpenAI Gymnasium<br/>Custom Environment]
        AGENT[Deep RL Agent]
        POLICY[Navigation Policy]
    end

    subgraph ROS["ROS2 Integration"]
        ODOM[Odometry Node]
        CTRL[Control Node]
        SIM[Gazebo Simulation]
    end

    TB3 --> LIDAR
    LIDAR --> GYM
    WORLD --> GYM
    GYM --> AGENT
    AGENT --> POLICY
    POLICY --> CTRL
    CTRL --> TB3
    ODOM --> GYM

    style TB3 fill:#e1f5ff
    style GYM fill:#fff4e1
    style AGENT fill:#d4edda

Demo Video

The result of this project cam be shown in the video below:

Odometry Model

For this project, since the focus is the control of a turtlebot3, not localization, so I assume the perfect odometry for this project.

To get the odometry from the wheel angle and velocity, it applied kinematics model from Modern Robotics to conduct the Forward Kinematics and Inverse Kinematics of the robot.

The geometry model of the robot can be present using $H$ matrix.

For the Turtlebot3, which is a diff-drive mobile robot. The parameters are defined as

So that \(H(0)\) can be calculated as

Where \(d\) is the one-half of the distance between the wheels and the \(r\) is the radius of the wheel.

Hence \(F\) matrix can be calculated by

Forward Kinematics

The forward kinematics is to find the tranform from current state to next state \(T_{bb'}\) given the change in wheel positions \(\left( \Delta\phi_l, \Delta\phi_r\right)\).

• Find the body twist \({\cal V}_b\) from the body frame of the current state to next state:

• Construct the 3d twist by

• Then, the transform from current state to next state \(T_{bb'}\) can be calculated by

Inverse Kinematics

Inverse kinematics is to find the wheel velocities \(\left(\dot{\phi_l}, \dot{\phi_r} \right)\) given the body twist \({\cal V}_b\).

The wheel velocities can be calculated as

Since this robot is non-holonomic so that it cannot move sideways. Hence \(v_y=0\), then the wheel velocities \(\left(\dot{\phi_l}, \dot{\phi_r} \right)\) are:

OpenAI Gymnasium

To start the training pipeline of the turtlebot3, I wrapped the ROS2 turtlebot3 simulation within the OpenAI Gymnasium environment, creating a standardized interface for Deep RL algorithms.

flowchart LR
    subgraph Gymnasium["OpenAI Gymnasium Environment"]
        OBS[Observation Space<br/>x, y, θ]
        ACT[Action Space<br/>v_x, ω_z]
        REW[Reward Function]
    end

    subgraph Robot["Turtlebot3 Simulation"]
        STATE[Robot State]
        SENSORS[LiDAR Sensors]
        ACTUATORS[Wheel Actuators]
    end

    STATE --> OBS
    SENSORS --> OBS
    ACT --> ACTUATORS
    ACTUATORS --> STATE
    STATE --> REW

    style OBS fill:#e1f5ff
    style ACT fill:#fff4e1
    style REW fill:#d4edda

Observation Space

The observation space for the environment can be defined as:

Action Space

The action space can be defined as:

Reward Function

The reward structure encourages collision-free goal-reaching behavior:

• Collision penalty: -1 (when robot hits a wall)
• Goal reward: +1 (when robot reaches the target)
• This sparse reward signal requires the agent to learn exploratory behavior

Training Process

The Deep Reinforcement Learning training follows an episodic approach where the agent learns through trial and error.

stateDiagram-v2
    [*] --> Initialize: Episode Start
    Initialize --> Observe: Reset to start position
    Observe --> SelectAction: Get current state
    SelectAction --> Execute: Choose action (RL policy)
    Execute --> GetReward: Apply action to robot

    GetReward --> CheckTerminal: Evaluate outcome

    CheckTerminal --> Observe: Continue (no collision, no goal)
    CheckTerminal --> Learn: Terminal state reached

    Learn --> UpdatePolicy: Collision (-1) or Goal (+1)
    UpdatePolicy --> Initialize: Next episode

    note right of CheckTerminal
        Terminal conditions:
        - Wall collision (-1)
        - Goal reached (+1)
        - Max steps exceeded
    end note

    note right of UpdatePolicy
        Policy updated using:
        - Experience replay
        - Neural network backprop
        - TD-learning
    end note

The training process is shown in the figure below, in which every robot will be reset to its original position whenever it hits a wall.

Training Algorithm

The agent learns through the following cycle:

1. Initialization: Robot spawns at the start position
2. Observation: Collect state information (position, orientation, LiDAR data)
3. Action Selection: Neural network policy outputs velocity commands
4. Execution: Robot moves according to the selected action
5. Reward Evaluation: System checks for collision or goal achievement
6. Learning: Update policy based on reward signal
7. Reset: Upon terminal state, reset environment and repeat

Packages

• nuturtle_description: Visualizing the pose of the turtlebot3 burger
• turtlelib: Call customized library functions for kinematics and slam calculations.
• nusim: Simulate the turtlebot in the real world.
• nuturtle_control: Performing the turtlebot kinematics calculation and parsing the command sent to the turtlebot3.
• nuslam: Performs the SLAM algorithm for correcting the error odometry.
• nuturtle_interfaces: All message interfaces for communication between the nodes
• nuturtle_slam: Integrate slam_toolbox for setting up learning environment
• nuturtle_deeprl: Wrapped with OpenAI Gymnasium for Deep Reinforcement Learning traing environment