Pen Thief

Python, Computer Vision, ROS2, MoveIt2, Interbotix Pincher PX100

Authors: Allen Liu

GitHub: View This Project on GitHub

Project Description

This project implements a vision-based robotic manipulation system that detects and retrieves a pen using computer vision and the Interbotix Pincher PX100 robot arm. The system uses color-based object detection with depth sensing for 3D localization.

System Architecture

graph TB
    subgraph Vision["Computer Vision Pipeline"]
        CAM[RealSense Camera<br/>RGB + Depth]
        HSV[HSV Color Filtering<br/>Pen Detection]
        DEPTH[Depth Processing<br/>3D Localization]
    end

    subgraph Transform["Coordinate Transformation"]
        TF[Camera to Robot TF<br/>Frame Conversion]
        POS[3D Position<br/>in Robot Frame]
    end

    subgraph Control["Robot Control"]
        API[Interbotix Control API]
        MOTION[Motion Planning]
        ARM[PX100 Robot Arm]
    end

    CAM --> HSV
    CAM --> DEPTH
    HSV --> TF
    DEPTH --> TF
    TF --> POS
    POS --> MOTION
    MOTION --> API
    API --> ARM

    style HSV fill:#e1f5ff
    style TF fill:#fff4e1
    style MOTION fill:#d4edda

System Workflow

flowchart TD
    START([System Ready]) --> CAPTURE[Capture RGB-D Image]
    CAPTURE --> FILTER[Apply HSV Color Filter<br/>Isolate Pen]

    FILTER --> DETECT{Pen<br/>Detected?}
    DETECT -->|No| CAPTURE

    DETECT -->|Yes| EXTRACT[Extract 2D Centroid]
    EXTRACT --> DEPTH_LOOKUP[Query Depth at Centroid]
    DEPTH_LOOKUP --> TRANSFORM[Transform to Robot Frame]

    TRANSFORM --> PLAN[Plan Grasp Trajectory]
    PLAN --> APPROACH[Approach Pen]
    APPROACH --> GRASP[Close Gripper]
    GRASP --> LIFT[Lift Object]
    LIFT --> RETURN[Return to Home Pose]
    RETURN --> RELEASE[Release Pen]
    RELEASE --> END([Task Complete])

    style DETECT fill:#fff4e1
    style GRASP fill:#e1f5ff
    style TRANSFORM fill:#d4edda

Technical Implementation

Computer Vision

  • HSV Color Filtering: OpenCV-based color space conversion and thresholding to segment the pen from the background
  • Depth Integration: RealSense depth sensor data fused with 2D color detection for accurate 3D position estimation

Robot Control

  • Frame Transformation: Conversion from camera coordinates to robot base frame using calibrated transformation matrix
  • Motion Execution: Interbotix control API for commanding the PX100 arm through pick-and-place sequence

Robot in Action