Pen Thief 笔小偷

Color-based object detection with depth sensing for robotic grasping 基于颜色检测和深度感知的机器人抓取

Python, Computer Vision, ROS2, MoveIt2, Interbotix Pincher PX100

Authors: Allen Liu

GitHub: View This Project on GitHub

Project Description

This project implements a vision-based robotic manipulation system that detects and retrieves a pen using computer vision and the Interbotix Pincher PX100 robot arm. The system uses color-based object detection with depth sensing for 3D localization.

System Architecture

graph TB
    subgraph Vision["Computer Vision Pipeline"]
        CAM[RealSense Camera<br/>RGB + Depth]
        HSV[HSV Color Filtering<br/>Pen Detection]
        DEPTH[Depth Processing<br/>3D Localization]
    end

    subgraph Transform["Coordinate Transformation"]
        TF[Camera to Robot TF<br/>Frame Conversion]
        POS[3D Position<br/>in Robot Frame]
    end

    subgraph Control["Robot Control"]
        API[Interbotix Control API]
        MOTION[Motion Planning]
        ARM[PX100 Robot Arm]
    end

    CAM --> HSV
    CAM --> DEPTH
    HSV --> TF
    DEPTH --> TF
    TF --> POS
    POS --> MOTION
    MOTION --> API
    API --> ARM

    style HSV fill:#e1f5ff
    style TF fill:#fff4e1
    style MOTION fill:#d4edda

System Workflow

flowchart TD
    START([System Ready]) --> CAPTURE[Capture RGB-D Image]
    CAPTURE --> FILTER[Apply HSV Color Filter<br/>Isolate Pen]

    FILTER --> DETECT{Pen<br/>Detected?}
    DETECT -->|No| CAPTURE

    DETECT -->|Yes| EXTRACT[Extract 2D Centroid]
    EXTRACT --> DEPTH_LOOKUP[Query Depth at Centroid]
    DEPTH_LOOKUP --> TRANSFORM[Transform to Robot Frame]

    TRANSFORM --> PLAN[Plan Grasp Trajectory]
    PLAN --> APPROACH[Approach Pen]
    APPROACH --> GRASP[Close Gripper]
    GRASP --> LIFT[Lift Object]
    LIFT --> RETURN[Return to Home Pose]
    RETURN --> RELEASE[Release Pen]
    RELEASE --> END([Task Complete])

    style DETECT fill:#fff4e1
    style GRASP fill:#e1f5ff
    style TRANSFORM fill:#d4edda

Technical Implementation

Computer Vision

  • HSV Color Filtering: OpenCV-based color space conversion and thresholding to segment the pen from the background
  • Depth Integration: RealSense depth sensor data fused with 2D color detection for accurate 3D position estimation

Robot Control

  • Frame Transformation: Conversion from camera coordinates to robot base frame using calibrated transformation matrix
  • Motion Execution: Interbotix control API for commanding the PX100 arm through pick-and-place sequence

Robot in Action

Python, 计算机视觉, ROS2, MoveIt2, Interbotix Pincher PX100

作者: Allen Liu

GitHub: 在 GitHub 上查看此项目

项目描述

该项目实现了一个基于视觉的机器人操作系统，使用计算机视觉和 Interbotix Pincher PX100 机械臂检测和抓取笔。系统使用基于颜色的目标检测结合深度传感进行3D定位。

技术实现

计算机视觉

  • HSV 颜色过滤: 基于 OpenCV 的颜色空间转换和阈值处理，从背景中分割笔
  • 深度集成: RealSense 深度传感器数据与2D颜色检测融合，实现精确的3D位置估计

机器人控制

  • 坐标变换: 使用标定的变换矩阵从相机坐标转换到机器人基座坐标系
  • 运动执行: 使用 Interbotix 控制 API 命令 PX100 机械臂完成拾取放置序列

机器人演示