In the last article, I explained the various visual sensors that you can equip on your robot. After comparing the various sensors, I decided to use the Intel Realsense D435 camera. It provides an RGB camera, a depth camera, has a perfect form factor, and comes with an officially maintained ROS wrapper.
So, I started the journey to get the D435 camera working with ROS2. The journey was long, unexpected, and ultimately failed. Using Ubuntu 20.04 LTS with latest ros2-foxy, and a combination of ros2-realsense v3.2.2 && librealsense-v2.48.0 or ros2-realsense v3.2.0 && librealsense-v2.45.0, I could not get it working. Nonetheless, I decided to document my approach in this article so that others might save time when trying to get it running.
First Approach: Installing and Running Ubuntu Packages
The first approach uses pre-build Ubuntu Packages. You will read how to install the SDK and the library, how to start the ROS2 node and the RViz visualization.
SDK and Library Installation
The official realsense-ros documentation explains how to install the librealsense SDK and the ros2-realsense ROS2 package by using prebuild packages.
Run the following commands:
sudo apt-get install ros-$ROS_DISTRO-realsense2-camera
sudo apt-get install ros-$ROS_DISTRO-realsense2-description
When its finished, connect your Realsense camera an execute dmesg. You should see an output similar to this:
[ 9985.717873] usb 1-1.3: new high-speed USB device number 3 using xhci_hcd
[ 9985.822961] usb 1-1.3: New USB device found, idVendor=8086, idProduct=0b07, bcdDevice=50.cd
[ 9985.822978] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 9985.822991] usb 1-1.3: Product: Intel(R) RealSense(TM) Depth Camera 435
[ 9985.823002] usb 1-1.3: Manufacturer: Intel(R) RealSense(TM) Depth Camera 435
[ 9985.826872] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[ 9985.827127] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[ 9985.830442] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3/1-1.3:1.0/input/input1
[ 9985.832009] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
This looks good. Let’s start the ROS nodes.
Starting the ROS2 node
The documentation lists several starting options. Let’s use the official one:
ros2 launch realsense2_camera rs_launch.py device_type:=d435
However, this command does not work and creates the following error:
realsense2_camera_node-1] 03/07 11:15:46,387 ERROR [281472753719712] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_ACCESS
[realsense2_camera_node-1] 03/07 11:15:46,387 ERROR [281472994937248] (sensor.cpp:526) acquire_power failed: failed to set power state
[realsense2_camera_node-1] 03/07 11:15:46,388 WARNING [281472994937248] (rs.cpp:306) null pointer passed for argument "device"
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[realsense2_camera_node-1] [INFO] [1625310949.972289490] [rclcpp]: signal_handler(signal_value=2)
The Github repository is well maintained and has a well-supported issue list. Searching here, I found this issue and saw the main error: The UDEV rules were missing. You can install them manually as outlined there, and then restart the ROS node.
This time, no errors popped up, and I could see see the topics.
$> ros2 topic list
/camera/color/camera_info
/camera/color/image_raw
/camera/depth/camera_info
/camera/depth/color/points
/camera/depth/image_rect_raw
/camera/extrinsics/depth_to_color
/clicked_point
/goal_pose
/initialpose
/parameter_events
/rosout
/tf
/tf_static
Visualization with RViz
When RVIZ is started, you can click on "Add", then "By Topic" and select the topic and the plugin you want to use. I added the Camera plugin to see images from the topic /camera/color/image_raw. However, as soon as I selected this topic, I could see several errors in the terminal.
[INFO] [1625329665.271601068] [rviz]: Message Filter dropping message: frame 'camera_color_optical_frame' at time 1625329664,860 for reason 'Unknown'
control_transfer returned error, index: 768, error: No data available
Searching for this RVIZ error, I found an related issue #1547. In this particular case, the solution involved using and explicit argument to reset the device during startup, and also to revert to an earlier version of the library.
The explicit reset is done with the following command:
ros2 launch realsense2_camera rs_launch.py device_type:=d435 initial_reset:=true --debug
....
[realsense2_camera_node-1] [INFO] [1625385699.954313972] [RealSenseCameraNode]: RealSense ROS v3.2.1
[realsense2_camera_node-1] [INFO] [1625385699.954474877] [RealSenseCameraNode]: Built with LibRealSense v2.45.0
[realsense2_camera_node-1] [INFO] [1625385699.954518413] [RealSenseCameraNode]: Running with LibRealSense v2.45.0
However, the problem persisted with librealsense-v2.45.0. Reading more documentation and issues on Github, I learned that the ros2-realsense v3.2.2 actually requires librealsense-v2.48.0. Since this version is not available as an Ubuntu package, I would need to manually compile it, or use ros2-realsense v3.2.0 in combination with librealsense-v2.45.0. I thought "The Newer, the Better" and started the manual installation.
Second Approach: Building, Installing and Running from Source
Librealsense SDK
The librealsense installation from source is documented on Github. The essential steps are:
- Install build essentials for your Linux system
- Clone the GitHub repository
- Run cmake
- Build and install
If you are using an Ubuntu version earlier then 20.04, you need to add additional packages and patch your Kernel. The configuration with cmake is important, you need to use appropriate cmake flags when building. For example on a Raspberry Pi it is advised to use FORCE_RSUSB_BACKEND . When you want to use the native commands for streaming realsense2 pictures, use -DBUILD_NETWORK_DEVICE=ON.
The complete compilation can take a while - on my QuadCore Linux Workstation, it was about 40 minutes.
ROS2 Realsense Package
The documentation only explains the step for ROS1, but i could figure out how to get it running on ROS1 as well.
Follow these steps ...
- install the latest Realsense SDK
- Create a ROS2 package with
ros2 pkg create --build-type ament_cmake ros2-realsense - Clone the desired ros2-librealsense v3.2.2 into the
srcfolder - Configure and compile the sources with
colcon build
... or execute these commands in your shell:
mkdir -p ~/foxy_ws/src
cd ~/foxy_ws/src/
git clone https://github.com/IntelRealSense/realsense-ros.git
cd realsense-ros/
git checkout tags/3.2.2 #Be sure to use latest tag!
cd ..
colcon build --symlink-install
It took me about two minutes to build the package.
Starting >>> realsense2_camera_msgs
[1.382s] WARNING:colcon.colcon_core.prefix_path.colcon:The path '/home/ubuntu/realsense-ros/install' in the environment variable COLCON_PREFIX_PATH doesn't exist
[1.382s] WARNING:colcon.colcon_ros.prefix_path.ament:The path '/home/ubuntu/realsense-ros/install/realsense-ros' in the environment variable AMENT_PREFIX_PATH doesn't exist
[1.383s] WARNING:colcon.colcon_ros.prefix_path.catkin:The path '/home/ubuntu/realsense-ros/install/realsense-ros' in the environment variable CMAKE_PREFIX_PATH doesn't exist
Finished <<< realsense2_camera_msgs [25.5s]
Starting >>> realsense2_camera
Starting >>> realsense2_description
Finished <<< realsense2_description [4.27s]
[Processing: realsense2_camera]
Finished <<< realsense2_camera [2min 18s]
Summary: 3 packages finished [2min 44s]
Starting the ROS2 node
The compilation was successful. We can now source the installation folder with . install/setup.bash and starting the librealsense-ros2 node.
ros2 launch realsense2_camera rs_launch.py enable_pointcloud:=true device_type:=d435 initial_reset:=true enable_sync:=true --debug
[realsense2_camera_node-1] [INFO] [1625394768.207055050] [RealSenseCameraNode]: RealSense ROS v3.2.2
[realsense2_camera_node-1] [INFO] [1625394768.207213882] [RealSenseCameraNode]: Built with LibRealSense v2.45.0
[realsense2_camera_node-1] [INFO] [1625394768.207256325] [RealSenseCameraNode]: Running with LibRealSense v2.48.0
[realsense2_camera_node-1] [WARN] [1625394768.207289788] [RealSenseCameraNode]: ***************************************************
[realsense2_camera_node-1] [WARN] [1625394768.207318417] [RealSenseCameraNode]:** running with a different librealsense version **
[DEBUG] [launch.launch_context]: emitting event synchronously: 'launch.events.process.ProcessStderr'
[DEBUG] [launch]: processing event: '<launch.events.process.process_stderr.ProcessStderr object at 0xffffa8320a60>'
[DEBUG] [launch]: processing event: '<launch.events.process.process_stderr.ProcessStderr object at 0xffffa8320a60>' ✓ '<launch.event_handlers.on_process_io.OnProcessIO object at 0xffffa8320e20>'
[realsense2_camera_node-1] [WARN] [1625394768.207345676] [RealSenseCameraNode]:** than the one the wrapper was compiled with! **
[realsense2_camera_node-1] [WARN] [1625394768.207373546] [RealSenseCameraNode]: ***************************************************
The error message confused me, but was quickly resolved: There still was ros-foxy-librealsense2 package was still installed, and during string the ros2-realsense node, it would load the older librealsense! After deinstallation and launching the node, I could continue.
ros2 launch realsense2_camera rs_launch.py device_type:=d435 initial_reset:=true --debug
[160431.153805] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[160431.174617] usb 2-1: New USB device found, idVendor=8086, idProduct=0b07, bcdDevice=50.cd
[160431.174622] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[160431.174625] usb 2-1: Product: Intel(R) RealSense(TM) Depth Camera 435
[160431.174628] usb 2-1: Manufacturer: Intel(R) RealSense(TM) Depth Camera 435
[160431.174630] usb 2-1: SerialNumber: 020223022670
[160431.180002] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[160431.181574] uvcvideo 2-1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[160431.181579] uvcvideo 2-1:1.0: Entity type for entity Processing 2 was not initialized!
[160431.181581] uvcvideo 2-1:1.0: Entity type for entity Camera 1 was not initialized!
[160431.181779] input: Intel(R) RealSense(TM) Depth Ca as /devices/pci0000:00/0000:00:15.0/usb2/2-1/2-1:1.0/input/input33
[160431.182552] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435 (8086:0b07)
[160431.183971] uvcvideo 2-1:1.3: Entity type for entity Processing 7 was not initialized!
[160431.183974] uvcvideo 2-1:1.3: Entity type for entity Extension 8 was not initialized!
[160431.183977] uvcvideo 2-1:1.3: Entity type for entity Camera 6 was not initialized!
The terminal output showed new warnings, but the node was finally up.
devcon@giga:/home/work/development/core$ ros2 node info /camera/camera
/camera/camera
Subscribers:
/parameter_events: rcl_interfaces/msg/ParameterEvent
Publishers:
/camera/color/camera_info: sensor_msgs/msg/CameraInfo
/camera/color/image_raw: sensor_msgs/msg/Image
/camera/depth/camera_info: sensor_msgs/msg/CameraInfo
/camera/depth/color/points: sensor_msgs/msg/PointCloud2
/camera/depth/image_rect_raw: sensor_msgs/msg/Image
Also, the rqt graph looked good.
Visualization with Rviz
Its time to see the results of the invested effort. Start RViz, add the camera plugin and configure it to listen to the topic /camera1/color/image_raw.
However, immediately after I could see the following message in the terminal:
Message Filter dropping message: frame 'camera_color_optical_frame'
And no data was shown in RViz.
At this time, I created the new issue #1970 on Github. Quickly, other community members reached out to me, and I learned that others were having similar issues ROS2 and the D435. And I gave up.
Conclusion
My first attempt to get the D435 running with ROS2 failed. I invested several hours searching for issues, re-reading documentation, re-compiling, and just trying different startup parameters. I could not get it to work. But because I wanted to use this particular sensor, I tried to get it running with ROS1. After not more than 2 hours, including getting and solving other error messages, I could finally see some pictures. This particular part will be explained in a future article.