Setup for DCS 2.0

Set up Stereo Vision System with DroneCore 2.0

Jetpack 5.1 - Orin NX - DCS 2.0 - Stereo Camera Configuration

1

Check if your system is configured for a stereo camera pair and IMX477

# Log into the device
sudo dmesg | grep ov

If the result from this command is not empty, please start at step 5.

2

Download the device tree from this link

3

Copy the new device tree and extlinux configuration from your desktop to the device (Nvidia Jetson)

scp kernel_tegra234-p3767-0000-p3509-a02.dtb \ 
[email protected]:/home/dcs_user

4

Log into the device, make backups, and apply the files

# Make backups

sudo cp /boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb \ 
/boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb.backup

# Apply new files

sudo cp kernel_tegra234-p3767-0000-p3509-a02.dtb \
/boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb

5

Turn off the device and connect the cameras to the CAM0 and CAM1 ports

6

Turn on the device and check if the patch was applied.

sudo dmesg | grep ov

If everything was applied successfully, you should see an output similar to this:

[   14.472618] ov9281_devel 1-0060: probing ov9281 v4l2 sensor at addr 0x60
[   14.472749] ov9281_devel 1-0060: tegracam sensor driver:ov9281_v2.0.6
[   14.541325] ov9281_devel 1-0060: ov9281 camera configured in free-run mode 
[   14.542460] ov9281_devel 1-0060: ov9281 sensor ID matches
[   14.545845] ov9281_devel 1-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   14.545850] ov9281_devel 1-0060: new driver version
[   14.553355] ov9281_devel 1-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   14.553365] ov9281_devel 1-0060: new driver version
[   14.553383] tegra-camrtc-capture-vi tegra-capture-vi: subdev ov9281_devel 1-0060 bound
[   14.557244] ov9281_devel 1-0060: detected ov9281 sensor
[   14.557428] ov9281_devel 2-0060: probing ov9281 v4l2 sensor at addr 0x60
[   14.557571] ov9281_devel 2-0060: tegracam sensor driver:ov9281_v2.0.6
[   14.625288] ov9281_devel 2-0060: ov9281 camera configured in free-run mode 
[   14.625671] ov9281_devel 2-0060: ov9281 sensor ID matches
[   14.626745] ov9281_devel 2-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   14.626748] ov9281_devel 2-0060: new driver version
[   14.628103] ov9281_devel 2-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   14.628110] ov9281_devel 2-0060: new driver version
[   14.628135] tegra-camrtc-capture-vi tegra-capture-vi: subdev ov9281_devel 2-0060 bound
[   14.629226] ov9281_devel 2-0060: detected ov9281 sensor

7

To display the camera stream, please follow these steps. This guide uses graphics forwarding via USB, so some interruptions or inconsistencies in the streams might occur

# Connect to the device via SSH -X
ssh -X [email protected]

# Install additional dependencies for setting camera mode and parameters
sudo apt update
sudo apt install v4l-utils

# Stream from video0 config mode 0 1280x800
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=0

gst-launch-1.0 -v v4l2src device="/dev/video0" ! "video/x-raw,width=1280,height=800,format=(string)GRAY8" ! videoconvert ! videoscale ! "video/x-raw,width=640,height=400" ! ximagesink sync=false

# You can tweak the camera's exposure and gain after it is running with these commands
v4l2-ctl -d /dev/video0 --set-ctrl exposure=20000
v4l2-ctl -d /dev/video0 --set-ctrl gain=50

# Stream from video0 config mode 2 640x400
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=2

gst-launch-1.0 -v v4l2src device="/dev/video0" ! "video/x-raw,width=640,height=400,format=(string)GRAY8" ! videoconvert ! videoscale ! "video/x-raw,width=640,height=400" ! ximagesink sync=false

# To display the other camera, please use the guide with video1

# GStreamer and v4lsrc are standard video mechanisms in Linux; it is possible to find applications or scripts utilizing these programmatically. Additionally, OpenCV supports streaming from v4l2src.

Jetpack 5.1 - Orin NX - DCS 2.0 - Stereo Camera Configuration in Trigger Mode

1

Check if your system is configured for a stereo camera pair and IMX477

# Log into the device
sudo dmesg | grep ov

If the result from this command is not empty, please start at step 5. Check if the output contains: ov9281 camera configured in fsync slave-trigger mode. If yes, start at step 5

2

Download the device tree from this link

3

Copy the new device tree and extlinux configuration from your desktop to the device (Nvidia Jetson)

scp kernel_tegra234-p3767-0000-p3509-a02-trigger.dtb \ 
[email protected]:/home/dcs_user

4

Log into the device, make backups, and apply the files

# Make backups

sudo cp /boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb \ 
/boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb.backup

# Apply new files

sudo cp kernel_tegra234-p3767-0000-p3509-a02-trigger.dtb \
/boot/dtb/kernel_tegra234-p3767-0000-p3509-a02.dtb

5

Turn off the device and connect the cameras to the CAM0 and CAM1 ports

6

Turn on the device and check if the patch was applied.

sudo dmesg | grep ov

If everything was applied successfully, you should see an output similar to this:

[   13.526070] ov9281_devel 1-0060: probing ov9281 v4l2 sensor at addr 0x60
[   13.526213] ov9281_devel 1-0060: tegracam sensor driver:ov9281_v2.0.6
[   13.595492] ov9281_devel 1-0060: ov9281 camera configured in fsync slave-trigger mode 
[   13.596614] ov9281_devel 1-0060: ov9281 sensor ID matches
[   13.602362] ov9281_devel 1-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   13.602368] ov9281_devel 1-0060: new driver version
[   13.607409] ov9281_devel 1-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   13.607416] ov9281_devel 1-0060: new driver version
[   13.607431] tegra-camrtc-capture-vi tegra-capture-vi: subdev ov9281_devel 1-0060 bound
[   13.608415] ov9281_devel 1-0060: detected ov9281 sensor
[   13.608570] ov9281_devel 2-0060: probing ov9281 v4l2 sensor at addr 0x60
[   13.608759] ov9281_devel 2-0060: tegracam sensor driver:ov9281_v2.0.6
[   13.676446] ov9281_devel 2-0060: ov9281 camera configured in fsync slave-trigger mode 
[   13.676832] ov9281_devel 2-0060: ov9281 sensor ID matches
[   13.678912] ov9281_devel 2-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   13.678920] ov9281_devel 2-0060: new driver version
[   13.680322] ov9281_devel 2-0060: ov9281_set_frame_rate: camera fps set to: 5  => TIMING_VTS: 28609
[   13.680328] ov9281_devel 2-0060: new driver version
[   13.680354] tegra-camrtc-capture-vi tegra-capture-vi: subdev ov9281_devel 2-0060 bound
[   13.682361] ov9281_devel 2-0060: detected ov9281 sensor

7

Set up system rules to have access to the device as a user

if [ ! -f /etc/udev/rules.d/61-jetson-common.rules ] ; then
    sudo touch /etc/udev/rules.d/61-jetson-common.rules
    echo 'KERNEL=="gpiochip*", SUBSYSTEM=="gpio", MODE="0660", GROUP="gpio"' | sudo tee -a /etc/udev/rules.d/61-jetson-common.rules
    echo 'KERNEL=="i2c*", SUBSYSTEM=="i2c", MODE="0660", GROUP="i2c"' | sudo tee -a /etc/udev/rules.d/61-jetson-common.rules
    echo 'KERNEL=="/dev/ttyTHS0*", SUBSYSTEM=="dialout", MODE="0660", GROUP="dialout"' | sudo tee -a /etc/udev/rules.d/61-jetson-common.rules
fi

sudo usermod -a -G i2c dcs_user
sudo usermod -a -G gpio dcs_user
sudo usermod -a -G dialout dcs_user
sudo udevadm control --reload-rules && udevadm trigger

# Reboot the device to properly apply changes

8

Setup and start hardware triggering

Install libgpiod on the device:

sudo apt update
sudo apt install automake libtool autoconf-archive v4l-utils build-essential

git clone git://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git

cd libgpiod
./autogen.sh --enable-tools=yes --enable-bindings-cxx --prefix=/usr/local
make -j4
sudo make install
sudo ldconfig

Start hardware triggering:

sudo gpioset --daemonize -t3ms,30ms PN.01=1

Please note that the triggering should be set to the required frequency (e.g., for 30FPS configuration -t3ms,30ms; for 20FPS -t5ms,45ms)

9

Verify the triggering configuration

v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=0
v4l2-ctl -d /dev/video1 --set-ctrl sensor_mode=0

# One camera
gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,width=1280,height=800,framerate=30/1,format=(string)GRAY8" ! fpsdisplaysink video-sink=fakesink --verbose

# Second camera

10

With this command, it is possible to list supported camera configurations

You can adjust the timing of gpioset and the framerate=30/1 parameter in gstreamer pipelines. The last interval is only a theoretical maximum; currently, this FPS is not supported.

v4l2-ctl --list-formats-ext

11

To display the camera stream, please follow these steps

This guide uses graphics forwarding via USB, so some interruptions or inconsistencies in the streams might occur

```
# Connect to the device via SSH -X
ssh -X [email protected]

# Install additional dependencies for setting camera mode and parameters
sudo apt update
sudo apt install v4l-utils

# Stream from video0 config mode 0 1280x800
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=0

gst-launch-1.0 -v v4l2src device="/dev/video0" ! "video/x-raw,width=1280,height=800,format=(string)GRAY8,framerate=30/1" ! videoconvert ! videoscale ! "video/x-raw,width=640,height=400" ! ximagesink sync=false

# You can tweak the camera's exposure and gain after it is running with these commands
v4l2-ctl -d /dev/video0 --set-ctrl exposure=20000
v4l2-ctl -d /dev/video0 --set-ctrl gain=50

# Stream from video0 config mode 2 640x400
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=2

gst-launch-1.0 -v v4l2src device="/dev/video0" ! "video/x-raw,width=640,height=400,format=(string)GRAY8,framerate=30/1" ! videoconvert ! videoscale ! "video/x-raw,width=640,height=400" ! ximagesink sync=false

# To display the other camera, please use the guide with video1

# GStreamer and v4lsrc are standard video mechanisms in Linux; it is possible to find applications or scripts utilizing these programmatically. Additionally, OpenCV supports streaming from v4l2src.
```