Streaming Cameras on Embedded Systems

System Configuration

These boards do not support Power over Ethernet (PoE), therefore all GigE models were powered externally via the GPIO. All GS3-U3 models were powered externally via the GPIO. The CM3 model was powered via the USB3 interface.

 

	ODROID-XU Board Specification
CPU	Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core
GPU	PowerVR SGX544MP3
RAM	2 Gbyte LPDDR3
Host Adaptor Driver	exynos-xhci
Operating System	Ubuntu 13.10 with 3.4.67 linux kernel
Software	FlyCap 2.6.3.3 Linux ARM
Camera and Firmware	GS3-U3-23S6M (firmware 2.3.3.0)

 

	Samsung Exynos 5250 Arndale Specification
CPU	Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core
GPU	ARM Mali T-604
RAM	2 Gbyte DDR3
Hard Drive	SanDisk Extreme II SSD 480GB (SDSSDXP-480G)
Hard Drive Enclosure	Oyen Digital MiniPro 2.5-inch eSATA 6 Gb/s, USB 3.1 hard drive enclosure
Host Adaptor Driver	xhci-hcd
Operating System	Arndale raring server release 13.09 with 3.12 Linux kernel
Software	FlyCap 2.6.3.3 Linux ARM
Camera and Firmware	GS3-U3-23S6M (firmware 2.3.3.0)

 

	NVIDIA Jetson TK1 Specification
CPU	Tegra K1 SOC
GPU	Kepler GPU with 192 CUDA cores
RAM	2 Gbyte
Host Adaptor System	xhci-hcd
Operating System	Ubuntu 14.04
Software	FlyCap 2.6.3.3 Linux ARM
Camera and Firmware	GS3-U3-23S6M (firmware 2.3.3.0) GS3-PGE-23S6C (firmware 1.6.3.0) FL3-GE-03S1M (firmware 1.26.3.0)

 

	NVIDIA Jetson TX1 Specification
CPU	Cortex A57
GPU	Maxwell GPU with 256 CUDA cores
RAM	4 Gbyte
Host Adaptor System	xhci-hcd
Operating System	Ubuntu 14.04
Software	FlyCap 2.9.3.43 Linux ARM
Camera and Firmware	GS3-U3-23S6M (firmware 2.22.3.0) GS3-PGE-23S6M (firmware 1.10.3.0)

 

	NVIDIA Jetson TX2 Specification
CPU	Cortex A57
GPU	Pascal GPU with 256 CUDA cores
RAM	8 Gbyte
Host Adaptor System	xhci-hcd
Operating System	Ubuntu 16.04
Software	FlyCap 2.11.3.121 Linux ARM
Camera and Firmware	GS3-U3-23S6M (firmware 2.25.3.0) GS3-PGE-23S6M (firmware 1.13.3.0) BFLY-PGE-23S6M (firmware 1.61.3.0)

 

	NVIDIA DRIVE PX Specification
CPU	Tegra X1 64-bit processor
GPU	Maxwell GPU with 256 CUDA cores
RAM	4 Gbyte
Host Adaptor System	xhci-hcd
Operating System	Ubuntu 15.04 with 3.18.21-tegra-g06aec38 Linux NVIDIA kernel
Software	FlyCap 2.9.3.43 Linux ARM64
Camera and Firmware	BFLY-PGE-50S5C-C (firmware 1.53.3.0) BFLY-PGE-23S6M-C (firmware 1.40.3.0) CM3-U3-50S5M-CS (firmware 1.10.3.0)
	Note: To achieve maximum bandwidth: GigE cameras require a Gigabit adapter connected directly to an on-board USB port. USB3 cameras require a custom kernel as described in NVIDIA DRIVE PX Getting Started.

  

 

System Setup

ODROID-XU Getting Started

1. To boot from the microSD card, ensure boot media SW1-1 is set to OFF and SW2-2 is set to ON.

2. Download xUbuntu 13.10 image on a Linux PC running Ubuntu.

3. Unpack the files. For an .xz extension, use xz -d image.img.xz on cmd line.

4. Insert the microSD card in the Ubuntu PC.

Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.

5. Clear the destination media by entering dd if=/dev/zero of=/dev/sdX bs=4M

Note: /dev/sdX where X denotes the microSD card’s file storage.

6. Flash your image: sudo dd if=image.img of=/dev/sdX bs=4M

7. Enter sync before removing the media to make sure that all data is written.

8. Download FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).

9. Retrieve dependency using command sudo apt-get install libglademm-2.4-dev

10. Extract FlycCapture2 ARM hard float package and run flycap2-conf as root.

11. Reboot ODROID-XU.

12. In the terminal, change to FlyCapture2 ARM hard float directory’s bin folder.

13. Run the command export LD_LIBRARY_PATH=./../lib

14. Start FlyCapture2 using command ./FlyCap2

Samsung Exynos 5250 Arndale Getting Started

1. Download the Linaro image for Arndale. This configuration used Version 13.11.

2. Insert the microSD card in the Ubuntu PC.

3. Write the image to a microSD card.

Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.

4. Download the FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).

5. Extract the Flycapture2 ARM hard float package and run flycap2-conf as root.

6. Reboot.

7. In the terminal, change to Flycapture2 ARM hard float directory’s bin folder.

8. Run the command export LD_LIBRARY_PATH=./../lib

9. Start FlyCapture2 using command ./FlyCap2

NVIDIA Jetson TK1 Getting Started

Make sure you have at least 15 GB of free space on your Linux machine. This file copying process takes approximately one hour. The flashing process ends with 'Reset the board to boot from internal eMMC.'

1. Follow NVIDIA’s readme file and boot up the pre-installed Ubuntu 14.04 on TK1.

2. By default, Jetson TK1 does not provide support for USB3 cameras. To enable USB3 support:

a. Go to https://developer.nvidia.com/linux-tegra-rel-19 and download both of these tar-balls:

• Driver Package: Jetson TK1

• Sample file system

b. On the Linux PC, unzip the tar files by running the command:

export RELEASE_NAME=Tegra124_Linux_R19.3.0_armhf.tbz2
sudo tar xpf ${RELEASE_NAME}
cd Linux_for_Tegra/rootfs/
sudo tar xpf ../../Tegra_Linux_Sample-Root-Filesystem_R19.3.0_armhf.tbz2
cd ../

Note: It’s important to use sudo to extract these files.

c. In the extracted Linux_for_Tegra folder, edit the jetson-tk1.conf file by uncomment the line (ODMDATA=0x6209C000) for USB2 and comment in the line for USB3 (ODMDATA=0x6009C000).

d. Enter sudo ./apply_binaries.sh

e. Connect the TK1 to the Linux machine using the micro-USB cable that comes with the TK1.

f. On the TK1, hold down the Recovery button and press the Reset button once. Wait two seconds and release the Recovery button.

g. Verify that the TK1 is connected to the Linux PC by issuing the lsusb command on the Linux PC and look for a connected usb device called NVIDIA Corp.

h. Copy the Linux_for_Tegra folder unzipped in step 2 to your Linux machine.

i. In the Linux machine, enter sudo ./flash.sh -S 14GiB jetson-tk1 mmcblk0p1

3. Reboot TK1 by pressing the Reset button.

The steps above will allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000


to this file:

/boot/extlinux/extlinux.conf


NVIDIA Jetson TX1 Getting Started

1. Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
   Note: This step overwrites your existing file system.
2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.
3. Install the g++ compiler in order to compile the FlyCapture2 SDK’s C++ samples

sudo apt-get update
sudo apt-get install build-essential

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

NVIDIA Jetson TX2 Getting Started

1. Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
   Note: This step overwrites your existing file system.
2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.

The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:

usbcore.usbfs_memory_mb=1000

to this file:

/boot/extlinux/extlinux.conf

NVIDIA DRIVE PX Getting Started

To obtain maximum bandwidth with DRIVE PX:

• USB3 cameras require the latest version of a custom kernel from GitHub, as described below.
• GigE cameras require a USB 3.1-to-Ethernet adapter connected directly to an on-board USB port.

For USB3 cameras:

1. Get the latest kernel from GitHub:
   
   # Clone source repo for mainline kernel
git clone https://github.com/torvalds/linux.git
# Get list of tags to determine appropriate release (I settled on v4.7.0-rc7 for specific technical reasons).
git tag -l

# Checkout specific tagged revision to local branch.
# git checkout tags/<tag_name> -b <branch_name>
git checkout tags/v4.7-rc7 -b nv_custom_branch
   
   
2. To acquire images greater than 2 MB in resolution, modify the os_args parameter of the KERNEL_PRIMARY partition in:
   
   vibrante-<platform|ver>-foundation/utils/scripts/bootburn/<cfg_file>
   
   Add the following to os_args:
   
   usbcore.usbfs_memory_mb=1000

3. After the kernel is built, flash the custom kernel onto the board. Because the process depends on the board and kernel version, consult the NVIDIA Vibrante Linux Development guide at /home/nvidia/PDK_Docs/ for specific instructions.
   Note: You only want to re-flash the kernel, not the complete file system. To avoid flashing the complete file system, add the "-s" option (skip filesystem) to the bootburn.sh command:

   "./utils/scripts/bootburn/bootburn.sh -b e2379b00a -s"

   The sample in the documentation does not use the -s.

For GigE and USB3 cameras:
(Note: Connect the GigE camera to the PC with a USB-to-Ethernet adapter.)

1. Install the g++ compiler in order to install the FlyCapture2 SDK's C++ samples:
   
   sudo apt-get update
sudo apt-get install build-essential
   
   
2. Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the README file for installation instructions.

 

Benchmark Results

The following tables benchmark the performance of running selected USB3 or GigE cameras on the following boards: ODROID-XU; Samsung Exynos 5250 Arndale, and NVIDIA Jetson TK1 sand TX1, and DRIVE PX. 

ODROID-XU

These results are from a console application that continuously captures images using GS3-U3-23S6M. The benchmark using the console application is shown below. 

 

Console application for ODROID-XU

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1280 x 960 Raw8	200 FPS	200 FPS	22%	234 MB/s
1792 x 960 Raw8	200 FPS	200 FPS	28%	328 MB/s

GS3-U3-23S6M benchmark results

Currently, ODROID-XU officially supports up to Linux kernel 3.4.67. Expect the CPU usage on the ODROID-XU board to reduce by 50% when running Linux kernel 3.6 or higher.

Samsung Exynos 5250 Arndale

Using FlyCapture2 to enumerate camera

 

These are the results from a console application that continuously captures images using GS3-U3-23S6M.

 

 

Console application for Arndale

This is a typical solution for an in-field system solution where the embedded board is portable and capable of performing basic processing as well as image/data saving. The benchmark results using the console application are shown below. 

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth	Image Save Speed
1280 x 960 Raw8	200 FPS	200 FPS	24%	234 MB/s	190 MB/s
1792 x 960 Raw8	200 FPS	200 FPS	28%	328 MB/s	170 MB/s
1920 x 1200 Raw8	162 FPS	162 FPS	29%	356 MB/s	176 MB/s

GS3-U3-23S6M benchmark results

 

GS3-U3-23S6M running at 1280 x 960 resolution, 24% CPU usage

GS3-U3-23S6M running at 1280 x 960 resolution, Raw8 pixel format, approximately 200 FPS

The speed of image saving is dependent on the hard drive. Typical hard drive write speed is advertised with sequential write speed. When benchmarking hard drives, one should use a file size similar to the image size.

GS3-U3-23S6M running at 1280 x 720 resolution, Raw8 pixel format, 24 FPS

NVIDIA Jetson TK1 

These are the results from a console application that continuously captures images using GS3-U3-23S6M, GS-PGE-23S6C, and FL3-GE-03S1M. The resulting benchmark using the console application is shown below. 

Console application for NVIDEA Jetson TK1

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1280 x 960 Raw8	200 FPS	200 FPS	24%	234 MB/s
1920 x 1200 Raw8	162 FPS	162 FPS	30%	356 MB/s

GS3-U3-23S6M benchmark results

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Raw8	45 FPS	45 FPS	25%	103 MB/s

GS3-PGE-23S6C benchmark results

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
640 x 480 Raw8	120 FPS	120 FPS	24%	36 MB/s

Flea3-GE-03S1M benchmark results

 

NVIDIA Jetson TX1 

These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M and Grasshopper3 GS-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.

Console application for NVIDEA Jetson TX1

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1280 x 960 Raw8	201 FPS	201 FPS	27%	234 MB/s
1920 x 1200 Raw8	163 FPS	163 FPS	27%	356 MB/s

GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Raw8	45 FPS	45 FPS	30%	104 MB/s

GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX1

NVIDIA Jetson TX2 

These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M, Grasshopper3 GS-PGE-23S6M, and Blackfly BFLY-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, for GigE cameras we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.

Console application for NVIDEA Jetson TX1

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Raw8	163 FPS	163 FPS	25%	375 MB/s

GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Raw8	45 FPS	46 FPS	25%	105 MB/s

GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX2

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Raw8	41 FPS	41 FPS	25%	94 MB/s

BFLY-PGE-23S6M benchmark result using NVIDIA Jetson TX2

NVIDIA DRIVE PX 

These are the results from a console application that continuously captures images using BLFY-PGE-23S6M, BFLY-PGE-50SM-C and CM3-U3-50SM. The resulting benchmark using the console application is shown below.

Console application for NVIDEA DRIVE PX

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
2448 x 2048 Mono16	18 FPS	18 FPS	80%	85 MB/s

BFLY-PGE-23S6M benchmark results

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
2448 x 2048 Mono16	9 FPS	9 FPS	80%	92 MB/s

BFLY-PGE-5056M benchmark results

 

	Requested Frame Rate	Processed Frame Rate	CPU Usage	Bandwidth
1920 x 1200 Mono16	19 FPS	19 FPS	40%	23 MB/s

CM3-U3-5056M benchmark results

 

Troubleshooting

Image Transfer Fails to Start when Image Size is Bigger than 2 MB

Cause: The USBFS buffer size is too small (16 MB by default). Use the following command to check usbfs_memory_mb size.

cat /sys/module/usbcore/parameters/usbfs_memory_mb

Solution: Increase the memory by adding the following to the APPEND line:

usbcore.usbfs_memory_mb=1000
to this file:
/boot/extlinux/extlinux.conf

If this method fails to set the memory limit, run the following command:
sudo sh -c 'echo 1000 > /sys/module/usbcore/parameters/usbfs_memory_mb'

Camera is Detected but Cannot Stream

Cause: The USB3 bus power provided by the ARM board is not sufficient to power the camera.

Solution: Power the USB3 camera using a GPIO cable.

GigE Camera Frame Rates are extremely low on Jetson TK1, TX1, or TX2 board

Cause: The socket buffer size by default is too low.

Solution: Increase socket buffer size by typing in the following in Terminal:

sudo sysctl –w net.core.rmem_max=33554432
sudo sysctl –w net.core.wmem_max=33554432
sudo sysctl –w net.core.rmem_default=33554432
sudo sysctl –w net.core.wmem_default=33554432