INTEL ATOM IMAGE SIGNAL PROCESSING (ISP) LINUX DRIVER

Last Updated On

2014-09-08

Introduction

The release is in the form of a kernel patch. Please note that the release patch is specific to BYT D0 CPU.

Prerequisites

  1. Intel Atom with ISP enabled.
  2. Imaging sensors, or cameras, attached via MIPI CSI interface for Atom ISP capturing.
  3. Supported OS:
    • Tizen3.0 with Linux kernel 3.10 LTSI (32-bit)
    • Tizen1.0 with Linux kernel 3.8 (32-bit)
    • Fedora18 with Linux kernel 3.8 (32-bit)
    • Fedora18 with Linux kernel 3.8 (64-bit)

Hardware Tested

Sensors:

Sensor Vendor Supported Max. Res. & Fps Supported Color Format
OV5640 OmniVision 1920x1080p @ 30fps (eff. 28fps) YUV422, RGB565
MT9M114 Aptina 1280x720p @ 30fps YUV422, RGB565, BA10

Interface: Intel Annville AIC via MIPI CSI connector.

Sensor Port Data Lanes Pin Count
OV5640 0 x2 40
OV5640 2 x2 40
MT9M114 1 x1 24

Building the Driver

To build the driver, you will need to patch the Linux kernel with both the ISP patch(es) provided here, and patches
from IO and graphics. The following detailed instructions give you the steps to build the kernel with all the required
patches put in:

https://intelpedia.intel.com/BYT_Kernel

While patching the IO patches, you will need to also patch ISP patch(es) to load ISP modules. Patch them like any
other patches found in the instructions.

Building the Driver as Built-in or as Loadable Modules

See “Building for Tizen3.0 with Kernel 3.10 LTSI” if you are building for kernel 3.10 LTSI.

While patching the kernel in the previous section, you may be prompted how the ISP modules to be build:

  1. If you decided to have ISP loaded as a built-in driver, press “y” and then Enter.
  2. If you decided to have ISP loaded as module, press “m”.
    • The following files will need to be loaded after a fresh boot:
      • videobuf2-core.ko (NOTE: may not be available.)
      • videobuf2-memops.ko (NOTE: may not be available.)
      • videobuf2-dma-contig.ko
      • ov5640_1.ko (Onmivision MIPI lane x2 sensor (Port 0))
      • ov5640_2.ko (Onmivision MIPI lane x2 sensor (Port 2))
      • mt9m114.ko (Aptina MIPI lane x1 sensor (Port 1))
      • atomisp.ko (Atom ISP CSS2.0 driver)

When prompted with how other drivers to be build, press “Enter” and continue.

Building for Tizen3.0 with Kernel 3.10 LTSI

The modules NEED to be build as Modules (“m”) due to an issue with UDEV not able to load the driver
as built-in.

Instead of using the git am command to apply the ISP provided patches, the normal patch -p command
is used. The command to use is

patch -p1 < [/path/to/ISP/patch/file]

The same steps applied in the previous section to build the modules.

Install ISP Firmware

  1. Copy the firmware files into /lib/firmware. The files required are:
    • iaisp_2400_css.bin.big
    • iaisp_2400_css.bin.small
  2. Reboot the machine.

Enable ISP for Linux in BIOS

When booting Bayley Bay target machine, the BIOS must be set with the right ISP PCI device.
In the BIOS settings under “Uncore Configuration” set:

  1. “ISP Enable/Disable” to Enable
  2. “ISP PCI Device” to B0D3F0

Load the Modules

If you have decided to build the ISP as modules, you will need to load them manually in
the following sequence:

  1. insmod videobuf2-core.ko (NOTE: if available)
  2. insmod videobuf2-memops.ko (NOTE: if available)
  3. insmod videobuf2-dma-contig.ko
  4. insmod ov5640_1.ko
  5. insmod ov5640_2.ko
  6. insmod mt9m114.ko
  7. insmod atomisp.ko

Check ISP Drivers

To check whether the ISP drivers were properly loaded:

  1. Type

dmesg | grep atomisp | grep success

You should see the ISP has been successfully loaded.

Atom ISP Fastboot 2-stage Firmware Load

To enable two stage load of the firmware for fastboot purposes please change the
Atomisp driver Makefile and enable ATOMISP_FASTBOOT option.

Enabling Streams with Resolution Beyond 1920x1080

For enabling streams of size greater than 1920x1080 please enable the following kernel config options:
- CONFIG_CMA
- set CMA_SIZE_MBYTES to 64MB

Supported Color Formats

The following are the supported and tested color formats by Atom ISP:

In (from Sensors to ISP) Out (from ISP to display)
YUV422 YV16
YUV422 NV12
RGB565 (RGBP) RGB565 (RGBP)
BA10 (Bayer Raw) NV12

The RGB888 (RGB3) color format is also supported, but the sensors used do not support RGB888 input.

Streaming Frames Using ISP

There are two ways to stream frames from ISP:

atomisp_testapp

This is a sample app written to stream frames from ISP. It uses the following APIs:

Please see the included atomisp_testapp directory for more details. It has README.md and README.md.html with better
details about how one can use the app.

gstreamer

Gstreamer has been tested by engineering team on Tizen 1.0. The color-format supported between GStreamer & AtomISP is
UYVY with maximum resolution of 720p. The following are sample GStreamer commands that were used for testing:

For remote streaming:

For saving a single buffer to a file on target: 

gst-launch v4l2src device="/dev/video0" num-buffers=1 ! video/x-raw-yuv,format=\(fourcc\)UYVY,width=640,height=480 ! jpegenc ! filesink location=atomisp.jpg

For gstreamer-1.x support please contact PED SW engineering team.

Known Issues and Limitations

  1. Aptina MT9M114 is a one lane (x1) sensor and thus can only support up to 1280x960 resolution.

  2. Only YV16, NV12, RGB565, and RGB888 are supported as output streams from the firmware.

  3. ISP FRAME BUFFER WIDTH 32 MULTIPLE: ISP requires a number of frame buffers for processing incoming images. The frame
    buffers can be allocated by an application and passed as buffer pointers to ISP firmware
    through AtomISP v4l2 driver. The frame memory size is calculated based on incoming image resolution width and height.
    Note that ISP frame memory width needs to be a multiple of 32.
    For example, when an incoming image width is x and height is y, the application is responsible to allocate a memory
    size:

    Frame buffer size = ( (int) x/32 + 32: 0 if x%32 > 0) * 32 * y * bytes/pixel

  4. When three simultaneous streams are running after about 20,000 frames all three streams timeout. The issue is
    suspected to be a performance issue due to memory reads in firmware and is under investigation.

  5. The Preview, or Viewfinder, is only supported with Aptina sensor on NV12 and BA10 colour format combination.

  6. Only EMGD driver dated August 5, 2014, or older, works in this release.

    • Preferred EMGD is July 22, 2014 release.

Release Notes

2014-09-08

2014-08-27

2014-08-22

2014-07-24