Picamera taking pictures fast and processing them

Question

I'm working on a school project where I have to make a robot move parallel to a wall and stop when it sees a big red square. For this I have the picamera module and the openCV library. I have working code but the camera takes too long to take a picture (3-4 seconds every time). The analyzing is done in a few milliseconds but I need to take pictures faster. The working code is this:

@staticmethod
def detect_red():
    # saving the picture to an in-program stream rather than a file
    stream = io.BytesIO()

    # to speed things up, lower the resolution of the camera
    camera_width = 320
    camera_height = 240
    #scale_down = 6
    red = False

    with picamera.PiCamera() as camera:
        camera.resolution = (camera_width, camera_height)
        # capture into stream
        camera.capture(stream, format='jpeg')
    # convert image into numpy array
    data = np.fromstring(stream.getvalue(), dtype=np.uint8)
    # turn the array into a cv2 image
    img = cv2.imdecode(data, 1)

    # Resizing the image, blur the image and convert it to HSV values for better recognition
    # img = cv2.resize(img, (len(img[0]) / scale_down, len(img) / scale_down))
    # img = cv2.GaussianBlur(img, (5,5), 0)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

    #Defining the red color range and calculating if these values lie in the range
    red_lower = np.array([0, 150, 0], np.uint8)
    red_upper = np.array([5, 255, 255], np.uint8)
    red_binary = cv2.inRange(img, red_lower, red_upper)

    # Dilates the red space, making it larger
    dilation = np.ones((15, 15), "uint8")
    red_binary = cv2.dilate(red_binary, dilation)

    contours, _ = cv2.findContours(red_binary, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

    if not contours == []:
        if not red:
            red = True
            print "Red surface detected!"
    else:
        print "No red surface  detected."

    return red

Now I've been trying to get a faster method using capture_continuous and I based my code on: take images in a short time using the Raspberry Pi camera module

def outputs(self):
    stream = io.BytesIO()
    for i in range(40):
        red = False
        data = np.fromstring(stream.getvalue(), dtype=np.uint8)
        img = cv2.imdecode(data, 1)
        print "a"
        img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
        red_lower = np.array([0, 150, 0], np.uint8)
        red_upper = np.array([5, 255, 255], np.uint8)
        red_binary = cv2.inRange(img, red_lower, red_upper)
        dilation = np.ones((15, 15), "uint8")
        red_binary = cv2.dilate(red_binary, dilation)

        contours, _ = cv2.findContours(red_binary, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

        if not contours == []:
            if not red:
                red = True
                print "Red surface detected!"
        else:
            print "No red surface  detected."


        stream.seek(0)
        stream.truncate()

def video_red(self):
    with picamera.PiCamera() as camera:
        camera.resolution = (320,240)
        camera.framerate = 5
        camera.start_preview()
        time.sleep(2)      
        camera.capture_continuous(self.outputs(), 'jpeg', use_video_port=True)

This is the new code but I keep getting openCV Error: Assertion failed (scn == 3 || scn == 4) in the cv2.cvtColor method. How can I fix this? I'm also not sure if what I'm doing with the stream is correct. Any help would be appreciated.

score 5 · Answer 1 · answered Nov 03 '14 at 13:57

Sounds like a fascinating project!

I'm going to ignore the second script for now as it's a bit confused; but there's plenty that can be done to speed up your first script, and it's probably more useful to concentrate on improving that as you've obviously had it working before (albeit too slowly). So, first things first:

If I weren't sure where the script was spending too much time, I'd profile it. There are several tools built into to Python for this and some nice GUIs for visualizing the results too. However, if you don't want to profile everything (say you've got a hunch as to where the slow-down is), you can simply use time.time() around statements to figure out how long they're taking. For example:

import picamera
import time

# How long does camera initialization take?
start = time.time()
with picamera.PiCamera() as camera:
    pass
print(time.time() - start)

In this particular case, though, I can take a pretty good guess where the major slow-downs are! Firstly, camera initialization takes a long time (at least a second as you'll see if you try the script above). So ideally you want to initialize the camera once in your script (outside your detect_red function) instead of doing it again and again.

Next, you probably want to have a read of the sections in the picamera manual about rapid capture; ignore the later bits about threading and all that complex stuff - the important thing is that using the video port to capture images is much faster than using the still port (because it doesn't involve a mode switch; you can read about the camera's various ports in the hardware chapter). So, the only change necessary here is to add use_video_port=True to your capture call.

With these two simple changes the first script now looks like this:

def detect_red(camera):
    # saving the picture to an in-program stream rather than a file
    stream = io.BytesIO()

    #scale_down = 6
    red = False

    # capture into stream
    camera.capture(stream, format='jpeg', use_video_port=True)
    # convert image into numpy array
    data = np.fromstring(stream.getvalue(), dtype=np.uint8)
    # turn the array into a cv2 image
    img = cv2.imdecode(data, 1)

    # Resizing the image, blur the image and convert it to HSV values for better recognition
    # img = cv2.resize(img, (len(img[0]) / scale_down, len(img) / scale_down))
    # img = cv2.GaussianBlur(img, (5,5), 0)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

    #Defining the red color range and calculating if these values lie in the range
    red_lower = np.array([0, 150, 0], np.uint8)
    red_upper = np.array([5, 255, 255], np.uint8)
    red_binary = cv2.inRange(img, red_lower, red_upper)

    # Dilates the red space, making it larger
    dilation = np.ones((15, 15), "uint8")
    red_binary = cv2.dilate(red_binary, dilation)

    contours, _ = cv2.findContours(red_binary, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

    if not contours == []:
        if not red:
            red = True
            print "Red surface detected!"
    else:
        print "No red surface  detected."

    return red


with picamera.PiCamera() as camera:
    # to speed things up, lower the resolution of the camera
    camera.resolution = (320, 240)
    detect_red(camera)
    detect_red(camera)
    detect_red(camera)
    detect_red(camera)
    # and so on...

You should find repeated calls to detect_red much faster in this version. If it's fast enough, great! If it's not there's another trick you can use to speed things up which is to avoid JPEG encoding; at the moment, your script is capturing an image from the camera, having the camera encode it as a JPEG, then passing that JPEG to OpenCV to decode back into RGB values before processing it (OpenCV's image format is just a numpy array in BGR order). It would be faster to avoid all that encoding and decoding by having the camera return the RGB values directly.

Good luck!

score 3 · Answer 2 · answered Nov 03 '14 at 19:57

You should also be able to directly set the size of the captured image, instead of doing a resize later; this will save a good chunk of compute time. Do this by setting: camera.resolution = (1024, 768) for whatever esolution you actually want.

Another trick which might be helpful would be just using the "red" channel of the RGB image as a greyscale image. Your captured image should be a numpy array with a row for each of the three color channels; you can just use the first channel as a greyscale image and save yourself some data processing.

score 1 · Answer 3 · edited Apr 13 '17 at 12:56

1

I have use this code for cv2.cvtColor.

https://raspberrypi.stackexchange.com/questions/24262/getting-image-data-from-raspberry-pi-camera-module-in-opencv-with-python#= http://picamera.readthedocs.org/en/latest/api_array.html#pirgbarray

import cv2
import picamera
import picamera.array
import time

with picamera.PiCamera() as camera:
    with picamera.array.PiRGBArray(camera) as stream:
        while True:
            camera.capture(stream, format='bgr')
            image = stream.array
            gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
            cv2.imshow('frame', gray)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
            # reset the stream before the next capture
            stream.seek(0)
            stream.truncate()
        cv2.destroyAllWindows()

edited Apr 13 '17 at 12:56

Community

1

answered Apr 08 '15 at 08:08

Philipz

11
2

Cool. What does it do? What are the benefits? Why do you prefer this one? – Bex Apr 08 '15 at 11:56
can you explain why this one? – thekiwi5000 Apr 08 '15 at 17:31
The new version of Picamera has solve Picamera array lib. The picamera.array module provides a set of classes which aid in constructing n-dimensional numpy arrays from the camera output. Don't forget the stream.seek(0) and stream.truncate() ~~~ – Philipz Apr 09 '15 at 09:10
Capturing to an OpenCV object – Philipz Apr 09 '15 at 09:24

Picamera taking pictures fast and processing them

3 Answers3

Linked