4

Can a system consisting of 10x Raspberry Pi's, 10x RPi Camera Modules + WiFi dongles be reliably & consistently synchronized over WiFi to take photos at the same time? Or are radio triggers a better option for simultaneous shooting. The system must remain scalable to 100+ Raspberries.

tiivik
  • 41
  • 1
  • 2
  • 2
    What do you mean by take photos at the same time? What criteria will you use to decide if the photos are taken at the same time? – joan Jul 08 '15 at 13:06
  • It is necessary to make all the RPi's to capture an image with the connected Camera Modules simultaneously. That means the time difference between 10 or even 100 photos captured by 10-100 RPi's should be as minimal as possible (preferrably under 1 millisecond). I would make the cameras capture a photo of a display of precise digital timer to define the synchronization. – tiivik Jul 08 '15 at 13:10
  • 2
    Similar: http://raspberrypi.stackexchange.com/q/32965/5538 Again, I would test w/ broadcast packets first to see if that can provide tight enough synchronization. 1 millisecond is not reasonable though -- you cannot guarantee that little latency even on a single pi while plugged directly into it, much less one that has to respond to a network signal. – goldilocks Jul 08 '15 at 13:12
  • 1
    Also have a look at http://raspberrypi.stackexchange.com/questions/24338/how-to-control-multiple-raspberry-pis-at-once-over-ethernet/24349#24349 where I give an MQTT example to trigger an event at the same time on multiple Pis. If we assume the Pis are all on NTP they should have a common time frame. If you can give advance notice to take the photos you may get closer to your millisecond synchronisation,i.e. rather than saying take photo now, say take photo at a timestamp (a few seconds into the future). – joan Jul 08 '15 at 13:22
  • Have a look at http://raspberrypi.stackexchange.com/questions/28113/raspberry-pi-camera-when-is-it-ready-for-next-frame/28223#28223 which outlines why in any sync situation (RTOS or not) there will always be up to a frame's worth of difference between the capture times (i.e. if your framerate is 30fps there'll be up to 33.333ms drift in your sync times because you don't command the camera to capture a frame, you command it to return the next available frame from an ongoing sequence). – Dave Jones Jul 08 '15 at 20:18
  • 1
  • @n.st Being aware of the limitations there are a good idea, but I think that specific answer is 1) A bit dated, and 2) A bit pessimistic, as anyone who's sat in a large airport waiting area or conference center might have observed, you can have significantly more than 25 devices on a single WLAN with the right equipment (as someone else observes, the logical limit per subnet is 255). Also, if this were based on a single broadcast packet, I think to the extent that you could fit all the devices within range of a single router it should work...maybe. – goldilocks Jul 09 '15 at 13:42
  • hi I came across this yesterday in an unrelated search, and thought I might share it with you as it could possibly help you hopefully I am allowed to post the link http://technabob.com/blog/2014/03/01/raspberry-pi-bullet-time-rig/ its using a piface if i remember correctly. hopefully you can glean some info to help in your project. – GSiklos Jul 09 '15 at 18:08
  • What about joining the network for NTP syncing and scheduling, and then disconnecting from the network? – EDP Aug 03 '15 at 17:08

4 Answers4

7

You may be able to get pretty close. My idea goes something like:

  • sync time across all RPi's using NTP
  • broadcast a network message to shoot in the future
  • a listener calculates how long to wait and then performs the action

Using NTP you can get all RPi's to be in pretty good time sync with each other. From this Wikipedia article NTP can have sub-1ms resolution on a local network:

NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions

Put all your RPi's on a 192.168.0.0/16 network and you'll be set for thousands of devices; e.g. 192.168.0.1 - 192.168.254.254. My home network is a 192.168.1.0/24 network, which would support 254 devices.

The listener waits for a message with a timestamp, sleeps until that time and executes the desired action. For example, here's an implementation in Python. If your timing requirements are more stringent, maybe drop to C:

#!/usr/bin/env python
# original code from:
# http://www.prodigyproductionsllc.com/articles/programming/write-a-udp-client-and-server-with-python/

import datetime
import logging
import re
import socket
import sys
import time

SYNC_RE = re.compile(r'sync at (?P<timestamp>.*)')


logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logger = logging.getLogger('listener')

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind(('', 5000))

logger.info('Listening for broadcasts...')

def do_action():
    now = datetime.datetime.utcnow()
    logger.info('syncd at {}'.format(now))


def sync_at(dt):
    delta = time.mktime(dt.timetuple()) - time.time()

    # broadcast blasts a few messages to ensure delivery
    # ignore broadcasts in the past
    if delta < 0:
        logger.debug('past')
        return

    time.sleep(delta)

    do_action()


while True:
    try:
        message, address = s.recvfrom(8192)

        matches = SYNC_RE.match(message)
        if not matches:
            continue

        dt = datetime.datetime.strptime(matches.group('timestamp'), "%Y-%m-%dT%H:%M:%S")
        logger.debug('dt={}, timestamp={}'.format(dt, matches.group('timestamp')))

        sync_at(dt)
    except (KeyboardInterrupt, SystemExit):
        break
    except Exception, exc:
        logger.exception(exc)

In the code above you can put your logic in do_action() and you're all set.

The broadcaster calculates a time in the future and blasts this on the network. I noticed sometimes the message would be missed, so I have it send it a few times. The listener above ignores any messages that appear in the past. The time delay is set to 2 seconds in the future.

#!/usr/bin/env python
# original code from:
# http://www.prodigyproductionsllc.com/articles/programming/write-a-udp-client-and-server-with-python/

import datetime
import logging
import socket
import sys
import time


logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logger = logging.getLogger('broadcast')

dest = ('<broadcast>', 5000)
delta = datetime.timedelta(seconds=2)

now = datetime.datetime.utcnow().replace(microsecond=0)

later = now + delta
later_iso = later.isoformat()

message = 'sync at {}\n'.format(later_iso)

logger.info(message)

for i in range(10):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
    s.sendto(message, dest)

    time.sleep(0.05)

In my quick tests, with a timestamp delay of 2 seconds, the devices are synced within a millisecond:

enter image description here

I like this approach because sync is handled by NTP and messaging is super basic networking. :)

berto
  • 1,231
  • 1
  • 9
  • 12
  • 1
    This is a great answer, but still beware that managing to take pictures with millisecond latency consistently cannot be guaranteed -- the OS will get in your way. I'm more optimistic than I was before though, and I think you could get consistent 10 or 100 ms latency this way...which realistically should be enough for almost anything. BTW, I think there is an advantage to using a passive timer with a calculated offset (like you are doing) vs. trying to trigger immediately if the time delay itself is quite long (seconds). – goldilocks Jul 09 '15 at 16:08
  • Thanks @goldilocks. You're right about taking pictures. I approached the question as relating to telling a cluster of RPi's to trigger something at approximately the same time. You make a good point about the OS getting in the way and in that respect turning off any and all unneeded services, e.g. webserver, GUI, etc, will help. – berto Jul 09 '15 at 16:28
  • I think it would be awesome if this did work out that tight, and this is the kind of approach I would try first. I've harped on about the granularity a bit much because I wanted the OP to beware there aren't going to be guarantees -- but I hope I did not create the impression it is not worth trying. Anyway, lots of good starting points here now. :) – goldilocks Jul 09 '15 at 16:55
  • Many thanks! NTP syncing & shooting in the future seems like a reasonable idea. – tiivik Jul 10 '15 at 12:53
  • In general your 192.168.1.0/24 network, supports 254 devices. 192.168.1.0 being your network identifier and 192.168.1.255 the network broadcast address. In general. – EDP Aug 03 '15 at 11:16
  • 100 ms sounds horrible. Can it really be that bad and how to improve it? – Lightsout Mar 05 '19 at 20:00
2

the time difference [...] should be as minimal as possible (preferrably under 1 millisecond)

That's not very feasible on a single pi sans networking, by which I mean, if you write a program to take a series of photographs spaced with millisecond granularity -- #1 at N, #2 at N + 1014 ms, #3 at N + 5135 ms, etc. -- this timing will fail badly. You might be able to get them within 10-100 ms of the target, but again, this is with regard to one pi, no network.

This is largely a constraint of the fact that the operating systems available for the pi are generally multi-tasking (as opposed to real-time).

However, before you go down the RTOS route, you might want to re-evaluate your requirements. Unless you are taking pictures of very fast moving objects, 1 ms is totally irrelevant:1 Consider an object moving 100 km/h * 1000 = 100000 m/h / 3600 = 27.777 m/s / 1000 = 2 cm/ms.

So, if this is for getting perfectly timed images at a racetrack for a science documentary, you probably want to invest in some much more expensive equipment. However, if 250-500 ms latency is at all feasible, you might be able to achieve that over wifi, although I think there will be some percentage of inconsistent shots. Using some kind of direct radio signal might get this down to 100 ms -- but I think a requirement for anything less than that is a requirement for completely different, very specialized, hardware.


1. Note that the shutter speed of the camera is probably going to be in excess of 1 ms.

goldilocks
  • 58,859
  • 17
  • 112
  • 227
1

Yes, no problem. It's done for creating 3d-scan of peaple.

Here is a site that shows how (and how exspensive it will be).

http://www.pi3dscan.com/

/Eric

1

I have built a system to do exactly this. I have a central Pi (called Pi0), with a bunch of slave Pi's (Pi2 - Pi12). Pi0 is set up as a wireless AP, and all the slave Pi's connect to Pi0 as stations on the same wireless network.

When a photo is triggered, Pi0 uses a network broadcast to send out a single udp packet to the entire subnet (to port 6789), to the subnet broadcast address. All of the slave Pi's are listening on that UDP port, so they all receive the trigger messages at the same time, and all take pictures.

  • Could you perhaps explain how you do this exactly? This is precisely what I want to achieve – crazjo Oct 17 '17 at 14:02