The Open Camera Project – $4,000 bounty for Open Firmware/Drivers for Raspberry Pi Camera Module

raspberrypi_camera

[Updated: June 19,2015]

Inspired by Adafruit’s Open Kinect bounty a few years back we are offering a [UPDATE JUNE 15, JUNE 19:] $1000  $3,000 $4000 bounty to the first person/team to deliver a working open source driver and FPGA MIPI design for the Raspberry Pi camera module. I am sure you can imagine what the Parallella could do in the world of robots and drones once there are multiple cameras and sensors connected to the Parallella FPGA.:-) To make things easy we have included a Raspberry Pi camera connector on the latest open source Porcupine board.

Raspberry Pi Camera Specs:

  • 5 megapixel native resolution
  • MIPI interface
  • Fixed focus lens on-board.
  • 2592 x 1944 pixel static images, and also supports 1080p30, 720p60 and 640x480p60/90 video.

Why?
The Parallella needs open eyes.

What we want?
Linux drivers and HDL code for implementing MIPI interface in FPGA that is completely documented and under an open source license working on the Parallella board. To demonstrate the driver you must be able to display pictures from the camera at 30 frames per second in an OpenCV program (or equivalent). If you are unsure about requirements, please post comments to this post.

Who gets the $4,000 bounty?
The first person/team to have their submission pull request accepted to the parallella-examples repository wins the bounty. The bounty is split into four “sub-bounties”.

  1. $1,000: First RAW picture taken [Sylvain getting close to claiming…] [UPDATE: Claimed by Sylvain]
  2. $1,000: First static picture displayed in OpenCV (1 frame/sec?) [UPDATE: Claimed by Sylvain]
  3. $1,000: First to capture video/images at 30 frames/sec (in user space) [UPDATE: Claimed by Sylvain]
  4. $1,000: First proper kernel driver patch that builds on work from (1-3)

Solutions must be useful to others to qualify.

All code/docs need to be open source and/or public domain. Please see the github repository for a list of acceptable licenses.

Happy hacking!

Andreas

30 Comments

  • j says:

    Can I donate a few bucks to bump the bounty up ?

  • Joey says:

    Can I donate stone money to increase the bounty? I feel bad these VERY smart people see getting such a low amount of money for using their brains?

  • Josy Boelen says:

    MIPI is a closed specification and bears a cost of US$ 8,000.- (per year). I don’t think you can incorporate any ‘derived’ works in an open source project.
    http://mipi.org/specifications/camera-interface
    http://mipi.org/join-mipi

    There are other image sensors with non-proprietary/closed-spec interfaces, but I don’t think the Raspberry Camera price …

    • Selling products is definitely an issue, but it’s hard to see how publishing source code is a problem? Protocols can be easily reverse engineered with an oscilloscope.

      • Nobody of Import says:

        Indeed Andreas. They may be able to restrict the specification documents under copyright and trademark, but code driving the stuff if it’s RE’d or the like is fair game on the part of the parties doing it, especially in the EU.

    • Marek Kraft says:

      How about this sensor module? The sample images look really good.

      http://www.arducam.com/camera-modules/5mp-ov5642/

      You can get it for less than the Pi camera (with free shipping):

      http://www.uctronics.com/mega-pixel-camera-module-ov5642-1080p-jpeg-output-p-1420l.html

      The signals are directly FPGA-friendly, and there’s full sensor docs available. The lens holder enables the use of macro/fisheye/whatever comes to your mind. Looks really good.

      • Nobody of Import says:

        The goal, I believe, is to take a very popular (due to the Raspberry PI foundation’s efforts) device and provide FOSS support proper on the Parallella for the same. It doesn’t accomplish the goals Andreas has for the whole thing. It’s the software/hardware engineering equivalent of an X-prize.

        Me? I guess I’ll be picking up a cam shortly. I’ve got several PiB2’s and I’ll be dinking with this. There’s absolutely no good reason to not have a FOSS driver for this beastie and any other properly compliant MIPI CSI cameras- it’s an industry Consortium playing gatekeeper. If they hold on to the specs, making them expensive to acquire, then only so many players will be able to actually and legitimately participate (Never mind that the Shenzen bunch have already reverse engineered it enough to make their OWN closed source answers themselves…

  • Chris Hemingway says:

    MIPI D-PHY (the physical layer) uses a combination of 1.2V single ended and 200mV differential signalling, generally requiring some form of level translation. A simple trick is to use a passive resistor network to combine/split to multiple FPGA pins at different voltages, but I cannot see such a design on your schematic (porcupine.pdf).
    XILINX have an app note on this very topic, see http://www.xilinx.com/support/documentation/application_notes/xapp894-d-phy-solutions.pdf for a solution for the ZYNC7, which I believe is what you are using.
    Without such level translation, reception of MIPI might be quite tricky, and it may not technically be possible to claim the bounty. I have only worked on transmission of MIPI though, so I cannot say for sure.

    • Thanks for the note! Definitely agree that it would have been better if we would have put the level translation on the board. Still, it “might be” possible to hack something together. This bounty doesn’t refer to a neat working solution, just a working solution. The challenge is really in the FPGA design and drivers.

      • Chris Hemingway says:

        So to clarify, does the bounty apply if some external hardware (small things like logic and misc resistors) is used? Or would a solution need to be using only Parallela and Raspi Camera?

        • Sylvain Munaut says:

          The Zynq differential receiver are pretty happy with just the raw CSI signal when in HS mode.
          However to detect the start of a packet, you need to know when the link is going from LP to HS to know when to start looking for the sync sequence.

          The current idea I had were :
          * The sensor has a ‘HS only’ bit in the mipi register. I had hoped it would then do one long HS session with multiple packets and using null packets for padded. Unfortunately, I tried that and it seems to do nothing … so there is still mixed HS/LP with one packet per burst and not one single long HS transaction with multiple packets.
          * Use external circuit to just detect the 1.2V level on the P (or N) line. Currently this is 2 * 1k resistor and a NPN BJT. Not perfect but could be done as a simple mod on the porcupine or small add-on board.

          It might be possible to keep track of packets purely in HS mode but I’ll try that later.

  • I also feel that the bounty is kind of low.

  • Jim Houston says:

    I would be great if offering a prize can make this happen. It is frustrating the way that MIPI locks up the interface. Their web page doesn’t seem friendly to an open source implementation.

    I have been playing with this on and off for 6 months. I had the silly idea that I could spend a few hours and hook my logic analyzer up to a Raspberry Pi and camera and see If I could understand the MIPI CSI-2 signaling. The good news is that I can hook up the analyzer and the camera still works. The bad news is that the SLVS-200 differential signals are not easy to probe. I tried using the LVDS inputs on a Spartan 3a board to translate to single ended signals but it wasn’t reliable I also tried using FIN1002 chips. I’m using adapter boards from Ebay to break out the signals from the cable and I have tried tapping into the cable between the PI and the camera and also repeating the I2C signals using an FPGA so that I could have a second camera with clean 50 ohm termination at the receivers. Reading the Xilinx apnote on CSI-2 cameras it seems like the latest LVDS interface should receive the SLVS signals. I thought I would try the Spartan 3a board first because was expecting smoke and didn’t want to loose my Parallella. It also seemed simpler to use a standalone FPGA rather than having to add something to the Parallella FPGA.

    I’m currently side tracked thinking about differential probes. I found Daniel Kramnik’s DIY Active Differential probe and I have been thinking of building something similar.

    Maybe this thread will lure others who have been lurking out of the shadows. It would be interesting to compare notes.

    Jim Houston – jim@overmoreover.com

  • GMA says:

    nice to be compatible with ZYNQ, any chance to get one, to test it, and write some drivers for Zedboard ( ZYNQ MPSoC ) ?

    thanks

    • Nobody of Import says:

      Go build you an equivalent board (They handed you schematics…)- and roll your own. The real rub’s going to be in figuring out the protocol proper. You can scope it out on an RPi or you can try to build your Parallella into a protocol analyzer for this purpose (I’d advise the latter. You can then make up something that exports the OLS (OpenWorkbench Logic Sniffer) protocol so you can use one of several analyzers and then extend the protocol analysis functions there- or use the Parallella as a bridge to a full-on analyzer- hint…most people with one of those won’t be doing that- they’ll be “scoping” out the RPi with it’s closed driver instead with the analyzer….and analyzers that work in the 1Gbps domain…heh…they’re not cheap.

  • Viktorio el Hakim says:

    Hello,

    Is it allowed to use a different evaluation board for part of the bounties? I have a ZYBO development board and I use it to develop the HDL and Linux driver for the Zynq. I plan to purchase the Parallella and GPIO board in the near future, but I’m a bit short on money right now…

    Regards,
    Viktorio

    • Yes, this is not a problem, but to qualify for the bounty the submission needs to be validated on the Parallella as well. Let me know if you need a free board? Just send a note to support@adapteva.com Sounds like you are well on your way!

      btw do check out Sylvain’s work so far!
      https://parallella.org/forums/viewtopic.php?f=10&t=2514&start=10

      • Viktorio says:

        Hey,

        Thanks so much for the reply! I would greatly appreciate it if I can get a free dev. board but I don’t really wanna sound like a beggar of some sorts.. It’s unfortunate that my university is not part of the parallella program, otherwise I probably would have gotten a board already (if I understand how the university program works anyways). Regardless, I will keep you posted with my progress and if I succeed with the ZYBO, I would definetly try to get my hands on a parallella (preferably with a Zynq 7020 😉 )

        I will also check out how Sylvain is doing and support his progress with what I have as well.

        Regards,
        Viktorio

  • DogP says:

    It looks like Sylvain has made some great progress (nice job!). One thing I wanted to point out in case you hadn’t considered it is that Lattice has hardware directly compatible with the MIPI interfaces (DSI and CSI-2), and reference designs available to bridge to a CMOS parallel interface, which work with their tiny (and cheap) MachXO2/3 FPGAs – http://www.latticesemi.com/en/Products/DesignSoftwareAndIP/IntellectualProperty/ReferenceDesigns/ReferenceDesign03/CSI2toParallel.aspx . They even have a “Custom Configuration” form, where they’ll send you an HDL netlist for your specific needs, if it differs from the reference design (needs company/project details, but I’d guess Parallella would be big enough that they’d provide a custom config).

    I haven’t personally used one to interface to any MIPI interfaces, but I do regularly use the MachXO2 and MachXO3 parts, and I find them to be great to work with. If I was personally going to attempt this, I’d probably toss one of them on an interface board and offload some of the complexity (for just a few bucks). I guess the disadvantage is that the parallel interface would eat up a lot of I/Os.

    Pat

    • Sylvain Munaut says:

      The Lattice part are not more compatible than the Xilinx ones. If you look at the schematic you’ll see they use 4 IO pins per lane because they don’t have IO that can both do LP/HS in a single pin. Xilinx has exactly the same app note with the same external resistor network.

  • delfare says:

    This looks really great 🙂
    I have been looking for informations about connecting csi-2 camera modules to fpga and it seems that this will be the first opensource code for doing it 🙂
    I didn’t buy parallella board because there was no camera capabilities, but now I’ll buy one 🙂
    Does the porcupine board sold by digikey already contains the raspberry pi camera connector or is it an old version?

  • […] the Raspberry Pi camera module to the Parallella board. More information on the bounty can be found HERE. Well, it doesn’t surprise me that hacker extraordinaire Sylvain Munaut has claimed the […]

  • Daniel says:

    Is this bounty still valid?

  • Maksims M. says:

    Sorry, is all 4 bounty steps still valid ??
    Or may be some are ready and other need implementation ?

  • Georgy says:

    It is possible to communicate with MIPI CSI/2 cameras using USB camera controller IC. There are many such ICs.
    Here is a list of such chips: http://mrfdsp.com/mipi-csi2/index.html
    Take a JPEG image snapshot over mipi from 13megapixels camera using such IC and send this image directly to USB port of some openWRT or similar architecture board. Many USB camera controller ICs have internal firmware which is able to send control registers to CMOS cameras.
    Although SDK may be difficult to obtain from manufacturer.

  • Richard Blakely says:

    A similar method to that described by Georgy, Dec. 21, 2016, is to use a Raspberry Pi Zero 1.3, which has a single processor and a CSI port, to communicate with a Raspberry Pi-type camera. Then, the Zero can send data to the Parallella. A Zero computer is very small and has many ports that could be useful to communicate and for other inputs and outputs. For example, two Zero boards could handle optical information before handing the data to the Parallella. Until someone comes up with a better method this is the method I will pursue.

Leave a Reply