Robot2 – an ARM based colour tracking robot
I’ve finally got around to writing up this project! What you see above is a small robot with a gooey ARM Cortex-M3 STM32 core, a teensy embedded camera from SparkFun, an OLED and an LCD screen, three LiPo batteries, some modified servos and a one-piece (unibody!) aluminium case. The robot uses the camera to track colour, moving towards it.
This project was actually a final year school project, so I didn’t even have to pay for it, which is good – the PCBs were ordered as a panel from China, I went through several of the ARMs, and some of the other parts did not come cheap. At the end of the day, though, it works! It took a long time to get there, though…
1. Prototyping
I’d never used an ARM before, which means I wanted to try one out before going for the real thing. However, I wanted to try with the chip I’d actually be using, and getting one on a premade PCB seemed like a pointless expense (how I regret thinking that…), so I made a simple breakout board and soldered one on. By hand.
Incredibly, this actually worked. It took a long time to get openocd installed and talking to the programmer (an ARM-USB-TINY from Olimex), and about as long again to actually program anything to the chip. Eventually I had that cracked and moved on to compiling my own code instead of just uploading a sample hex file. Many hours of struggling later I had a working Makefile using the Codesourcery GCC port. It was time for a more complicated Hello, World:
With this out of the way, I moved on to:
2. Design
Since this is a school project, design is important – I had to actually write this stuff up! (a 69-page A3-size Powerpoint file.) First up was schematic capture, which I did in EAGLE and involved separate designs for the main board, the camera, the OLED carrier and the SD card carrier. This is where I made a few crucial and stupid mistakes, like wiring the ARM’s analog ground to Vcc and the analogue supply to GND:
The PCB design took a few days but eventually I’d made up a design for each of the boards, and panelised these with GerbMerge to be sent off to Gold Pheonix. I even got to pick black soldermask!
The case design, while straightforward, was fun. Normally we’d use vacuum formed plastic or MDF at school, so stretching to some sheet aluminium was exciting. It’s just normal aluminium with some holes drilled into it for the PCBs to mount to, though. The design was done in ProDesktop.
Schematic and PCB designs are linked at the end of the post.
3. Manufacture
Once the PCBs and components arrived I decided to try out reflow soldering, instead of just hand soldering all these components. It worked really well – I got some solder paste, put a little blob on each pad, placed the components and then shoved it under the grill on full heat until it reflowed.
In the end I used up every one of the control boards until I finally got it right at the end – there were a few problems with the reflow soldering after someone moved the tray down one position to grill a steak (and yes, I’ve heard all the steak and chips jokes 
) and that threw off a few boards. The required fix for the swapped AVcc/AGND pins on the ARM wasted a few more. In the end, I got one right and that’s the one that’s now in the robot.
4. Programming
Programming this thing essentially involved an awful lot of C calling the libraries that ST provide and a bit of assembler for that speed-critical reading data from the camera. This part took ages to develop, as the camera basically just sends data as fast as it pleases and there’s no simple way to manage this when your microchip has less RAM than one image from the camera. I ended up getting a logic analyser to help out, and was soon able to pick out what the camera was sending:
A logic trace from the camera
I was trying to see what the camera was showing, but this was really difficult to accomplish – there was no way to send data to the OLED screen fast enough, and no way to store the entire image in memory. I suddenly realised I could use one of the tiny LCDs from SparkFun – they work over SPI, which I’d already broken out for an SD card (that ended up unusued), and they even took the same data format the camera was sending! They turned out to be absolutely perfect for the job. All I had to do was read in each line of data, store it in memory, then trigger the DMA controller to copy that out over the SPI port. It only took a few lines of assembler and suddenly the LCD was showing exactly what the camera sent. Perfect!
From there I was quickly able to add the colour tracking part – for each pixel, it checked if it was close enough to red, and if so it used a simple centre-of-mass calculation to determine the average red position in the camera’s field of view. This technique worked pretty well.
I added a simple menu on the OLED – you can toggle turning, driving and lights.
The nav switch is a handy little SparkFun switch that works really well for this application.
5. Summary
There was a lot more to this thing’s development – like how to route power from 3 batteries to the two servos and the main logic, making the small breakout PCB for the LCD that included its own vreg and some status LEDs, endless cursing of various bits and pieces of the build environment, trying to affix two servos to a flat piece of aluminium and playing with supporting FAT on the SD card (harder than it might appear), to name just a few. Talking about all of them would take forever, though, so instead I’ve written what I hope is a more interesting to read summary.
6. Resources
The C/ASM code is available on GitHub: http://github.com/adamgreig/followingrobot which includes the build environment, makefile, etc etc. This might be pretty useful if you were trying to program one of these chips.
The EAGLE sch/brd files are available here: https://randomskk.net/projects/robot2/robot2_eagle.zip which also includes the SparkFun LCD breakout board and footprint, which could come in handy.
If you’re really interested, the humangous PPT that contains a lot more detail on pretty much every stage of making this thing, as well as more photographs and logic traces and such (and also a few boring waffle pages I have to include) is available here: https://randomskk.net/projects/robot2/robot2_coursework.ppt
Finally, here’s the Flickr photo set: http://www.flickr.com/photos/randomskk/sets/72157607851550306/
Please do post a comment if you have any questions!
And just to whet your appetite: the nixie clock PCBs have been sent off for manufacture and all the components have arrived! I should have a writeup of that in a few weeks once it’s all been put together.
[...] used an STM32 for the first time in Robot2, and soon discovered how cool they were. Each one is packed full of neat peripherals, runs really [...]
Negative Acknowledge » Blog Archive » STM32 Prototyping Boards said this on September 14th, 2009 at 10:19 pm
Hi, I was just curious what college you went to and what was this class like. Working with and using micro controllers to there full advantage is what I would like to be doing, but I am having a hard time finding out what school and classes I should be looking for if there even are any. Justin M, justindmickelsen@gmail.com
Justin said this on January 1st, 2010 at 12:49 pm
Hey, I would also be interested in your choice in college, I’m at that point in life when I must choose one, and I think I’ve found some I like, but I am always looking for more. I do have some experience with micro’s(I’m trying to get your work with the sparkfun camera working this summer). Anyway, great work your doing(though the posts have been slowing down lately). Thanks.
Tom McLeod said this on June 14th, 2010 at 1:27 am
Hey there, I’m also doing a project with tcm8230 camera, and I have some questions I’d like to ask you, could you drop me a mail at o_eoin@hotmail.com. Thanks in advance
Eoin said this on September 6th, 2010 at 11:59 am
merry xmas, can you send me the eagle library of the camera TMD8230MD please?
Omar said this on December 25th, 2010 at 3:04 am
I’m afraid I don’t have the eagle library for this any more – I lost it a while ago and haven’t had to use that part since.
Random said this on December 26th, 2010 at 5:10 pm
ooohhh u_u well that’s too bad anyway I used your eagle proyect to copy the footprint part, thanks.
Omar said this on January 6th, 2011 at 1:08 am
Nack,
I don’t see your project on GitHub anymore. Did the robot wander off…?
++tkx,
-e
ewertz said this on February 6th, 2011 at 2:53 am
Figures — found it over at
https://github.com/adamgreig/followingrobot
ewertz said this on February 6th, 2011 at 2:59 am
Hello, I didn’t understand so well what did you do to recognize the red color, did you use the camera, right? how did you use it?, how did you obtain the information of color from the camera with your memory limitations? and what was the reference of this camera?… Thank you very much, this is very useful for me now.
Carol said this on June 19th, 2011 at 5:33 am
Hi, very nice job! I`ve got a question why didn`t you put whole image in uC memory? Correct me if I wrong but if you used QVGA format which is 160×120 so it consumes only 19kB per channel, so 19 * 3 = 57kB for RGB. STM32F1xx has 64kB of SRAM. I`m asking because I`m starting similiar project with image processing on STM32 Cortex. And the biggest problem is the memory
Adrian said this on June 21st, 2011 at 5:24 pm
Small correction of small mistake: QQVGA (160×120) not QVGA (320×240)
Adrian said this on June 21st, 2011 at 5:51 pm
It won’t actually fit – RAM in the STM32F1xx series varies depending on density and this medium density device only has 20kB of RAM, so I had to analyse as I go. Good luck with your project!
Random said this on July 6th, 2011 at 7:34 pm
Hello,
I’m trying to interface the TCM8230MD with an Arduino (ATMega328P @16MHZ) and I’m having some problems with the camera start-up. Can you gave me some advice about how to deal with this?
Thanks.
Lara said this on October 3rd, 2011 at 10:56 am
Heya! I think you’ll be very hard pressed to get the camera working with an Arduino – it simply can’t run fast enough to do anything useful, nor does it have enough memory to storage even a tiny image. What are you trying to do with it exactly?
Random said this on October 3rd, 2011 at 10:57 am
Hi, I came across this page randomly, and I go to the same school that you went to. I would just like to say I’m very impressed. I’m doing a similar project to yours (but not at all copying – don’t worry), with the same ARM processor – it took ages to finally get the dev environment working! I’m trying to use a pretty hefty camera as well – 752*480 pixels. Not a lot of luck at the moment – I think that the minimum 12MHz clock in needs (that the STM32 is generating at 72MHz) doesn’t have rise/fall times low enough. It would be annoying if it didn’t work because Mr. P spent top dollar on it! I am planning to make an autonomous quadcopter for my A2 project, and was wondering how well the Sharp IR sensors behave in indoor lighting (these may be useful) – is there a lot of interference (honestly)? I thought that I had to say something as I came across this website
Matt B said this on December 22nd, 2011 at 9:29 pm
Sorry wrong e-mail.
Matt B said this on December 22nd, 2011 at 9:53 pm
Hi Matt! Hah, what a coincidence. Probably a bit late now but did you see this, this and this? The first two are everything you need to get the dev environment working (on Linux and with command line build tools, at least…) while the third is all the source and build support stuff for my project. Might still be helpful for you.
A 752*480px camera sounds fairly hefty indeed! Are you storing the data coming off it, or trying to process it a line at a time? If you’re storing the data, what on? If you’re clocking it at 12MHz into the STM32 running at 72MHz that really gives you very, very, very few clock cycles to accomplish anything useful, it sounds like you’d really better be using DMA from the GPIO peripheral to stand a chance. Alternatively the new STM32F4 line includes onboard camera peripherals that would interface with many embedded cameras directly, streaming image data into RAM (and have a lot more RAM to boot, too) which might make your life a lot easier!
The Sharp sensors should be fine indoors and are indeed a fairly good way of getting precise small-distance ranges, but an ultrasonic module might be better for distances in the range of a couple of metres — less affected by interference, too.
It sounds like a really cool and ambitious project, I bet Mr P’s excited for it! I’ve toyed with autonomous quads too, and it’s definitely not an easy undertaking. Best of luck! I’d love to hear how it goes.
Random said this on December 23rd, 2011 at 9:56 pm
Thanks for the links, unfortunately I had the environment up and running quite a while ago (and on windows…), but they may be useful in the future.
After days of trying to get the camera to work, I had almost given up. But just out of luck, I reduced the clock speed to 6MHz and now it seems to work perfectly! It’s strange how the ARM can’t seem to output 12MHZ (maybe rise/fall times are too high).
I’m not planning to use DMA, nor to store the image data. Each pixel will be processed on-the-fly (I’m making a laser triangulation depth sensor) – each in exactly 14 cycles. It is a bit risky, because I’m not using a pixel clock to know when the data is valid, but I’ve cycle-timed my processing code block and it is consistently 13 cycles (plus one NOP as a buffer). There will be 752 copies of the code inline (!!!) to process one row. Also the interrupt latency time is far too high and unpredictable to use (I think this is what you used for robot2). The brightest pixel’s position in each column will be stored in the SRAM. Then, when the frame is over, will be converted into depth measurements and sent to the sensor’s host (maybe another ARM chip) for SLAM or something. The camera will be running at about 14FPS so I think the sensor will be good enough for some pretty intelligent robots
I’ve looked into quad-copters and it doesn’t seem too bad, but something will probably crop up (although will hopefully crop back down again safely).
If you have any advice or think you can see any problems I’d gladly absorb it
Matt B said this on December 27th, 2011 at 5:34 pm
That is unusual. Have you tried using an oscilloscope to check the clock output? It’s definitely capable of a 12MHz output, but maybe there’s too much capacitance on the line for it to drive it effectively. Still, 6MHz should suffice if the camera will operate.
Processing each pixel on the fly seems reasonable. Are you writing the code for that in assembler or C? You might want to think about things like the caching system and how likely your code is to cause cache hits…
I was able to read in one line of data by having an interrupt fire on the HSYNC line and then reading bytes into memory by polling the pixel clock line. Once I’d read an entire line, I had time to do quick processing of it (finding the centre of mass of red pixels on that line) before the next line started, an approach that sounds like it could also work for you. If you’ve got per-pixel processing working well there’s no point changing though. If interrupt latency is being an issue check out the various priority configuration options for the STM32′s interrupts — it’s one of the most configurable and powerful interrupt systems in microchips these days and can probably work with what you need (I think they should be able to work down to just a couple of clock cycles…).
Doing SLAM sounds like fun. Is this laser just facing down, so you build a height map of the terrain you fly over? Trying to do localisation with such a narrow field of view (just one height measurement…) sounds like it might be tricky, though — have you got a proof of concept or a simulation or anything working?
Some of the hardest bits of quadcopters is stabalisation: ideally you need sensor fusion (a Kalman filter or something) with a degree of freedom for, at least, three axis of gyro, magno and accel, and ideally also further DoFs for bias; plus you need control systems to keep the thing level and flying where you want it to go (a tuned PID or such) that work without a human in the loop to correct errors. That’s not easy to do, though it’s a problem other people have solved — are you using (or considered using) things like OpenPilot or Paparazzi for your sensor fusion and control loops? Using their code might let you focus on the (already not trivial) problem of making the thing autonomous.
At any rate it sounds like a very exciting project, good luck!
Random said this on December 27th, 2011 at 7:11 pm
I have not tried a ‘scope because I don’t have one! I’ll try it at school. It may be that there is quite a lot of capacitance because I used a jumper on that line on the PCB, which wasn’t the greatest of ideas… But if the camera works at 6MHz, that doubles the processing time so I’m not complaining, and 15 FPS is good enough.
I’m writing everything in assembler at the moment. I have tried to find actual information about the cortex m3′s caches, but I really don’t know where to look. I find the SYSTICK timer useful for cycle-counting – but a reasonable amount of it is just guessing.
Unfortunately I cannot just poll the pixel clock line because each poll loop takes at minimum 8 or 9 cycles, which is not fast enough. My approach is to synchronise with the pixel clock line, so that my inline assembler will read each pixel at the centre of the PIXCLK waveform. The synchronisation will occur once before each line. I’m pretty sure that the pixel reading code will take consistently the same amount of time, because I tested it billions of times.
I did not know that interrupts were that responsive on the cortex m3, but using them would still reduce the FPS by a lot. Perhaps I will try them if I can’t get this CPU approach to work.
It is not just one point – it is a line laser pulsed at the exposure time of the camera, and I was thinking more for inside the house – navigating and building maps of walls etc, not a height map (ultrasonic sensors will probably be used for landing/ceiling awareness). But that will be next year!
I have found 3, 3 axis sensors (acc, gyro and magneto), and will probably just try making up some simple stuff to start with (complimentary filters and things) and just improve on that, a PID loop would also be used.
Thanks for your suggestions!
Matt B said this on December 28th, 2011 at 2:50 am
If your CPU approach is working out, and it seems like it is, I’d stick with it.
It sounds like you’ve really got it all under control! Your project sounds really impressive, I can’t wait to hear how it turns out. Doing proper SLAM with a line laser would be incredibly cool (basically LIDAR) and in theory can get you great results for indoor maps. Getting something basic working for stabalisation will probably suffice, especially for indoor flying or such. That’s a really ambitious A2 project you’ve got going! I bet Mr P’s loving it. Good luck again!
Random said this on December 28th, 2011 at 1:18 pm
Hi,i m interfacing a 5.17 mega pixel camera with stm32f100….using i2c….i just need to take a pic & store it in memory card…..can you pls helpme out what part of ur code would help me…Thanx
Rocky said this on February 16th, 2012 at 11:29 am
I think you might struggle to get the image, but if you can set up the SD card to stream data over SPI then a lot of my code (mostly in main.c and camera.c) should be helpful. I’d suggest you need to read the data into RAM one line at a time, then set up a DMA transfer to shove that line out to the memory card once it has been read. It won’t be easy, though — the F100s have even lower clock speeds.
Random said this on February 16th, 2012 at 12:48 pm
ok Thanx a lot…..1 more thing….if i need to display the image directly to oled as u did….not only red or green but the whole image as u did previously…..is it possible to get it through ur code…??
Rocky said this on February 16th, 2012 at 2:13 pm