PiWars 2024: The Challenges and Their Challenges

There are seven challenges at the in-person event for PiWars 2024, each with their own problems to solve. They are split in to three categories: Autonomous only, Autonomous/Remote, Remote Only. For the second of those, if you are in the Advanced/Professional category you have to attempt the challenge autonomously.

State of the Onion

NE-Five exists as a robot, the base hardware is all there and everything has met the minimum requirements as I’ve set them, the devil is in the detail though and integration hell is totally a thing. Also, perfect is the enemy of done and I really need to get a focus on what needs doing now vs what would be nice to have.

Motor control

Motor control has been overhauled with the switch from Red Robotics RedBoard hat to the Pimoroni Yukon. The big difference is the latter has support for encoders, but it also has on board processing which means that processing load is taken away from the overhead on the Raspberry Pi. I also have a ROS node for the Yukon so it can send and receive ROS messages such as motor velocity commands and odometry.

Servo Control

I’ve implemented a new ROS node that not only takes commands to move the robots arms but also provides joint state feedback to the wider ROS system, this also includes the neck servos as they are the same Dynamixel smart servos used in the wrists. The linear actuator that adjusts how high the robot is standing has also been hooked up to the Yukon, it has a feedback line that is factored in too. Each of the smart servos is independent of the Pi too as you give them a command and they do it, handling any PID loops and monitoring internally as needed.

Camera System

I’m still using the Luxonis FFC-3P camera system, with two wide angle global shutter cameras and a narrower field rolling shutter camera in the centre position. Luxonis have recently released a big update that includes on-device pointcloud generation. Previously I was trying to do this on the Pi and it was basically taking all the resources just to do that. Having this board run tasks itself and only providing the data the Pi needs is a big win for sure.

I’ve also been playing around with object recognition, it works and you can run custom neural networks on their which, again, means the Pi doesn’t have to do anything but use the data it produces.

Back to the Challenges…

The robot works great in remote control mode, it’s currently the only mode however which isn’t ideal. There are five challenges I’ll have to tackle autonomously so I’ll concentrate on those for now.

Lava Palava

This is a line following drag race, there is a course with a black floor and white strip down the middle which the robot has to follow as quickly as possible. The course has a chicane in it from previous years but this year will also have a speedhump.

With the motor encoders providing feedback with regards to distance travelled and the camera system able to detect objects, I intend to combine the two to have it aim for a goal that’s X meters in front of the robot and have it follow the line until it’s travelled that far. Or until I push the estop button if it tries to run away…

Eco-Disaster

In this challenge you have to sort a number of red and green barrels in to blue or yellow areas of the arena. Starting in one corner of the arena, NE-Five should be tall enough to be able to detect all the objects it needs to look out for, the start position and two sorting zones are also known locations which helps. Using a similar setup to Lava Palava I should be able to use the detected barrels to position the robot relative to them so it can pick them up. Using odometry and being able to see the coloured sorting zones, it should be able to navigate to them to drop them as needed.

Escape Route

This challenge has to be run without the robot’s operator being able to see the arena directly. For remote control this means using cameras or have someone shouting out commands, for autonomous the operator needs to be behind a screen still but only press “go” and hope for the best.

The arena will be in a randomly selected configuration out of six possible layouts. There are three coloured blocks, each having known dimensions. The plan is that as soon as the challenge begins, the robot scans to see which block is closest and add a waypoint to get passed it. After it gets there, or while en route, it can look for the next block and figure out its next steps there too.

Similar to Lava Palava, where it’s aiming for a point a certain distance ahead, the end goal will be passed the yellow line with intermediate steps to get around each block. The depth camera already has an option to convert a depth image to laser scan so should be relatively easy to detect a clear path.

Minesweeper

For this one, the robot will have to look for an illuminated red square and move to it. Once it has visited that square another will light up, the process repeats. As with Escape Route and Lava Palava I’ll be looking for the specific colour of the square but this time I’ll be setting it as the waypoint to move towards. Once it detects it’s on top of a red square, it’ll stop. Once the red switches off it’ll start looking around for anything of the same colour, with the wide angle stereo cameras this should give a good field of view for this and the odometry once again comes in to play.

Zombie Apocalypse

This one is currently the biggest unknown as I don’t have a projectile launcher for the robot yet, I do have a pile of parts however… Sample designs for the zombie targets have been released though, so I’m planning on trying to detect those to use for the coordinates. I have parts from an electrically fired Nerf gun so will mount that on a pair of servos for pan/tilt and use them to aim at the target. I also have a green laser for this, so hopefully will be able to detect when the laser is within an area in the centre of the target before firing.

If’s, But’s, And Maybe’s…

Other than for the last challenge, I pretty much have everything in place. The devil’s in the details with these things but I’m in a considerably better position than any previous competition which is a great feeling. What are the priorities though?

The Toad List

What needs doing?

Nerf gun and mounting hardware
Camera to provide coordinates of:
- A white line, it’ll have length rather than being point data so probably just “make sure white line is in the middle of the view”
- A zombie, there will be multiple at different heights, the higher ones having more points available. Primarily we’ll need the X,Y coordinates but detecting distance will help ensure were detecting the right things as they’ll all be on a plane.
- Coloured boxes, these will be used as signposts and will need to be avoided. Depth to laserscan for obstacle detection.
- Coloured barrels to pick up and navigate around, this will need pose estimation.
- Coloured flooring, for both mine sweeper and Eco Disaster
Arm control to ensure coordinates are in the same system as the camera, this is for picking up the barrels
Waypoint system, hook in to odometry to have the robot follow a path.
Robot pose estimation, where is the robot and which way is it pointing?

There is a common theme in that a lot of the challenges have overlapping needs but there’s a lot of work to do.

Load’s of time though, right?

NE-Five Upgraded? Yukon Bet On It!

Upgrades galore in this latest post! Raspberry Pi 5, Pimoroni Yukon, and a banging sound system?!

I’ve been hard at work with a few upgrades on the robot over the past few months and thought I’d share a long overdue update, and if you’re here from the Raspberry Pi blog, welcome! In this post I’ll talk about some of the upgrades to the robot I’ve not covered yet, and will cover each in more detail in future posts. This includes motor control using PID loops, odometry from the encoders, and using running a ROS node on a Raspberry Pi Pico/RP2040 based board.

The key upgrades? A Raspberry Pi 5, Pimoroni Yukon, a banging sound system! Wait, soundsystem?!

Raspberry Pi 5

The Raspberry Pi 4 was a massive upgrade from the model 3, the extra RAM being the biggest winner for me, but the CPU was still a bit slow compared to some other single board computers on the market. When the Raspberry Pi 5 was announced it sounded like it was the upgrade I needed! I managed to get on the waiting list only a few hours after the announcement and a month or so later the board arrived.

Plenty of others have covered the Pi5 in more detail so I’ll not give a review here, needless to say though it’s made on-robot development *much* easier. I was running VS Code on the robot before which allowed me to develop on it remotely, this worked but had a big overhead that can swamp the Pi’s CPU and RAM if you don’t keep an eye on it. The extra power of the Raspberry Pi 5 makes the experience more responsive and makes it feel like I’m coding directly on my desktop. I’ve not scratched the surface of what it can do yet, but I’m getting closer!

Pimoroni Yukon

One of the primary issues to deal with in robotics is controlling motors, interfacing with sensors, and all the other things required to interact with the real world. Typically you have a computer of some kind coordinating all the components of a robot along with dedicated systems that handle the actual hardware involved. Think of a robot as if it’s a ship, you have the captain in overall control but they delegate specific tasks to others who only interrupt them when something needs attention. They’ll tell the engine room what speed to go at, and they’ll deal with it unless there’s a problem with the engines. They may also have a lookout who’s job it is to watch radar, sonar, or other sensors, and only tell the captain when something will cause a problem.

In this instance, the Raspberry Pi 5 is the captain, the Luxonis FFC-3P is the lookout, each servo in the arms and head look after themselves, but the motors were still controlled by the Pi. The Yukon has helped fix this by taking up the role of the engineers. It has a Raspberry Pi RP2040 chip on board, same as the Pi Pico, and modules that allow for control of specific bits of hardware. In my case, a quad servo direct module, an LED strip module, four big motor modules. Code runs on the Yukon that listens for commands from the Raspberry Pi and actions them, it gives feedback to the code running on the Pi but does the heavy lifting itself. This means I can improve speed control of the motors using encoder feedback, and also use that feedback to provide position data so the robot knows how far it’s moved from its starting position.

Banging Soundsystem

I’ve wanted to have microphones and speakers on the robot since day one, for voice interaction and teleoperation purposes, but also fun. When I saw Eagle Prime throw its first punch at MakerFaire Bay Area years ago, they had some issues to work through with the code so used its PA to blast out some tunes to keep themselves and the crowd occupied while they got thing fixed.

I ended up going with a HifiBerry Miniamp and pair of Tectonic speakers for this, Matt Perks (of DIY Perks fame) recommends them and that’s good enough for me. I also have the Raspotify service installed on the robot so I’m able to use it as a Spotify speaker, works a treat!

Mostly though, it’s the easter eggs I can add…

I’ll be going through each of the above in more detail in future posts, so stay tuned for updates!

NE-Five Mk4 – It’s All About That Base…

An update on the design of NE-Five Mk4

The road to PiWars continues! Most important of all is a solid base to build from so that’s where we’re starting.

A render of the new design for the robot base. It has a mecanum wheel at each corner, these wheels have rollers around their circumference rather than a solid rim.

This is the first iteration of the base design, it’s the same width but slightly longer to give more room inside the enclosure. Another big improvement, I hope I least, is that I’ve added suspension to each motor.

As this robot uses mecanum wheels it’s incredibly important that all four always have contact with the ground as all four wheels work together to allow the robot to move in any direction, if one isn’t in contact then the effect that wheel would have won’t be present and it’ll veer off course. I’ve added a hinge at the bottom of each mount and the black part will be printed in flexible filament. By varying the wall thickness and infill I should be able to control how much travel each wheel has. That’s the hope at least…

Another improvement is for quality of life more than anything and that’s the method by which the upper part of the base (not pictured, or designed yet…) attaches. On the previous iterations of NE-Five these parts have been attached using tabs that are simply screwed in place, this makes working on the robot tricky as if I need to work on the wiring it’s not designed for it.

I’ve also made the switch from Red Robotics RedBoard to the Pimoroni Yukon, the RedBoard has served me very well but the lack of encoder support is a problem. There’s ways around it, like using the Pi to Pico adapter that Neil developed, but the Yukon has a motor controller and encoder module all in one. It’ll also allow me to control the torso actuator and LED lights which is another issue on the Pi.

The NeoPixel library on the Pi requires you to run it as root, this makes running it as part of a ROS launch file a bit of a pain. By handing off control of this to the Yukon that problem goes away.

The other big benefit of switching to the Yokun is that I can send it messages to do something and it’ll do it rather than using CPU cycles on the Pi. Splitting hardware up between real-time and scheduled systems like this is very common and should work a treat here. The Yukon runs MicroPython too so I should be able to use ROS Serial to connect to and have it act like a ROS node, which it will be but running on the hardware.

All of this is theory at the minute and there’s always little problems I miss until at least the third iteration, stay tuned to find out what mistakes I’ve made this time! 😅

NE-Five (Re)design, Lessons Learned

One common questions I get asked about NE-Five is “is it open source” and I always give the answer “mostly…”, there’s a reason for that. The code has been available for a long while, along with a basic simulation that you can run in Gazebo, but the CAD model has never been shared. That’s becuase it’s a mess, to say the least! I hadn’t used Fusion 360 in anger until I started working on MacFeegle Prime, the v1 of NE-Five. As such, it’s an absolute mess of deadends, redundant designs, and terrible usage of the component system…

The Road to NE-Five Mk4

There have been various versions of NE-Five over the years, the biggest changes being from MacFeegle Prime to NE-Five, in the early days they had tank treads for the true Johnny Five aesthetic. Treads were a bit unreliable, at least the way I made them, so I switched to wheels for a later version rather than getting bogged down trying to make them work, all the while the rest of the robot being a mess. I think, bizarrely, that the first NE-Five was actually the Mk2, I guess I considered NE-Five the class of robot and MacFeegle Prime was just the name of the first one? Sound’s plausible, it’ll do as an explanation at least.

The Mk1 introduced an aesthetic which we’ve pretty much stuck with ever since, it also moved the motor and servo control from the head to an enclosure on his caboose. You can see the mess of cables coming out of the back of his head in the photo below. The bundle was that tight that if he tried to look up too far, his head would pop off as the servo couldn’t move the wires…

The rear of the old robot, showing how messy the wiring was. There is a circuit board on it's back, but there is a large bundle of wires coming out of the back of his head which was the big problem.

Mk2 still used a Stereo Pi board in his head, but the RedBoard was moved to the caboose and hosted on a PiZero connected over USB. This meant that there was only a USB cable and wire for the NeoPixels between head and the rest of the body which was much less stressful on the neck mechanism.

Both of these designs still used hobby servos for the arms and neck mechanisms, along with the Stereo Pi for control and vision, so the Mk3 was the biggest leap from a technology perspective.

NE-Five Mk3

Aesthetically the Mk2 and Mk3 look very similar, but there is an awful lot changed internally.

The overall form-factor has been retained but significant upgrades were made in the arms and vision system. The Stereo Pi was replaced with a Luxonis FFC-3P and the Pi Zero in the caboose replaced with a Raspberry Pi 4, the RedBoard is retained for power supply, NeoPixel, and motor control. The “hobby” servos were replaced with Robotis Dynamixels, the most obvious benefit of which is that they daisy chain together so the wiring is much tidier. More importantly though, they give position and status feedback so you can detect if they’ve stalled for example, they are more expensive but they’re worth it purely in savings compared to the number of servos I burned out without realising it!

The Luxonis FFC-3P vision board is also a game changer, where with the StereoPi I was able to generate a depthmap at 320×240 at around 18fps, the 3P can run at around 900x600px and 30 fps. This is as well as running neural networks for object recognition using it’s three cameras as well, it has dedicated hardware for the task, it also compresses images for preview on the Pi or over wifi to massively reduce bandwidth needed. This takes basically all the stress of computer vision away from the Pi and only sends over the data we need over USB, rather than swamping the Pi with frames that will mostly be discarded.

A screenshot showing three different images from the same point in time. Top left is a wide angle view of the room with two cats eating the treats I bribed them with. Top right is a depth image where close objects are red and change to blue as they get futher away. Bottom middle is a closer view of one of the cats from the narrow field camera. — An example of the output from the Luxonis FFC-3P. Top left, a view from one of the two wide-angle cameras, top right is a coloured disparity image showing the distance of various objects, and bottom middle is the view from the narrower-fov centre camera.

The 3P is configured with three cameras: Left and right are both AR0234 sensors with 110degree wide angle lenses, these also have global shutter sensors so ideal for reducing shutter effects when moving. The centre camera is an IMX378 with auto-focus, this has a 69 degree FOV so much narrower. It also has a rolling shutter so more for use to get closer views of objects when stationary. Imagery from all three can be used simultaneously, I’ve barely scratched the surface of what these can do.

NE-Five Mk4, and Future

So, with all those lessons learned and improvements in mind, I’m going to design the Mk4 from scratch in Fusion 360. The reason for this is to get rid of all the baggage from the last four years of designs in the files so that they will be much easier to maintain, and more importantly to share. There’s also some cool scripts for Fusion 360 that can generate URDF (universal robot definition files) but they only work if it’s laid out in a certain way.

Internally and externally it’ll look much like the Mk3 does now, but with what I hope to be a more “production” quality of finish. There will also be a new motor controller board I’m developing that will include reading encoders and IMU to generate odometry.

The future part is an aspiration, I’m planning on developing this in to an actual product and sell as kits or preassembled, I am going to be releasing the CAD models and BOM still though in case people want to build their own. For those in teaching or research who just want one that comes assembled and with a warranty though, buying one may be preferrable. It’s very early days on that at the minute but I’m hoping it’ll be ready for sale towards the end of 2024. I was always intending for this to be a portfolio peice for the company to help me get robotics work but that hasn’t worked out too well. A few folks have said it’s almost a product already so I figured I’d try and get it the rest of the way there.

Stay tuned for more info on this, this is the precursor to me documenting the build of the Mk4 so a lot more info will be coming soon.

Making the RedBoard Work on the Raspberry Pi 5

I’ve been using Red Robotics excellent RedBoard for years now, and while upgrading NE-Five to use a Raspberry Pi 5 I discovered breaking changes which means the available libraries wont work.

Basically, the way that GPIO works on the Raspberry Pi 5 is different as it uses its shiny new RP1 chip to wrangle its devices. As such, legacy code that used the old way needs updating. In our case that’s pigpiod, and the develop has said it’s non-trivial and will take time. This is entirely reasonable, it’s an open source project and depends on the good will and free time of the developer.

I’ve been using Approximate Engineering’s RedBoard library which abstracts a lot of the underlying bits away, so I thought I’d take a stab at updating it to use gpiozero instead. I’ve managed to get enough features working for NE-Five, the rest I will revisit when I’ve recovered a few spoons from the effort…

Long story short, you need to specify the pin factory for gpiozero to work in a virtual environment, and for whatever reason lgpio doesn’t work if you install it in a venv from pip. Instead, you need to install it from source, ignoring their instructions… This took an awful lot of trial and error to figure out, hence the spoon defecit.

If you go to my fork, there are instructions on installing everything which hopefully will get you working. Just make sure you’re on the develop branch!

PiWars 2022!

So, for those who follow me on Twitter will know I’ve been accepted to PiWars 2022! The theme for this year is agritech, something I have a bit of experience with from my time at my old company.

Here is a list of challenges, there’s three arena challenges plus a freeform obstacle course of our choosing. For the latter, the only rules are you have to start and finish at the same location and you’ve five minutes total.

I’ll be using NE Five for this competition, it’ll basically be the same robot as now but with an added axis in the waist so it can twist, and also I’ll get everything working that I didn’t get to last year… More on that in an upcoming post.

Shepherd’s Pi

For Shepherd’s Pi we have to herd six wayward sheep in to an enclosure, we have to do this three times in five minutes for maximum points. Also, bonus points if you can control the robot using a whistle!

Each sheep has to be 75x75x150mm in size, they have to stay on their feet, and all six need to be in the enclosure. More bonus points are available if the enclosure has a latching gate.

The three red crosses are wolves, these can be moved but also need to stay upright. No killing allowed!

My vague plan for this is to use the hands to best advantage, ideal to pick one up in each hand then hold a third between them. This will result in two round trips to the enclosure followed by a return to the barn while the arena resets.

Nature’s Bounty

For Nature’s Bounty we have an apple tree, full of apples, that need to be harvested. The arena setup for this is with the same barn as above with the tree in the middle spot.

Each branch will hold a single apple, with there being four branches at each level. The diagram to the left is the side view, from the top it looks like a cross.

Each apple has to be 40mm sphere in diameter, handily this is the size of a ping pong ball. They can be attached using magnets too, though magnets can’t be used to pick them up.

For this one I’m hoping to be able to use the arms to best advantage, picking apples with both, then stowing the apples in a basket of some kind before delivering them to the barn. Possibly each load in a different basket?

As before, five minutes max and aiming for three attempts.

Hungry Cattle

Hungry Cattle sees us filling the feed troughs for our herd of cows, three in total. We have to fill each trough at least half way with dried “food” of some kind, then return to the barn. We can either carry enough food on the robot for all runs or refill each time we return.

Five minutes and three attempts, as per the others.

I’m not entirely sure how I’ll do this yet but some kind of scoop to pour the food in to the troughs?

I think the new axis in the torso will help here, the robot will be able to turn left and right for better axis which should help?

Farmyard Tours

https://youtu.be/A_PmidIsUC0

This one is the least constrained but due to that is the trickiest as you have too many options!

Basically, you have to show some visitors around your farm and hope nothing goes wrong…

You have five minutes to show off what your robot is capable of, you have to change level by at least the height of your robot, and move across three different types of terrain. It has to be a circuit, finishing roughly where you started.

Stay tuned for more info, from improvements to my robot to how I solve the challenges, no doubt a few daft surprises too…

MacFeegle Prime, Lessons Learned, and PiWars 2021!

Using The RedBoard+ From Within a Docker Container…

…because why do anything the easy way?

I’ve switched to using the StereoPi image for MacFeegle Prime as it offers the lowest latency for streaming video, the downside is I can’t get ROS to build on it so I’m using Docker which needs to interface with hardware…

I’m using the RedBoard+ by Red Robotics, it uses the pigpio library which can work over a socket or pipe. I’m trying to figure out how to have the pigpio deamon running on the Pi host and access it from a docker container that’s running ROS.

Security Note Of Doom!

The following changes have security implications, only do this if you’re running on a private, trusted network. I don’t really know what I’m doing so assume the worst and double check everything I suggest!

Things To Tweak

Upon some digging, this wasn’t as evil as I thought it would be. For pigpiod you need to change the service definition to remove the “-l” from the start command:

sudo nano /lib/systemd/system/pigpiod.service

Remove the “-l” from the ExecStart line.

This is needed as by default pigpiod runs in localhost mode which means that it’ll only accept connections from the local machine. When connecting from inside of a docker container, it’s considered another host.

In order for the docker container to access the hardware it’ll need to have privileged status, add this to the command when you create your container. Eg:

docker run --priviledged -d ros

From inside the docker container, and in the same shell instance you’re running the code that calls the RedBoard library, you need to run this:

export PIGPIO_ADDR=[PI IP ADDRESS]

This sets the address used by the pigpio library that RedBoard uses to the host machine.

Conclusion

After doing the above I was able to use the RedBoard keyboard_control.py script to control the robot from inside a docker container. I’ve not tried anything further yet but that proved the concept enough for today!

Raspberry Pi – Docker on USB Drive

As per my last post I’m using Docker on my robot with ROS. The last task is to get docker running from a dedicated USB drive to split resources between that and the SD card the OS is running from. A good guide to mounting a USB drive can be found here.

Note, rather than using umask=000 for mounting you need to mount, then change the permissions of the “host” directory to 777. For example, mount to /media/usb as per the article then chmod 777 /media/usb **WHILE MOUNTED**. This should allow you to mount, then set to automount on boot.

If you are running headless and there is a problem with the fstab file it can get annoying so to test in advance of a reboot run “sudo mount -a” to mount all volumes as per that file. If it succeeds, you can reboot.

I was having a problem mounting with fstab, I could manually mount the usb folder every time but not using “mount -a”. The penny dropped when I did “df -h” to see how much space was free and noticed /media was itself a mount point. I created a new root folder called “docker” and it worked a treat.

Following this answer I moved the location of the docker containers and such to /docker.

I’ve run “docker pull ros” and it’s now happily using the usb drive instead of the SD card. 🙂

StereoPi Image Mods

I’ve been using the Ubiquity Robotics Raspberry Pi ROS image to run both the robot and controller, it seemed the easiest way to get ROS running, but now I’m trying to get low latency streaming working from the cameras it is proving tricky.

New Plan, use the Stereo Pi image with Docker to host ROS images/containers.

Some Mods Needed…

In order to lower the latency as much as possible the Stereo Pi image is a heavily modified version of Raspbian, this includes custom partitions and the file system set to read only. Here are the steps I followed to get it to a state where I can restart development.

1. Get and Modify the Image

Head here and follow the steps to get and install the image, follow the steps to get it on the wifi and under the Advanced section you’ll see details on how to SSH to the Pi afterwards. Once logged in you’ll need to temporarily set the file system to read/write then edit the fstab file.

Under the “SLP Structure Details” section you’ll find this command:

 sudo mount -o rw,remount /

This will set the file system to read/write, at which point you can open the fstab and edit it to make this permanent by changing the ro to rw for the root.

nano /etc/fstab

2. Resize the Partitions

I was trying to figure out how to do this on Windows then realised I could just install gparted on the controller and remote to it… I put the micro-sd card in a USB card reader and followed these instructions. The 1.8G root partition was expanded to around 25GB and the recording drive slimmed down accordingly.

screenshot of gparted with the final sizes of the partitions

3. Install Docker

This next bit is trivial, run this command, as taken from the Raspberry Pi blog:

 curl -sSL https://get.docker.com | sh

4. Get ROS Images

Next bit is easy too:

 docker pull ros

Considerations

One of the reasons StereoPi is very quick is because it is tailored not to use the filesystem on the SD card, to that end it may be worth moving all of the docker containers and images over to a USB drive. There’s more information on how to do that here but I’ve not tried it yet:

https://stackoverflow.com/questions/32070113/how-do-i-change-the-default-docker-container-location

Next Steps

The next things I need to do is convert my code to work in a docker container, this shouldn’t be too tricky but as the RedBoard library will need to talk to the hardware there will likely be complications.