BrickSortingMachine

Sat, 23 Aug 2025 05:00:00 +0000

To make the build of the Brick Sorting Machine (see build instructions) easy to reproduce, I decided to deliver the image based LEGO® brick identification for the machine as a remote web-service.

As a hobby project with super limited development time, creating a remote service was the most time efficient path forward. It bypasses the need for me to dive into optimizing and quantizing neural networks for AI accelerator hardware. To continuously enhance its accuracy, the classifier training learns from the images this service processes. Every image submitted will be used to improve the model’s performance.

Public Dataset Releases

After manual labeling of the data my plan is to publicly release the training data sets in regular intervals for training of everybody’s own classifier projects

SW Interface

The brick sorting machine software stack will be ready to connect to this classification backend within the next days. The service currently is in private beta test phase. If you are interested and building a compatible machine please reach out to me via Discord to get an API-KEY.

Backgrounds

The image classification backend is reachable via api.bricksortingmachine.com. Its neural network is fully tuned to work best as a classifier for the Brick Sorting Machine with its ability to capture two viewing perspectives simultaneously. It classifies parts into one of 21 part classes, which perfectly fits the machine’s 22 bins (with one bin for unclassified parts).

The classification extracts the two known regions of interest from the provided image and feeds them through the visual feature encoders. At this point this mid-level feature information is fused from both perspectives and provided jointly to the classification head. The network thus always works on image pairs both at training and inference time. This way the the classifier can make maximum use of the mutual multi-view information which often is key to uniquely identify part classes.

Currently available part classes:

    bar
    brick1x
    brick2x
    brick_modified
    car
    hinge
    human
    multiple_parts
    plane
    plate
    plate1x
    plate2x
    plate_modified
    plate_shaped
    round
    round_slope
    slope
    slope1x
    small
    tile
    window

Sat, 23 Aug 2025 05:00:00 +0000

← Back to Build Instructions Overview Page

Following the part separation stage, individual LEGO bricks are conveyed to the scanner for recognition and classification. The scanning system is solely camera-based, featuring a conveyor belt that transports the LEGO bricks past the camera.

View through the scanner towards the v-channel with mirror and camera

The design of the scanner adopts a key idea from Johann Rocholl’s Conveyor belt for LEGO sorting. The solution consists of a single camera paired with a mirror, allowing the machine to capture two different perspectives simultaneously. The core motivation behind this is to capture the LEGO parts from multiple perspectives, as some parts are difficult to identify when viewed from certain angles. Choosing a mirror over multiple cameras not only simplifies the overall design but also ensures synchronization between the captured images. Additionally, it significantly reduces the computational and bus load.

Lighting is another crucial element in each scanning system. The machine utilizes strong, non-flickering illumination that is isolated from external environmental factors. This approach allows the camera to operate with very short exposure times, effectively minimizing motion blur and thereby enhancing the accuracy of the part identification process.

Object detection is realized via OpenCV’s MOG background subtraction. The moment the detected part reaches a pre-defined position in the image, two cut-outs are taken. One of the direct view and one via the indirect mirror view. Both cutouts are presented simultaneously to a multi input CNN, which consists of two weight sharing instances of the same pre-trained MobileNetV2 encoder plus a dense classification head. The network thus always works on image pairs. Both during training and inference.

Frame

At this time, BrickLink Studio natively does not support duplo bricks. Because of this, the build instructions use a large amount of 3001 2x4 bricks. In practice I prefer using duplo to speed-up the build.

Conveyor Belt Roller

Conveyor Belt

For the conveyor belt material I made very good experience with 100mm wide Woven Elastic Band. It is made from a blend of Polyester and Elastane and commonly used in sewing and clothing making e.g. as a waistband.

It is elastic and thus easy to put under tension but at the same time it has a certain stiffness so it stays in line with the conveyor and does not run off the rollers. Even without a crowned pulley this belt material stays on track very well.

To close the belt into a loop, I experimented with two methods: stapling and sewing.

Closing the belt to a loop by stapling (left) and sewing (right)

Sat, 23 Aug 2025 05:00:00 +0000

← Back to Build Instructions Overview Page

The electronics of the Brick Sorting Machine are currently under heavy re-work. My goal is to make the build easier to reproduce and a key element of this is to reduce the electronics to a 1-box setup (having moth the machine control and the image processing on the Raspberry). The following electronics is the future setup after the re-work is completed. Thus currently this list is untested:

Compute and Camera

Raspberry Pi 5 8GB ~83€
Active Cooler ~6€
Raspberry Pi Camera Module 3 Wide 12MP ~40€
(not the NoIR version)
27W USB-C Power Supply ~13€
Housing ~5€
- Needs space for active cooler and venting channels
- Needs outlet for Camera Serial Interface (CSI) port

Sum: approx. 150€

Motor Controllers & Servos

2x Servo Keyestudio 270° compatible to LEGO® bricks ~9€
1x Arduino Nano for the rotating Slide 1-5€
DC Motor Controller for 9V LEGO® compatible motors
- As of now I did not find a zero-soldering, off-the-shelf solution for this. That’s why I am using several L293D dual H-bridges connected via GPIO to the Raspberry. As I am not an electronics professional I can neither recommend nor advise against using this solution. For this approach you need to do a small amount of soldering, use a pegboard or custom PCB.
- There is many small scale DC motor drivers on the market even with USB support. If you found a good solution I’d love to get a suggestion and open to integrate it with the software.

Sat, 09 Aug 2025 06:00:00 +0000

In this article series I publish the plans and DIY instructions for building your own LEGO® brick sorting machine. Feedback on the build or any questions are highly encouraged. As I am not an electronics professional, please feel free to reach out via Discord or Instagram if you identify problems.

The 4 modules of a typical LEGO sorting machine: (1) Bulk Storage (2) Part Separation (3) Scanner (4) Sorted Storage

Article Overview

The instructions follow the 4-stage design common to many sorters:

Material/Electronics - List of Materials needed for the build
Software - The open source software stack for LEGO sorting machines on GitHub
Main Modules
- Bulk Storage - Build instructions of the bulk part storage
- Part Separation - Vibration Feeder Build Instructions
- Scanner - Plans to build the scanner module
- Sorted Storage - Blog article on the build of the sorting slide

BL-Studio Files

Rendering of the LEGO sorting machine in BrickLink Studio

Here are the BrickLink studio files to assist with the LEGO brick part of the build:

BL studio files on Github
Part List on BrickLink approx. cost 150EUR

Sat, 09 Aug 2025 05:00:00 +0000

← Back to Build Instructions Overview Page

The task of part separation is to take the entangled stream of bricks and separate it into individual parts which can be passed on to the scanner strictly one by one at a time. Separation is especially challenging with LEGOs, since the parts have a huge variety in size (e.g. 1x1 plates stuck inside a window) and also friction (e.g. bricks vs. rubber wheels). When building a LEGO Sorting Machine this is one of the key components to get right since large parts of the efficiency of the overall machine depend on it.

There is a big variety of potential designs and many earlier machines implemented this component. The approach I chose is a pair of v-channel vibration feeders and this article provides building instructions as well as bricklink studio CAD files.

The Build

Additional Information

Why 2 channels in series - Having multiple vibration channels behind each other has 2 key advantages. The first is that channel 2 is pulsed twice as often as channel 1. The parts thus move at higher speeds. Having two different speeds “pulls” the parts apart from each other longitudinally. The second advantage is the vertical step between both channels. The small fall gives another chance to separate strongly entangled parts e.g. two window elements.

Rubber Bands - The v-channel is suspended from 2 rubber bands at each corner. 1 band to carry the vertical load of the v-channel and 1 band to center the longitudinal position. The offset weight of the vibration motor moves the v-channel in the freedom the rubber bands provide. Only the lateral position is fully constrained. I will post a photo as soon as possible.

Elastic mount of the v-channel using rubber bands. The mounting is flexible in vertical and longitudinal direction but constrained laterally (sry for the dust - the machine is located in a former wood shop 😊)

Tuning - This v-channel has several adjustable parameters. 1) the inclination of the channel 2) the motor speed 3) the mass of the offset-weight. Alls these parameters need to be tuned to your specific machine. Over very long periods of time I also noticed the optimal parameters to slightly change. I chose the old 9V motors for their high RPMs but high speed is not always best. I tuned their speed via the motor driver to hit the resonance frequency of my v-channel. This way the motion is strong enough to disentangle the lego parts. Also the inclination needs to be calibrated to a sweet spot. It needs to be steep enough to still transport the parts forward. It needs to be shallow enough so that entangled pieces do not pass through the v-channel too quickly.

Surface - To reduce surface friction of the plates I added paper sticky tape and it works very well. Adding tiles on top of the plates would increase the mass of the v-channel and thus require a bigger vibration motor. Using large tiles instead of the plates could be another solution, they then require a different technique to be joined at the bottom.

Est. Cost - The pure material cost of a v-channel is approx €20 according to the part list on BrickLink.

Tue, 22 Apr 2025 11:00:00 +0000

To keep up with the community of LEGO® sorting machine makers, I want to continue showcasing recently published work. Today’s highlight is lego.sorting.machine. It was built in public on Instagram since August 2023 and very recently completed its end-to-end sorting milestone.

First Impressions

What first comes to mind when looking at this machine is its super clean design, utilizing almost purely white parts. It is built from a mixture of standard LEGO® parts and many custom 3D printed LEGO® compatible components. Its overall size fits nicely on a table-top. The usage is primarily targeted towards efficiently sorting smaller LEGO® parts.

The overall setup follows the classic 4-stage design common to many sorters:

Bulk Storage - Holding the unsorted bulk
Part Separation / Unification - Isolating individual bricks
Scanner - Camera-based part identification
Sorted Storage - Directing parts to the respective bins

However, that’s where the standard design ends, as the rest of the machine is packed with innovative features.

Left: Bulk part storage. Middle: Two sequential v-channel vibration feeders. Right: Rotating and pitching slide to guide identified parts into custom, stacked storage boxes.

Unique V-Channel Monitoring

One particularly unique aspect is the use of a second camera for fill state monitoring of the vibration feeders. Both the bulk storage belt and the v-channels themselves can be adaptively controlled based on the fill state of the downstream stage. Since bricks often behave chaotically in the feeder, this is a clever automation step to ensure a steady, well-separated stream of parts. This is key for an efficient machine running at high parts per minute.

Innovative Scanner Design

The scanner stage also incorporates unique elements:

Transparent Belt - The machine uses a transparent conveyor belt. This is a very unique choice, which enables the scanning of the underside of LEGO® parts.
Mirror - Complementing the transparent belt, a mirror is positioned within the scanner. This can allow the overall system to capture top, front and bottom perspectives to disambiguate part classification.

Rotating Slide

The final sorting stage features a modern rotating slide mechanism:

Yaw and Pitch Control - The slide shows both yaw (rotation) and pitch (angling) capabilities. This allows it to access two levels of sorting bins, pushing the sortable class count to 16 while keeping a compact footprint.
User-Friendly Storage - The sorting chute leads to stacked boxes. This design offers great usability, as the boxes can be easily removed, emptied and replaced without disrupting the machine’s operation.

A custom, stackable sorting tray for efficient part unloading

Overall, the lego.sorting.machine presents a highly refined and innovative take on LEGO® sorting, introducing several novel ideas. Congratulations for reaching this big milestone, and I hope you will continue building!

Sat, 05 Apr 2025 11:00:00 +0000

The motivation is straightforward: LEGO so far has produced approximately 20,000 different parts (including variations such as color). To advance sorting machines beyond handling part classes and enable sorting by individual part IDs, the number of discrete storage locations must strongly increase.

Storage approaches typically scale hardware needs linearly in the number of bins. For example, a belt-and-gate system requires one dedicated gate and servo for each storage location. This quickly becomes infeasible as the number of bins increases, leading to complex and expensive machines.

A Scalable Approach: Automated Storage and Retrieval Systems (AS/RS)

To overcome this limitation, I’m exploring an approach with more favorable scaling properties. It draws inspiration from large-scale warehouse storage, specifically Automated Storage and Retrieval Systems (AS/RS). These systems efficiently manage vast quantities of goods using automated cranes or shuttles to retrieve items from dense storage racks.

An automated highbay warehouse (photo by Arno Senoner on Unsplash)

As a proof of concept, I plan to integrate a small-scale, DIY AS/RS with my existing LEGO brick sorting machine. The sorter will continue classifying parts as usual but will now instruct the storage system to retrieve the appropriate bin for depositing each part. The core idea is simple: If the DIY automated storage system can reliably manage 100 bins, scaling up to 1,000 bins—or even more—should be achievable by either increasing its physical size or deploying multiple AS/RS units side by side.

More importantly, this approach shifts the complexity from handling individual parts to moving entire bins, which not only simplifies automation but also enables instant part retrieval. For BrickLink sellers and advanced LEGO builders, this means dramatically faster access to specific parts. Instead of manually searching through drawers or containers, the system can fetch and deliver the required bin in seconds, streamlining the entire building and order fulfillment process.

Let me summarize the key requirements for this project:

Scale - The system must support ≥100 storage bins, with the option for further internal subdivisions to accommodate smaller parts.
Efficient Operation - The system should enable ≥10 storage operations per minute without introducing additional latency.

Design

The design and construction of the AS/RS will revolve around three major components:

Storage Rack - A structured shelving system to hold each storage bin in a 2D grid layout.
Moving Gantry - A mechanism providing precise X/Y motion across the storage rack to access bins.
Telescopic Fork - A retrieval mechanism that extends and retracts along the Z-axis, allowing bins to be pulled from / pushed to the rack.

Storage Rack

The storage rack serves as the structural backbone of the system, organizing bins in a grid layout for efficient retrieval. For storage bins, I have chosen off-the-shelf polystyrene sorting bins. Priced at just €1.70 per bin, they are cheap and at the same time available quickly. The bins measure 162mm x 108mm x 45mm. They are thus large enough to accommodate even bigger LEGO parts while at the same time allowing them to be further subdivided to maximize storage density for smaller parts.

Storage rack featuring 10x8 bins and polystyrene sorting tray

For the first prototype setup, I am targeting a 10-by-8 grid of bins. To ensure smooth retrieval, I’ve accounted for small gaps between bins, bringing the total estimated dimensions of the storage system to approximately 1.65m x 0.8m. This compact footprint is ideal, as it allows the unit to fit neatly behind the existing sorting machine while still providing high storage capacity.

Gantry

The moving gantry is responsible for precise 2D motion across the storage rack, enabling the system to position itself over any bin. To achieve smooth and reliable movement, I drew inspiration from DIY CNC machines and 3D printers, which commonly rely on linear rails for accurate motion. However, full-length linear rails at the scale required for this project are prohibitively expensive, prompting me to explore more cost-effective alternatives.

For the vertical Y-axis, I opted for round linear bearings, which provide a reasonable balance between precision and affordability. These bearings allow smooth movement while keeping costs lower than premium linear rail systems.

X/Y motion mechanics using linear bearings and aluminium extrusions

For the horizontal X-axis, which spans a significant length, I needed an even more budget-friendly yet effective solution. Drawing inspiration from entry-level 3D printers like the Ender 3, I chose 20mm x 20mm V-slot aluminium extrusions paired with matching rubber wheels that fit precisely into the V-slot profile. These components are readily available as off-the-shelf replacement parts for 3D printers, making them both cost-effective and easy to source. The wheels come as complete assemblies with pre-mounted bearings, screws, and spacers, simplifying integration into the gantry design.

The gantry’s wheel configuration is designed to ensure rigidity and maintain precise vertical alignment:

Two wheels on top of the extrusion rail
Two wheels below, running along the underside of the rail

This setup effectively prevents unwanted movement or wobble, ensuring the gantry remains stable as it moves across the storage rack. The two upper wheels bear the majority of the load, supporting the weight of the moving gantry assembly. To ensure long-term durability, I will closely monitor these wheels, their bearings, and their mounting points for any signs of wear. If necessary, I will reinforce the structure or integrate additional wheels to better distribute the load and enhance long-term reliability.

Beyond load-bearing considerations, the gantry must be exceptionally rigid, particularly against torsion and racking forces, which could cause misalignment. To address this, I designed it with a wide stance and rigid diagonal supports, improving structural integrity. The stainless steel rods of the vertical Y-axis are securely mounted at the gantry ends using press fittings, which are tensioned with small screws to ensure a stable connection. This setup enhances overall rigidity and minimizes flex during rapid movements.

Gantry-end element reinforced for shearing forces

The telescopic fork, serving as the end-effector that interacts with the storage bins, requires an onboard stepper motor for its operation. Consequently, cables must be routed to the fork while allowing for unrestricted movement along both the X and Y axes.

To achieve clean and reliable cable management, I plan to incorporate miniature drag chains along the linear rails. These chains will guide and protect the cables, ensuring they move smoothly without tangling or excessive strain. However, in the current CAD design, these drag chains have not yet been included. My approach is to first build a prototype without them, allowing me to gain practical experience with the system before finalizing the cable routing.

The same iterative approach applies to end-stop switches, which are critical for automated homing and safe operation. These switches will enable the system to accurately reset its position upon startup, preventing misalignment or unintended movements. They will be added in later iterations as I refine the design based on real-world testing results.

Telescopic Fork

The telescopic fork serves as the end-effector responsible for handling storage bins. Its retrieval process follows a three-step sequence:

The fork extends beneath the designated bin, reaching into the storage rack.
The gantry lifts the fork, raising the bin from its slot.
The fork then retracts, carrying the bin along with it.

There is a wide range of DIY inspiration available showcasing various approaches to building a telescopic fork. The most common design mounts the motor on the stationary base, using a system of belts and pulleys to coordinate the movement of the telescoping segments. However, the Awesomemakes YouTube channel presents a more elegant solution: By mounting the motor to the middle segment of the fork, the design achieves a much simpler motion coupling. This approach reduces mechanical complexity and minimizes the number of moving parts.

Telescopic forks of existing DIY AS/RS projects: Youtube/awesomemakes, Youtube/Maker Tutor, Youtube/Nathan Koflanovich, Youtube/Mr Jos

For my own design, I decided to combine both approaches: I will mount the motor to the middle segment to benefit from the simplified mechanics, but instead of using gears, I plan to use one belt to achieve synchronized motion between the elements.

CAD drawing of the telescopic fork

The fork is constructed from five 3D-printed components: the base, middle, and top sliding segments, a motor mounting plate (orange), and a pulley holder (pink). The motor is rigidly attached to the mounting plate, which is itself fastened to the middle segment of the fork using elongated screw holes. These slots allow for precise adjustment of the motor position to tension the belt. The pulley holder (pink) is printed separately and is only glued in place after the three sliding segments have been nested together.

At this stage, the design does not incorporate any linear rails or bearings. Instead, it relies solely on PLA plastic surfaces sliding directly against each other. While this keeps the construction lightweight and simple, I’m uncertain about its long-term durability. This aspect will need to be closely monitored for wear over time.

In addition, the fit and tolerances between the sliding elements are critical: The parts must move with minimal friction while also maintaining tight alignment to prevent play. This is especially important when the fork is fully extended backward, as the motor (being the heaviest component) is positioned far behind the mounting point. This could introduce imbalance or sagging.

The telescopic fork allows extension in both directions. This is enabled by the motor being mounted to the underside and a matching cutout in the base segment.

It is crucial to the overall design that the fork can extend in both directions—forward into the storage rack and backward toward the sorting machine. This bidirectional capability offers two key advantages:

Future Expandability – It opens the possibility of adding a second storage rack on the opposite side of the gantry. This would double the storage capacity without requiring any mechanical changes to the fork or gantry system. The same hardware could serve both sides, maximizing efficiency within the same footprint.
Precise Bin Positioning – Extending the fork toward the sorting machine also allows the system to precisely adjust the position of the bin during part insertion. This fine-grained control over where the LEGO piece lands within the bin will make it possible to implement internal subdivisions. Especially for small parts, this would significantly increase storage density by allowing multiple sub-compartments within a single bin, each dedicated to a different part or color variant.

Wrapping Up

This concludes the overview of the current AS/RS design work at the CAD level. Nearly all core components are designed, with some details still left open.

In parallel, I’ve already begun 3D printing for the physical prototype, testing key subsystems like the gantry mechanics, motor controllers and the telescopic fork in real-world conditions. These tests will provide lots of important early learnings.

I’ll be sharing an update on these tests very soon, here on the blog. Until then, feel free to reach out! I’m always happy to connect with fellow builders, makers, and curious minds — whether you have questions, suggestions, or just want to chat about LEGO sorter projects. The easiest way to reach me is via Discord or Instagram. Looking forward to hearing from you!

CAD drawing of the automated storage and retrieval system (without z-axis)

Thu, 08 Aug 2024 11:00:00 +0000

The goal of this rebuild is to maximize the machine’s operation time by allowing it to run unattended for longer periods without human support. This includes simplifying and accelerating human maintenance, such as refilling unsorted parts and emptying storage containers for sorted LEGO® bricks. There are many possible options for designing the sorted storage: Conveyor belts, horizontal/vertical gates, pneumatic/mechanical approaches, etc. To reduce complexity, space and hardware requirements, I decided to follow the simple rotational slide design principle.

Bricks are falling on a slide which sits on top of a rotating platform. Based on the brick classification results, the platform rotates in order to direct each brick into its target bin. This design is very hardware efficient since it merely uses two servos to serve a large number of densely packed storage bins. As a bonus, timing is not particularly critical with this system.

First Designs

My very first design iteration of the slide was built mostly from Lego (unfortunately no pictures were taken at that time yet). The construction was quite large, thus resulting in a pretty high slide mass. The high mass far off the slide’s center of rotation created a large moment of inertia. The servo was under high load and during acceleration and deceleration there were oscillations.

Thus I learned to keep the mass low which led to the second design iteration built from a mixture of LEGO bricks and cardboard. It immediately performed much better. The oscillations were greatly reduced. The servo was able to position the slide quickly and accurately above the bins. During this time I was using hardware storage containers as storage bins (after all this whole machine is being built in a woodshop).

First variants of the cardboard slide

On the electronics side, the slide is actuated by 360° Geekservos. Out of the box they are LEGO compatible so no adapters had to be made. The servos are controlled by an Arduino Nano. The code of the slide controller (and the full machine) is available on GitHub. It uses the excellent ServoEasing library which enables smooth acceleration curves. The Arduino receives its motion commands via USB/serial by the serial service of the sorting machine stack (sorting machine architecture).

Going 2D

Right from the beginning I wanted the slide to operate in 2 dimensions. Thus not just rotate but also pitch. This would at least double the amount of reachable storage bins. To realize this, both the slide and the storage containers had to be modified.

On the slide, the change was simple. I mounted a second servo onboard the rotating platform. Via a gear reduction the servo drives two arms which pitch the slide up and down. The motion is quick and precise and the mechanics integrate nicely into the existing design.

Slide pitch mechanics

The work on the storage bins was more extensive. The primary goal was to pack as many bins as possible around the slide in an almost full circle. To achieve this, the bins needed to be as narrow as possible while still allowing parts to fall in reliably. The initial prototypes of these 2-level chutes were designed in FreeCAD and built from cardboard. This approach enabled rapid iterations on the dimensions, yielding first promising results.

Cardboard prototype of the 2-level chutes (see video)

After the design was validated in cardboard, the work took a little bit of a detour since I started getting into 3D printing. This allowed the design of the chutes to be much more refined. It still features two channels which now separate at the bottom and change shape into round tubes. They are round because it is intended to hang round cloth tubes (they look almost like socks) beneath them in order to store sorted parts. The chute design also features mounting holders to mount them inside a cutout of the tabletop board. The CAD design files are available here.

Onshape drawing, printing and arrangement of the redesigned chutes

For the mounting of the chutes the tabletop board had to be modified heavily. The board is made from OSB wood. It features a C-shape cutout into which the chutes get mounted from below. The center wood circle is supported from one side only, thus utilizing approximately 300° of the full circle.

Mounting of the 2-level chutes inside the tabletop board — cloth storage “socks” added on the bottom right

After the chutes were mounted, the cloth “socks” could be added as well. They just fit over the round end of the chutes. A rubber band holds them in place while locking on to the rim.

View of the complete machine as of today

This is the final view of the completed new design. I am super happy with it because it reduces manual handling efforts significantly and thus causes the machine to be running more often. The 22 bins fit our 20 bin sorting concept perfectly (+1 bin for parts with low confidence classification). I’ll make sure to create a video soon, until then you can reach me via DM as BrickSortingMachine on Instagram. If you are interested in LEGO® sorting machines in general you might also want to join the fabulous Discord server of the Great Brick Lab where frequent exchange is happening.

Sat, 16 Dec 2023 11:00:00 +0000

Physical Setup

The LEGO sorting machine is driven by two main compute units. A Raspberry Pi 3 as interface to the L293D motor drivers and a Intel® NUC Kit D34010WYK handling all image processing, classification and HMI. Images are acquired by a ELP 5MP 120° Wide FOV USB camera module.

Logical Architecture

All system units are implemented as independent services (python processes) communicating via TCP sockets using a very simple text based message passing protocol. The Raspberry Pi is acting as a relay server routing certain message types to the intended receivers.

Logical Architecture

Raspberry Pi / Machine Controller

The machine controller is driven by a state machine, which handles the operation of the two conveyor belts and the two v-channel vibration feeders. While the scanner belt is running continuously, both the storage belt and the vibration feeders switch activation states based on a fixed timing sequence. Whenever an object is detected on the scanner belt, the vibration feeders are stopped.

Brick Scanner with Camera Module and Mirror

NUC / Vision Service

The vision service handles all image processing and part detection. Images are captured via USB and the OpenCV background subtraction is detecting bricks against the white background of the scanner belt. Whenever a part is detected on the scanner a belt busy message is sent to stop the vibration feeders. The moment the part reaches the trigger area, a image is stored and a request is sent to the classification service.

Sorting Machine HMI with a car part detected and classified

NUC / Classification Service

The classification service receives stored camera images and creates image cutouts for both direct perspective (side view) and mirror perspective (top view). Then CNN inference is triggered on both cutouts simultaneously for part class prediction. The prediction results are again distributed via a TCP message.

NUC / Serial Service

The serial service is a gateway between the TCP socket connection and the USB/Serial interface for Arduinos. As of now, only a single Arduino is part of the machine. It is controlling the servos of the rotational slide. After a part class prediction message is received, the serial service forwards a slide positioning message to the Arduino. The Arduino then controls the servos to smoothly rotate the slide to the corresponding storage bin location.

NUC / Notification Service

The notification service is handling non-visual human machine interaction. This includes both audio feedback on part class prediction results as well as notifications on soft e-stop states of the machine. Soft e-stops are currently only triggered if no part was detected within the last 60s. This can be caused by the storage conveyor running empty or a part blocking the stream. In future, optical encoders are planned to be added to monitor belt speeds / detect blocked conditions. Based on the suggestion of a dear friend, error conditions are now forwarded to smartphones via the Pushover service 😉.

This concludes today’s view into the sorter architecture. I am always interested to get in contact with other sorting machine builders. Feel free to contact me via DM on Instagram.

Wed, 13 Dec 2023 11:00:00 +0000

Probably the best-known LEGO sorter is Daniel West’s Universal LEGO Sorting Machine. His work was inspired by Akiyuki’s NXT Vision Guided Brick Sorter and Jacques Mattheij’s Sorting two metric tons of lego. Focusing on hardware, all three machines show a similar four stage design:

Bulk Storage → Part Separation → Classification → Sorted Storage

The following overview will look at a larger field of machines and show the creative variety in designing each stage.

Stage 1 — Bulk Storage

In bulk storage you have a large amount of unsorted, entangled bricks and the goal is to feed it to the machine progressively. There are two typical designs used for bulk brick storage. Nr. 1 is what I call the “belt slowly moving under pile of bricks” type. Two nice examples can be seen with Daniel’s and Jacques Mattheij’s machines.

From Daniel West’s The WORLD’S FIRST Universal LEGO Sorting Machine

From Jacques Mattheij’s Sorting two metric tons of lego

While in Daniel’s Design, we have a narrow bulk storage where the full width of the ground is transporting bricks, Jacques’s machine features small 1x4 tiles being glued to the belt. As a result only the small middle part is feeding a small amount of bricks to the machine. If you want to learn more about these two machines, both Daniel (here and here) and Jacques published additional background information (Part 1, Part 2 and Part 3).

The bulk storage design Nr. 2 is based on a “step feeder”, where a step is moving vertically through the storage pile and feeds a small portion of bricks into the machine. Akiyuki’s, Francisco Garcia’s and Johann Rocholl’s machines employ this design idea.

From Akiyuki’s LEGO Mindstorms NXT Vision Guided Brick Sorter ver1

From Francisco Garcia’s Lego Sorter with TensorFlow on Raspberry Pi

From Johann Rocholl’s Brick feeder prototype

Stage 2 — Part separation

The task of part separation is to take the entangled stream of bricks and separate it into individual parts which can be passed on to the scanner strictly one by one at a time. Separation is especially challenging with LEGOs, since the parts have a huge variety in size (e.g. 1x1 plates stuck inside a window) and also friction (e.g. bricks vs. rubber wheels). On the machines I saw so far, part separation is built up from one of the following tools.

Chicaning Conveyors are belts with diagonal barriers which force the parts on the conveyor to align sequentially. The delicious video below shows the chicanes being used in the food industry.

Here are nice examples of chicaning conveyors in Lego sorting machines.

From Francisco Garcia’s Lego Sorter with TensorFlow on Raspberry Pi

From Peter Backx’s Deep Learning Lego Sorter

Two-speed belt steps are another possible component to achieve part separation. They consist of a slow moving conveyor belt, which drops parts onto a second, much faster moving belt. This feature takes parts which are already sequentially aligned and spreads them out longitudinally.

From Akiyuki’s LEGO Mindstorms NXT Vision Guided Brick Sorter

From Francisco Garcia’s Lego Sorter with TensorFlow on Raspberry Pi

The vibration feeder is the most frequently used part separator. It strongly shakes parts inside a v-shaped channel. As a result it forces the parts to align sequentially. Vibration feeders have the advantage of being able to separate even entangled parts. The amplitude and frequency of the vibrations as well as the slope of the feeder needs to be tuned well to achieve good separation performance.

From Daniel West’s Universal LEGO Sorting Machine

From Jacques Mattheij’s Sorting two metric tons of lego

From Chris James’s Big Robot LEGO Sorting Machine

From Johann Rocholl’s Brick feeder prototype

From Peter V’s Lego Sorting machine close-up

Also combinations of these designs are possible. The LegoLAS system by Jörn Schlingensiepen runs two vibration feeders at different slopes and thus effectively different transport speeds. This combines the sequential alignment of the vibration feeder with the longitudinal spreading of the belt step.

From Jörn Schlingensiepen’s Lego Automatic Sorting LegoLAS2.0

Stage 3 — Classification

After part separation, the bricks are transported one by one into the classification stage. Its task is to recognize/classify parts or part categories. Most classification stages rely on visual recognition via camera images. There are different approaches in how camera based scanners are designed. Driving motivation is to capture parts from multiple perspectives because sometimes parts are hard to distinguish from certain angles. Also strong, non-flickering lighting, which is decoupled from the environment is important. It allows cameras to operate on very short exposure times to minimize motion blur.

Multiple Poses — Daniel’s machine achieves capturing of multiple perspectives by letting the parts run towards the camera. The angle of observation and the scale of the part thus changes slightly while the parts are approaching. A specialty of this setup is that the classifier needs to become scale invariant. It thus can’t use absolute image dimensions as a classification cue.

From Daniel West’s Universal LEGO Sorting Machine

Multiple Cameras — Gijs van Haeff’s sorting machine instead uses multiple cameras to capture multiple perspectives simultaneously.

From Gijs van Haeff’s Universal LEGO Sorting Machine

Mirror — Johann Rocholl’s sorter features a single camera and a mirror to capture two perspectives at once. This design reduces complexity, ensures synchronization between the captured images and greatly reduces bus/CPU load compared to grabbing multiple cameras.

From Johann Rocholl’s Conveyor belt for LEGO sorting

Smartphone — Spencer Hunber’s Nexus sorting machine is using a smartphone for high quality camera and potential CNN accelerator hardware.

From Spencer Hubert’s Nexus

Scale — Part weights are a great complementary feature to standard visual input for classification. Akiyuki fitted a digital scale to his Lego sorting machine. This can be achieved using either a USB scale or via a camera looking at the display of the scale and doing basic character recognition. However, the classifier needs to be adapted to accept multimodal input. A mechanism is also needed for pushing parts onto and off the scale. The weight feature could also be useful for detecting multiple parts erroneously presented to the camera at the same time. By combining visual input with part weights, sorting machines potentially can achieve higher classification accuracy.

From Akiyuki’s LEGO Mindstorms NXT Vision Guided Brick Sorter

Stage 4 — Sorted Storage

After the correct part type has been identified inside the classification stage. The part can be directed to its correct storage location. This can be achieved using one of the following approaches.

Horizontal Gates — Usually a conveyor belt with mechanical arms which direct the part into the intended bucket. The two examples below don’t just guide the part but even actively push the parts into the bucket while closing. This allows parts to follow each other more closely on the belt.

From NealAnthoons’s Lego Sorting Machine

From Gijs van Haeff’s Universal LEGO Sorting Machine

Vertical Gates turn the principle around and replace the conveyor with a vertical channel through which the parts are falling. A gate directly above the target bin then directs them to the right place. The speed of the falling part is high thus allowing parts to be sorted quickly.

From Johann Rocholl’s Sorting Machine Flap Design

From Spencer Hubert’s Nexus

Pneumatic Sorters use a short, directed blast of air to push the parts off the belt and into the storage bin. They feature very little moving parts and typically use a solenoid valve which can be triggered from GPIO ports. Timing is crucial in order for the parts not to fly sideways into the wrong bin. A belt position encoder (e.g. optically via the wiggly line in Jacques’s design) helps measuring the exact belt and thus part position. A belt with lateral cleats (see Eppos Vision System below) helps to keep the parts on course when they fly with the air stream. Jacques Mattheij’s machine apparently even adjusts the dose of air based on the respective part size. Finally it is very helpful to use a silent air compressor, since normal ones are quite loud for long term usage. Apart from the compressor noise, personally I think pneumatic sorters at work are super relaxing to watch 😉

From Jacques Mattheij’s Sorting two metric tons of lego

From Peter V’s The Shape Sifter — First working demonstration

From Eppos Vision System’s Automatic LEGO sorting machine

Rotary Slides again use mechanical servos but instead of just binarily opening or closing a gate they choose one out of many storage locations by pointing a slide towards the target. This design is very efficient in the number of servos required. In theory multiple such slides could even be cascaded.

From Francisco Garcia’s Lego Sorter with TensorFlow on Raspberry Pi

From Claus Christiansen’s Automated AI LEGO Sorting machine

From Jörn Schlingensiepen’s Lego Automatic Sorting LegoLAS2.0

From Michael Talarczyk’s LEGO Sortierer (Color Sorter) mit raspberry pi

Picking Robots are the final variant of sorting systems. A robotic arm is grabbing the parts, moving over a 2D set of storage locations and dropping it into the respective bin. While such part handling takes more time, the two-dimensional storage matrix allows for an extremely large amount of bins.

From Tampere University’s Robotic LEGO sorting robotwith machine learning

From RBTX’s First automatic Lego™ bricks sorting system

Recent Updates

In the time since I wrote this article new machines continue to be in the making. In this section I am trying to keep up with recently published machines.

2023–12–27

360er0/awesome-lego-machine-learning gives a great overview on LEGO machine learning projects including many sorting machines.

2024–03–13

Freakstuff published a video on his automatic sorting machine. What’s special about this project is, that it is purely mechanical. It sorts LEGO parts by size. This is both very useful for subsequent manual sorting and as a preceding step for automated shape sorting. In shape sorting, part separation is a very hard task which gets much easier if parts are pre-sorted to be similar in size.

From Freakstuff’s automatic sorting machine (German)

Looking at the design in detail, the 4 typical stages are still apparent even though being grouped in an interesting way.

Bulk storage is organized as a wooden crate with a step feeder which ensures a constant in-flow of parts. The step is oriented parallel to a conveyor belt. With each stroke the step releases a well defined portion of bricks in longitudinal sequence onto the conveyor. This realizes the required part separation.

Freakstuff’s step feeder

Part classification (in terms of part size) is implemented using a sequence of sieve tubes featuring progressively larger openings. Small parts fall through the openings in the early stages, larger parts get transported further on to fall into a later bin.

The sieve openings are designed in a spiral shape. This is a very interesting choice which according to freakstuff is key for the parts to progress through the tubes in a continuous motion.

Freakstuff’s spiral shaped sieve tubes

According to Freakstuff, his machine is already used in production in his BrickLink store, followed by manual sorting for individual parts. He is planning to extend the setup by a visual classification based auto sorting stage. Classification shall be realized via the the brickognize web service. In general it will remain interesting to follow the project on his channel.

2024–05–21

In a parallel development, Smyrnoff Specialist Lego Parts has also been working on a fully mechanical sorter since January of this year. The design is similar to Freakstuff’s approach, utilizing tube modules with progressively larger dimensions for part sorting. Each module incorporates a spiral shape for efficient part transport. This machine is constructed from a combination of LEGO elements as well as 3D printed components.

Sorter for LEGO® bricks by Smyrnoff Specialist Lego Parts

Feedback and Contact

If you are also currently in the process of building a sorting machine I would love to hear from you. Also if you would like to add a missing machine/design to this list please contact me. In all cases I am easiest to reach via DM as BrickSortingMachine on Instagram.