Shared Construction Toy Building System

An experimental IoT system that enables two people to remotely collaborate on the construction of an object from interlocking bricks. The system comprises a data gathering build plate interface, computer vision algorithms to interpret the additions and removals of bricks, a mobile application to monitor the state of the virtual shared model, and a server system which synchronises data between multiple build plates and application instances.

When collaborating on a design, each participant uses a build plate interface and mobile device running the application. The participant has a selection of toy construction bricks which can be placed on the build plate interface based upon their design suggestions or to match other collaborators' design suggestions.

Interface Hardware

The build plate interface is comprised of a 12 by 12 unit construction area on which the user can construct single level interlocking brick creations using different shaped and coloured components. A built-in camera mounted above the build plate captures the building process which is then processed using a computer vision system running on the Raspberry Pi housed in the base of the unit. The system posts updated brick configuration data to the server utilising its wireless network connection.

The 3D printed base of the unit is slightly wider and deeper than the construction area. A 3D printed tower extends from the back of the base and curves toward the front such that the camera is held parallel to, and above the centre of, the construction area.

Image processing

For the system to understand changes to the brick configuration on the build plate, each image captured is processed in two stages. Stage 1 converts an image into a stud configuration which is a 12 by 12 sized array containing the colour ID for each unit of the build plate including areas which are uncertain (for example shadows or obscured regions). In stage 2, the brick configuration is updated based upon the changes to the stud configuration calculated in stage 1.

The image processing begins with a camera image of the build plate captured by the build plate interface in BGR format. This includes some of the surrounding area. Next a perspective warp and crop is applied such that the build plate corners are the corners of the image. The image is re-sized to be 12 by 12 pixels in HSV format. At this stage, pixels located where part of a brick is located on the build plate have the colour of the brick, pixels located where there are no bricks are the colour of the build plate, and pixels which are next to bricks may be slightly coloured by the brick as the pixel grid does not align directly with the brick grid due to the camera perspective. Finally, this 12 by 12 image is converted into a stud configuration which can be represented as a grid of discrete values indicating, for each unit on the build plate, if there is part of a brick present, no brick present, or uncertain.
Stage 1: Image processing. a) Camera image in BGR format, b) Perspective warp and crop applied to align build plate corners, c) Image re-sized to 12x12 pixels and converted to HSV format, d) Stud configuration where each colour represents the colour ID.

Application

The IOS application developed using Unity is used to visualise the virtual version of the collaborative creation. Changes made on different collaborators build plate interfaces are reflected in the digital model in real-time. When collaborators make changes to the shared model these changes are seen by all users. Within the app, users can select a model they want to work on by entering the model name and enable or disable editing mode. Once they have selected a model they will then see a visualisation of it showing the different collaborators contributions. Bricks for which multiple users ‘agree’ the position are shown as solid coloured bricks. Bricks for which only one user has placed are labelled to indicate which contributor suggested it. By matching the suggestion of another contributor, the brick will become solid coloured without a label indicated that it is now synchronised.

The app presents the virtual model on a rendered build plate of the same unit size as the build plate of the physical interface. The app presents additional information including the users name, the editing mode status (on or off), the current model name, and the estimated cost.

Server

The server is utilised by the interface hardware and application to synchronise changes and store model data via its API. This was developed using Python, Flask, and PyMongo and is run on a network connected computer. The API provides the functionalities needed for the interface unit to upload new brick configuration data and for the client app to get the latest data for a specific model and switch the editing mode status. The server estimates the total model cost using placeholder values for the brick costs. This could be replaced with cost estimations from a source such as the BrickLink API in the future.

Specifications

Manufacturing methods 3D printing
Development tools Python, Flask, OpenCV, Unity
Materials PLA filament, Raspberry Pi Zero, Raspberry Pi Camera, Lego bricks and build plate