Capture real time images of chess games and convert them to into FEN strings for stockfish analysis
A good paper: https://mdpi-res.com/d_attachment/jimaging/jimaging-07-00094/article_deploy/jimaging-07-00094-v2.pdf?version=1622796868
There is a major problem evident in this paper: See page 14 figure 12 Mean inference time on CPU: 2.11 +- 0.64 s Mean inference time on GPU: 0.35 +- 0.06 s The CPU inference time is slow, and we can assume that the rpi4 will be even slower. The CPU used was a 3.20 GHz Intel Core i5-6500 CPU. Rpi is not this good. Most of this time is taken up by the piece classification CNN.
Current State:
-
Start with image of chess board:
-
Use Canny edge detection to find true edges. This finds the edges in the image but they are 'wobbly' and also don't extend all the way to the end of the image which could be useful for cropping.
-
To improve this I used statistical methods to find the pixel locations where the edges occured and created an image of these edges extended through the whole image. Again, this could be useful for cropping and defining board coordinate systems. It also may be helpful for removing the extraneous stuff like the row/column letter number labels. This is NOT a Hough transform but achieves a very similiar end result.
Methodology:
First we find the average gray scale pixel value in each row and column of the image:
The overall mean pixel value of the image:
And the standard deviations of rows and columns:
We consider a row/column to be an edge if the following condition is true:
-
We can overlay this new grid ontop of the original image:
-
Next, we can use this grid to find corners in the image using some algorithm. I tried out Harris Corner detection for now but there are a few options
6.Finally, we can overlay these corners onto the original image:
The above methodology will fail for skewed images or images where the board is rotated in frame. The failure occurs at step 3 since the statistical analysis assumes the camera is top down view. The error will scale with rotation angle or skew. Example of this effect:
We can fix this problem by using a Hough transform instead.