Scrabble Board Letter Recognition

Project Overview

For my Numerical Linear Algebra course (MTH 472), I built an end-to-end system that recognizes the letter layout on a Scrabble board from a single photo. The goal was to automatically convert a real-world board state into a machine-readable 15×15 character grid—useful for Scrabble-solver applications and for eliminating manual transcription errors.

The core algorithm is based on Eigenletters, a PCA/SVD-driven analogue of the Eigenfaces method: letter images are treated as high-dimensional vectors, projected into a low-dimensional PCA subspace, then classified using k-nearest neighbors (k-NN). This creates a compact and interpretable feature representation built directly from linear algebra.

The Challenge

Real-world Scrabble board photos present many challenges:

• Uneven lighting across the board (bright vs shadowed regions)
• Glare/reflections and textured surfaces
• Tiles are 3D objects with shading and perspective distortion
• Background clutter can dominate edge maps and contour detection
• Many non-letter squares (bonus squares / empty tiles) look “structured” enough to confuse detectors

This made preprocessing and filtering as critical as the classifier itself and turned into a mini computer vision project in addition to learn about NLA methods.

Implementation

The CV pipeline consisted of: photo → preprocessing → grid detection & segmentation → tile normalization

To classify the extracted letters, I created a synthetic dataset that mimics the normalized versions of letter tiles extracted from real photos after the CV pipeline. The next step is to classify these normalized tiles using PCA + k-NN.

The full pipeline is described in detail in the linked slides.

Results & Future Work

Achieved an accuracy of 90-094% on test boards. Most common misclassifications were between visually similar letters (O/Q, E/F, P/R, S/B) and not filtering out non-letter tiles completely.

They CV pipeline can also be improved to handle boards more robustly because if the board detection fails, the grid will be misaligned.

Reflections

This project was my first combining classical linear algebra techniques with practical computer vision methods. It also applied machine learning concepts like feature extraction and k-NN classification in a real-world setting.

In a modern context, deep learning OCR might outperform PCA, but it requires far more data and compute. Eigenletters remain an excellent learning exercise in how linear algebra can power interpretable recognition systems.