Artificial Intelligence (AI) continues to astound with its adaptability and potential as researchers from the University of Surrey and Stanford University have developed a new method to teach AI to understand human line drawings, even from non-artists.

The latest model exhibits an impressive capacity to identify scene sketches at close to human level. As explained by Dr. Yulia Gryaditskaya, a lecturer at the University of Surrey’s Centre for Vision, Speech, and Signal Processing:

"Sketching is a powerful language of visual communication, sometimes proving even more expressive and flexible than spoken language. Developing AI tools to comprehend sketches is an essential step towards enhanced human-computer interaction and more efficient design workflows. This includes the ability to search for or create images by sketching them."

Despite frequent utilisation of drawings to communicate and explore new ideas by people of all backgrounds and age groups, AI systems have generally struggled to interpret sketches. AI has had to learn to comprehend images, typically involving a laborious and time-consuming process of collecting labels for each pixel within an image.

However, the team from Surrey and Stanford have approached this problem innovatively, teaching the AI using a combination of sketches and written descriptions. The AI learned to group pixels, correlating them with categories within a given description, resulting in a more human-like comprehension of drawings compared to earlier models. The AI successfully identified and labelled objects as diverse as kites, trees, and giraffes with an 85% accuracy rate, outperforming models reliant on labelled pixels.

Alongside recognizing objects in complex scenes, the AI has the ability to determine which pen strokes pertain to each object. Remarkably, this method works effectively with informal sketches drawn by non-artists as well as drawings of objects it hasn't explicitly been trained on.

Centred in the heart of Surrey's Institute for People-Centred AI, the research seeks to comprehend the way humans see the world via the way we draw. As pointed out by the Co-director of the Institute, Prof. Yi-Zhe Song:

"This research is a prime example of how AI can enhance fundamental human activities like sketching. In terms of understanding raw drawings with near-human accuracy, this technology holds immense potential to amplify natural human creativity, regardless of artistic ability."

To share the findings, a presentation is scheduled at the international IEEE/CVF Conference on Computer Vision and Pattern Recognition set to take place in Seattle between June 17 and 21, 2024.