Computer vision and the Science Museum Group Collection
Although museums have extensive displays and exhibition programmes, it is usual for them also have significant numbers of collection objects in storage. These objects are available for loan and for research purposes. As the Science Museum Group moves over 300,000 objects to a new storage facility, we are photographing, cataloguing and publishing these objects online.
Because of the need for a rapid digitisation programme, the approach is necessarily one of breadth rather than depth. We have therefore begun to explore the opportunities for artificial intelligence to add descriptive metadata keywords for the digitised objects.
Amazon’s Rekognition service was used to add keyword tags and these are now available as an experimental features within our collection API although they are not currently used in the functionality of the collection website. As a light-weight demonstrator, we have built What the Machine Saw, a webpage that finds a random object in the collection with keyword tags and displays them along with the object’s image and title along with links to other objects with the same tags.
What the Machine Saw was created for The Museums + AI Network conveening at the Pratt Institute, New York, 16–17 September 2019. The open source code and more detailed information are available.
It is interesting to explore these keyword tags and see the areas where:
- they generally do well but also misidentifies some images (e.g. phone, electronics, computer, toothbrush, bicycle, dress, cat, dog, etc.)
- the tags group together objects which are otherwise distributed across different areas of the collection (e.g. Barbie, mask, disk, toaster, sack, sign, etc.)
- the tags use keywords which are unlikely to be included in any formal curatorial description (e.g. hole, triangle, treasure, cross, etc.)
- the tags are fundamentally mislabeling objects but nonetheless draw together interesting groupings of objects (e.g. weapon, gold, interior design, game, modern art, etc.)
There are over 1,800 unique keyword tags applied and they form an L-shaped curve with a small number applied to several thousands of objects and a huge longtail of tags applied to hundreds of objects.
Over the coming months we will continue to explore and post about the opportunities that these tags offer.
Further reading
The following links cover other cultural heritage collections exploring the possibilities around machine-generated keyword tags:
- AI Explorer, Harvard Art Museums, displays computer vision tags from five different services and displays them side-by-side exposing the different results from each.
- RECOGNITION, which won the Tate’s 2016 IK Prize, used computer vision to pair artworks with images from news feeds.
- At the 2017 National Digital Forum conference Paul Rowe presented a talk entitled R2-D2 analysed your collection images and here’s what he found that gives an overview of a variety of approaches to computer vision.
- As part of her PhD research (pdf), Olivia Vane created themed timelines using computer generated tags from the Cooper Hewitt collection.
- SMK (National Gallery of Denmark) is exploring ways of incorporating machine generated keyword tags into their online collection search.
- Sarjeant Gallery Te Whare o Rehua Whanganui are also exploring similar approaches to collection search.
- The Barnes Foundation have also posted about their experiments with using computer vision to enable new forms of collection discovery.
