Projects:Sketchy recognition

From DiVersions
Revision as of 17:55, 19 November 2020 by Sarah (talk | contribs)
Jump to navigation Jump to search

Sketchy recognition

Nicolas Malevé & Michael Murtaugh

Sketch-recognition.pdf [1] Bread, Nose, Kangaroo or Teddy Bear?

A photograph from the collection of the Museum of Musical Instrument is processed by a contour detector algorithm. The algorithm draws the lines it found on the image sequentially. While it is tracing the contours, another algorithm, a sketch detector, tries to guess what is being drawn. Is it bread? A kangaroo? It is a teddy bear.

Sketchy Recognition is an attempt to provoke a dialogue with, and between, algorithms, visitors and museum collections.

Collection: Musical Instruments Museum (MIM)

Sketching at piano.jpg
Drawings at piano.jpg
A close reading by Anne Laforet, October 2020

Sketchy Recognition takes the form of an illustrated essay/story describing the work process of the Institute of Computational Vandalism for DiVersions. The essay is structured around a succession of documents and time-spaces, some of which were experiments by Nicolas Malevé and Michael Murtaugh.

The story starts with a conversation with Saskia Willaerts, curator of the African collections in the Musical Instruments Museum. The curator reveals some insights about the specific classification of musical instruments, which originated in 19th century Germany and was inspired by the classification of plants. The museum database is also accessible on a web portal for museums with similar collections, the aim being to call upon the African museums in this network to provide the exact names of these instruments and the people who created them. Willaerts points out the creation of a common vocabulary between the institutions and stresses that “the good thing about computers and digitization is that you don’t have to limit yourself to one specific term, you can use as many as you want to.”

The Institute of Computational Vandalism considers these algorithms as interfaces to approach volumes of archived documents. The artists have executed scraping scripts that extract data from outside of a system, in this case the database of the Musical Instruments Museum. It is an ad hoc, partial process, but it allows for autonomy and sequencing of the data according to its own use. Starting with pictures of the instruments, they have used algorithms for contour recognition which produced drawings that were subsequently processed with sketch recognition tools. This type of algorithm explains the title of the project. By adding a letter however (the ‘y’ in Sketchy Recognition), a DiVerted meaning arises: an unfinished, superficial, questionable recognition. This play on words underlines the experiments and thoughts of the collective around the names that are given to objects: by whom, through which social and technical modalities, with which power balances?

The sketch recognition algorithm they chose was QuickDraw, developed by Google, that is presented to the public as a game but in fact represents a machine learning tool that is always on the lookout for training data. New names are then given to these drawings of museum objects, names that no longer pertain to the scientific vocabulary of the museum but are derived from daily life through a North American perspective on the world: the oud is recognised by the machine as a guitar, two bells are turned into binoculars, the statue of a figurine becomes an ice cream cone... Sketchy Recognition shows us this process through short animations in which the modes of recognition and interpretation of museums and algorithms are juxtaposed and possibly collide. The Musical Instruments Museum uses taxonomies to recognise the instruments, their names, origin etc. and to produce data, meta data and images. QuickDraw uses algorithms to compare vast volumes of images to models and offers a series of hypotheses as the drawing gets clearer, in a conversational way, as if the programme were trying to guess the name of the object. (For instance, for the oud, you get: Is it a cup? A panda? A guitar? A guitar!) This triangular relation between images, museum and algorithms was expanded with an audience during the first DiVersions exhibit, where visitors were asked to colour the drawings based on the images of the collections. Thanks to a capturing device for the drawings and a screen, the drawing person experienced the real time recognition process of QuickDraw and its dialogue function, and he or she could then decide whether to follow the hints of the algorithm or, on the contrary, to adopt a critical position.

This entertaining DiVersion, exploring the offset between two pieces, the difference in approach between a museum and a machine, makes us think about the fissures, frictions and frontiers between recognition, interpretation and identification, and the ensuing regimes of authority and truth. The Institute of Computational Vandalism answers its own questions with experiments that diffract and divert recognition systems. They create absurd and joyful probes where both informational systems collide and mix, for instance when the terms identified by QuickDraw for the museum objects end up in a cloned interface of the Musical Instruments Museum (Find the Error), or when the names of musical instruments are haphazardly displayed alongside words that were recognised by the algorithms to identify them (Class Roulette).

Schetsmatige herkenning

Sketch-recognition.pdf [2]


Brood, Neus, Kangoeroe of Teddybeer?

Een foto uit de collectie van het Museum of Musical Instrument wordt verwerkt door een algoritme om contouren te detecteren. Het algoritme tekent de lijnen die het op de foto heeft gevonden een voor een. Terwijl het de contouren traceert, probeert een ander algoritme, een schetsdetector, te raden wat er wordt getekend. Is het brood? Een kangoeroe? Het is een teddybeer. Schetsmatige herkenning (werktitel) is een poging om een dialoog uit te lokken met en tussen algoritmes, bezoekers en museumcollecties.

Collectie: Musical Instruments Museum (MIM)

Sketching at piano.jpg
Drawings at piano.jpg
Een aandachtige lezing van Anne Laforet, oktober 2020

Sketchy Recognition takes the form of an illustrated essay/story describing the work process of the Institute of Computational Vandalism for DiVersions. The essay is structured around a succession of documents and time-spaces, some of which were experiments by Nicolas Malevé and Michael Murtaugh.

The story starts with a conversation with Saskia Willaerts, curator of the African collections in the Musical Instruments Museum. The curator reveals some insights about the specific classification of musical instruments, which originated in 19th century Germany and was inspired by the classification of plants. The museum database is also accessible on a web portal for museums with similar collections, the aim being to call upon the African museums in this network to provide the exact names of these instruments and the people who created them. Willaerts points out the creation of a common vocabulary between the institutions and stresses that “the good thing about computers and digitization is that you don’t have to limit yourself to one specific term, you can use as many as you want to.”

The Institute of Computational Vandalism considers these algorithms as interfaces to approach volumes of archived documents. The artists have executed scraping scripts that extract data from outside of a system, in this case the database of the Musical Instruments Museum. It is an ad hoc, partial process, but it allows for autonomy and sequencing of the data according to its own use. Starting with pictures of the instruments, they have used algorithms for contour recognition which produced drawings that were subsequently processed with sketch recognition tools. This type of algorithm explains the title of the project. By adding a letter however (the ‘y’ in Sketchy Recognition), a DiVerted meaning arises: an unfinished, superficial, questionable recognition. This play on words underlines the experiments and thoughts of the collective around the names that are given to objects: by whom, through which social and technical modalities, with which power balances?

The sketch recognition algorithm they chose was QuickDraw, developed by Google, that is presented to the public as a game but in fact represents a machine learning tool that is always on the lookout for training data. New names are then given to these drawings of museum objects, names that no longer pertain to the scientific vocabulary of the museum but are derived from daily life through a North American perspective on the world: the oud is recognised by the machine as a guitar, two bells are turned into binoculars, the statue of a figurine becomes an ice cream cone... Sketchy Recognition shows us this process through short animations in which the modes of recognition and interpretation of museums and algorithms are juxtaposed and possibly collide. The Musical Instruments Museum uses taxonomies to recognise the instruments, their names, origin etc. and to produce data, meta data and images. QuickDraw uses algorithms to compare vast volumes of images to models and offers a series of hypotheses as the drawing gets clearer, in a conversational way, as if the programme were trying to guess the name of the object. (For instance, for the oud, you get: Is it a cup? A panda? A guitar? A guitar!) This triangular relation between images, museum and algorithms was expanded with an audience during the first DiVersions exhibit, where visitors were asked to colour the drawings based on the images of the collections. Thanks to a capturing device for the drawings and a screen, the drawing person experienced the real time recognition process of QuickDraw and its dialogue function, and he or she could then decide whether to follow the hints of the algorithm or, on the contrary, to adopt a critical position.

This entertaining DiVersion, exploring the offset between two pieces, the difference in approach between a museum and a machine, makes us think about the fissures, frictions and frontiers between recognition, interpretation and identification, and the ensuing regimes of authority and truth. The Institute of Computational Vandalism answers its own questions with experiments that diffract and divert recognition systems. They create absurd and joyful probes where both informational systems collide and mix, for instance when the terms identified by QuickDraw for the museum objects end up in a cloned interface of the Musical Instruments Museum (Find the Error), or when the names of musical instruments are haphazardly displayed alongside words that were recognised by the algorithms to identify them (Class Roulette).

Reconnaissance esquissée

Sketch-recognition.pdf [3]


Du pain, un nez, un kangourou ou un ours en peluche?

Une photo de la collection du Musée des Instruments de Musique est traitée par un algorithme de détection de contours. L'algorithme dessine les lignes qu'il trouve dans l'image l’une après l'autre. Pendant qu’il trace les lignes, un autre algorithme, un détecteur de croquis, essaie de deviner ce que les lignes représentent. Est-ce du pain? Un kangourou? C'est un ours en peluche.

Reconnaissance Esquissée (titre provisoire) essaie de provoquer un dialogue entre et avec les algorithmes, les visiteur.euse.s et les collections de musées.


Collection: Musical Instruments Museum (MIM)

Sketching at piano.jpg
Drawings at piano.jpg
Une lecture attentive par Anne Laforet, octobre 2020

Reconnaissance Esquissée (Sketchy Recognition) se présente sous la forme d'un essai-récit illustré qui décrit le processus de travail de l'Institute of Computational Vandalism pour DiVersions. Cet essai s'organise autour d'une succession de documents et d'espaces-temps, dont certains sont des expérimentations de Nicolas Malevé et Michael Murtaugh.

Le récit commence par un entretien avec la conservatrice Saskia Willaerts chargée des collections africaines au Musée des instruments de musique. La conservatrice donne quelques clés sur la classification spécifique aux instruments de musique, créée en Allemagne au 19ème siècle, classification inspirée par celle des plantes. La base de données du musée est également disponible sur un portail muséal ayant des collections similaires, avec le but de faire intervenir les musées africains de ce réseau pour indiquer les noms exacts des instruments et des personnes qui les ont créés. Willaerts met en avant la création d'un vocabulaire commun entre les institutions et souligne que " The good thing about computers and digitization is that you don't have to limit yourself to one specific term, you can use as many as you want to."

L'Institute of Computational Vandalism considère les algorithmes comme des interlocuteurs pour approcher des volumes de documents archivés. Les artistes ont effectué des opérations de scraping (du verbe anglais scrape pour gratter la surface), des scripts qui extraient les données depuis l'extérieur d'un système, sur la base de données du Musée des instruments de musique. C'est un processus ad hoc, partial, mais qui permet une autonomie et un séquençage des données selon son propre usage. À partir des photos des instruments, ils ont utilisé des algorithmes de reconnaissance de contours qui ont généré des dessins qui sont alors passé au crible des outils de reconnaissance de croquis (Sketch Recognition en anglais). Ce type d’algorithme explicite le titre du projet : en ajoutant une lettre (le titre original de leur projet est Sketchy Recognition), un sens DiVergent apparaît : une reconnaissance inachevée, superficielle, douteuse. Un jeu de mots qui souligne les expérimentations et réflexions du collectif autour des noms qui sont donnés à ces objets, par qui, sous quels modalités socio-techniques, avec quel rapport de forces.

Comme algorithme de reconnaissance des croquis, ils ont choisi QuickDraw, développé par Google, qui est proposé au public comme un jeu mais est en fait une démonstration d’un outil d'apprentissage automatique (machine learning) en perpétuelle recherche de données d'entraînement. De nouveaux noms sont alors donnés aux dessins d’objets de musée, qui ne relève plus du vocabulaire scientifique du musée mais de termes du quotidien perçu à travers une expérience nord-américaine du monde : un oud est vu par la machine comme une guitare, deux cloches comme des jumelles, la statuette comme un cornet de glace... Reconnaîssance esquissée montre ce processus sous forme de courtes animations où sont mis en dialogue, voire en collision, les modes de reconnaissance et d'interprétation muséaux et algorithmiques. Le Musée des instruments de musique utilise des taxinomies pour reconnaître les instruments, leurs noms, provenances, etc., et produire des données et métadonnées ainsi que des images. Quick draw emploie des algorithmes pour comparer des volumes importants d'images à des modèles, et propose une série d'hypothèses au fur et à mesure que le dessin se précise, de manière conversationnelle comme si le programme devinait le nom de l'objet (pour l’oud, cela donne : est-ce une tasse, un panda, une guitare ? Une guitare !). À ce triangle images, musée, algorithme, s'ajoute le public qui lors de la première exposition de DiVersions fut invité à colorier les dessins issus des images de la collection. Grâce à un dispositif de capture des dessins et à un écran, le·a dessinateur·rice était confronté·e à la reconnaissance en temps réel de QuickDraw et son mode de dialogue, et pouvait décider de suivre les indices de l'algorithme, ou au contraire adopter une position critique.

Ce jeu (employé ici autant dans le sens de DiVertissement que de marge entre deux pièces) entre regarder comme un musée et regarder comme une machine interroge sur les interstices, frictions et frontières entre reconnaissance, interprétation et identification, et les régimes d'autorité et de vérité qui en découlent. À ses propres questions, l'Institute of Computational Vandalism répond par des expérimentations qui diffractent et dévient les systèmes de reconnaissance. Ils créent des sondes absurdes et joyeuses où les deux systèmes informationnels se heurtent et se mélangent. Comme dans Find the Error où les termes identifiés par QuickDraw pour les objets de musée se retrouvent dans une interface clonée du Musée des instruments de musique ou comme dans Class Roulette, où les noms d'instruments sont affichés aléatoirement à côté de mots reconnus par les algorithmes pour identifier les mêmes instruments .