Projects:Sketchy recognition

Sketchy recognition

v2: 10.10.2019

Nicolas Malevé & Michael Murtaugh

A conversation with Nicolas and Michael, June 2020

[1]

Bread, Nose, Kangaroo or Teddy Bear?

A photograph from the collection of the Museum of Musical Instrument is processed by a contour detector algorithm. The algorithm draws the lines it found on the image sequentially. While it is tracing the contours, another algorithm, a sketch detector, tries to guess what is being drawn. Is it bread? A kangaroo? It is a teddy bear.

Sketchy Recognition (working title) is an attempt to provoke a dialogue with, and between, algorithms, visitors and museum collections.

Cast:

• Musical instruments: MIM collection, Brussels.
• Line detector: The Hough algorithm in the OpenCV toolbox, originally developed to analyse bubble chamber photographs.
• Sketch recognizer: an algorithm based on the research of Mathias Eitz, James Hays and Marc Alexa (2012), and the code and models by Jean-Baptiste Alayrac.
• Data: from the hands of the many volunteers who contributed to Google's Quick, Draw! Dataset.
• Special sauce, bugs and fixes: Michael and Nicolas

(Re)sources

• Code for this project
• You were asked to draw an angel, Working notes from the Scandinavian Institute for Computational Vandalism (April 2017)
• Assisted drawing, Working notes from the Scandinavian Institute for Computational Vandalism (January 2016) + Assisted drawing: Exploring Augmented Creativity, original blogpost by Samim (December 2015)
• How Do Humans Sketch Objects?, Mathias Eitz, James Hays and Marc Alexa (2012) + C/C++ and Python/Jupyter implementations
• sketch-recognizer, Jean-Baptist Alayrac's working Python code that we ended up using

Collection: Musical Instruments Museum (MIM)

v1: 10.10.2019

Michael Murtaugh & Nicolas Malevé

A close reading by Anne Laforet, October 2020

Sketchy Recognition takes the form of an illustrated essay/story describing the work process of the Institute of Computational Vandalism for DiVersions. The essay is structured around a succession of documents and time-spaces, some of which were experiments by Nicolas Malevé and Michael Murtaugh.

The story starts with a conversation with Saskia Willaerts, curator of the African collections in the Musical Instruments Museum. The curator reveals some insights about the specific classification of musical instruments, which originated in 19th century Germany and was inspired by the classification of plants. The museum database is also accessible on a web portal for museums with similar collections, the aim being to call upon the African museums in this network to provide the exact names of these instruments and the people who created them. Willaerts points out the creation of a common vocabulary between the institutions and stresses that “the good thing about computers and digitization is that you don’t have to limit yourself to one specific term, you can use as many as you want to.”

The Institute of Computational Vandalism considers these algorithms as interfaces to approach volumes of archived documents. The artists have executed scraping scripts that extract data from outside of a system, in this case the database of the Musical Instruments Museum. It is an ad hoc, partial process, but it allows for autonomy and sequencing of the data according to its own use. Starting with pictures of the instruments, they have used algorithms for contour recognition which produced drawings that were subsequently processed with sketch recognition tools. This type of algorithm explains the title of the project. By adding a letter however (the ‘y’ in Sketchy Recognition), a DiVerted meaning arises: an unfinished, superficial, questionable recognition. This play on words underlines the experiments and thoughts of the collective around the names that are given to objects: by whom, through which social and technical modalities, with which power balances?

The sketch recognition algorithm they chose was QuickDraw, developed by Google, that is presented to the public as a game but in fact represents a machine learning tool that is always on the lookout for training data. New names are then given to these drawings of museum objects, names that no longer pertain to the scientific vocabulary of the museum but are derived from daily life through a North American perspective on the world: the oud is recognised by the machine as a guitar, two bells are turned into binoculars, the statue of a figurine becomes an ice cream cone... Sketchy Recognition shows us this process through short animations in which the modes of recognition and interpretation of museums and algorithms are juxtaposed and possibly collide. The Musical Instruments Museum uses taxonomies to recognise the instruments, their names, origin etc. and to produce data, meta data and images. QuickDraw uses algorithms to compare vast volumes of images to models and offers a series of hypotheses as the drawing gets clearer, in a conversational way, as if the programme were trying to guess the name of the object. (For instance, for the oud, you get: Is it a cup? A panda? A guitar? A guitar!) This triangular relation between images, museum and algorithms was expanded with an audience during the first DiVersions exhibit, where visitors were asked to colour the drawings based on the images of the collections. Thanks to a capturing device for the drawings and a screen, the drawing person experienced the real time recognition process of QuickDraw and its dialogue function, and he or she could then decide whether to follow the hints of the algorithm or, on the contrary, to adopt a critical position.

This entertaining DiVersion, exploring the offset between two pieces, the difference in approach between a museum and a machine, makes us think about the fissures, frictions and frontiers between recognition, interpretation and identification, and the ensuing regimes of authority and truth. The Institute of Computational Vandalism answers its own questions with experiments that diffract and divert recognition systems. They create absurd and joyful probes where both informational systems collide and mix, for instance when the terms identified by QuickDraw for the museum objects end up in a cloned interface of the Musical Instruments Museum (Find the Error), or when the names of musical instruments are haphazardly displayed alongside words that were recognised by the algorithms to identify them (Class Roulette).

Schetsmatige herkenning

v2: 10.10.2019

Nicolas Malevé & Michael Murtaugh

Een gesprek met Nicolas en Michael, juni 2020

[2]

Schetsmatige herkenning

Brood, Neus, Kangoeroe of Teddybeer?

Een foto uit de collectie van het Museum of Musical Instrument wordt verwerkt door een algoritme om contouren te detecteren. Het algoritme tekent de lijnen die het op de foto heeft gevonden een voor een. Terwijl het de contouren traceert, probeert een ander algoritme, een schetsdetector, te raden wat er wordt getekend. Is het brood? Een kangoeroe? Het is een teddybeer. Schetsmatige herkenning (werktitel) is een poging om een dialoog uit te lokken met en tussen algoritmes, bezoekers en museumcollecties.

Cast:

• Muziekinstrumenten: MIM-collectie, Brussel.
• Lijndetector: Het Hough algoritme in de OpenCV toolbox, oorspronkelijk ontwikkeld voor het analyseren van foto's van van subatomaire deeltjes gegenereerd in bellenvaten.
• Schetsherkenner: een algoritme gebaseerd op het onderzoek van Mathias Eitz, James Hays en Marc Alexa (2012), en de code en modellen van Jean-Baptiste Alayrac.
• Data: uit de handen van de vele vrijwilligers die hebben bijgedragen aan Google's Quick, Draw! Dataset.
• Speciale saus, bugs en fixes: Michael en Nicolas

Bronnen

• Code voor dit project
• You were asked to draw an angel, Notities van het Scandinavian Institute for Computational Vandalism (april 2017)
• Assisted drawing, Notities van het Scandinavian Institute for Computational Vandalism (januari 2016) + Assisted drawing: Exploring Augmented Creativity, oorspronkelijke blogpost van Samim (december 2015)
• How Do Humans Sketch Objects?, Mathias Eitz, James Hays en Marc Alexa (2012) + C/C++ en Python/Jupyter implementaties
• sketch-recognizer, Jean-Baptist Alayrac's Python code waar we uiteindelijk mee aan de slag gingen.

Collectie: Musical Instruments Museum (MIM)

v1: 10.10.2019

Michael Murtaugh & Nicolas Malevé

Een aandachtige lezing van Anne Laforet, oktober 2020

Sketchy Recognition takes the form of an illustrated essay/story describing the work process of the Institute of Computational Vandalism for DiVersions. The essay is structured around a succession of documents and time-spaces, some of which were experiments by Nicolas Malevé and Michael Murtaugh.

The story starts with a conversation with Saskia Willaerts, curator of the African collections in the Musical Instruments Museum. The curator reveals some insights about the specific classification of musical instruments, which originated in 19th century Germany and was inspired by the classification of plants. The museum database is also accessible on a web portal for museums with similar collections, the aim being to call upon the African museums in this network to provide the exact names of these instruments and the people who created them. Willaerts points out the creation of a common vocabulary between the institutions and stresses that “the good thing about computers and digitization is that you don’t have to limit yourself to one specific term, you can use as many as you want to.”

The Institute of Computational Vandalism considers these algorithms as interfaces to approach volumes of archived documents. The artists have executed scraping scripts that extract data from outside of a system, in this case the database of the Musical Instruments Museum. It is an ad hoc, partial process, but it allows for autonomy and sequencing of the data according to its own use. Starting with pictures of the instruments, they have used algorithms for contour recognition which produced drawings that were subsequently processed with sketch recognition tools. This type of algorithm explains the title of the project. By adding a letter however (the ‘y’ in Sketchy Recognition), a DiVerted meaning arises: an unfinished, superficial, questionable recognition. This play on words underlines the experiments and thoughts of the collective around the names that are given to objects: by whom, through which social and technical modalities, with which power balances?

The sketch recognition algorithm they chose was QuickDraw, developed by Google, that is presented to the public as a game but in fact represents a machine learning tool that is always on the lookout for training data. New names are then given to these drawings of museum objects, names that no longer pertain to the scientific vocabulary of the museum but are derived from daily life through a North American perspective on the world: the oud is recognised by the machine as a guitar, two bells are turned into binoculars, the statue of a figurine becomes an ice cream cone... Sketchy Recognition shows us this process through short animations in which the modes of recognition and interpretation of museums and algorithms are juxtaposed and possibly collide. The Musical Instruments Museum uses taxonomies to recognise the instruments, their names, origin etc. and to produce data, meta data and images. QuickDraw uses algorithms to compare vast volumes of images to models and offers a series of hypotheses as the drawing gets clearer, in a conversational way, as if the programme were trying to guess the name of the object. (For instance, for the oud, you get: Is it a cup? A panda? A guitar? A guitar!) This triangular relation between images, museum and algorithms was expanded with an audience during the first DiVersions exhibit, where visitors were asked to colour the drawings based on the images of the collections. Thanks to a capturing device for the drawings and a screen, the drawing person experienced the real time recognition process of QuickDraw and its dialogue function, and he or she could then decide whether to follow the hints of the algorithm or, on the contrary, to adopt a critical position.

This entertaining DiVersion, exploring the offset between two pieces, the difference in approach between a museum and a machine, makes us think about the fissures, frictions and frontiers between recognition, interpretation and identification, and the ensuing regimes of authority and truth. The Institute of Computational Vandalism answers its own questions with experiments that diffract and divert recognition systems. They create absurd and joyful probes where both informational systems collide and mix, for instance when the terms identified by QuickDraw for the museum objects end up in a cloned interface of the Musical Instruments Museum (Find the Error), or when the names of musical instruments are haphazardly displayed alongside words that were recognised by the algorithms to identify them (Class Roulette).

Reconnaissance esquissée

v2: 10.10.2019

Nicolas Malevé & Michael Murtaugh

Une conversation avec Nicolas et Michael, juin 2020

[3]

Reconnaissance Esquissée

Du pain, un nez, un kangourou ou un ours en peluche?

Une photo de la collection du Musée des Instruments de Musique est traitée par un algorithme de détection de contours. L'algorithme dessine les lignes qu'il trouve dans l'image l’une après l'autre. Pendant qu’il trace les lignes, un autre algorithme, un détecteur de croquis, essaie de deviner ce que les lignes représentent. Est-ce du pain? Un kangourou? C'est un ours en peluche.

Reconnaissance Esquissée (titre provisoire) essaie de provoquer un dialogue entre et avec les algorithmes, les visiteur.euse.s et les collections de musées.

Distribution:

• Instruments de musique: MIM collection, Bruxelles.
• Détecteur de contour: L'algorithme de Hough dans la boîte à outils OpenCV, développé à l'origine pour détecter les lignes dans les photographies de chambre à bulles.
• Reconnaissance de croquis: une algorithme basé sur la recherche de Mathias Eitz, James Hays et Marc Alexa (2012), et le code et les modèles de Jean-Baptiste Alayrac.
• Données: dessinées par les nombreux volontaires qui ont contribué à Google’s Quick, Draw! Dataset.
• Sauce spéciale, bugs et corrections: Michael et Nicolas

(Re)sources

• Code pour ce projet
• You were asked to draw an angel, Notes de travail du Scandinavian Institute for Computational Vandalism (avril 2017)
• Assisted drawing, Notes de travail du Scandinavian Institute for Computational Vandalism (janvier 2016) + Assisted drawing: Exploring Augmented Creativity, blogpost original par Samim (décembre 2015)
• How Do Humans Sketch Objects?, Mathias Eitz, James Hays et Marc Alexa (2012) + C/C++ et Python/Jupyter implémentations
• sketch-recognizer, le code Python de Jean-Baptist Alayrac que nous avons fini par utiliser

Collection: Musical Instruments Museum (MIM)

v1: 10.10.2019

Michael Murtaugh & Nicolas Malevé

Une lecture attentive par Anne Laforet, octobre 2020

Reconnaissance Esquissée (Sketchy Recognition) se présente sous la forme d'un essai-récit illustré qui décrit le processus de travail de l'Institute of Computational Vandalism pour DiVersions. Cet essai s'organise autour d'une succession de documents et d'espaces-temps, dont certains sont des expérimentations de Nicolas Malevé et Michael Murtaugh.

Le récit commence par un entretien avec la conservatrice Saskia Willaerts chargée des collections africaines au Musée des instruments de musique. La conservatrice donne quelques clés sur la classification spécifique aux instruments de musique, créée en Allemagne au 19ème siècle, classification inspirée par celle des plantes. La base de données du musée est également disponible sur un portail muséal ayant des collections similaires, avec le but de faire intervenir les musées africains de ce réseau pour indiquer les noms exacts des instruments et des personnes qui les ont créés. Willaerts met en avant la création d'un vocabulaire commun entre les institutions et souligne que " The good thing about computers and digitization is that you don't have to limit yourself to one specific term, you can use as many as you want to."

L'Institute of Computational Vandalism considère les algorithmes comme des interlocuteurs pour approcher des volumes de documents archivés. Les artistes ont effectué des opérations de scraping (du verbe anglais scrape pour gratter la surface), des scripts qui extraient les données depuis l'extérieur d'un système, sur la base de données du Musée des instruments de musique. C'est un processus ad hoc, partial, mais qui permet une autonomie et un séquençage des données selon son propre usage. À partir des photos des instruments, ils ont utilisé des algorithmes de reconnaissance de contours qui ont généré des dessins qui sont alors passé au crible des outils de reconnaissance de croquis (Sketch Recognition en anglais). Ce type d’algorithme explicite le titre du projet : en ajoutant une lettre (le titre original de leur projet est Sketchy Recognition), un sens DiVergent apparaît : une reconnaissance inachevée, superficielle, douteuse. Un jeu de mots qui souligne les expérimentations et réflexions du collectif autour des noms qui sont donnés à ces objets, par qui, sous quels modalités socio-techniques, avec quel rapport de forces.

Comme algorithme de reconnaissance des croquis, ils ont choisi QuickDraw, développé par Google, qui est proposé au public comme un jeu mais est en fait une démonstration d’un outil d'apprentissage automatique (machine learning) en perpétuelle recherche de données d'entraînement. De nouveaux noms sont alors donnés aux dessins d’objets de musée, qui ne relève plus du vocabulaire scientifique du musée mais de termes du quotidien perçu à travers une expérience nord-américaine du monde : un oud est vu par la machine comme une guitare, deux cloches comme des jumelles, la statuette comme un cornet de glace... Reconnaîssance esquissée montre ce processus sous forme de courtes animations où sont mis en dialogue, voire en collision, les modes de reconnaissance et d'interprétation muséaux et algorithmiques. Le Musée des instruments de musique utilise des taxinomies pour reconnaître les instruments, leurs noms, provenances, etc., et produire des données et métadonnées ainsi que des images. Quick draw emploie des algorithmes pour comparer des volumes importants d'images à des modèles, et propose une série d'hypothèses au fur et à mesure que le dessin se précise, de manière conversationnelle comme si le programme devinait le nom de l'objet (pour l’oud, cela donne : est-ce une tasse, un panda, une guitare ? Une guitare !). À ce triangle images, musée, algorithme, s'ajoute le public qui lors de la première exposition de DiVersions fut invité à colorier les dessins issus des images de la collection. Grâce à un dispositif de capture des dessins et à un écran, le·a dessinateur·rice était confronté·e à la reconnaissance en temps réel de QuickDraw et son mode de dialogue, et pouvait décider de suivre les indices de l'algorithme, ou au contraire adopter une position critique.

Ce jeu (employé ici autant dans le sens de DiVertissement que de marge entre deux pièces) entre regarder comme un musée et regarder comme une machine interroge sur les interstices, frictions et frontières entre reconnaissance, interprétation et identification, et les régimes d'autorité et de vérité qui en découlent. À ses propres questions, l'Institute of Computational Vandalism répond par des expérimentations qui diffractent et dévient les systèmes de reconnaissance. Ils créent des sondes absurdes et joyeuses où les deux systèmes informationnels se heurtent et se mélangent. Comme dans Find the Error où les termes identifiés par QuickDraw pour les objets de musée se retrouvent dans une interface clonée du Musée des instruments de musique ou comme dans Class Roulette, où les noms d'instruments sont affichés aléatoirement à côté de mots reconnus par les algorithmes pour identifier les mêmes instruments .