publications

2023

1. How games can make behavioural science better Long, Bria, Simson, Jan, Buxó-Lugo, Andrés, Watson, Duane G., and Mehr, Samuel A. Nature 2023 [abstract] [doi] [www.nature.com]

   Wordle, Minecraft and Scrabble are played online by millions. Gamifying experiments can make behavioural research more inclusive, rigorous and reproducible — if it’s done right.

2. occupationMeasurement: A Comprehensive Toolbox for Interactive Occupation Coding in Surveys Simson, Jan, Kononykhina, Olga, and Schierholz, Malte Journal of Open Source Software 2023 [abstract] [doi] [joss.theoj.org]

   Simson et al., (2023). occupationMeasurement: A Comprehensive Toolbox for Interactive Occupation Coding in Surveys. Journal of Open Source Software, 8(88), 5505, https://doi.org/10.21105/joss.05505

3. Everything, Everywhere All in One Evaluation: Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness Simson, Jan, Pfisterer, Florian, and Kern, Christoph arXiv:2308.16681 [cs, stat] 2023 [abstract] [doi] [arxiv.org]

   A vast number of systems across the world use algorithmic decision making (ADM) to (partially) automate decisions that have previously been made by humans. When designed well, these systems promise more objective decisions while saving large amounts of resources and freeing up human time. However, when ADM systems are not designed well, they can lead to unfair decisions which discriminate against societal groups. The downstream effects of ADMs critically depend on the decisions made during the systems’ design and implementation, as biases in data can be mitigated or reinforced along the modeling pipeline. Many of these design decisions are made implicitly, without knowing exactly how they will influence the final system. It is therefore important to make explicit the decisions made during the design of ADM systems and understand how these decisions affect the fairness of the resulting system. To study this issue, we draw on insights from the field of psychology and introduce the method of multiverse analysis for algorithmic fairness. In our proposed method, we turn implicit design decisions into explicit ones and demonstrate their fairness implications. By combining decisions, we create a grid of all possible "universes" of decision combinations. For each of these universes, we compute metrics of fairness and performance. Using the resulting dataset, one can see how and which decisions impact fairness. We demonstrate how multiverse analyses can be used to better understand variability and robustness of algorithmic fairness using an exemplary case study of predicting public health coverage of vulnerable populations for potential interventions. Our results illustrate how decisions during the design of a machine learning system can have surprising effects on its fairness and how to detect these effects using multiverse analysis.

2022

1. Acoustic regularities in infant-directed speech and song across cultures Hilton, Courtney B., Moser, Cody J., Bertolo, Mila, Lee-Rubin, Harry, Amir, Dorsa, Bainbridge, Constance M., Simson, Jan, Knox, Dean, Glowacki, Luke, Alemu, Elias, Galbarczyk, Andrzej, Jasienska, Grazyna, Ross, Cody T., Neff, Mary Beth, Martin, Alia, Cirelli, Laura K., Trehub, Sandra E., Song, Jinqi, Kim, Minju, Schachner, Adena, Vardy, Tom A., Atkinson, Quentin D., Salenius, Amanda, Andelin, Jannik, Antfolk, Jan, Madhivanan, Purnima, Siddaiah, Anand, Placek, Caitlyn D., Salali, Gul Deniz, Keestra, Sarai, Singh, Manvir, Collins, Scott A., Patton, John Q., Scaff, Camila, Stieglitz, Jonathan, Cutipa, Silvia Ccari, Moya, Cristina, Sagar, Rohan R., Anyawire, Mariamu, Mabulla, Audax, Wood, Brian M., Krasnow, Max M., and Mehr, Samuel A. Nature Human Behaviour 2022 [abstract] [doi] [www.nature.com]

   When interacting with infants, humans often alter their speech and song in ways thought to support communication. Theories of human child-rearing, informed by data on vocal signalling across species, predict that such alterations should appear globally. Here, we show acoustic differences between infant-directed and adult-directed vocalizations across cultures. We collected 1,615 recordings of infant- and adult-directed speech and song produced by 410 people in 21 urban, rural and small-scale societies. Infant-directedness was reliably classified from acoustic features only, with acoustic profiles of infant-directedness differing across language and music but in consistent fashions. We then studied listener sensitivity to these acoustic features. We played the recordings to 51,065 people from 187 countries, recruited via an English-language website, who guessed whether each vocalization was infant-directed. Their intuitions were more accurate than chance, predictable in part by common sets of acoustic features and robust to the effects of linguistic relatedness between vocalizer and listener. These findings inform hypotheses of the psychological functions and evolution of human communication. Across 21 societies, people alter their speech and song when interacting with infants. These infant-directed vocalizations are recognized by listeners. This suggests that forms of human vocalizations may be shaped by their functions.

2020

1. Infants relax in response to unfamiliar foreign lullabies Bainbridge, Constance M., Bertolo, Mila, Youngers, Julie, Atwood, S., Yurdum, Lidya, Simson, Jan, Lopez, Kelsie, Xing, Feng, Martin, Alia, and Mehr, Samuel A. Nature Human Behaviour 2020 [abstract] [doi] [www.nature.com]

   Music is characterized by acoustic forms that are predictive of its behavioural functions. For example, adult listeners accurately identify unfamiliar lullabies as infant-directed on the basis of their musical features alone. This property could reflect a function of listeners’ experiences, the basic design of the human mind, or both. Here, we show that US infants (N = 144) relax in response to eight unfamiliar foreign lullabies, relative to matched non-lullaby songs from other foreign societies, as indexed by heart rate, pupillometry and electrodermal activity. They do so consistently throughout the first year of life, suggesting that the response is not a function of their musical experiences, which are limited relative to those of adults. The infants’ parents overwhelmingly chose lullabies as the songs that they would use to calm their fussy infant, despite their unfamiliarity. Together, these findings suggest that infants may be predisposed to respond to common features of lullabies found in different cultures.

2019

1. Bright: an augmented reality assistive platform for visual impairment Bakshi, A. M., Simson, J., Castro, C. de, Yu, C. C., and Dias, A. 2019 IEEE Games, Entertainment, Media Conference (GEM) 2019 [abstract] [doi]

   283 million people suffer from moderate to severe vision impairment or blindness; many today rely on lens-based optical aids of limited use. Electronic pass-through zoom headsets are extremely expensive and provide narrow feature sets with compromised comfort and ease-of-use, while mobile app-based aids provide useful functionality but in an inconvenient form factor. The present project explores the development of an aid for the visually impaired, designed as a software solution running on off-the-shelf optical see-through augmented reality head-mounted display devices such as the Microsoft HoloLens, to deliver a broad feature set while reducing cost and optimizing comfort and usability.

2. Universality and diversity in human song Mehr, Samuel A., Singh, Manvir, Knox, Dean, Ketter, Daniel M., Pickens-Jones, Daniel, Atwood, S., Lucas, Christopher, Jacoby, Nori, Egner, Alena A., Hopkins, Erin J., Howard, Rhea M., Hartshorne, Joshua K., Jennings, Mariela V., Simson, Jan, Bainbridge, Constance M., Pinker, Steven, O’Donnell, Timothy J., Krasnow, Max M., and Glowacki, Luke Science 2019 [abstract] [doi] [science.sciencemag.org]

   What is universal about music, and what varies? We built a corpus of ethnographic text on musical behavior from a representative sample of the world’s societies, as well as a discography of audio recordings. The ethnographic corpus reveals that music (including songs with words) appears in every society observed; that music varies along three dimensions (formality, arousal, religiosity), more within societies than across them; and that music is associated with certain behavioral contexts such as infant care, healing, dance, and love. The discography—analyzed through machine summaries, amateur and expert listener ratings, and manual transcriptions—reveals that acoustic features of songs predict their primary behavioral context; that tonality is widespread, perhaps universal; that music varies in rhythmic and melodic complexity; and that elements of melodies and rhythms found worldwide follow power laws.

1. What If? Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness Simson, Jan, Pfisterer, Florian, and Kern, Christoph Proceedings of the 2nd European Workshop on Algorithmic Fairness 2023 [ceur-ws.org]
2. A Holistic Toolbox for Interactive Occupation Coding in Surveys Simson, Jan, Kononykhina, Olga, and Schierholz, Malte Presentation at ESRA 2023 Conference 2023 [abstract] [www.europeansurveyresearch.org]

   There are a multitude of different ways to earn a living which is why people’s occupations are almost as diverse as people themselves. Therefore, one of the classic issues one encounters when working with occupational data is the vast heterogeneity of occupations people have. To address this problem, a variety of different classifications have been developed, such as the International Standard Classification of Occupations (ISCO) or the German Klassifikation der Berufe (KldB), narrowing down the number of occupation categories into more manageable numbers. This leads to a different problem: The coding of occupations into standardized categories is a time-intensive process plagued by reliability issues. To date the standard approach to code occupations is by collecting free text responses and having specifically trained personnel sit down with the classification manual, possibly assisted by computer software, to assign each category by hand post-hoc. Since coding typically occurs after data collection and with limited information, the assignment of categories is often ambiguous and unreliable. Here we present a new instrument which implements a faster, more convenient and interactive occupation coding workflow where interviewees are included in the coding process. Using the best performing algorithm from a previous comparison of machine learning models, a list of suggested occupation categories from the Auxiliary Classification of Occupations (AuxCO) is generated, one of which is to be selected by the interviewee. Issues of ambiguity within occupational categories are addressed with clarifying follow-up questions. The instrument is implemented as part of a flexible toolbox, covering the whole process from data collection and questionnaire design to the final coding of occupations into KldB and ISCO. Anonymized training data and pre-trained machine-learning models for Germany are provided as part of the toolbox. The open-source toolbox is available at https://github.com/occupationMeasurement/occupationMeasurement.

3. A Holistic Toolbox for Interactive Occupation Coding in Surveys Simson, Jan, Kononykhina, Olga, and Schierholz, Malte Presentation at Perspectives on (Un-)Employment - 14th Interdisciplinary Ph.D. Workshop 2023 [abstract] [iab.de]

   The coding of occupations into standardised categories such as the International Standard Classification of Occupations (ISCO) or German Klassifikation der Berufe (KldB) is a time-intensive process plagued by reliability issues. To date the standard approach to code occupations is by collecting free text responses and having someone manually sit down with a classification manual, possibly assisted by computer software, to assign each category by hand post-hoc. Here we present a new instrument which implements a faster and more convenient, interactive occupation coding workflow where interviewees are part of the coding process. This workflow uses supervised machine learning and is implemented as part of a flexible toolbox covering the whole process from data collection to final occupation coding. The toolbox comes with pre-trained machine-learning models and anonymized training data from German surveys.

4. What If? Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness Simson, Jan, Pfisterer, Florian, and Kern, Christoph Presentation at BigSurv23. Winner of Student Paper Award. 2023 [abstract] [www.bigsurv.org]

   A vast number of systems in Europe and beyond currently use algorithmic decision making (ADM) to (partially) automate decisions that have previously been done by humans. When designed well, these systems promise both more accurate and more efficient decisions all the while saving large amounts of resources and freeing up human time. When ADM systems are not designed well, however, they can lead to unfair algorithms which discriminate against parts of the population under the guise of objectivity and legitimacy. Many examples of both fair and helpful as well as discriminatory algorithms exist in the wild to date. The group they fall into typically depends on the decisions made during their design. It is therefore clearly important to properly understand the decisions that go into the design of ADM systems and how these decisions affect the fairness of the resulting system. To study this, we introduce the method of multiverse analysis for algorithmic fairness.

1. Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness Simson, Jan, Pfisterer, Florian, and Kern, Christoph Frontiers in Artificial Intelligence and Applications. Poster presented at HHAI2023 2023 [abstract] [doi] [ebooks.iospress.nl]

   A vast number of systems across the world currently use algorithmic decision making (ADM) to augment human decision making or even automate decisions that have previously been done by humans. When designed well, these systems promise both more accurate and more efficient decisions all the while saving large amounts of resources and freeing up human time. When ADM systems are not designed well, however, they can lead to unfair algorithms which discriminate against societal groups under the guise of objectivity and legitimacy. Whether systems are ultimately fair or not typically depends on the decisions made during the systems’ design. It is therefore important to properly understand the decisions that go into the design of ADM systems and how these decisions affect the fairness of the resulting system. To study this, we introduce the method of multiverse analysis for algorithmic fairness. During the creation and design of an ADM system one needs to make a multitude of different decisions. Many of these decisions are made implicitly without knowing exactly how they will impact the final system and whether or not they will lead to fair outcomes. In our proposed adaptation of multiverse analysis for ADM we plan to turn these implicit decisions made during the design of an ADM system into explicit ones. Using the resulting decision space, we create a grid of all possible “universes” of decision-combinations. For each of these universes, a fairness metric is computed. Using the resulting dataset of possible decisions and fairness one can see how and which decisions impact fairness. We demonstrate how multiverse analyses can be used to better understand variability and robustness of algorithmic fairness using an exemplary case study of predicting public health coverage. We show preliminary results illustrating how small decisions during the design of an ADM system can have surprising effects on its fairness and how to detect them using multiverse analysis.

2. Motivation and music: Does motivation matter? Jessani, Ghazal, Simson, Jan, Bainbridge, Constance, and Mehr, Samuel A Poster. Presented at the Harvard Graduate School of Education Student Research Symposium. 2020
3. Infants relax in response to unfamiliar foreign lullabies Bainbridge, Constance, Bertolo, Mila, Youngers, Julie, Atwood, S., Yurdum, Lidya, Simson, Jan, Lopez, Kelsie, Xing, Feng, Martin, Alia, and Mehr, Samuel A Virtual conference due to COVID-19. Proceedings of the 42nd Annual Meeting of the Cognitive Science Society. 2020
4. Cultural invariance in infant responses to world music. Bainbridge, Constance, Bertolo, Mila, Youngers, Julie, Atwood, S., Yurdum, Lidya, Simson, Jan, Lopez, Kelsie, Xing, Feng, Martin, Alia, and Mehr, Samuel A Virtual conference due to COVID-19. Poster presented at the biennial meeting of the International Congress of Infant Studies 2020