My research interests revolve around AI robustness, a problem that is inherently technical and deeply social. The problem lies in between the information space where computers typically operate and the social space where humans act and interact. Robust AI entails AI that can be reliably used by humans in social contexts; furthermore, it entails that human inclusion in computation must be a valuable approach (and perhaps a necessary approach to AGI if anyhow possible :).

My research has focused on making AI more robust by developing methods that enable human inclusion in computation. These include methods to support human workers and domain experts in analysing model knowledge, and to support users and other stakeholders in assessing model behaviour. Going further, I’m also interested in understanding people, particularly social factors that influence the practice of developers and users around AI robustness, e.g., in debugging AI models and taking AI advice.

My work has quite an interdisciplinary flavor, mixing quantitative and qualitative research. It mixes empirical studies, data creation, algorithm/interface/system design, and evaluation; together, they make a cohesive cycle. My work is inspired by researchers and practitioners in design, philosophy, and social sciences. Through projects, I have worked with designers, engineers, biologists, doctors, architects, etc. Seeing others use my work marks some of the most exciting moments in my research.

A little note about humans and AGI: there can be several reasons why human inclusion (as in the integration of human perception and cognition in computation) is essential for AGI: 1) there is something unique in our human brain/body; 2) there are some fundamental limits of computation; and in addition, 3) the world is too complex and indeterministic. Which one(s) do you believe?

Selected Publications

I publish in AI-related venues spanning information systems, human-computer interaction, and AI ethics. These include WWW/TheWebConf, CHI, AIES, COLM, ICLR, SIGIR, HCOMP, etc. To read my work, below is a selected list of publications. For a full and up-to-date list, please check my Google Scholar page.

Overview of human-centered perspective to AI robustness, especially about the relationships between data, model, and humans:

  • Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M. and Yang, J. A.I. Robustness: a human-centered perspective on technological challenges and opportunities. ACM Computing Surveys (CSUR) 57, no. 6 (2025): 1-38. (On arXiv since 2022) (ACM link, arXiv link with supplimentary material)
  • Yang, J., Drake, T., Damianou, A. and Maarek, Y. Leveraging crowdsourcing data for deep active learning an application: Learning intents in Alexa. In Proceedings of the Web Conference (TheWebConf), pp. 23-32, 2018. (link)

On analysing model knowledge:

  • Lippmann, P. and Yang, J. Style over Substance: Distilled language models reason via stylistic replication. In Proceedings of the Second Conference on Language Modeling (COLM), 2025. (link)
  • Sharifi Noorian, S., Qiu, S., Gadiraju, U., Yang, J. and Bozzon, A. What Should You Know? A human-in-the-loop approach to unknown unknowns characterization in image recognition. In Proceedings of the ACM Web Conference (TheWebConf), pp. 882-892, 2022. (link)

On assessing model behaviour:

  • Arzberger, A., Buijsman, S., Lupetti, M.L., Bozzon, A. and Yang, J. Nothing Comes Without Its World---practical challenges of aligning LLMs to situated human values through RLHF. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), pp. 61-73, 2024. (link)
  • Lammerts, P., Lippmann, P., Hsu, Y.C., Casati, F. and Yang, J. How Do You Feel? measuring user-perceived value for rejecting machine decisions in hate speech detection. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES), pp. 834-844, 2023. (link)

On understanding people (and interaction):

  • Corti, L., Oltmans, R., Jung, J., Balayn, A., Wijsenbeek, M. and Yang, J. ``It Is a Moving Process": Understanding the evolution of explainability needs of clinicians in pulmonary medicine. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), pp. 1-21, 2024. (link)
  • Balayn, A., Rikalo, N., Lofi, C., Yang, J. and Bozzon, A. How can explainability methods be used to support bug identification in computer vision models? In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), pp. 1-16, 2022. (link)

Data & Code: many of my publications come with data and code. If you are looking for them, please check this page.

Some Awards

My work has received many “best paper” awards/nominations:

  • Best paper award nomination at ACM SIGIR 2024 for "Adaptive In-Context Learning with Large Language Models for Bundle Generation" (link).
  • Best paper award nomination at WWW/TheWebConf 2023 for "HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale" (link).
  • Best student paper award at AAAI/ACM AIES 2023 for "Fairness Toolkits, A Checkbox Culture? On the Factors that Fragment Developer Practices in Handling Algorithmic Harms" (link).
  • Best paper award at AAAI HCOMP 2022 for "It is Like Finding a Polar Bear in the Savannah! Concept-level AI Explanations with Analogical Inference from Commonsense Knowledge" (link).
  • Best paper award nomination at WWW/TheWebConf 2022 for Ready Player One! Eliciting Diverse Knowledge Using A Configurable Game" (link).
  • Best blue-sky ideas paper at AAAI HCOMP 2021 for" The Science of Rejection: A Research Area for Human Computation" (link).
  • Best demo award at AAAI HCOMP 2021 for "FindItOut: A Multiplayer GWAP for Collecting Plural Knowledge".
  • Best paper award at ACM HyperText 2017 for "Clarity is a Worthwhile Quality - On the Role of Task Clarity in Microtask Crowdsourcing" (link).

Activities & Services

I promote important ideas around humans and AI in our research communities and actively serve the communities.

For activities related to human-cetered computing and robust AI, please check our AAAI Sympsium Bi-directionality in Human-AI Collaborative Systems at Stanford, the Personalized Generative AI workshop with mostly researchers from industry (e.g., Amazon and Meta), and the Human-in-the-loop Data Curation workshop with active participation from both academia and industry.

I regularly serve as an area chair or on the senior program committee of ACL Rolling Review (ARR), The Web Conference (WWW) (received the “best reviewer” award in 2023), AAAI Conference on Artificial Intelligence (AAAI), and ACM Conference on Information and Knowledge Management (CIKM). I was a program chair for the 25th International Conference on Web Engineering (ICWE).

I’m an associate editor for Frontiers of Artificial Intelligence (NLP section) and serve on the editorial board of the Journal of Human Computation. I was a guest editor for the ACM Journal of Data and Information Quality.

I review for the Dutch Research Council (NWO), Swiss National Science Foundation (SNSF), and European Research Council (ERC).