(Last Updated: Nov 2024)

I am a strong proponent of collaborative research work. Throughout my research experience (so far, much more to go!), I am extremely fortunate to have found supportive mentors and amazing collaborators. I thank them for the opportunities and advice, as none of these projects would’ve been possible without them! If you have thoughts/comments on any of the projects I am currently working on, I’d love to hop on a quick call to chat about them!

Table of Contents

• Broad Research Interests
  • Are LLM explanations effective for all users?
  • Aligned Personas using Folk Psychology-driven Post-hoc Rationalization 🧠 ➡ 📖
• Past Work and Side Projects
  • Social Networks
  • Music and Machine Learning

What am I working on right now? ⬆️

Broadly, my research interests lie in human-centred perspectives in understanding and evaluating foundation models. To this end, I am interested in two primary directions:

• How can LLMs be useful? In this age, we use LLMs for most tasks. This includes us consuming some knowledge provided by them (search), writing and refining emails (writing assistance) or even teaching complex concepts to diverse audiences (education). I am particularly interested in working on problems in the broader context of human-LM collaboration for human improvement. LLMs built specifically for these domains can potentially help ‘teach’ lay humans – about a new problem area, or research domain. This `knowledge transfer’ setup can be challenging to model and evaluate as they often require controlled human experiments for evaluations and need to be grounded in interdisciplinary approaches. Furthermore, these signals can be sent back to the LLM as feedback to improve upon.

• Can LLMs estimate human behaviour by adopting personas? Among the remarkable emergent abilities of LLMs is generating steered responses; they can now be made to assume a ‘persona’, with a cheap and easy solution of just prompting them to do so. This has led to LLMs being used to mimic user responses in surveys and behavioural tests, create realistic simulation communities and generate vast amount of diverse, synthetic data. I am interested in working on problems that evaluate or utilize LLM personas in downstream tasks.

Are LLM explanations effective for all users?

This project is ongoing with the NLP Group and HCI Groups at USC. LLMs are being widely used in educational scenarios such as answering difficult questions, simplifying complex material in classrooms and automatically generating examination questions. This has led to a gradual move towards replacing traditional teaching mechanisms, such as peer tutoring, with LLMs due to the ease of personalizing content, which improves accessibility and inclusivity in education, while also considerably saving educator time and effort. Prior work has explored the tailoring of LLM responses to the target audience in a number of tasks. However, evaluating these techniques has shown that there is no uniform consensus on whether this tailoring is actually successful. To this end, we propose a human-centric evaluation of LLMs in tailoring content to people with diverse educational backgrounds. We scope it to LLMs explaining answers to Why-questions. For example, consider the question: Why is the sky blue?. The answer may vary when the information-seeker is an elementary school student (‘because sunlight hits the air and splits into multiple parts, which scatter around’) or someone with a graduate degree in Physics (‘selective scattering is proportional to the inverse fourth power of the wavelength, making blue light much more prominent in the observer’s line of sight’). Motivated by this, we pose two research questions: (1) Can LLMs explain to diverse users? We aim to provide a detailed analysis of the explanations generated by LLMs to a large set of Why-questions. (2) Are these LLM explanations truly useful to the intended audience? We propose a human-centric evaluation of LLM explanations to quantify the utility of LLMs as educators.

Aligned Personas using Folk Psychology-driven Post-hoc Rationalization 🧠 ➡ 📖

This project is being continued from my Summer 2024 Internship with Apple. Early approaches in aligning language models to certain ‘personas’ made use of fine-tuning LLMs on user-specific data. However, obtaining large scale user-specific data is often a bottleneck and adapting to a new users’ preferences would imply that the model would be fine-tuned again. Recent approaches have used zero-shot prompting, by populating a system prompt with the desired persona for the LLM to follow. These approaches made use of user demographic attributes and/or opinion history to model a specific user. However, personas created from these prior approaches are heavily underspecified and not generalizable across user behaviour. Personas created from demographics alone often display inconsistency, stereotypical and biased behaviour. Some prior work has suggested incorporating more user context in terms of opinion histories, or creating abstractions between multiple users to gather more context. While these methods can help the model understand what the user believes, they do not aid it in explicitly reason about why the user holds the particular beliefs they do. Rationalizing behavior is a standard process which humans use to understand one another’s behavior. This rationalization is often guided by so-called “folk” psychological theories – informal justifications from lived experience of “common sense”

Past Work and Side Projects ⬆️

Social Networks

In what I consider past life now 😄, I spent considerable time working on topics in anomaly detection on social networks like Twitter. As a part of 2 year long undergraduate thesis work, I worked on a novel problem of collusion detection, where malicious entities come together to trade appraisals on social media platforms. Along with my team members Aditya Chetan, Dr. Hridoy Sankar Dutta and Prof. Tanmoy Chakraborty, we worked on a series of projects surrounding Collusion detection:

1. In a preliminary study (ASONAM’18), we established the differences between collusion detection and bot/spam detection methods, followed by a supervised, feature-based classification system to identify collusive retweeters over Twitter. This was an important work to actively probe collusive users and collect labelled data in that manner, along with identifying characteristics pertaining to their behaviour.

2. In a follow up work (WSDM’19), we devised an unsupervised approach in identifying collusive users, owing to the scarcity of labelled data. We made use of a learning-to-rank approach, coupled with features that were intrinsic to collusive behaviour. I am extremely proud of this work, particularly because everything from approach design to implementation was conducted by a bunch of undergrads (my collaborator Aditya and I), and provided me with the necessary motivation to pursue a PhD! 🙌🏽

3. In a spin-off project (ACM TIST) of the first work, Udit Arora and our team designed a multi-view approach to looking at collusive users – from a follower/followee, tweet and content perspective, to enrich the signals for effective classification.

Music and Machine Learning

I have also dabbled with a diverse set of problems in music (audio) and machine learning. Some problems that I have worked on:

1. What are the textual ingredients necessary to make a podcast popular? (NLP4MusA)

2. Understanding the difference between emotions conveyed and induced by hindustani classical music. (ICMI’20 Late Breaking Reports)

3. Beatbox transcription to symbolic notations: from generating a diverse dataset to a first approach in solving this task. (PAKDD’22)