LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity

Simple item page
dc.contributor.authorArunkumar, Anjanaen_US
dc.contributor.authorSharma, Shubhamen_US
dc.contributor.authorAgrawal, Rakhien_US
dc.contributor.authorChandrasekaran, Sriramen_US
dc.contributor.authorBryan, Chrisen_US
dc.contributor.editorBujack, Roxanaen_US
dc.contributor.editorArchambault, Danielen_US
dc.contributor.editorSchreck, Tobiasen_US
dc.date.accessioned2023-06-10T06:17:34Z
dc.date.available2023-06-10T06:17:34Z
dc.date.issued2023
dc.description.abstractCross-task generalization is a significant outcome that defines mastery in natural language understanding. Humans show a remarkable aptitude for this, and can solve many different types of tasks, given definitions in the form of textual instructions and a small set of examples. Recent work with pre-trained language models mimics this learning style: users can define and exemplify a task for the model to attempt as a series of natural language prompts or instructions. While prompting approaches have led to higher cross-task generalization compared to traditional supervised learning, analyzing 'bias' in the task instructions given to the model is a difficult problem, and has thus been relatively unexplored. For instance, are we truly modeling a task, or are we modeling a user's instructions? To help investigate this, we develop LINGO, a novel visual analytics interface that supports an effective, task-driven workflow to (1) help identify bias in natural language task instructions, (2) alter (or create) task instructions to reduce bias, and (3) evaluate pre-trained model performance on debiased task instructions. To robustly evaluate LINGO, we conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions, spanning 55 different languages. For both user groups, LINGO promotes the creation of more difficult tasks for pre-trained models, that contain higher linguistic diversity and lower instruction bias. We additionally discuss how the insights learned in developing and evaluating LINGO can aid in the design of future dashboards that aim to minimize the effort involved in prompt creation across multiple domains.en_US
dc.description.number3
dc.description.sectionheadersVisualization and Machine Learning
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume42
dc.identifier.doi10.1111/cgf.14840
dc.identifier.issn1467-8659
dc.identifier.pages409-421
dc.identifier.pages13 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.14840
dc.identifier.urihttps://diglib.eg.org:443/handle/10.1111/cgf14840
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.subjectCCS Concepts: Human-centered computing -> Visual analytics; Text input; Computing methodologies -> Natural language processing
dc.subjectHuman centered computing
dc.subjectVisual analytics
dc.subjectText input
dc.subjectComputing methodologies
dc.subjectNatural language processing
dc.titleLINGO : Visually Debiasing Natural Language Instructions to Support Task Diversityen_US
Files
Original bundle
Now showing 1 - 3 of 3
Thumbnail Image
Name:
v42i3pp409-421_cgf14840.pdf
Size:
4.67 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
1207-file-i7.mp4
Size:
84.42 MB
Format:
Unknown data format
Download
Thumbnail Image
Name:
1207-file-i8.pdf
Size:
27.28 KB
Format:
Adobe Portable Document Format
Download
Collections
42-Issue 3