+
+ + Computation and Language 43 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ ImageBind-LLM: Multi-modality Instruction Tuning +
+
+
+
+
+
+
+
+ Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, Yu Qiao
+
+
+ We present ImageBind-LLM, a multi-modality instruction tuning method of large
+language models (LLMs) via ImageBind. Existing works mainly focus on language
+and image instruction tuning, different from which, our ImageBind-LLM can
+respond to multi-modality conditions, including audio, 3D point clouds, video,
+and their embedding-space arithmetic by only image-text alignment training.
+During training, we adopt a learnable bind network to align the embedding space
+between LLaMA and ImageBind's image encoder. Then, the image features
+transformed by the bind network are added to word tokens of all layers in
+LLaMA, which progressively injects visual instructions via an attention-free
+and zero-initialized gating mechanism. Aided by the joint embedding of
+ImageBind, the simple image-text training enables our model to exhibit superior
+multi-modality instruction-following capabilities. During inference, the
+multi-modality inputs are fed into the corresponding ImageBind encoders, and
+processed by a proposed visual cache model for further cross-modal embedding
+enhancement. The training-free cache model retrieves from three million image
+features extracted by ImageBind, which effectively mitigates the
+training-inference modality discrepancy. Notably, with our approach,
+ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
+significant language generation quality. Code is released at
+https://github.com/OpenGVLab/LLaMA-Adapter.
+
+
+
+ comment: Code is available at https://github.com/OpenGVLab/LLaMA-Adapter
+
+
+
+ + ☆ A Function Interpretation Benchmark for Evaluating Interpretability + Methods +
+
+
+
+
+
+
+
+ Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, Antonio Torralba
+
+
+ Labeling neural network submodules with human-legible descriptions is useful
+for many downstream tasks: such descriptions can surface failures, guide
+interventions, and perhaps even explain important model behaviors. To date,
+most mechanistic descriptions of trained networks have involved small models,
+narrowly delimited phenomena, and large amounts of human labor. Labeling all
+human-interpretable sub-computations in models of increasing size and
+complexity will almost certainly require tools that can generate and validate
+descriptions automatically. Recently, techniques that use learned models
+in-the-loop for labeling have begun to gain traction, but methods for
+evaluating their efficacy are limited and ad-hoc. How should we validate and
+compare open-ended labeling tools? This paper introduces FIND (Function
+INterpretation and Description), a benchmark suite for evaluating the building
+blocks of automated interpretability methods. FIND contains functions that
+resemble components of trained neural networks, and accompanying descriptions
+of the kind we seek to generate. The functions are procedurally constructed
+across textual and numeric domains, and involve a range of real-world
+complexities, including noise, composition, approximation, and bias. We
+evaluate new and existing methods that use language models (LMs) to produce
+code-based and language descriptions of function behavior. We find that an
+off-the-shelf LM augmented with only black-box access to functions can
+sometimes infer their structure, acting as a scientist by forming hypotheses,
+proposing experiments, and updating descriptions in light of new data. However,
+LM-based descriptions tend to capture global function behavior and miss local
+corruptions. These results show that FIND will be useful for characterizing the
+performance of more sophisticated interpretability methods before they are
+applied to real-world models.
+
+
+
+ comment: 25 pages, 7 figures
+
+
+
+ + ☆ Zero-Shot Audio Captioning via Audibility Guidance +
+ +
+ The task of audio captioning is similar in essence to tasks such as image and
+video captioning. However, it has received much less attention. We propose
+three desiderata for captioning audio -- (i) fluency of the generated text,
+(ii) faithfulness of the generated text to the input audio, and the somewhat
+related (iii) audibility, which is the quality of being able to be perceived
+based only on audio. Our method is a zero-shot method, i.e., we do not learn to
+perform captioning. Instead, captioning occurs as an inference process that
+involves three networks that correspond to the three desired qualities: (i) A
+Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A
+model that provides a matching score between an audio file and a text, for
+which we use a multimodal matching network called ImageBind, and (iii) A text
+classifier, trained using a dataset we collected automatically by instructing
+GPT-4 with prompts designed to direct the generation of both audible and
+inaudible sentences. We present our results on the AudioCap dataset,
+demonstrating that audibility guidance significantly enhances performance
+compared to the baseline, which lacks this objective.
+
+
+
+
+ + ☆ DoLa: Decoding by Contrasting Layers Improves Factuality in Large + Language Models +
+ +
+ Despite their impressive capabilities, large language models (LLMs) are prone
+to hallucinations, i.e., generating content that deviates from facts seen
+during pretraining. We propose a simple decoding strategy for reducing
+hallucinations with pretrained LLMs that does not require conditioning on
+retrieved external knowledge nor additional fine-tuning. Our approach obtains
+the next-token distribution by contrasting the differences in logits obtained
+from projecting the later layers versus earlier layers to the vocabulary space,
+exploiting the fact that factual knowledge in an LLMs has generally been shown
+to be localized to particular transformer layers. We find that this Decoding by
+Contrasting Layers (DoLa) approach is able to better surface factual knowledge
+and reduce the generation of incorrect facts. DoLa consistently improves the
+truthfulness across multiple choices tasks and open-ended generation tasks, for
+example improving the performance of LLaMA family models on TruthfulQA by
+12-17% absolute points, demonstrating its potential in making LLMs reliably
+generate truthful facts.
+
+
+
+ comment: The source code is available at https://github.com/voidism/DoLa
+
+
+
+ + ☆ On Large Language Models' Selection Bias in Multi-Choice Questions +
+ +
+ Multi-choice questions (MCQs) serve as a common yet important task format in
+the research of large language models (LLMs). Our work shows that LLMs exhibit
+an inherent "selection bias" in MCQs, which refers to LLMs' preferences to
+select options located at specific positions (like "Option C"). This bias is
+prevalent across various LLMs, making their performance vulnerable to option
+position changes in MCQs. We identify that one primary cause resulting in
+selection bias is option numbering, i.e., the ID symbols A/B/C/D associated
+with the options. To mitigate selection bias, we propose a new method called
+PriDe. PriDe first decomposes the observed model prediction distribution into
+an intrinsic prediction over option contents and a prior distribution over
+option IDs. It then estimates the prior by permutating option contents on a
+small number of test samples, which is used to debias the subsequent test
+samples. We demonstrate that, as a label-free, inference-time method, PriDe
+achieves a more effective and computation-efficient debiasing than strong
+baselines. We further show that the priors estimated by PriDe generalize well
+across different domains, highlighting its practical potential in broader
+scenarios.
+
+
+
+ comment: Work in progress
+
+
+
+ + ☆ Introducing "Forecast Utterance" for Conversational Data Science +
+ +
+ Envision an intelligent agent capable of assisting users in conducting
+forecasting tasks through intuitive, natural conversations, without requiring
+in-depth knowledge of the underlying machine learning (ML) processes. A
+significant challenge for the agent in this endeavor is to accurately
+comprehend the user's prediction goals and, consequently, formulate precise ML
+tasks. In this paper, we take a pioneering step towards this ambitious goal by
+introducing a new concept called Forecast Utterance and then focus on the
+automatic and accurate interpretation of users' prediction goals from these
+utterances. Specifically, we frame the task as a slot-filling problem, where
+each slot corresponds to a specific aspect of the goal prediction task. We then
+employ two zero-shot methods for solving the slot-filling task, namely: 1)
+Entity Extraction (EE), and 2) Question-Answering (QA) techniques. Our
+experiments, conducted with three meticulously crafted data sets, validate the
+viability of our ambitious goal and demonstrate the effectiveness of both EE
+and QA techniques in interpreting Forecast Utterances.
+
+
+
+
+ + ☆ OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs +
+ +
+ Instruction-tuned Large Language Models (LLMs) have recently showcased
+remarkable ability to generate fitting responses to natural language
+instructions. However, an open research question concerns the inherent biases
+of trained models and their responses. For instance, if the data used to tune
+an LLM is dominantly written by persons with a specific political bias, we
+might expect generated answers to share this bias. Current research work seeks
+to de-bias such models, or suppress potentially biased answers. With this
+demonstration, we take a different view on biases in instruction-tuning: Rather
+than aiming to suppress them, we aim to make them explicit and transparent. To
+this end, we present OpinionGPT, a web demo in which users can ask questions
+and select all biases they wish to investigate. The demo will answer this
+question using a model fine-tuned on text representing each of the selected
+biases, allowing side-by-side comparison. To train the underlying model, we
+identified 11 different biases (political, geographic, gender, age) and derived
+an instruction-tuning corpus in which each answer was written by members of one
+of these demographics. This paper presents OpinionGPT, illustrates how we
+trained the bias-aware model and showcases the web application (available at
+https://opiniongpt.informatik.hu-berlin.de).
+
+
+
+ comment: 6 pages, 1 figure, 3 tables
+
+
+
+ + ☆ FLM-101B: An Open LLM and How to Train It with $100K Budget +
+
+
+
+
+
+
+
+ Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, Yequan Wang
+
+
+ Large language models (LLMs) have achieved remarkable success in NLP and
+multimodal tasks. Despite these successes, their development faces two main
+challenges: (i) high computational cost; and (ii) difficulty in conducting fair
+and objective evaluations. LLMs are prohibitively expensive, making it feasible
+for only a few major players to undertake their training, thereby constraining
+both research and application opportunities. This underscores the importance of
+cost-effective LLM training. In this paper, we utilize a growth strategy to
+significantly reduce LLM training cost. We demonstrate that an LLM with 101B
+parameters and 0.31TB tokens can be trained on a $100K budget. We also adopt a
+systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
+existing evaluations that focus more on knowledge-oriented abilities. We
+introduce our benchmark including evaluations on important aspects of
+intelligence including symbolic mapping, itrule understanding, pattern mining,
+and anti-interference. Such evaluations minimize the potential impact of
+memorization. Experimental results show that our model FLM-101B, trained with a
+budget of $100K, achieves comparable performance to powerful and well-known
+models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
+contexts unseen in training data. The checkpoint of FLM-101B will be
+open-sourced at https://huggingface.co/CofeAI/FLM-101B.
+
+
+
+
+ + ☆ Uncovering Drift in Textual Data: An Unsupervised Method for Detecting + and Mitigating Drift in Machine Learning Models +
+
+
+
+
+
+
+
+ Saeed Khaki, Akhouri Abhinav Aditya, Zohar Karnin, Lan Ma, Olivia Pan, Samarth Marudheri Chandrashekar
+
+
+ Drift in machine learning refers to the phenomenon where the statistical
+properties of data or context, in which the model operates, change over time
+leading to a decrease in its performance. Therefore, maintaining a constant
+monitoring process for machine learning model performance is crucial in order
+to proactively prevent any potential performance regression. However,
+supervised drift detection methods require human annotation and consequently
+lead to a longer time to detect and mitigate the drift. In our proposed
+unsupervised drift detection method, we follow a two step process. Our first
+step involves encoding a sample of production data as the target distribution,
+and the model training data as the reference distribution. In the second step,
+we employ a kernel-based statistical test that utilizes the maximum mean
+discrepancy (MMD) distance metric to compare the reference and target
+distributions and estimate any potential drift. Our method also identifies the
+subset of production data that is the root cause of the drift. The models
+retrained using these identified high drift samples show improved performance
+on online customer experience quality metrics.
+
+
+
+ comment: 8 pages, Accepted in 2023 Amazon Internal Machine Learning Conference
+
+
+
+ + ☆ USA: Universal Sentiment Analysis Model & Construction of Japanese + Sentiment Text Classification and Part of Speech Dataset +
+ +
+ Sentiment analysis is a pivotal task in the domain of natural language
+processing. It encompasses both text-level sentiment polarity classification
+and word-level Part of Speech(POS) sentiment polarity determination. Such
+analysis challenges models to understand text holistically while also
+extracting nuanced information. With the rise of Large Language Models(LLMs),
+new avenues for sentiment analysis have opened. This paper proposes enhancing
+performance by leveraging the Mutual Reinforcement Effect(MRE) between
+individual words and the overall text. It delves into how word polarity
+influences the overarching sentiment of a passage. To support our research, we
+annotated four novel Sentiment Text Classification and Part of Speech(SCPOS)
+datasets, building upon existing sentiment classification datasets.
+Furthermore, we developed a Universal Sentiment Analysis(USA) model, with a
+7-billion parameter size. Experimental results revealed that our model
+surpassed the performance of gpt-3.5-turbo across all four datasets,
+underscoring the significance of MRE in sentiment analysis.
+
+
+
+ comment: 10 pages, 5 figures. Model and Dataset will release soon
+
+
+
+ + ☆ Enhancing Pipeline-Based Conversational Agents with Large Language + Models +
+
+
+
+
+
+
+
+ Mina Foosherian, Hendrik Purwins, Purna Rathnayake, Touhidul Alam, Rui Teimao, Klaus-Dieter Thoben
+
+
+ The latest advancements in AI and deep learning have led to a breakthrough in
+large language model (LLM)-based agents such as GPT-4. However, many commercial
+conversational agent development tools are pipeline-based and have limitations
+in holding a human-like conversation. This paper investigates the capabilities
+of LLMs to enhance pipeline-based conversational agents during two phases: 1)
+in the design and development phase and 2) during operations. In 1) LLMs can
+aid in generating training data, extracting entities and synonyms,
+localization, and persona design. In 2) LLMs can assist in contextualization,
+intent classification to prevent conversational breakdown and handle
+out-of-scope questions, auto-correcting utterances, rephrasing responses,
+formulating disambiguation questions, summarization, and enabling closed
+question-answering capabilities. We conducted informal experiments with GPT-4
+in the private banking domain to demonstrate the scenarios above with a
+practical example. Companies may be hesitant to replace their pipeline-based
+agents with LLMs entirely due to privacy concerns and the need for deep
+integration within their existing ecosystems. A hybrid approach in which LLMs'
+are integrated into the pipeline-based agents allows them to save time and
+costs of building and running agents by capitalizing on the capabilities of
+LLMs while retaining the integration and privacy safeguards of their existing
+systems.
+
+
+
+
+ + ☆ The Daunting Dilemma with Sentence Encoders: Success on Standard + Benchmarks, Failure in Capturing Basic Semantic Properties +
+ +
+ In this paper, we adopted a retrospective approach to examine and compare
+five existing popular sentence encoders, i.e., Sentence-BERT, Universal
+Sentence Encoder (USE), LASER, InferSent, and Doc2vec, in terms of their
+performance on downstream tasks versus their capability to capture basic
+semantic properties. Initially, we evaluated all five sentence encoders on the
+popular SentEval benchmark and found that multiple sentence encoders perform
+quite well on a variety of popular downstream tasks. However, being unable to
+find a single winner in all cases, we designed further experiments to gain a
+deeper understanding of their behavior. Specifically, we proposed four semantic
+evaluation criteria, i.e., Paraphrasing, Synonym Replacement, Antonym
+Replacement, and Sentence Jumbling, and evaluated the same five sentence
+encoders using these criteria. We found that the Sentence-Bert and USE models
+pass the paraphrasing criterion, with SBERT being the superior between the two.
+LASER dominates in the case of the synonym replacement criterion.
+Interestingly, all the sentence encoders failed the antonym replacement and
+jumbling criteria. These results suggest that although these popular sentence
+encoders perform quite well on the SentEval benchmark, they still struggle to
+capture some basic semantic properties, thus, posing a daunting dilemma in NLP
+research.
+
+
+
+
+ + ☆ Word segmentation granularity in Korean +
+ +
+ This paper describes word {segmentation} granularity in Korean language
+processing. From a word separated by blank space, which is termed an eojeol, to
+a sequence of morphemes in Korean, there are multiple possible levels of word
+segmentation granularity in Korean. For specific language processing and corpus
+annotation tasks, several different granularity levels have been proposed and
+utilized, because the agglutinative languages including Korean language have a
+one-to-one mapping between functional morpheme and syntactic category. Thus, we
+analyze these different granularity levels, presenting the examples of Korean
+language processing systems for future reference. Interestingly, the
+granularity by separating only functional morphemes including case markers and
+verbal endings, and keeping other suffixes for morphological derivation results
+in the optimal performance for phrase structure parsing. This contradicts
+previous best practices for Korean language processing, which has been the de
+facto standard for various applications that require separating all morphemes.
+
+
+
+ comment: Accepted for publication in Korean Linguistics (Benjamins)
+
+
+
+ + ☆ Exploring an LM to generate Prolog Predicates from Mathematics Questions +
+ +
+ Recently, there has been a surge in interest in NLP driven by ChatGPT.
+ChatGPT, a transformer-based generative language model of substantial scale,
+exhibits versatility in performing various tasks based on natural language.
+Nevertheless, large language models often exhibit poor performance in solving
+mathematics questions that require reasoning. Prior research has demonstrated
+the effectiveness of chain-of-thought prompting in enhancing reasoning
+capabilities. Now, we aim to investigate whether fine-tuning a model for the
+generation of Prolog codes, a logic language, and subsequently passing these
+codes to a compiler can further improve accuracy. Consequently, we employ
+chain-of-thought to fine-tune LLaMA7B as a baseline model and develop other
+fine-tuned LLaMA7B models for the generation of Prolog code, Prolog code +
+chain-of-thought, and chain-of-thought + Prolog code, respectively. The results
+reveal that the Prolog generation model surpasses the baseline in performance,
+while the combination generation models do not yield significant improvements.
+The Prolog corpus based on GSM8K and the correspondingly finetuned Prolog
+generation model based on LLaMA7B are released to the research community.
+
+
+
+ comment: 6 pages, 3 figures
+
+
+
+ + ☆ BNS-Net: A Dual-channel Sarcasm Detection Method Considering + Behavior-level and Sentence-level Conflicts +
+ +
+ Sarcasm detection is a binary classification task that aims to determine
+whether a given utterance is sarcastic. Over the past decade, sarcasm detection
+has evolved from classical pattern recognition to deep learning approaches,
+where features such as user profile, punctuation and sentiment words have been
+commonly employed for sarcasm detection. In real-life sarcastic expressions,
+behaviors without explicit sentimental cues often serve as carriers of implicit
+sentimental meanings. Motivated by this observation, we proposed a dual-channel
+sarcasm detection model named BNS-Net. The model considers behavior and
+sentence conflicts in two channels. Channel 1: Behavior-level Conflict Channel
+reconstructs the text based on core verbs while leveraging the modified
+attention mechanism to highlight conflict information. Channel 2:
+Sentence-level Conflict Channel introduces external sentiment knowledge to
+segment the text into explicit and implicit sentences, capturing conflicts
+between them. To validate the effectiveness of BNS-Net, several comparative and
+ablation experiments are conducted on three public sarcasm datasets. The
+analysis and evaluation of experimental results demonstrate that the BNS-Net
+effectively identifies sarcasm in text and achieves the state-of-the-art
+performance.
+
+
+
+ comment: 11 pages, 5 figures
+
+
+
+ + ☆ Evaluating ChatGPT as a Recommender System: A Rigorous Approach +
+
+
+
+
+
+
+
+ Dario Di Palma, Giovanni Maria Biancofiore, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia, Eugenio Di Sciascio
+
+
+ Recent popularity surrounds large AI language models due to their impressive
+natural language capabilities. They contribute significantly to
+language-related tasks, including prompt-based learning, making them valuable
+for various specific tasks. This approach unlocks their full potential,
+enhancing precision and generalization. Research communities are actively
+exploring their applications, with ChatGPT receiving recognition. Despite
+extensive research on large language models, their potential in recommendation
+scenarios still needs to be explored. This study aims to fill this gap by
+investigating ChatGPT's capabilities as a zero-shot recommender system. Our
+goals include evaluating its ability to use user preferences for
+recommendations, reordering existing recommendation lists, leveraging
+information from similar users, and handling cold-start situations. We assess
+ChatGPT's performance through comprehensive experiments using three datasets
+(MovieLens Small, Last.FM, and Facebook Book). We compare ChatGPT's performance
+against standard recommendation algorithms and other large language models,
+such as GPT-3.5 and PaLM-2. To measure recommendation effectiveness, we employ
+widely-used evaluation metrics like Mean Average Precision (MAP), Recall,
+Precision, F1, normalized Discounted Cumulative Gain (nDCG), Item Coverage,
+Expected Popularity Complement (EPC), Average Coverage of Long Tail (ACLT),
+Average Recommendation Popularity (ARP), and Popularity-based Ranking-based
+Equal Opportunity (PopREO). Through thoroughly exploring ChatGPT's abilities in
+recommender systems, our study aims to contribute to the growing body of
+research on the versatility and potential applications of large language
+models. Our experiment code is available on the GitHub repository:
+https://github.com/sisinflab/Recommender-ChatGPT
+
+
+
+
+ + ☆ Loquacity and Visible Emotion: ChatGPT as a Policy Advisor +
+ +
+ ChatGPT, a software seeking to simulate human conversational abilities, is
+attracting increasing attention. It is sometimes portrayed as a groundbreaking
+productivity aid, including for creative work. In this paper, we run an
+experiment to assess its potential in complex writing tasks. We ask the
+software to compose a policy brief for the Board of the Bank of Italy. We find
+that ChatGPT can accelerate workflows by providing well-structured content
+suggestions, and by producing extensive, linguistically correct text in a
+matter of seconds. It does, however, require a significant amount of expert
+supervision, which partially offsets productivity gains. If the app is used
+naively, output can be incorrect, superficial, or irrelevant. Superficiality is
+an especially problematic limitation in the context of policy advice intended
+for high-level audiences.
+
+
+
+ comment: 33 pages
+
+
+
+ + ☆ Evaluating the Efficacy of Supervised Learning vs Large Language Models + for Identifying Cognitive Distortions and Suicidal Risks in Chinese Social + Media +
+
+
+
+
+
+
+
+ Hongzhi Qi, Qing Zhao, Changwei Song, Wei Zhai, Dan Luo, Shuo Liu, Yi Jing Yu, Fan Wang, Huijing Zou, Bing Xiang Yang, Jianqiang Li, Guanghui Fu
+
+
+ Large language models, particularly those akin to the rapidly progressing GPT
+series, are gaining traction for their expansive influence. While there is keen
+interest in their applicability within medical domains such as psychology,
+tangible explorations on real-world data remain scant. Concurrently, users on
+social media platforms are increasingly vocalizing personal sentiments; under
+specific thematic umbrellas, these sentiments often manifest as negative
+emotions, sometimes escalating to suicidal inclinations. Timely discernment of
+such cognitive distortions and suicidal risks is crucial to effectively
+intervene and potentially avert dire circumstances. Our study ventured into
+this realm by experimenting on two pivotal tasks: suicidal risk and cognitive
+distortion identification on Chinese social media platforms. Using supervised
+learning as a baseline, we examined and contrasted the efficacy of large
+language models via three distinct strategies: zero-shot, few-shot, and
+fine-tuning. Our findings revealed a discernible performance gap between the
+large language models and traditional supervised learning approaches, primarily
+attributed to the models' inability to fully grasp subtle categories. Notably,
+while GPT-4 outperforms its counterparts in multiple scenarios, GPT-3.5 shows
+significant enhancement in suicide risk classification after fine-tuning. To
+our knowledge, this investigation stands as the maiden attempt at gauging large
+language models on Chinese social media tasks. This study underscores the
+forward-looking and transformative implications of using large language models
+in the field of psychology. It lays the groundwork for future applications in
+psychological research and practice.
+
+
+
+ comment: 15 pages
+
+
+
+ + ☆ All Labels Together: Low-shot Intent Detection with an Efficient Label + Semantic Encoding Paradigm AACL 2023 +
+ +
+ In intent detection tasks, leveraging meaningful semantic information from
+intent labels can be particularly beneficial for few-shot scenarios. However,
+existing few-shot intent detection methods either ignore the intent labels,
+(e.g. treating intents as indices) or do not fully utilize this information
+(e.g. only using part of the intent labels). In this work, we present an
+end-to-end One-to-All system that enables the comparison of an input utterance
+with all label candidates. The system can then fully utilize label semantics in
+this way. Experiments on three few-shot intent detection tasks demonstrate that
+One-to-All is especially effective when the training resource is extremely
+scarce, achieving state-of-the-art performance in 1-, 3- and 5-shot settings.
+Moreover, we present a novel pretraining strategy for our model that utilizes
+indirect supervision from paraphrasing, enabling zero-shot cross-domain
+generalization on intent detection tasks. Our code is at
+https://github.com/jiangshdd/AllLablesTogethe.
+
+
+
+ comment: Accepted by IJCNLP-AACL 2023
+
+
+
+ + ☆ An Anchor Learning Approach for Citation Field Learning +
+ +
+ Citation field learning is to segment a citation string into fields of
+interest such as author, title, and venue. Extracting such fields from
+citations is crucial for citation indexing, researcher profile analysis, etc.
+User-generated resources like academic homepages and Curriculum Vitae, provide
+rich citation field information. However, extracting fields from these
+resources is challenging due to inconsistent citation styles, incomplete
+sentence syntax, and insufficient training data. To address these challenges,
+we propose a novel algorithm, CIFAL (citation field learning by anchor
+learning), to boost the citation field learning performance. CIFAL leverages
+the anchor learning, which is model-agnostic for any Pre-trained Language
+Model, to help capture citation patterns from the data of different citation
+styles. The experiments demonstrate that CIFAL outperforms state-of-the-art
+methods in citation field learning, achieving a 2.83% improvement in
+field-level F1-scores. Extensive analysis of the results further confirms the
+effectiveness of CIFAL quantitatively and qualitatively.
+
+
+
+ comment: Under review
+
+
+
+ + ☆ Machine Learning for Tangible Effects: Natural Language Processing for + Uncovering the Illicit Massage Industry & Computer Vision for Tactile Sensing +
+ +
+ I explore two questions in this thesis: how can computer science be used to
+fight human trafficking? And how can computer vision create a sense of touch?
+ I use natural language processing (NLP) to monitor the United States illicit
+massage industry (IMI), a multi-billion dollar industry that offers not just
+therapeutic massages but also commercial sexual services. Employees of this
+industry are often immigrant women with few job opportunities, leaving them
+vulnerable to fraud, coercion, and other facets of human trafficking.
+Monitoring spatiotemporal trends helps prevent trafficking in the IMI. By
+creating datasets with three publicly-accessible websites: Google Places,
+Rubmaps, and AMPReviews, combined with NLP techniques such as bag-of-words and
+Word2Vec, I show how to derive insights into the labor pressures and language
+barriers that employees face, as well as the income, demographics, and societal
+pressures affecting sex buyers. I include a call-to-action to other researchers
+given these datasets. I also consider how to creating synthetic financial data,
+which can aid with counter-trafficking in the banking sector. I use an
+agent-based model to create both tabular and payee-recipient graph data.
+ I then consider the role of computer vision in making tactile sensors. I
+report on a novel sensor, the Digger Finger, that adapts the Gelsight sensor to
+finding objects in granular media. Changes include using a wedge shape to
+facilitate digging, replacing the internal lighting LEDs with fluorescent
+paint, and adding a vibrator motor to counteract jamming. Finally, I also show
+how to use a webcam and a printed reference marker, or fiducial, to create a
+low-cost six-axis force-torque sensor. This sensor is up to a hundred times
+less expensive than commercial sensors, allowing for a wider range of
+applications. For this and earlier chapters I release design files and code as
+open source.
+
+
+
+ comment: PhD thesis
+
+
+
+ + ☆ XGen-7B Technical Report +
+
+
+
+
+
+
+
+ Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojciech Kryściński, Lidiya Murakhovs'ka, Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Joty, Caiming Xiong
+
+
+ Large Language Models (LLMs) have become ubiquitous across various domains,
+transforming the way we interact with information and conduct research.
+However, most high-performing LLMs remain confined behind proprietary walls,
+hindering scientific progress. Most open-source LLMs, on the other hand, are
+limited in their ability to support longer sequence lengths, which is a key
+requirement for many tasks that require inference over an input context. To
+address this, we have trained XGen, a series of 7B parameter models on up to 8K
+sequence length for up to 1.5T tokens. We have also finetuned the XGen models
+on public-domain instructional data, creating their instruction-tuned
+counterparts (XGen-Inst). We open-source our models for both research
+advancements and commercial applications. Our evaluation on standard benchmarks
+shows that XGen models achieve comparable or better results when compared with
+state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
+modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
+open-source LLMs.
+
+
+
+
+ + ☆ Improving Open Information Extraction with Large Language Models: A + Study on Demonstration Uncertainty +
+
+
+
+
+
+
+
+ Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoyu Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, Liang Zhao
+
+
+ Open Information Extraction (OIE) task aims at extracting structured facts
+from unstructured text, typically in the form of (subject, relation, object)
+triples. Despite the potential of large language models (LLMs) like ChatGPT as
+a general task solver, they lag behind state-of-the-art (supervised) methods in
+OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant
+context from relevant relations and generate structured output due to the
+restrictions on fine-tuning the model. Second, LLMs generates responses
+autoregressively based on probability, which makes the predicted relations lack
+confidence. In this paper, we assess the capabilities of LLMs in improving the
+OIE task. Particularly, we propose various in-context learning strategies to
+enhance LLM's instruction-following ability and a demonstration uncertainty
+quantification module to enhance the confidence of the generated relations. Our
+experiments on three OIE benchmark datasets show that our approach holds its
+own against established supervised methods, both quantitatively and
+qualitatively.
+
+
+
+
+ + ☆ From Base to Conversational: Japanese Instruction Dataset and Tuning + Large Language Models +
+ +
+ Instruction tuning is essential for large language models (LLMs) to become
+interactive. While many instruction tuning datasets exist in English, there is
+a noticeable lack in other languages. Also, their effectiveness has not been
+well verified in non-English languages. We construct a Japanese instruction
+dataset by expanding and filtering existing datasets and apply the dataset to a
+Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning
+on both Japanese and English existing models using our instruction dataset. We
+evaluated these models from both quantitative and qualitative perspectives. As
+a result, the effectiveness of Japanese instruction datasets is confirmed. The
+results also indicate that even with relatively small LLMs, performances in
+downstream tasks would be improved through instruction tuning. Our instruction
+dataset, tuned models, and implementation are publicly available online.
+
+
+
+
+ + ☆ Large Language Models as Optimizers +
+
+
+
+
+
+
+
+ Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen
+
+
+ Optimization is ubiquitous. While derivative-based algorithms have been
+powerful tools for various problems, the absence of gradient imposes challenges
+on many real-world applications. In this work, we propose Optimization by
+PROmpting (OPRO), a simple and effective approach to leverage large language
+models (LLMs) as optimizers, where the optimization task is described in
+natural language. In each optimization step, the LLM generates new solutions
+from the prompt that contains previously generated solutions with their values,
+then the new solutions are evaluated and added to the prompt for the next
+optimization step. We first showcase OPRO on linear regression and traveling
+salesman problems, then move on to prompt optimization where the goal is to
+find instructions that maximize the task accuracy. With a variety of LLMs, we
+demonstrate that the best prompts optimized by OPRO outperform human-designed
+prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
+
+
+
+
+ + ♻ ☆ Transformers as Support Vector Machines +
+ +
+ Since its inception in "Attention Is All You Need", transformer architecture
+has led to revolutionary advancements in NLP. The attention layer within the
+transformer admits a sequence of input tokens $X$ and makes them interact
+through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where
+$(K,Q)$ are the trainable key-query parameters. In this work, we establish a
+formal equivalence between the optimization geometry of self-attention and a
+hard-margin SVM problem that separates optimal input tokens from non-optimal
+tokens using linear constraints on the outer-products of token pairs. This
+formalism allows us to characterize the implicit bias of 1-layer transformers
+optimized with gradient descent: (1) Optimizing the attention layer with
+vanishing regularization, parameterized by $(K,Q)$, converges in direction to
+an SVM solution minimizing the nuclear norm of the combined parameter
+$W=KQ^\top$. Instead, directly parameterizing by $W$ minimizes a Frobenius norm
+objective. We characterize this convergence, highlighting that it can occur
+toward locally-optimal directions rather than global ones. (2) Complementing
+this, we prove the local/global directional convergence of gradient descent
+under suitable geometric conditions. Importantly, we show that
+over-parameterization catalyzes global convergence by ensuring the feasibility
+of the SVM problem and by guaranteeing a benign optimization landscape devoid
+of stationary points. (3) While our theory applies primarily to linear
+prediction heads, we propose a more general SVM equivalence that predicts the
+implicit bias with nonlinear heads. Our findings are applicable to arbitrary
+datasets and their validity is verified via experiments. We also introduce
+several open problems and research directions. We believe these findings
+inspire the interpretation of transformers as a hierarchy of SVMs that
+separates and selects optimal tokens.
+
+
+
+ comment: minor edits and update global convergence figure
+
+
+
+ + ♻ ☆ Max-Margin Token Selection in Attention Mechanism +
+ +
+ Attention mechanism is a central component of the transformer architecture
+which led to the phenomenal success of large language models. However, the
+theoretical principles underlying the attention mechanism are poorly
+understood, especially its nonconvex optimization dynamics. In this work, we
+explore the seminal softmax-attention model $f(\boldsymbol{X})=\langle
+\boldsymbol{Xv}, \texttt{softmax}(\boldsymbol{XWp})\rangle$, where
+$\boldsymbol{X}$ is the token sequence and
+$(\boldsymbol{v},\boldsymbol{W},\boldsymbol{p})$ are trainable parameters. We
+prove that running gradient descent on $\boldsymbol{p}$, or equivalently
+$\boldsymbol{W}$, converges in direction to a max-margin solution that
+separates $\textit{locally-optimal}$ tokens from non-optimal ones. This clearly
+formalizes attention as an optimal token selection mechanism. Remarkably, our
+results are applicable to general data and precisely characterize
+$\textit{optimality}$ of tokens in terms of the value embeddings
+$\boldsymbol{Xv}$ and problem geometry. We also provide a broader
+regularization path analysis that establishes the margin maximizing nature of
+attention even for nonlinear prediction heads. When optimizing $\boldsymbol{v}$
+and $\boldsymbol{p}$ simultaneously with logistic loss, we identify conditions
+under which the regularization paths directionally converge to their respective
+hard-margin SVM solutions where $\boldsymbol{v}$ separates the input features
+based on their labels. Interestingly, the SVM formulation of $\boldsymbol{p}$
+is influenced by the support vector geometry of $\boldsymbol{v}$. Finally, we
+verify our theoretical findings via numerical experiments and provide insights.
+
+
+
+ comment: minor edits and update convergence analysis figure
+
+
+
+ + ♻ ☆ LM-Infinite: Simple On-the-Fly Length Generalization for Large Language + Models +
+ +
+ In recent years, there have been remarkable advancements in the performance
+of Transformer-based Large Language Models (LLMs) across various domains. As
+these LLMs are deployed for increasingly complex tasks, they often face the
+need to conduct longer reasoning processes or understand larger contexts. In
+these situations, the length generalization failure of LLMs on long sequences
+becomes more prominent. Most pre-training schemes truncate training sequences
+to a fixed length. LLMs often struggle to generate fluent and coherent texts,
+let alone carry out downstream tasks, after longer contexts, even with relative
+positional encoding designed to cope with this problem. Common solutions such
+as finetuning on longer corpora often involve daunting hardware and time costs
+and require careful training process design. To more efficiently leverage the
+generation capacity of existing LLMs, we theoretically and empirically
+investigate the main out-of-distribution (OOD) factors contributing to this
+problem. Inspired by this diagnosis, we propose a simple yet effective solution
+for on-the-fly length generalization, LM-Infinite. It involves only a
+$\Lambda$-shaped attention mask (to avoid excessive attended tokens) and a
+distance limit (to avoid unseen distances) while requiring no parameter updates
+or learning. We find it applicable to a variety of LLMs using relative-position
+encoding methods. LM-Infinite is computationally efficient with $O(n)$ time and
+space, and demonstrates consistent text generation fluency and quality to as
+long as 32k tokens on ArXiv and OpenWebText2 datasets, with 2.72x decoding
+speedup. On downstream tasks such as passkey retrieval, it continues to work on
+inputs much longer than training lengths where vanilla models fail immediately.
+
+
+
+ comment: 9 pages, 4 figures
+
+
+
+ + ♻ ☆ BigText-QA: Question Answering over a Large-Scale Hybrid Knowledge Graph +
+ +
+ Answering complex questions over textual resources remains a challenge,
+particularly when dealing with nuanced relationships between multiple entities
+expressed within natural-language sentences. To this end, curated knowledge
+bases (KBs) like YAGO, DBpedia, Freebase, and Wikidata have been widely used
+and gained great acceptance for question-answering (QA) applications in the
+past decade. While these KBs offer a structured knowledge representation, they
+lack the contextual diversity found in natural-language sources. To address
+this limitation, BigText-QA introduces an integrated QA approach, which is able
+to answer questions based on a more redundant form of a knowledge graph (KG)
+that organizes both structured and unstructured (i.e., "hybrid") knowledge in a
+unified graphical representation. Thereby, BigText-QA is able to combine the
+best of both worlds$\unicode{x2013}$a canonical set of named entities, mapped
+to a structured background KB (such as YAGO or Wikidata), as well as an open
+set of textual clauses providing highly diversified relational paraphrases with
+rich context information. Our experimental results demonstrate that BigText-QA
+outperforms DrQA, a neural-network-based QA system, and achieves competitive
+results to QUEST, a graph-based unsupervised QA system.
+
+
+
+
+ + ♻ ☆ ToolAlpaca: Generalized Tool Learning for Language Models with 3000 + Simulated Cases +
+ +
+ Enabling large language models to utilize real-world tools effectively is
+crucial for achieving embodied intelligence. Existing approaches to tool
+learning have either primarily relied on extremely large language models, such
+as GPT-4, to attain generalized tool-use abilities in a zero-shot manner, or
+utilized supervised learning to train limited scopes of tools on compact
+models. However, it remains uncertain whether smaller language models can
+achieve generalized tool-use abilities without tool-specific training. To
+address this question, this paper introduces ToolAlpaca, a novel framework
+designed to automatically generate a diverse tool-use corpus and learn
+generalized tool-use abilities on compact language models with minimal human
+intervention. Specifically, ToolAlpaca first automatically creates a highly
+diversified tool-use corpus by building a multi-agent simulation environment.
+The corpus contains 3938 tool-use instances from more than 400 real-world tool
+APIs spanning 50 distinct categories. Subsequently, the constructed corpus is
+employed to fine-tune compact language models, resulting in two models, namely
+ToolAlpaca-7B and ToolAlpaca-13B, respectively. Finally, we evaluate the
+ability of these models to utilize previously unseen tools without specific
+training. Experimental results demonstrate that ToolAlpaca achieves effective
+generalized tool-use capabilities comparable to those of extremely large
+language models like GPT-3.5, demonstrating that learning generalized tool-use
+ability is feasible for compact language models.
+
+
+
+
+ + ♻ ☆ Structured Chain-of-Thought Prompting for Code Generation +
+ +
+ Large Language Models (LLMs) (e.g., ChatGPT) have shown impressive
+performance in code generation. LLMs take prompts as inputs, and
+Chain-of-Thought (CoT) prompting is the state-of-the-art prompting technique.
+CoT prompting asks LLMs first to generate CoTs (i.e., intermediate natural
+language reasoning steps) and then output the code. However, CoT prompting is
+designed for natural language generation and has low accuracy in code
+generation.
+ In this paper, we propose Structured CoTs (SCoTs) and present a novel
+prompting technique for code generation, named SCoT prompting. Our motivation
+is source code contains rich structural information and any code can be
+composed of three program structures (i.e., sequence, branch, and loop
+structures). Intuitively, structured intermediate reasoning steps make for
+structured source code. Thus, we ask LLMs to use program structures to build
+CoTs, obtaining SCoTs. Then, LLMs generate the final code based on SCoTs.
+Compared to CoT prompting, SCoT prompting explicitly constrains LLMs to think
+about how to solve requirements from the view of source code and further the
+performance of LLMs in code generation. We apply SCoT prompting to two LLMs
+(i.e., ChatGPT and Codex) and evaluate it on three benchmarks (i.e., HumanEval,
+MBPP, and MBCPP). (1) SCoT prompting outperforms the state-of-the-art baseline
+- CoT prompting by up to 13.79% in Pass@1. (2) Human evaluation shows human
+developers prefer programs from SCoT prompting. (3) SCoT prompting is robust to
+examples and achieves substantial improvements.
+
+
+
+ comment: arXiv admin note: text overlap with arXiv:2303.17780
+
+
+
+ + ♻ ☆ ZC3: Zero-Shot Cross-Language Code Clone Detection +
+ +
+ Developers introduce code clones to improve programming productivity. Many
+existing studies have achieved impressive performance in monolingual code clone
+detection. However, during software development, more and more developers write
+semantically equivalent programs with different languages to support different
+platforms and help developers translate projects from one language to another.
+Considering that collecting cross-language parallel data, especially for
+low-resource languages, is expensive and time-consuming, how designing an
+effective cross-language model that does not rely on any parallel data is a
+significant problem. In this paper, we propose a novel method named ZC3 for
+Zero-shot Cross-language Code Clone detection. ZC3 designs the contrastive
+snippet prediction to form an isomorphic representation space among different
+programming languages. Based on this, ZC3 exploits domain-aware learning and
+cycle consistency learning to further constrain the model to generate
+representations that are aligned among different languages meanwhile are
+diacritical for different types of clones. To evaluate our approach, we conduct
+extensive experiments on four representative cross-language clone detection
+datasets. Experimental results show that ZC3 outperforms the state-of-the-art
+baselines by 67.12%, 51.39%, 14.85%, and 53.01% on the MAP score, respectively.
+We further investigate the representational distribution of different languages
+and discuss the effectiveness of our method.
+
+
+
+ comment: Accepted by the 38th IEEE/ACM International Conference on Automated
+ Software Engineering (ASE 2023)
+
+
+
+ + ♻ ☆ EditSum: A Retrieve-and-Edit Framework for Source Code Summarization +
+ +
+ Existing studies show that code summaries help developers understand and
+maintain source code. Unfortunately, these summaries are often missing or
+outdated in software projects. Code summarization aims to generate natural
+language descriptions automatically for source code. Code summaries are highly
+structured and have repetitive patterns. Besides the patternized words, a code
+summary also contains important keywords, which are the key to reflecting the
+functionality of the code. However, the state-of-the-art approaches perform
+poorly on predicting the keywords, which leads to the generated summaries
+suffering a loss in informativeness. To alleviate this problem, this paper
+proposes a novel retrieve-and-edit approach named EditSum for code
+summarization. Specifically, EditSum first retrieves a similar code snippet
+from a pre-defined corpus and treats its summary as a prototype summary to
+learn the pattern. Then, EditSum edits the prototype automatically to combine
+the pattern in the prototype with the semantic information of input code. Our
+motivation is that the retrieved prototype provides a good start-point for
+post-generation because the summaries of similar code snippets often have the
+same pattern. The post-editing process further reuses the patternized words in
+the prototype and generates keywords based on the semantic information of input
+code. We conduct experiments on a large-scale Java corpus and experimental
+results demonstrate that EditSum outperforms the state-of-the-art approaches by
+a substantial margin. The human evaluation also proves the summaries generated
+by EditSum are more informative and useful. We also verify that EditSum
+performs well on predicting the patternized words and keywords.
+
+
+
+ comment: Accepted by the 36th IEEE/ACM International Conference on Automated
+ Software Engineering (ASE 2021)
+
+
+
+ + ♻ ☆ Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection +
+ +
+ Stance detection aims to identify the attitude expressed in a document
+towards a given target. Techniques such as Chain-of-Thought (CoT) prompting
+have advanced this task, enhancing a model's reasoning capabilities through the
+derivation of intermediate rationales. However, CoT relies primarily on a
+model's pre-trained internal knowledge during reasoning, thereby neglecting the
+valuable external information that is previously unknown to the model. This
+omission, especially within the unsupervised reasoning process, can affect the
+model's overall performance. Moreover, while CoT enhances Large Language Models
+(LLMs), smaller LMs, though efficient operationally, face challenges in
+delivering nuanced reasoning. In response to these identified gaps, we
+introduce the Ladder-of-Thought (LoT) for the stance detection task.
+Constructed through a dual-phase Progressive Optimization Framework, LoT
+directs the small LMs to assimilate high-quality external knowledge, refining
+the intermediate rationales produced. These bolstered rationales subsequently
+serve as the foundation for more precise predictions - akin to how a ladder
+facilitates reaching elevated goals. LoT achieves a balance between efficiency
+and performance. Our empirical evaluations underscore LoT's efficacy, marking a
+16% improvement over GPT-3.5 and a 10% enhancement compared to GPT-3.5 with CoT
+on stance detection task.
+
+
+
+ comment: 5 pages, 2 figures, 2 tables
+
+
+
+ + ♻ ☆ Claim Optimization in Computational Argumentation +
+ +
+ An optimal delivery of arguments is key to persuasion in any debate, both for
+humans and for AI systems. This requires the use of clear and fluent claims
+relevant to the given debate. Prior work has studied the automatic assessment
+of argument quality extensively. Yet, no approach actually improves the quality
+so far. To fill this gap, this paper proposes the task of claim optimization:
+to rewrite argumentative claims in order to optimize their delivery. As
+multiple types of optimization are possible, we approach this task by first
+generating a diverse set of candidate claims using a large language model, such
+as BART, taking into account contextual information. Then, the best candidate
+is selected using various quality metrics. In automatic and human evaluation on
+an English-language corpus, our quality-based candidate selection outperforms
+several baselines, improving 60% of all claims (worsening 16% only). Follow-up
+analyses reveal that, beyond copy editing, our approach often specifies claims
+with details, whereas it adds less evidence than humans do. Moreover, its
+capabilities generalize well to other domains, such as instructional texts.
+
+
+
+ comment: Accepted as a long paper at INLG 2023
+
+
+
+ + ♻ ☆ Layout and Task Aware Instruction Prompt for Zero-shot Document Image + Question Answering +
+ +
+ Layout-aware pre-trained models has achieved significant progress on document
+image question answering. They introduce extra learnable modules into existing
+language models to capture layout information within document images from text
+bounding box coordinates obtained by OCR tools. However, extra modules
+necessitate pre-training on extensive document images. This prevents these
+methods from directly utilizing off-the-shelf instruction-tuning language
+foundation models, which have recently shown promising potential in zero-shot
+learning. Instead, in this paper, we find that instruction-tuning language
+models like Claude and ChatGPT can understand layout by spaces and line breaks.
+Based on this observation, we propose the LAyout and Task aware Instruction
+Prompt (LATIN-Prompt), which consists of layout-aware document content and
+task-aware instruction. Specifically, the former uses appropriate spaces and
+line breaks to recover the layout information among text segments obtained by
+OCR tools, and the latter ensures that generated answers adhere to formatting
+requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning
+(LATIN-Tuning) to improve the performance of small instruction-tuning models
+like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot
+performance of Claude and ChatGPT to be comparable to the fine-tuning
+performance of SOTAs on document image question answering, and LATIN-Tuning
+enhances the zero-shot performance of Alpaca significantly. For example,
+LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by 263%
+and 20% respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA
+by 87.7%. Quantitative and qualitative analyses demonstrate the effectiveness
+of LATIN-Prompt and LATIN-Tuning. We provide the code in supplementary and will
+release it to facilitate future research.
+
+
+
+ comment: Add the LATIN-Tuning for Alapca. Code is available at
+ https://github.com/WenjinW/LATIN-Prompt
+
+
+
+ + ♻ ☆ A Survey on Large Language Model based Autonomous Agents +
+
+
+
+
+
+
+
+ Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, Ji-Rong Wen
+
+
+ Autonomous agents have long been a prominent research focus in both academic
+and industry communities. Previous research in this field often focuses on
+training agents with limited knowledge within isolated environments, which
+diverges significantly from human learning processes, and thus makes the agents
+hard to achieve human-like decisions. Recently, through the acquisition of vast
+amounts of web knowledge, large language models (LLMs) have demonstrated
+remarkable potential in achieving human-level intelligence. This has sparked an
+upsurge in studies investigating LLM-based autonomous agents. In this paper, we
+present a comprehensive survey of these studies, delivering a systematic review
+of the field of LLM-based autonomous agents from a holistic perspective. More
+specifically, we first discuss the construction of LLM-based autonomous agents,
+for which we propose a unified framework that encompasses a majority of the
+previous work. Then, we present a comprehensive overview of the diverse
+applications of LLM-based autonomous agents in the fields of social science,
+natural science, and engineering. Finally, we delve into the evaluation
+strategies commonly used for LLM-based autonomous agents. Based on the previous
+studies, we also present several challenges and future directions in this
+field. To keep track of this field and continuously update our survey, we
+maintain a repository of relevant references at
+https://github.com/Paitesanshi/LLM-Agent-Survey.
+
+
+
+ comment: 35 pages, 5 figures, 3 tables
+
+
+
+ + ♻ ☆ Halo: Estimation and Reduction of Hallucinations in Open-Source Weak + Large Language Models +
+
+
+
+
+
+
+
+ Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, Shizhu Liu, Pingchuan Tian, Yuping Wang, Yuxuan Wang
+
+
+ Large Language Models (LLMs) have revolutionized Natural Language Processing
+(NLP). Although convenient for research and practical applications, open-source
+LLMs with fewer parameters often suffer from severe hallucinations compared to
+their larger counterparts. This paper focuses on measuring and reducing
+hallucinations in BLOOM 7B, a representative of such weaker open-source LLMs
+that are publicly available for research and commercial applications. We
+introduce HaloCheck, a lightweight BlackBox knowledge-free framework designed
+to quantify the severity of hallucinations in LLMs. Additionally, we explore
+techniques like knowledge injection and teacher-student approaches to alleviate
+hallucinations in low-parameter LLMs. Our experiments effectively demonstrate
+the reduction of hallucinations in challenging domains for these LLMs.
+
+
+
+
+ + ♻ ☆ Prompting Multilingual Large Language Models to Generate Code-Mixed + Texts: The Case of South East Asian Languages +
+
+
+
+
+
+
+
+ Zheng-Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Samuel Cahyawijaya, Holy Lovenia, Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Long Phan, Yin Lin Tan, Thamar Solorio, Alham Fikri Aji
+
+
+ While code-mixing is a common linguistic practice in many parts of the world,
+collecting high-quality and low-cost code-mixed data remains a challenge for
+natural language processing (NLP) research. The recent proliferation of Large
+Language Models (LLMs) compels one to ask: how capable are these systems in
+generating code-mixed data? In this paper, we explore prompting multilingual
+LLMs in a zero-shot manner to generate code-mixed data for seven languages in
+South East Asia (SEA), namely Indonesian, Malay, Chinese, Tagalog, Vietnamese,
+Tamil, and Singlish. We find that publicly available multilingual
+instruction-tuned models such as BLOOMZ and Flan-T5-XXL are incapable of
+producing texts with phrases or clauses from different languages. ChatGPT
+exhibits inconsistent capabilities in generating code-mixed texts, wherein its
+performance varies depending on the prompt template and language pairing. For
+instance, ChatGPT generates fluent and natural Singlish texts (an English-based
+creole spoken in Singapore), but for English-Tamil language pair, the system
+mostly produces grammatically incorrect or semantically meaningless utterances.
+Furthermore, it may erroneously introduce languages not specified in the
+prompt. Based on our investigation, existing multilingual LLMs exhibit a wide
+range of proficiency in code-mixed data generation for SEA languages. As such,
+we advise against using LLMs in this context without extensive human checks.
+
+
+
+
+ + ♻ ☆ Aligning Large Language Models for Clinical Tasks +
+ +
+ Large Language Models (LLMs) have demonstrated remarkable adaptability,
+showcasing their capacity to excel in tasks for which they were not explicitly
+trained. However, despite their impressive natural language processing (NLP)
+capabilities, effective alignment of LLMs remains a crucial challenge when
+deploying them for specific clinical applications. The ability to generate
+responses with factually accurate content and to engage in non-trivial
+reasoning steps are crucial for the LLMs to be eligible for applications in
+clinical medicine. Employing a combination of techniques including
+instruction-tuning and in-prompt strategies like few-shot and chain-of-thought
+prompting has significantly enhanced the performance of LLMs. Our proposed
+alignment strategy for medical question-answering, known as
+'expand-guess-refine', offers a parameter and data-efficient solution. A
+preliminary analysis of this method demonstrated outstanding performance,
+achieving a score of 70.63% on a subset of questions sourced from the USMLE
+dataset.
+
+
+
+ comment: 10 pages, 3 figures
+
+
+
+ + ♻ ☆ Zero-shot information extraction from radiological reports using ChatGPT +
+ +
+ Electronic health records contain an enormous amount of valuable information,
+but many are recorded in free text. Information extraction is the strategy to
+transform the sequence of characters into structured data, which can be
+employed for secondary analysis. However, the traditional information
+extraction components, such as named entity recognition and relation
+extraction, require annotated data to optimize the model parameters, which has
+become one of the major bottlenecks in building information extraction systems.
+With the large language models achieving good performances on various
+downstream NLP tasks without parameter tuning, it becomes possible to use large
+language models for zero-shot information extraction. In this study, we aim to
+explore whether the most popular large language model, ChatGPT, can extract
+useful information from the radiological reports. We first design the prompt
+template for the interested information in the CT reports. Then, we generate
+the prompts by combining the prompt template with the CT reports as the inputs
+of ChatGPT to obtain the responses. A post-processing module is developed to
+transform the responses into structured extraction results. We conducted the
+experiments with 847 CT reports collected from Peking University Cancer
+Hospital. The experimental results indicate that ChatGPT can achieve
+competitive performances for some extraction tasks compared with the baseline
+information extraction system, but some limitations need to be further
+improved.
+
+
+
+
+ + ♻ ☆ HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models +
+
+
+
+
+
+
+
+ Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, Songseong Kim
+
+
+ Large Language Models (LLMs) pretrained on massive corpora exhibit remarkable
+capabilities across a wide range of tasks, however, the attention given to
+non-English languages has been limited in this field of research. To address
+this gap and assess the proficiency of language models in the Korean language
+and culture, we present HAE-RAE Bench, covering 6 tasks including vocabulary,
+history, and general knowledge. Our evaluation of language models on this
+benchmark highlights the potential advantages of employing Large
+Language-Specific Models(LLSMs) over a comprehensive, universal model like
+GPT-3.5. Remarkably, our study reveals that models approximately 13 times
+smaller than GPT-3.5 can exhibit similar performance levels in terms of
+language-specific knowledge retrieval. This observation underscores the
+importance of homogeneous corpora for training professional-level
+language-specific models. On the contrary, we also observe a perplexing
+performance dip in these smaller LMs when they are tasked to generate
+structured answers.
+
+
+
+
+ + ♻ ☆ Automating Behavioral Testing in Machine Translation +
+ +
+ Behavioral testing in NLP allows fine-grained evaluation of systems by
+examining their linguistic capabilities through the analysis of input-output
+behavior. Unfortunately, existing work on behavioral testing in Machine
+Translation (MT) is currently restricted to largely handcrafted tests covering
+a limited range of capabilities and languages. To address this limitation, we
+propose to use Large Language Models (LLMs) to generate a diverse set of source
+sentences tailored to test the behavior of MT models in a range of situations.
+We can then verify whether the MT model exhibits the expected behavior through
+matching candidate sets that are also generated using LLMs. Our approach aims
+to make behavioral testing of MT systems practical while requiring only minimal
+human effort. In our experiments, we apply our proposed evaluation framework to
+assess multiple available MT systems, revealing that while in general
+pass-rates follow the trends observable from traditional accuracy-based
+metrics, our method was able to uncover several important differences and
+potential bugs that go unnoticed when relying only on accuracy.
+
+
+