Drishti Sharma

M-REWARDBENCH: Evaluating Reward Models in Multilingual Settings
Accepted to ACL 2025 (Main Conference) | Rated 4/5
Srishti Gureja, Lester J. Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata
Advisors: Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee
Paper / Code / Dataset / Leaderboard
This work presents M-REWARDBENCH, the first large-scale benchmark for evaluating reward models (RMs) in multilingual settings, covering 23 languages across 8 scripts and 5 language families. The benchmark assesses four core capabilities: (1) chat, (2) safety, (3) reasoning, and (4) translation, using 2.87k human-aligned preference instances. A total of 25 reward models—spanning Classifier, Generative, and Implicit (DPO-trained) types—are evaluated. Key findings include: (1) Generative RMs (e.g., GPT-4 Turbo) achieve the highest multilingual performance and cross-lingual consistency, with only a 3% average performance drop compared to English, (2) Classifier and Implicit RMs exhibit larger drops (~8–13%) and greater volatility, especially in subjective categories like chat and safety, and (3) translation quality, language resource availability, and script type (e.g., Latin, Cyrillic) significantly impact RM performance. For translation, the benchmark incorporates easy and hard subsets from the MAPLE dataset across four directions (en↔zh, en↔de), showing that harder tasks consistently reduce accuracy. Performance improves by 1–3% when using higher-quality translations (Google Translate over NLLB-3.3B).

Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory QA
4th place @ COLING 2025 (RegNLP Track)
Jebish Purbey, Drishti Sharma, Siddhant Gupta, Khawaja Murad, Siddartha Pullakhandam, Ram Mohan Rao Kadiyala
Paper
It introduces LeSeR (Lexical Reranking of Semantic Retrieval), a hybrid method that first performs dense semantic retrieval using fine-tuned embedding models on query-passage pairs, then reranks results with BM25, a classical lexical retrieval method. This two-stage decoupled pipeline boosts both recall and precision, outperforming standalone dense or lexical models. We experimented with multiple embedding models—Stella, BGE, CDE, and MPNet—applying fine-tuning using Multiple Negative Symmetric Ranking (MNSR) Loss. Among these, BGE_MNSR integrated with LeSeR (BGE_LeSeR) achieved the best retrieval performance, with Recall@10 of 0.8201 and mAP@10 of 0.6655, outperforming both dense-only and lexical-only baselines. For answer generation, the top-performing combination was BGE_LeSeR with Qwen2.5 7B, which achieved the highest RePASs score (0.4340), demonstrating strong entailment, obligation coverage, and low contradiction.

Behind Maya: Building a Multilingual Vision Language Model
Accepted @ VLMs4ALL Workshop, CVPR 2025
Drishti Sharma et al.
Paper
This work introduces Maya, an open-source multilingual VLM designed to address the underperformance of existing VLMs in low-resource languages and diverse cultural contexts. It is built on two major contributions: (1) a multilingual image-text pretraining dataset with 4.4M samples across 8 languages (English, Chinese, French, Spanish, Russian, Hindi, Japanese, Arabic), created via a hybrid translation framework using tools like Aya 35B, rigorous prompt engineering (BLEU/N-gram scoring), and balanced sampling from the LLaVA dataset; (2) a multilingual multimodal model architecture that uses SigLIP as the vision encoder (replacing CLIP for its multilingual adaptability and variable patch size support) and Aya-23 8B as the LLM (supporting 23 languages with an 8K context window). Visual features are projected via a 2-layer MLP with GELU to align with language space, inspired by LLaVA 1.5.Maya outperforms PALO-7B on LLaVA-Bench-In-The-Wild, offering a strong multilingual alternative to LLaVA.

Detection of Language, Hate Speech, and Targets using LLMs in Devanagari Script
Accepted @ COLING 2025 (CHiPSAL Track)
Drishti Sharma et al.
Paper
This work presents a modular multilingual NLP system for five Devanagari-script languages—Hindi, Nepali, Marathi, Sanskrit, and Bhojpuri—addressing three tasks: (1) language identification, (2) hate speech detection, and (3) hate speech target classification. Each task is tackled using a distinct set of models and strategies: (1) for language identification, an ensemble of fine-tuned IndicBERT V2, MuRIL, and Gemma-2 9B models achieved high accuracy via majority voting with fallback logic; (2) for hate speech detection, a separate ensemble combined IndicBERT V2, Gemma-2 9B, and Gemma-2 27B fine-tuned using Odds Ratio Preference Optimization (ORPO), with BERT-based models trained using focal loss (α = 0.35, γ = 4.0) to address severe class imbalance; and (3) for target classification, the best performance was achieved by a single Gemma-2 27B model fine-tuned with ORPO, without ensembling. All decoder-only models were fine-tuned using 4-bit LoRA to ensure efficient training. The Sub-task A ensemble achieved an F1 score of 0.9980, the Sub-task B ensemble reached 0.7652, and the standalone model in Sub-task C attained 0.6804, demonstrating the value of task-specific model design for low-resource, script-sensitive language understanding.

Sequential Learning for Claim Verification and Explanation in Financial Domains
3rd place @ COLING 2025 (FINLP Track)
Drishti Sharma et al.
Paper
This work presents SeQwen, our sequential learning-based system developed for the COLING 2025 Financial Misinformation Detection (FMD) challenge, which focuses on two tasks: (1) classifying financial claims as True, False, or Not Enough Information (NEI), and (2) generating coherent explanations for each classification. We used the FIN-FACT dataset and evaluated multiple open-source LLMs—Qwen2.5, Mistral, Llama3, Gemma-2, and Phi-3. We adopted a two-stage sequential fine-tuning approach, where models were first fine-tuned for classification and then for joint classification and explanation generation. Key findings include: (1) Qwen2.5 7B emerged as the best-performing model, (2) sequential fine-tuning (SeQwen) consistently outperformed single-stage joint training, yielding a 7.1% improvement in overall score and substantial gains in explanation quality, and (3) simply extending joint training epochs led to smaller improvements than our staged approach.

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
✨ Spotlight @ ICLR 2025
Drishti Sharma et al.
Paper
This work presents INCLUDE, a large-scale multilingual benchmark designed to evaluate LLMs on regional knowledge and cultural grounding across 44 languages and 15 scripts. INCLUDE comprises 197,243 multiple-choice questions sourced from 1,926 real-world exams administered in 52 countries, covering diverse subjects such as law, history, culture, and general knowledge. It includes two subsets—INCLUDE-BASE and INCLUDE-LITE—for scalable evaluation across a wide range of LLMs. Models evaluated include GPT-4o, GPT-3.5, Claude, Gemini, Mistral, Yi, and others. GPT-4o achieves the highest overall accuracy, but all models exhibit substantial performance degradation in underrepresented languages and unfamiliar scripts. Prompt language (English vs. native) and instruction tuning have limited impact on overall accuracy.

Robust and Fine-Grained Detection of AI-Generated Texts
Submitted to ACL 2025
Drishti Sharma et al.
Paper
Trained and benchmarked models for fine-grained AI-generated content detection across 2.4M multilingual samples...

Improving Multilingual Capabilities with Cultural and Local Knowledge in LLMs
Submitted to ACL 2025
Drishti Sharma et al.
Paper
Instruction-tuned 7 LLMs with 485K culturally-informed English-Hindi pairs, yielding consistent improvements...