AIMS-RAIL/RAIL

NeurIPS 2026

Hongyu Jin^1,*, Siyi Wang^1,*, Yang Xiao^1,*, Jiaheng Dong^1,*, Shihong Tan⁴, Kaiyuan Peng¹, Georgiana Juravle², Shanquan Chen³, Gongping Huang⁴, Hong Jia⁵, Eun-Jung Holden¹, James Bailey⁶, Ting Dang^1,†

¹The University of Melbourne, ²Alexandru Ioan Cuza University of Iași, ³The University of Hong Kong, ⁴Wuhan University, ⁵The University of Auckland, ⁶Monash University

^*Equal contribution. ^†Corresponding author.

Contact: Hongyu Jin (hongyuj1@student.unimelb.edu.au), Ting Dang (ting.dang@unimelb.edu.au)

Project page · arXiv · Paper on Hugging Face · Dataset on Hugging Face

TL;DR: RAIL evaluates large audio-language models the way cognitive science assesses human listeners. Grounded in Cattell–Horn–Carroll (CHC) theory, it organizes 5,306 audio questions into five core auditory capabilities and 32 subcapabilities, and compares 26 LALMs with a human baseline. Models do well on knowledge inherited from text pretraining, while fine-grained auditory perception and processing efficiency remain below human level.

data/<capability>/<TASK>.jsonl    32 task files, 5,306 items
audio/<capability>/<TASK>/        6,485 audio clips (Git LFS)
eval/                             evaluation toolkit: HF inference, scoring, English report
scripts/build_rail_v2.py          rebuilds data/ and audio/ from the v1 release
metadata.json                     Croissant metadata with RAI fields
build_report.json                 build checks, statistics and flagged items

Clone with Git LFS to get the audio:

git lfs install
git clone https://github.com/AIMS-RAIL/RAIL.git

Items are organised in three levels that follow the paper: 5 core capabilities (CHC broad abilities), 32 tasks (CHC narrow abilities, named and coded as in the paper) and, where a task has more than one design, subtasks. Each task is one file, data/<capability>/<TASK>.jsonl (U1/U9 is stored as U1_U9).

Capability folders: auditory_processing, reasoning, memory, processing_efficiency, knowledge.

Every line of a task file is one item:

{"id": "gsm-ma-0001", "capability": "Memory", "task": "Associative Memory", "task_code": "MA",
 "answer_format": "multiple_choice", "question": "Which word means 'sync data'?",
 "choices": ["noik", "niok", "noem", "slaiel"], "answer": "noik", "answer_label": "A",
 "audio_paths": ["audio/memory/MA/gsm-ma-0001.wav"], "duration_s": 17.491, "...": "..."}

Loading a clone with datasets:

import sys
from datasets import load_dataset

sys.path.insert(0, "eval")
from rail_eval.data import hf_features  # explicit schema: some files have empty `choices`

rail = load_dataset("json", data_files="data/*/*.jsonl", features=hf_features(), split="train")
memory = load_dataset("json", data_files="data/memory/*.jsonl", features=hf_features(), split="train")

eval/ runs a Hugging Face audio-language model on every item, scores each item with the rule for its answer format, and writes an English report. The prompts and scoring rules are the ones used for the paper; re-scoring the paper's Qwen2-Audio-7B-Instruct outputs with this code reproduces the published numbers.

pip install torch transformers librosa numpy          # openai only for the optional LLM judge
cd eval

# full benchmark on one GPU: inference, scoring, report
python run_rail_eval.py all --model Qwen/Qwen2-Audio-7B-Instruct --out runs/qwen2_audio

# smoke test with 5 items per task
python run_rail_eval.py all --model Qwen/Qwen2-Audio-7B-Instruct --out runs/smoke --limit-per-task 5

# re-score existing predictions, optionally with the GPT-5.4 judge
python run_rail_eval.py score --out runs/qwen2_audio --judge-model gpt-5.4

• Models are loaded with transformers from_pretrained (Hub id or local checkpoint). Qwen2-Audio has a dedicated adapter; other models go through a generic chat-template adapter (--adapter generic, --trust-remote-code, --merge-multi-audio for single-audio models).
• Prompting: every prompt is wrapped so the model answers Reason: ...; Answer: ... (reason capped at 20 words, except Reasoning items); decoding is greedy.
• Scoring: strict option matching for multiple choice, label matching for closed-label tasks, numeric matching for open-ended math, token recall for free recall (M6), and B-AUC over reason length (0–50 tokens) for Processing Efficiency. Capability scores are pooled as in the paper's main table.
• Outputs: predictions*.jsonl, scored.jsonl, summary.json and report.md in the run folder.

Runs resume after interruption and can be sharded over GPUs (--num-shards, --shard-index). A Slurm template is in eval/slurm/. See eval/README.md for details. The original scoring scripts used for the paper are kept in eval/legacy/.

data/ and audio/ are generated from the v1 release (commit 8ef268a) by scripts/build_rail_v2.py, which relabels items with the paper's taxonomy, normalises the answer fields, rebuilds the evaluated memory prompts and checks the result against the paper's statistics:

git worktree add ../rail-v1 8ef268a
python scripts/build_rail_v2.py --v1-root ../rail-v1 --out /path/to/release

The output folder also contains the Hugging Face dataset card (README.md).

• Items are labelled capability → task → subtask as in the paper; the v1 fields benchmark, subset, task and ability are replaced, and files are split by task instead of by source manifest.
• Every id is unique (v1 had 35 duplicates); v1 ids are kept in legacy_id.
• Memory prompts and options are the exact ones used in evaluation (v1 lacked the Memory Span prompts, the options of Working and Meaningful Memory prompts, and the prosodic-matching options); speaker-tracking items use the evaluated wording.
• Options carry no letter prefixes; Processing Efficiency items list their allowed labels.
• duration_s is measured from the audio files for every item.
• Audio is stored as audio/<capability>/<TASK>/<id>[_k].<ext>.
• The evaluation toolkit in eval/ replaces the standalone scoring scripts.

@inproceedings{jin2026rail,
  title     = {RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models
               with a CHC-Grounded Benchmark},
  author    = {Jin, Hongyu and Wang, Siyi and Xiao, Yang and Dong, Jiaheng and
               Tan, Shihong and Peng, Kaiyuan and Juravle, Georgiana and Chen, Shanquan and
               Huang, Gongping and Jia, Hong and Holden, Eun-Jung and Bailey, James and
               Dang, Ting},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.11260}
}

The dataset is released under CC BY 4.0. Some items are derived from existing corpora; use them within the terms of those sources.