Zheng Lian

News

Job Openings: I am recruiting visiting students in Institute of Automation, Chinese Academy of Sciences.

• [Nov, 2025] Three papers are accepted by AAAI 2026 (including 2 Oral papers), congratulations to Liyun and Jinyang.

• [Oct, 2025] I will co-organize MER'25 Challenge and MRAC'25 Workshop@ACM Multimedia [link]

• [Oct, 2025] One paper receives Best Paper Mention Award in PRCV, congratulations to Yazhou.

• [Oct, 2025] IRNet is accepted by IEEE TPAMI

• [Oct, 2025] I will serve as Area Chair at ACL ARR 2025

• [Sep, 2025] I am selected to Stanford University's Top 2% Scientists list.

• [Sep, 2025] Two papers are accepted by NeurIPS 2025, congratulations to Ziyang and Ziyu.

• [Sep, 2025] Congratulations on successful promotion to CCF Senior Member.

• [Sep, 2025] One paper is accepted by IEEE TAFFC, congratulations to Yazhou.

• [Aug, 2025] Congratulations on successful promotion to IEEE Senior Member.

• [Jul, 2025] Three papers are accepted by ACM Multimedia 2025.

• [Jun, 2025] I will serve as Associate Editor at IEEE TASLP

• [May, 2025] One paper is accepted by ACL25(Finding), congratulations to Zhuofan.

• [May, 2025] OV-MER and AffectGPT are accepted by ICML25, including one Oral (Top 1%) paper.

• [Mar, 2025] I will serve as Area Chair at ACM Multimedia 2025.

• [Mar, 2025] Two paper are accepted by ICASSP2025

• [Feb, 2025] I will serve as Area Editor at Information Fusion.

• [Jan, 2025] I will serve as Associate Editor at IEEE TAFFC.

• [Sep, 2024] Emotion-LLaMA is accepted by NeurIPS24, congratulations to Zebang.

• [Aug, 2024] We organize MEIJU'25 Challenge@ICASSP [link]

• [Jul, 2024] SVFAP is accepted by TAFFC, congratulations to Licai.

• [Apr, 2024] We organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM Multimedia [link]

• [Mar, 2024] HiCMAE is accepted by Information Fusion, congratulations to Licai.

• [Mar, 2024] GPT-4V with Emotion is accepted by Information Fusion

• [Jan, 2024] We build MERBench, a unified evaluation benchmark for multimodal emotion recognition

• [Nov, 2023] Two papers are accepted by NeurIPS23

• [Oct, 2023] MAE-DFER is accepted by ACM MM, congratulations to Licai.

• [Apr, 2023] We organize MER'23 Challenge and MRAC'23 Workshop@ACM Multimedia [link]

• [May, 2023] EMT-DLFR is accepted by IEEE TAFFC, congratulations to Licai.

• [Jan, 2023] GCNet is accepted by IEEE TPAMI

• [Jul, 2022] PIRNet is accepted by IEEE TNNLS

• [Jan, 2022] SMIN is accepted by IEEE TAFFC

• [Jan, 2021] CTNet is accepted by IEEE TASLP

Research

1. IRNet: Iterative Refinement Network for Noisy Partial Label Learning
   Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, Lei Feng, Jianhua Tao
   Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2025 | paper

2. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
   Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, Jianhua Tao
   ICML 2025 (Oral, Top 1%) | paper

3. OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
   Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, Jianhua Tao
   ICML 2025 | paper

4. AStar: Boosting Multimodal Reasoning with Automated Structured Thinking
   Jinyang Wu, Mingkuan Feng, Guocheng Zhai, Shuai Zhang, Zheng Lian, Fangrui Lv, Pengpeng Shao, Ruihan Jin, Zengqi Wen, Jianhua Tao
   AAAI (Oral) 2026

5. QuMAB: Query-based Multi-annotator Behavior Pattern Learning
   Liyun Zhang, Zheng Lian, Hong Liu, Takanori Takebe, Yuta Nakashima
   AAAI (Oral) 2026

6. SimLabel: Similarity-Weighted Semi-supervision for Multi-annotator Learning with Missing Labels
   Liyun Zhang, Zheng Lian, Hong Liu, Takanori Takebe, Yuta Nakashima
   AAAI 2026

7. SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding
   Yazhou Zhang, Chunwang Zou, Zheng Lian†, Prayag Tiwari†, Jing Qin
   IEEE Transactions on Affective Computing (IEEE TAFFC) 2025 | paper

8. Are MLMs Trapped in the Visual Room?
   Yazhou Zhang, Chunwang Zou, Qimeng Liu, Lu Rong, Ben Yao, Zheng Lian†, Qiuchi Li†, Peng Zhang, Jing Qin.
   PRCV (Best Paper Mention Award) 2025 | paper

9. REFED: A Subject Real-time Dynamic Labeled EEG-fNIRS Synchronized Recorded Emotion Dataset
   Xiaojun Ning, Jing Wang, Zhiyang Feng, Tianzuo Xin, Shuo Zhang, Shaoqi Zhang, Zheng Lian, Yi Ding, Youfang Lin, Ziyu Jia
   NeurIPS (D&B Track) 2025| paper

10. MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
    Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, etc.
    NeurIPS (D&B Track) 2025 | paper

11. MER 2025: When affective computing meets large language models
    Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng, Haolin Zuo, Ziyang Ma, Xiaojiang Peng, Xie Chen, Ya Li, Erik Cambria, Guoying Zhao, Björn W Schuller, Jianhua Tao
    ACM Multimedia 2025 | paper

12. Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities
    Rui Liu, Haolin Zuo, Zheng Lian, Hongyu Yuan, Qi Fan
    ACM Multimedia 2025 | paper

13. ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection
    Hao Gu, Jiangyan Yi, Chenglong Wang, Jianhua Tao, Zheng Lian, Jiayi He, Yong Ren, Yujie Chen, Zhengqi Wen
    ACM Multimedia 2025 | paper

14. Listen, Watch, and Learn to Feel: Retrieval-Augmented Emotion Reasoning for Compound Emotion Generation
    Zhuofan Wen, Zheng Lian, Shun Chen, Hailiang Yao, Longjiang Yang, Bin Liu, Jianhua Tao
    ACL (Findings) 2025 | paper

15. MEIJU-The 1st Multimodal Emotion and Intent Joint Understanding Challenge
    Rui Liu, Xiaofen Xing, Zheng Lian, Haizhou Li, Björn W Schuller, Haolin Zuo
    ICASSP 2025 | paper

16. Adversarial Training and Gradient Optimization for Partially Deepfake Audio Localization
    Siding Zeng, Jiangyan Yi, Jianhua Tao, Jiayi He, Zheng Lian, Shan Liang, Chuyuan Zhang, Yujie Chen, Xiaohui Zhang
    ICASSP 2025 | paper

17. Explainable Multimodal Emotion Recognition
    Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, etc.
    Arxiv 2024 | paper

18. MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
    Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, etc.
    Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing 2024 | paper

19. GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao
    Information Fusion 2024 | paper

20. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann
    NeurIPS 2024 | paper

21. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
    Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
    Information Fusion 2024 | paper

22. SVFAP: Self-supervised Video Facial Affect Perceiver
    Licai Sun, Zheng Lian†, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu†, Jianhua Tao†
    IEEE Transactions on Affective Computing (IEEE TAFFC) 2024 | paper

23. Pseudo Labels Regularization for Imbalanced Partial-Label Learning
    Mingyu Xu, Zheng Lian, Bin Liu, Zerui Chen, Jianhua Tao
    ICASSP 2024 | paper

24. NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption
    Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan
    LREC-COLING 2024 | paper

25. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
    Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, Haizhou Li
    IEEE Transactions on Affective Computing (IEEE TAFFC) 2024 | paper

26. MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition
    Haiyang Sun, Fulin Zhang, Yingying Gao, Zheng Lian, Shilei Zhang, Junlan Feng
    Interspeech 2024 | paper

27. MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
    Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
    Arxiv 2023 | paper

28. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
    Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao
    Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023 | paper

29. MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
    Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, etc.
    ACM Multimedia (Organize Grand Challenge) 2023 | paper

30. ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
    Mingyu Xu*, Zheng Lian*, Lei Feng, Bin Liu, Jianhua Tao
    NeurIPS 2023 | paper

31. VRA: Variational Rectified Activation for Out-of-distribution Detection
    Mingyu Xu, Zheng Lian†, Bin Liu, Jianhua Tao
    NeurIPS 2023 | paper

32. MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
    Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
    ACM Multimedia 2023 | paper

33. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis
    Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
    IEEE Transactions on Affective Computing (IEEE TAFFC) 2023 | paper

34. EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition
    Haiyang Sun*, Zheng Lian*, Bin Liu, Ying Li, Licai Sun, Cong Cai, Jianhua Tao, Meng Wang, Yuan Cheng
    Interspeech 2023 | paper

35. Integrating VideoMAE based model and Optical Flow for Micro-and Macro-expression Spotting
    Ke Xu, Kang Chen, Licai Sun, Zheng Lian, Bin Liu, Gong Chen, Haiyang Sun, Mingyu Xu, Jianhua Tao
    ACM Multimedia 2023 | paper

36. PIRNet: Personality-enhanced Iterative Refinement Network for Emotion Recognition in Conversation
    Zheng Lian, Bin Liu, Jianhua Tao
    IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022 | Early Access

37. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition
    Zheng Lian, Bin Liu, Jianhua Tao
    IEEE Transactions on Affective Computing (IEEE TAFFC) 2022 | paper

38. AMOA: Global acoustic feature enhanced modal-order-aware network for multimodal sentiment analysis
    Ziming Li, Yan Zhou, Weibo Zhang, Yaxin Liu, Chuanpeng Yang, Zheng Lian, Songlin Hu
    COLING 2022 | paper

39. DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition
    Zheng Lian, Bin Liu, Jianhua Tao
    Neurocomputing 2021 | paper

40. CTNet: Conversational Transformer Network for Emotion Recognition
    Zheng Lian, Bin Liu, Jianhua Tao
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) 2021 | paper

41. Towards Fine-Grained Prosody Control for Voice Conversion
    Zheng Lian, Rongxiu Zhong, Zhengqi Wen, Bin Liu, Jianhua Tao
    Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2021 | paper

42. Investigation of multimodal features, classifiers and fusion methods for emotion recognition
    Zheng Lian, Ya Li, Jianhua Tao, Jian Huang
    National Conference Man-Machine Speech Communication, NCMMSC 2021 | paper

43. Multimodal Cross-and Self-Attention Network for Speech Emotion Recognition
    Licai Sun, Bin Liu, Jianhua Tao, Zheng Lian
    ICASSP 2021 | paper

44. Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks
    Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
    Interspeech 2020 | paper

45. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
    Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
    Interspeech 2020 | paper

46. ARVC: An Auto-Regressive Voice Conversion System Without Parallel Training Data
    Zheng Lian, Zhengqi Wen, Xinyong Zhou, Songbai Pu, Shengkai Zhang, Jianhua Tao
    Interspeech 2020 | paper

47. Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition
    Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian
    Interspeech 2020 | paper

48. Multimodal Transformer Fusion for Continuous Emotion Recognition
    Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, Mingyue Niu
    ICASSP 2020 | paper

49. Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
    Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
    IEEE Transactions on Affective Computing (IEEE TAFFC) 2020 | paper

50. Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
    Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
    Interspeech 2019 | paper

51. Conversational Emotion Analysis via Attention Mechanisms
    Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
    Interspeech 2019 | paper

52. Discriminative video representation with temporal order for micro-expression recognition
    Mingyue Niu, Jianhua Tao, Ya Li, Jian Huang, Zheng Lian
    ICASSP 2019 | paper

53. End-to-End Continuous Emotion Recognition from Video Using 3D ConvLSTM Networks
    Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Jiangyan Yi
    ICASSP 2018 | paper

54. Speech Emotion Recognition from Variable-Length Inputs with Triplet Loss Function
    Jian Huang, Ya Li, Jianhua Tao, Zheng Lian
    Interspeech 2018 | paper

Awards

• 2025, Select to Stanford University's Top 2% Scientists list.

• 2025, Best Paper Mention Award in PRCV

• 2024, Winner in the MuSe Challenge@ACM MM

• 2023, Best Poster in the First CSIG Conference on Emotional Intelligence

• 2023, Winner in the MEGC Challenge@ACM MM

• 2023, Outstanding Reviewer of ICASSP

• 2022, Winner in the MuSe Challenge@ACM MM

• 2021, The First Prize for Technological Invention of Chinese Institute of Electronics

• 2021, Best Paper for National Conference on Man-Machine Speech Communication (NCMMSC)

• 2021, Winner in the MuSe Challenge@ACM MM

• 2021, Climbing First-class Scholarship of Institute of Automation, Chinese Academy of Sciences

• 2020, Winner in the MuSe Challenge@ACM MM

• 2020, Interspeech Travel Grand

• 2020, Winner in Voice Conversion Challenge@Interspeech

• 2019, 2nd Place in the AVEC Challenge@ACM MM

• 2018, 2nd Place in the AVEC Challenge@ACM MM

Academic Services

• Co-organize MER'25 Challenge and MRAC'25 Workshop@ACM MM, 2025

• Co-organize MEIJU'25 Challenge@ICASSP, 2025

• Co-organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM MM, 2024

• Co-organize MER'23 Challenge@ACM MM and MRAC'23 Workshop@ACM MM, 2023

• Executive Committee Member, Speech Dialogue and Auditory Professional Committee, China Computer Federation (CCF)（2024-）

• Committee Member, Chinese Information Processing Society of China, Emotional Computing Committee（2021-）

• Committee Member, Chinese Society of Image and Graphics, Emotional Computing and Understanding Committee（2021-）

• Committee Member, Chinese Association for Artificial Intelligence, Emotional Intelligence Committee（2021-）

• Associate Editor: IEEE TAFFC, IEEE TASLP

• Area Editor: Information Fusion

• Area Chair: ACM Multimedia 2025, ACL ARR 2025

• Conference Program Committee: NeurIPS, ICML, AAAI, etc.

• Journal Reviewer: IEEE TPAMI, IJCV, IEEE TAFFC, IEEE TNNLS, IEEE TASLP, etc.

Projects

1. National Natural Science Foundation of China, Youth Science Fund Project, 2023/1~2025/12, Host ￥300,000

2. Chinese Academy of Sciences, 2023/1~2023/12, Host ￥500,000 (Total ￥9,600,000)

3. National Natural Science Foundation of China, 2023/1~2026/12, Host ￥60,000 (Total ￥560,000)

4. Outstanding Youth Fund of State Key Laboratory of Multimodal Artificial Intelligence Systems, 2024/12~2025/12, Host ￥100,000

Patents

1. Dialogue emotion correction method based on graph neural network (US Patent No. 12100418), 2024/09/24 Link
   Jianhua Tao, Zheng Lian, Bin Liu, Xuefei Liu
   

2. Automatic lie detection method and apparatus for interactive scenarios, device and medium (US Patent No. 11238289), 2022/02/01 Link
   Jianhua Tao, Zheng Lian, Bin Liu, Licai Sun
   

3. Multimodal dimensional emotion recognition method (US Patent No. 11281945), 2022/03/22 Link
   Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian
   

4. Multi-modal lie detection method and apparatus, and device (US Patent No. 11244119), 2022/02/08 Link
   Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian
   

5. Expression recognition method under natural scene (US Patent No. 11216652), 2022/01/04 Link
   Jianhua Tao, Mingyuan Xiao, Bin Liu, Zheng Lian