Zheng Lian
≡
profile photo

Zheng Lian

I am an Associate Professor (IEEE/CCF Senior Member) at Institute of Automation, Chinese Academy of Sciences. My research interest primarily centers on human-centric AI and affective computing. In this field, I co-organize a series of challenges and workshops (MER@IJCAI, MRAC@ACM Multimedia, MEIJU@ICASSP), establish benchmark (MERBench) and toolbox (MERTools) , and propose new tasks to enhance accuracy and reliability (EMER, OV-MER, AffectGPT, AffectGPT-R1). I also serve as Associate Editor at IEEE TAFFC and IEEE TASLP, Area Chair at ACM Multimedia 2025 and ACL ARR 2025, Area Editor at Information Fusion. My short-term goal is to establish a new pathway toward more reliable and accurate emotion recognition techniques. My long-term goal is to enhance real-world human-AI interaction, ensuring robustness, trust, privacy, efficiency, and responsible deployment in areas such as healthcare, education, finance, etc.

Email  /  Google Scholar  /  Github  /  CV(Eng)  /  CV(Chi)

News

Job Openings: I am recruiting visiting students in Institute of Automation, Chinese Academy of Sciences.

  • [Oct, 2025] I will co-organize MER'25 Challenge and MRAC'25 Workshop@ACM Multimedia [link]

  • [Oct, 2025] One paper receives Best Paper Mention Award in PRCV, congratulations to Yazhou.

  • [Oct, 2025] IRNet is accepted by IEEE TPAMI

  • [Oct, 2025] I will serve as Area Chair at ACL ARR 2025

  • [Sep, 2025] I am selected to Stanford University's Top 2% Scientists list.

  • [Sep, 2025] Two papers are accepted by NeurIPS 2025, congratulations to Ziyang and Ziyu.

  • [Sep, 2025] Congratulations on successful promotion to CCF Senior Member.

  • [Sep, 2025] One paper is accepted by IEEE TAFFC, congratulations to Yazhou.

  • [Aug, 2025] Congratulations on successful promotion to IEEE Senior Member.

  • [Jul, 2025] Two papers are accepted by ACM Multimedia 2025, congratulations to Haolin and Hao Gu.

  • [Jun, 2025] I will serve as Associate Editor at IEEE TASLP

  • [May, 2025] One paper is accepted by ACL25(Finding), congratulations to Zhuofan.

  • [May, 2025] OV-MER and AffectGPT are accepted by ICML25, including one Oral (Top 1%) paper.

  • [Mar, 2025] I will serve as Area Chair at ACM Multimedia 2025.

  • [Mar, 2025] Two paper are accepted by ICASSP2025

  • [Feb, 2025] I will serve as Area Editor at Information Fusion.

  • [Jan, 2025] I will serve as Associate Editor at IEEE TAFFC.

  • [Sep, 2024] Emotion-LLaMA is accepted by NeurIPS24, congratulations to Zebang.

  • [Aug, 2024] We organize MEIJU'25 Challenge@ICASSP [link]

  • [Jul, 2024] SVFAP is accepted by TAFFC, congratulations to Licai.

  • [Apr, 2024] We organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM Multimedia [link]

  • [Mar, 2024] HiCMAE is accepted by Information Fusion, congratulations to Licai.

  • [Mar, 2024] GPT-4V with Emotion is accepted by Information Fusion

  • [Jan, 2024] We build MERBench, a unified evaluation benchmark for multimodal emotion recognition

  • [Nov, 2023] Two papers are accepted by NeurIPS23

  • [Oct, 2023] MAE-DFER is accepted by ACM MM, congratulations to Licai.

  • [Apr, 2023] We organize MER'23 Challenge and MRAC'23 Workshop@ACM Multimedia [link]

  • [May, 2023] EMT-DLFR is accepted by IEEE TAFFC, congratulations to Licai.

  • [Jan, 2023] GCNet is accepted by IEEE TPAMI

  • [Jul, 2022] PIRNet is accepted by IEEE TNNLS

  • [Jan, 2022] SMIN is accepted by IEEE TAFFC

  • [Jan, 2021] CTNet is accepted by IEEE TASLP

    • Research

    (* Equal contribution; † Corresponding author)

    1. IRNet: Iterative Refinement Network for Noisy Partial Label Learning
      Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, Lei Feng, Jianhua Tao
      Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2025 | paper

    2. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
      Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, Jianhua Tao
      ICML 2025 (Oral, Top 1%) | paper

    3. OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, Jianhua Tao
      ICML 2025 | paper

    4. SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding
      Yazhou Zhang, Chunwang Zou, Zheng Lian†, Prayag Tiwari†, Jing Qin
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2025 | paper

    5. Are MLMs Trapped in the Visual Room?
      Yazhou Zhang, Chunwang Zou, Qimeng Liu, Lu Rong, Ben Yao, Zheng Lian†, Qiuchi Li†, Peng Zhang, Jing Qin.
      PRCV (Best Paper Mention Award) 2025 | paper

    6. REFED: A Subject Real-time Dynamic Labeled EEG-fNIRS Synchronized Recorded Emotion Dataset
      Xiaojun Ning, Jing Wang, Zhiyang Feng, Tianzuo Xin, Shuo Zhang, Shaoqi Zhang, Zheng Lian, Yi Ding, Youfang Lin, Ziyu Jia
      NeurIPS 2025

    7. MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
      Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, etc.
      NeurIPS 2025 | paper

    8. Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities
      Rui Liu, Haolin Zuo, Zheng Lian, Hongyu Yuan, Qi Fan
      ACM Multimedia 2025 | paper

    9. ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection
      Hao Gu, Jiangyan Yi, Chenglong Wang, Jianhua Tao, Zheng Lian, Jiayi He, Yong Ren, Yujie Chen, Zhengqi Wen
      ACM Multimedia 2025 | paper

    10. Listen, Watch, and Learn to Feel: Retrieval-Augmented Emotion Reasoning for Compound Emotion Generation
      Zhuofan Wen, Zheng Lian, Shun Chen, Hailiang Yao, Longjiang Yang, Bin Liu, Jianhua Tao
      ACL (Findings) 2025 | paper

    11. MEIJU-The 1st Multimodal Emotion and Intent Joint Understanding Challenge
      Rui Liu, Xiaofen Xing, Zheng Lian, Haizhou Li, Björn W Schuller, Haolin Zuo
      ICASSP 2025 | paper

    12. Adversarial Training and Gradient Optimization for Partially Deepfake Audio Localization
      Siding Zeng, Jiangyan Yi, Jianhua Tao, Jiayi He, Zheng Lian, Shan Liang, Chuyuan Zhang, Yujie Chen, Xiaohui Zhang
      ICASSP 2025 | paper

    13. Explainable Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, etc.
      Arxiv 2024 | paper

    14. MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, etc.
      Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing 2024 | paper

    15. GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
      Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    16. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
      Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann
      NeurIPS 2024 | paper

    17. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    18. SVFAP: Self-supervised Video Facial Affect Perceiver
      Licai Sun, Zheng Lian†, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu†, Jianhua Tao†
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2024 | paper

    19. Pseudo Labels Regularization for Imbalanced Partial-Label Learning
      Mingyu Xu, Zheng Lian, Bin Liu, Zerui Chen, Jianhua Tao
      ICASSP 2024 | paper

    20. NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption
      Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan
      LREC-COLING 2024 | paper

    21. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
      Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, Haizhou Li
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2024 | paper

    22. MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition
      Haiyang Sun, Fulin Zhang, Yingying Gao, Zheng Lian, Shilei Zhang, Junlan Feng
      Interspeech 2024 | paper

    23. MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
      Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
      Arxiv 2023 | paper

    24. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
      Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao
      Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023 | paper

    25. MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
      Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, etc.
      ACM Multimedia (Organize Grand Challenge) 2023 | paper

    26. ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
      Mingyu Xu*, Zheng Lian*, Lei Feng, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    27. VRA: Variational Rectified Activation for Out-of-distribution Detection
      Mingyu Xu, Zheng Lian†, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    28. MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      ACM Multimedia 2023 | paper

    29. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2023 | paper

    30. EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition
      Haiyang Sun*, Zheng Lian*, Bin Liu, Ying Li, Licai Sun, Cong Cai, Jianhua Tao, Meng Wang, Yuan Cheng
      Interspeech 2023 | paper

    31. Integrating VideoMAE based model and Optical Flow for Micro-and Macro-expression Spotting
      Ke Xu, Kang Chen, Licai Sun, Zheng Lian, Bin Liu, Gong Chen, Haiyang Sun, Mingyu Xu, Jianhua Tao
      ACM Multimedia 2023 | paper

    32. PIRNet: Personality-enhanced Iterative Refinement Network for Emotion Recognition in Conversation
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022 | Early Access

    33. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2022 | paper

    34. AMOA: Global acoustic feature enhanced modal-order-aware network for multimodal sentiment analysis
      Ziming Li, Yan Zhou, Weibo Zhang, Yaxin Liu, Chuanpeng Yang, Zheng Lian, Songlin Hu
      COLING 2022 | paper

    35. DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      Neurocomputing 2021 | paper

    36. CTNet: Conversational Transformer Network for Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) 2021 | paper

    37. Towards Fine-Grained Prosody Control for Voice Conversion
      Zheng Lian, Rongxiu Zhong, Zhengqi Wen, Bin Liu, Jianhua Tao
      Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2021 | paper

    38. Investigation of multimodal features, classifiers and fusion methods for emotion recognition
      Zheng Lian, Ya Li, Jianhua Tao, Jian Huang
      National Conference Man-Machine Speech Communication, NCMMSC 2021 | paper

    39. Multimodal Cross-and Self-Attention Network for Speech Emotion Recognition
      Licai Sun, Bin Liu, Jianhua Tao, Zheng Lian
      ICASSP 2021 | paper

    40. Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    41. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    42. ARVC: An Auto-Regressive Voice Conversion System Without Parallel Training Data
      Zheng Lian, Zhengqi Wen, Xinyong Zhou, Songbai Pu, Shengkai Zhang, Jianhua Tao
      Interspeech 2020 | paper

    43. Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian
      Interspeech 2020 | paper

    44. Multimodal Transformer Fusion for Continuous Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, Mingyue Niu
      ICASSP 2020 | paper

    45. Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
      Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
      IEEE Transactions on Affective Computing (IEEE TAFFC) 2020 | paper

    46. Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    47. Conversational Emotion Analysis via Attention Mechanisms
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    48. Discriminative video representation with temporal order for micro-expression recognition
      Mingyue Niu, Jianhua Tao, Ya Li, Jian Huang, Zheng Lian
      ICASSP 2019 | paper

    49. End-to-End Continuous Emotion Recognition from Video Using 3D ConvLSTM Networks
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Jiangyan Yi
      ICASSP 2018 | paper

    50. Speech Emotion Recognition from Variable-Length Inputs with Triplet Loss Function
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian
      Interspeech 2018 | paper

    Awards

  • 2025, Select to Stanford University's Top 2% Scientists list.

  • 2025, Best Paper Mention Award in PRCV

  • 2024, Winner in the MuSe Challenge@ACM MM

  • 2023, Best Poster in the First CSIG Conference on Emotional Intelligence

  • 2023, Winner in the MEGC Challenge@ACM MM

  • 2023, Outstanding Reviewer of ICASSP

  • 2022, Winner in the MuSe Challenge@ACM MM

  • 2021, The First Prize for Technological Invention of Chinese Institute of Electronics

  • 2021, Best Paper for National Conference on Man-Machine Speech Communication (NCMMSC)

  • 2021, Winner in the MuSe Challenge@ACM MM

  • 2021, Climbing First-class Scholarship of Institute of Automation, Chinese Academy of Sciences

  • 2020, Winner in the MuSe Challenge@ACM MM

  • 2020, Interspeech Travel Grand

  • 2020, Winner in Voice Conversion Challenge@Interspeech

  • 2019, 2nd Place in the AVEC Challenge@ACM MM

  • 2018, 2nd Place in the AVEC Challenge@ACM MM

    • Academic Services

  • Co-organize MER'25 Challenge and MRAC'25 Workshop@ACM MM, 2025

  • Co-organize MEIJU'25 Challenge@ICASSP, 2025

  • Co-organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM MM, 2024

  • Co-organize MER'23 Challenge@ACM MM and MRAC'23 Workshop@ACM MM, 2023

  • Executive Committee Member, Speech Dialogue and Auditory Professional Committee, China Computer Federation (CCF)（2024-）

  • Committee Member, Chinese Information Processing Society of China, Emotional Computing Committee（2021-）

  • Committee Member, Chinese Society of Image and Graphics, Emotional Computing and Understanding Committee（2021-）

  • Committee Member, Chinese Association for Artificial Intelligence, Emotional Intelligence Committee（2021-）

  • Associate Editor: IEEE TAFFC, IEEE TASLP

  • Area Editor: Information Fusion

  • Area Chair: ACM Multimedia 2025, ACL ARR 2025

  • Conference Program Committee: NeurIPS, ICML, AAAI, etc.

  • Journal Reviewer: IEEE TPAMI, IJCV, IEEE TAFFC, IEEE TNNLS, IEEE TASLP, etc.

    • Projects

    1. National Natural Science Foundation of China, Youth Science Fund Project, 2023/1~2025/12, Host ￥300,000

    2. Chinese Academy of Sciences, 2023/1~2023/12, Host ￥500,000 (Total ￥9,600,000)

    3. National Natural Science Foundation of China, 2023/1~2026/12, Host ￥60,000 (Total ￥560,000)

    4. Outstanding Youth Fund of State Key Laboratory of Multimodal Artificial Intelligence Systems, 2024/12~2025/12, Host ￥100,000

    Patents

    1. Dialogue emotion correction method based on graph neural network (US Patent No. 12100418), 2024/09/24 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Xuefei Liu

    2. Automatic lie detection method and apparatus for interactive scenarios, device and medium (US Patent No. 11238289), 2022/02/01 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Licai Sun

    3. Multimodal dimensional emotion recognition method (US Patent No. 11281945), 2022/03/22 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    4. Multi-modal lie detection method and apparatus, and device (US Patent No. 11244119), 2022/02/08 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    5. Expression recognition method under natural scene (US Patent No. 11216652), 2022/01/04 Link
      Jianhua Tao, Mingyuan Xiao, Bin Liu, Zheng Lian