Zheng Lian
≡
profile photo

Zheng Lian

I am an Associate Professor at State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences. My research interest primarily centers on human-centric AI and affective computing. In this field, I co-organize a series of challenges and workshops (MER@IJCAI, MRAC@ACM Multimedia, MEIJU@ICASSP), establish benchmark (MERBench) and toolbox (MERTools) , and propose new tasks to enhance accuracy and reliability (EMER, OV-MER). I also serve as Associate Editor at IEEE Transactions on Affective Computing (TAC), Area Chair at ACM Multimedia 2025, Area Editor at Information Fusion, and Session Chair at ISCSLP 2024. My short-term goal is to establish a new pathway toward more reliable and accurate emotion recognition techniques. My long-term goal is to enhance real-world human-AI interaction, ensuring robustness, trust, privacy, efficiency, and responsible deployment in areas such as healthcare, education, finance, etc.

Email  /  Google Scholar  /  Github  /  CV

News

Job Openings: I am recruiting visiting students in Institute of Automation, Chinese Academy of Sciences.

  • [May, 2025] OV-MER and AffectGPT are accepted by ICML25

  • [Apr, 2025] I will co-organize MER'25 Challenge and MRAC'25 Workshop@ACM Multimedia [link]

  • [Mar, 2025] I will serve as Area Chair at ACM Multimedia 2025.

  • [Mar, 2025] Two paper are accepted by ICASSP2025

  • [Feb, 2025] I will serve as Area Editor at Information Fusion.

  • [Jan, 2025] I will serve as Associate Editor at IEEE Transactions on Affective Computing (TAC).

  • [Nov, 2024] I will serve as Session Chair at ISCSLP 2024.

  • [Oct, 2024] We extend the EMER task to Open-vocabulary MER [link]

  • [Sep, 2024] Emotion-LLaMA is accepted by NeurIPS24

  • [Aug, 2024] We organize MEIJU'25 Challenge@ICASSP [link]

  • [Jul, 2024] SVFAP is accepted by TAC

  • [Jul, 2024] We update AffectGPT and build EMER-Corase dataset to promote the development of EMER [link]

  • [Apr, 2024] We organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM Multimedia [link]

  • [Mar, 2024] HiCMAE is accepted by Information Fusion

  • [Mar, 2024] GPT-4V with Emotion is accepted by Information Fusion

  • [Jan, 2024] We build MERBench, a unified evaluation benchmark for multimodal emotion recognition

  • [Nov, 2023] Two papers are accepted by NeurIPS23

  • [Oct, 2023] MAE-DFER is accepted by ACM MM

  • [Aug, 2023] We propose EMER and AffectGPT for more reliable affective computing techniques

  • [Apr, 2023] We organize MER'23 Challenge and MRAC'23 Workshop@ACM Multimedia [link]

  • [May, 2023] EMT-DLFR is accepted by TAC

  • [Jan, 2023] GCNet is accepted by TPAMI

  • [Jul, 2022] PIRNet is accepted by TNNLS

  • [Jan, 2022] SMIN is accepted by TAC

  • [Jan, 2021] CTNet is accepted by TASLP

    • Research

    (* Equal contribution; † Corresponding author)

    1. OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, Jianhua Tao
      ICML 2025 | paper

    2. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
      Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, Jianhua Tao
      ICML 2025 | paper

    3. QuMATL: Query-based Multi-annotator Tendency Learning
      Liyun Zhang, Zheng Lian, Hong Liu, Takanori Takebe, Yuta Nakashima
      Arxiv 2025 | paper

    4. EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models
      He Hu, Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, Laizhong Cui
      Arxiv 2025 | paper

    5. P2Mark: Plug-and-play Parameter-intrinsic Watermarking for Neural Speech Generation
      Yong Ren, Jiangyan Yi, Tao Wang, Jianhua Tao, Zheng Lian, Zhengqi Wen, Chenxing Li, Ruibo Fu, Ye Bai, Xiaohui Zhang
      Arxiv 2025 | paper

    6. MEIJU-The 1st Multimodal Emotion and Intent Joint Understanding Challenge
      Rui Liu, Xiaofen Xing, Zheng Lian, Haizhou Li, Björn W Schuller, Haolin Zuo
      ICASSP 2025 | paper

    7. Adversarial Training and Gradient Optimization for Partially Deepfake Audio Localization
      Siding Zeng, Jiangyan Yi, Jianhua Tao, Jiayi He, Zheng Lian, Shan Liang, Chuyuan Zhang, Yujie Chen, Xiaohui Zhang
      ICASSP 2025 | paper

    8. Explainable Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, etc.
      Arxiv 2024 | paper

    9. MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, etc.
      Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing 2024 | paper

    10. GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
      Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    11. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
      Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann
      NeurIPS 2024 | paper

    12. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    13. SVFAP: Self-supervised Video Facial Affect Perceiver
      Licai Sun, Zheng Lian, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2024 | paper

    14. Pseudo Labels Regularization for Imbalanced Partial-Label Learning
      Mingyu Xu, Zheng Lian, Bin Liu, Zerui Chen, Jianhua Tao
      ICASSP 2024 | paper

    15. NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption
      Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan
      LREC-COLING 2024

    16. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
      Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, Haizhou Li
      IEEE Transactions on Affective Computing (TAC) 2024 | paper

    17. MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition
      Haiyang Sun, Fulin Zhang, Yingying Gao, Zheng Lian, Shilei Zhang, Junlan Feng
      Interspeech 2024 | paper

    18. MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
      Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
      Arxiv 2023 | paper

    19. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
      Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao
      Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023 | paper

    20. MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
      Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, etc.
      ACM Multimedia (Organize Grand Challenge) 2023 | paper

    21. ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
      Mingyu Xu*, Zheng Lian*, Lei Feng, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    22. VRA: Variational Rectified Activation for Out-of-distribution Detection
      Mingyu Xu, Zheng Lian†, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    23. MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      ACM Multimedia 2023 | paper

    24. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2023 | paper

    25. EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition
      Haiyang Sun*, Zheng Lian*, Bin Liu, Ying Li, Licai Sun, Cong Cai, Jianhua Tao, Meng Wang, Yuan Cheng
      Interspeech 2023 | paper

    26. PIRNet: Personality-enhanced Iterative Refinement Network for Emotion Recognition in Conversation
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022 | Early Access

    27. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2022 | paper

    28. DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      Neurocomputing 2021 | paper

    29. CTNet: Conversational Transformer Network for Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) 2021 | paper

    30. Towards Fine-Grained Prosody Control for Voice Conversion
      Zheng Lian, Rongxiu Zhong, Zhengqi Wen, Bin Liu, Jianhua Tao
      Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2021 | paper

    31. Multimodal Cross-and Self-Attention Network for Speech Emotion Recognition
      Licai Sun, Bin Liu, Jianhua Tao, Zheng Lian
      ICASSP 2021 | paper

    32. Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    33. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    34. ARVC: An Auto-Regressive Voice Conversion System Without Parallel Training Data
      Zheng Lian, Zhengqi Wen, Xinyong Zhou, Songbai Pu, Shengkai Zhang, Jianhua Tao
      Interspeech 2020 | paper

    35. Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian
      Interspeech 2020 | paper

    36. Multimodal Transformer Fusion for Continuous Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, Mingyue Niu
      ICASSP 2020 | paper

    37. Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
      Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
      IEEE Transactions on Affective Computing (TAC) 2020 | paper

    38. Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    39. Conversational Emotion Analysis via Attention Mechanisms
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    40. Discriminative video representation with temporal order for micro-expression recognition
      Mingyue Niu, Jianhua Tao, Ya Li, Jian Huang, Zheng Lian
      ICASSP 2019 | paper

    41. End-to-End Continuous Emotion Recognition from Video Using 3D ConvLSTM Networks
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Jiangyan Yi
      ICASSP 2018 | paper

    42. Speech Emotion Recognition from Variable-Length Inputs with Triplet Loss Function
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian
      Interspeech 2018 | paper

    Awards

  • 2024, Winner in the MuSe Challenge@ACM MM

  • 2023, Best Poster in the First CSIG Conference on Emotional Intelligence

  • 2023, Winner in the MEGC Challenge@ACM MM

  • 2023, Outstanding Reviewer of ICASSP

  • 2022, Winner in the MuSe Challenge@ACM MM

  • 2021, The First Prize for Technological Invention of Chinese Institute of Electronics

  • 2021, Best Paper for National Conference on Man-Machine Speech Communication (NCMMSC)

  • 2021, Winner in the MuSe Challenge@ACM MM

  • 2021, Climbing First-class Scholarship of Institute of Automation, Chinese Academy of Sciences

  • 2020, Winner in the MuSe Challenge@ACM MM

  • 2020, Interspeech Travel Grand

  • 2020, Winner in Voice Conversion Challenge@Interspeech

  • 2019, 2nd Place in the AVEC Challenge@ACM MM

  • 2018, 2nd Place in the AVEC Challenge@ACM MM

    • Academic Services

  • Co-organize MER'25 Challenge and MRAC'25 Workshop@ACM MM, 2025

  • Co-organize MEIJU'25 Challenge@ICASSP, 2025

  • Co-organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM MM, 2024

  • Co-organize MER'23 Challenge@ACM MM and MRAC'23 Workshop@ACM MM, 2023

  • Executive Committee Member, Chinese Information Processing Society of China, Emotional Computing Committee（2024-）

  • Executive Committee Member, Speech Dialogue and Auditory Professional Committee, China Computer Federation (CCF)（2024-）

  • Committee Member, Chinese Society of Image and Graphics, Emotional Computing and Understanding Committee（2021-）

  • Committee Member, Chinese Association for Artificial Intelligence, Emotional Intelligence Committee（2021-）

  • Associate Editor: IEEE Transactions on Affective Computing

  • Area Editor: Information Fusion

  • Area Chair: ACM Multimedia 2025

  • Session Chair: ISCSLP 2024

  • Conference Program Committee: NeurIPS, ICLR, ICML, AAAI, ACL ARR, ACM Multimedia, ICASSP, Interspeech, ISCSLP, etc.

  • Journal Reviewer: TPAMI, IJCV, TAC, TNNLS, TASLP, TOMM, TCSVT, TALLIP, IEEE Signal Processing Magazine, Pattern Recognition, etc

    • Projects

    1. National Natural Science Foundation of China, Youth Science Fund Project, 2023/1~2025/12, Host ￥300,000

    2. Chinese Academy of Sciences, 2023/1~2023/12, Host ￥500,000 (Total ￥9,600,000)

    3. National Natural Science Foundation of China, 2023/1~2026/12, Host ￥60,000 (Total ￥560,000)

    4. Outstanding Youth Fund of State Key Laboratory of Multimodal Artificial Intelligence Systems, 2024/12~2025/12, Host ￥100,000

    Patents

    1. Dialogue emotion correction method based on graph neural network (US Patent No. 12100418), 2024/09/24 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Xuefei Liu

    2. Automatic lie detection method and apparatus for interactive scenarios, device and medium (US Patent No. 11238289), 2022/02/01 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Licai Sun

    3. Multimodal dimensional emotion recognition method (US Patent No. 11281945), 2022/03/22 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    4. Multi-modal lie detection method and apparatus, and device (US Patent No. 11244119), 2022/02/08 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    5. Expression recognition method under natural scene (US Patent No. 11216652), 2022/01/04 Link
      Jianhua Tao, Mingyuan Xiao, Bin Liu, Zheng Lian