Zheng Lian
≡
profile photo

Zheng Lian

I am an Associate Professor at the Institute of Automation, Chinese Academy of Sciences. My research interest primarily centers on human-centric AI and affective computing. My short-term goal is to establish a new pathway toward more reliable and accurate emotion recognition techniques. Please refer to our proposed EMER, AffectGPT, OV-MER for more details. My long-term goal is to enhance real-world human-AI interaction, ensuring robustness, trust, privacy, and efficiency for responsible deployment.

Email  /  Google Scholar  /  Github  /  CV

News

Job Openings: I am recruiting visiting students in Institute of Automation, Chinese Academy of Sciences.

  • [Oct, 2024] We extend the EMER task to Open-vocabulary MER [link]

  • [Sep, 2024] Emotion-LLaMA is accepted by NeurIPS2024

  • [Aug, 2024] We organize MEIJU'25 Challenge@ICASSP [link]

  • [Jul, 2024] SVFAP is accepted by TAC

  • [Jul, 2024] We update AffectGPT and build EMER-Corase dataset to promote the development of EMER [link]

  • [Apr, 2024] We organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM Multimedia [link]

  • [Mar, 2024] HiCMAE is accepted by Information Fusion

  • [Mar, 2024] GPT-4V with Emotion is accepted by Information Fusion

  • [Jan, 2024] We build MERBench, a unified evaluation benchmark for multimodal emotion recognition

  • [Dec, 2023] We evaluate GPT-4V on 6 tasks and 21 datasets for multimodal emotion understanding [link]

  • [Nov, 2023] Two papers are accepted by NeurIPS

  • [Oct, 2023] MAE-DFER is accepted by ACM MM

  • [Aug, 2023] We propose EMER and AffectGPT for more reliable affective computing techniques

  • [Apr, 2023] We organize MER'23 Challenge and MRAC'23 Workshop@ACM Multimedia [link]

  • [May, 2023] EMT-DLFR is accepted by TAC

  • [Jan, 2023] GCNet is accepted by TPAMI

  • [Jul, 2022] PIRNet is accepted by TNNLS

  • [Jan, 2022] SMIN is accepted by TAC

  • [Jan, 2021] CTNet is accepted by TASLP

    • Research

    (* Equal contribution; † Corresponding author)

    1. Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark
      Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, Mingyu Xu, Kang Chen, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, Jianhua Tao
      Arxiv 2024 | paper

    2. AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, Jianhua Tao
      Arxiv 2024 | paper

    3. Explainable Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, etc.
      Arxiv 2024 | paper

    4. MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
      Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, etc.
      Arxiv 2024 | paper

    5. GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
      Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    6. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
      Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann
      NeurIPS 2024 | paper

    7. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    8. SVFAP: Self-supervised Video Facial Affect Perceiver
      Licai Sun, Zheng Lian, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2024 | paper

    9. Can Deception Detection Go Deeper? Dataset, Evaluation, and Benchmark for Deception Reasoning
      Kang Chen*, Zheng Lian*, Haiyang Sun, Bin Liu, Jianhua Tao
      Arxiv 2024 | paper

    10. Pseudo Labels Regularization for Imbalanced Partial-Label Learning
      Mingyu Xu, Zheng Lian, Bin Liu, Zerui Chen, Jianhua Tao
      ICASSP 2024 | paper

    11. NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption
      Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan
      LREC-COLING 2024

    12. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
      Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, Haizhou Li
      IEEE Transactions on Affective Computing (TAC) 2024 | paper

    13. MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
      Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
      Arxiv 2023 | paper

    14. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
      Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao
      Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023 | paper

    15. MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
      Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, etc.
      ACM Multimedia (Organize Grand Challenge) 2023 | paper

    16. ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
      Mingyu Xu*, Zheng Lian*, Lei Feng, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    17. VRA: Variational Rectified Activation for Out-of-distribution Detection
      Mingyu Xu, Zheng Lian†, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    18. MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      ACM Multimedia 2023 | paper

    19. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2023 | paper

    20. EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition
      Haiyang Sun*, Zheng Lian*, Bin Liu, Ying Li, Licai Sun, Cong Cai, Jianhua Tao, Meng Wang, Yuan Cheng
      Interspeech 2023 | paper

    21. PIRNet: Personality-enhanced Iterative Refinement Network for Emotion Recognition in Conversation
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022 | Early Access

    22. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2022 | paper

    23. DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      Neurocomputing 2021 | paper

    24. CTNet: Conversational Transformer Network for Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) 2021 | paper

    25. Towards Fine-Grained Prosody Control for Voice Conversion
      Zheng Lian, Rongxiu Zhong, Zhengqi Wen, Bin Liu, Jianhua Tao
      Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2021 | paper

    26. Multimodal Cross-and Self-Attention Network for Speech Emotion Recognition
      Licai Sun, Bin Liu, Jianhua Tao, Zheng Lian
      ICASSP 2021 | paper

    27. Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    28. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    29. ARVC: An Auto-Regressive Voice Conversion System Without Parallel Training Data
      Zheng Lian, Zhengqi Wen, Xinyong Zhou, Songbai Pu, Shengkai Zhang, Jianhua Tao
      Interspeech 2020 | paper

    30. Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian
      Interspeech 2020 | paper

    31. Multimodal Transformer Fusion for Continuous Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, Mingyue Niu
      ICASSP 2020 | paper

    32. Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
      Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
      IEEE Transactions on Affective Computing (TAC) 2020 | paper

    33. Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    34. Conversational Emotion Analysis via Attention Mechanisms
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    35. Discriminative video representation with temporal order for micro-expression recognition
      Mingyue Niu, Jianhua Tao, Ya Li, Jian Huang, Zheng Lian
      ICASSP 2019 | paper

    36. End-to-End Continuous Emotion Recognition from Video Using 3D ConvLSTM Networks
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Jiangyan Yi
      ICASSP 2018 | paper

    37. Speech Emotion Recognition from Variable-Length Inputs with Triplet Loss Function
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian
      Interspeech 2018 | paper

    Awards

  • 2024, Winner in the MuSe Challenge@ACM MM

  • 2023, Best Poster in the First CSIG Conference on Emotional Intelligence

  • 2023, Winner in the MEGC Challenge@ACM MM

  • 2023, Outstanding Reviewer of ICASSP

  • 2022, Winner in the MuSe Challenge@ACM MM

  • 2021, The First Prize for Technological Invention of Chinese Institute of Electronics

  • 2021, Best Paper for National Conference on Man-Machine Speech Communication (NCMMSC)

  • 2021, Winner in the MuSe Challenge@ACM MM

  • 2021, Climbing First-class Scholarship of Institute of Automation, Chinese Academy of Sciences

  • 2020, Winner in the MuSe Challenge@ACM MM

  • 2020, Interspeech Travel Grand

  • 2020, Winner in Voice Conversion Challenge@Interspeech

  • 2019, 2nd Place in the AVEC Challenge@ACM MM

  • 2018, 2nd Place in the AVEC Challenge@ACM MM

    • Academic Services

  • Co-organize MEIJU'25 Challenge@ICASSP, 2025

  • Co-organize MER'24 Challenge@IJCAI and MRAC'24 Workshop@ACM MM, 2024

  • Co-organize MER'23 Challenge@ACM MM and MRAC'23 Workshop@ACM MM, 2023

  • Committee Member, Chinese Society of Image and Graphics, Emotional Computing and Understanding Committee（2021-）

  • Committee Member, Chinese Information Processing Society of China, Emotional Computing Committee（2021-）

  • Committee Member, Chinese Association for Artificial Intelligence, Emotional Intelligence Committee（2021-）

  • Executive Committee Member, Speech Dialogue and Auditory Professional Committee, China Computer Federation (CCF)（2024-）

  • Session Chair: ISCSLP

  • Conference Program Committee: NeurIPS, ICLR, AAAI, ACL ARR, ACM Multimedia, ICASSP, Interspeech, etc

  • Journal Reviewer: IJCV, TAC, TNNLS, TASLP, TOMM, TCSVT, TALLIP, IEEE Signal Processing Magazine, Information Fusion, Pattern Recognition, etc

    • Projects

    1. National Natural Science Foundation of China, Youth Science Fund Project, 2023/1~2025/12, Host ￥300,000

    2. Chinese Academy of Sciences, 2023/1~2023/12, Host ￥500,000 (Total ￥9,600,000)

    3. National Natural Science Foundation of China, 2023/1~2026/12, Host ￥60,000 (Total ￥560,000)

    4. Outstanding Youth Fund of State Key Laboratory of Multimodal Artificial Intelligence Systems, 2024/12~2025/12, Host ￥100,000

    Patents

    1. Dialogue emotion correction method based on graph neural network (US Patent No. 12100418), 2024/09/24 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Xuefei Liu

    2. Automatic lie detection method and apparatus for interactive scenarios, device and medium (US Patent No. 11238289), 2022/02/01 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Licai Sun

    3. Multimodal dimensional emotion recognition method (US Patent No. 11281945), 2022/03/22 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    4. Multi-modal lie detection method and apparatus, and device (US Patent No. 11244119), 2022/02/08 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    5. Expression recognition method under natural scene (US Patent No. 11216652), 2022/01/04 Link
      Jianhua Tao, Mingyuan Xiao, Bin Liu, Zheng Lian