Zheng Lian
≡
profile photo

Zheng Lian

I am an Assistant Professor at Institute of Automation, Chinese Academy of Sciences (CASIA). I received the Ph.D degree from CASIA in 2021. My research primarily centers on affective computing, multimodal learning and noisy label learning. Recently, we aim to establish a new pathway toward more reliable and accurate affective computing techniques. See EMER and AffectGPT for more details.

Email  /  Google Scholar

News

  • We will continue to organize MER24 Challenge@IJCAI and MRAC24 Workshop@ACM Multimedia [link]

  • We extend the EMER task to open-vocabulary emotion recognition [link]

  • GPT-4V with Emotion is accepted by Information Fusion 2024

  • HiCMAE is accepted by Information Fusion 2024

  • Code of MERBench is merged into MERTools

  • We build MERBench, a unified evaluation benchmark for multimodal emotion recognition

  • We evaluate GPT-4V on 6 tasks and 21 datasets for multimodal emotion understanding [link]

  • We propose EMER and AffectGPT, aiming to establish more reliable affective computing techniques

  • Two papers are accepted by NeurIPS 2023

  • We organize MER23 Challenge and MRAC23 Workshop@ACM Multimedia [link]

  • GCNet is accepted by TPAMI 2023

  • MAE-DFER is accepted by ACM MM 2023

  • One paper is accepted by TAC 2023 [link]

  • PIRNet is accepted by TNNLS 2022

  • SMIN is accepted by TAC 2022

  • CTNet is accepted by TASLP 2021

    • Research

    (* Equal contribution; † Corresponding author)

    1. Explainable Multimodal Emotion Reasoning
      Zheng Lian, Licai Sun, Haiyang Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Lan Chen, Bin Liu, Jianhua Tao
      Arxiv 2024 | paper

    2. GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition
      Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    3. HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      Information Fusion 2024 | paper

    4. Can Deception Detection Go Deeper? Dataset, Evaluation, and Benchmark for Deception Reasoning
      Kang Chen*, Zheng Lian*, Haiyang Sun, Bin Liu, Jianhua Tao
      Arxiv 2024 | paper

    5. Pseudo Labels Regularization for Imbalanced Partial-Label Learning
      Mingyu Xu, Zheng Lian, Bin Liu, Zerui Chen, Jianhua Tao
      ICASSP 2024 | paper

    6. NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption
      Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan
      LREC-COLING 2024

    7. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
      Rui Liu, Haolin Zuo, Zheng Lian, Björn W. Schuller, Haizhou Li
      IEEE Transactions on Affective Computing 2024 | paper

    8. MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
      Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
      Arxiv 2023 | paper

    9. GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
      Zheng Lian, Lan Chen, Licai Sun, Bin Liu, Jianhua Tao
      Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023 | paper

    10. MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
      Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, etc.
      ACM Multimedia (Organize Grand Challenge) 2023 | paper

    11. ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
      Mingyu Xu*, Zheng Lian*, Lei Feng, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    12. VRA: Variational Rectified Activation for Out-of-distribution Detection
      Mingyu Xu, Zheng Lian†, Bin Liu, Jianhua Tao
      NeurIPS 2023 | paper

    13. MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      ACM Multimedia 2023 | paper

    14. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis
      Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2023 | paper

    15. EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition
      Haiyang Sun*, Zheng Lian*, Bin Liu, Ying Li, Licai Sun, Cong Cai, Jianhua Tao, Meng Wang, Yuan Cheng
      Interspeech 2023 | paper

    16. PIRNet: Personality-enhanced Iterative Refinement Network for Emotion Recognition in Conversation
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022 | Early Access

    17. SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE Transactions on Affective Computing (TAC) 2022 | paper

    18. DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      Neurocomputing 2021 | paper

    19. CTNet: Conversational Transformer Network for Emotion Recognition
      Zheng Lian, Bin Liu, Jianhua Tao
      IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) 2021 | paper

    20. Towards Fine-Grained Prosody Control for Voice Conversion
      Zheng Lian, Rongxiu Zhong, Zhengqi Wen, Bin Liu, Jianhua Tao
      Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2021 | paper

    21. Multimodal Cross-and Self-Attention Network for Speech Emotion Recognition
      Licai Sun, Bin Liu, Jianhua Tao, Zheng Lian
      ICASSP 2021 | paper

    22. Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    23. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, Rongjun Li
      Interspeech 2020 | paper

    24. ARVC: An Auto-Regressive Voice Conversion System Without Parallel Training Data
      Zheng Lian, Zhengqi Wen, Xinyong Zhou, Songbai Pu, Shengkai Zhang, Jianhua Tao
      Interspeech 2020 | paper

    25. Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian
      Interspeech 2020 | paper

    26. Multimodal Transformer Fusion for Continuous Emotion Recognition
      Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, Mingyue Niu
      ICASSP 2020 | paper

    27. Multimodal Spatiotemporal Representation for Automatic Depression Level Detection
      Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
      IEEE Transactions on Affective Computing (TAC) 2020 | paper

    28. Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    29. Conversational Emotion Analysis via Attention Mechanisms
      Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang
      Interspeech 2019 | paper

    30. Discriminative video representation with temporal order for micro-expression recognition
      Mingyue Niu, Jianhua Tao, Ya Li, Jian Huang, Zheng Lian
      ICASSP 2019 | paper

    31. End-to-End Continuous Emotion Recognition from Video Using 3D ConvLSTM Networks
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Jiangyan Yi
      ICASSP 2018 | paper

    32. Speech Emotion Recognition from Variable-Length Inputs with Triplet Loss Function
      Jian Huang, Ya Li, Jianhua Tao, Zheng Lian
      Interspeech 2018 | paper

    Awards

  • 2023, Best Poster in the First CSIG Conference on Emotional Intelligence

  • 2023, 1st Place in the MEGC Challenge@ACM MM

  • 2023, Outstanding Reviewer of ICASSP

  • 2022, 1st Place in the MuSe Challenge@ACM MM

  • 2021, The First Prize for Technological Invention of Chinese Institute of Electronics

  • 2021, Best Paper for National Conference on Man-Machine Speech Communication (NCMMSC)

  • 2021, 1st Place in the MuSe Challenge@ACM MM

  • 2021, Climbing First-class Scholarship of Institute of Automation, Chinese Academy of Sciences

  • 2020, 1st Place in the MuSe Challenge@ACM MM

  • 2020, Interspeech Travel Grand

  • 2020, 2nd Place in the Voice Conversion Challenge@Interspeech

  • 2019, 2nd Place in the AVEC Challenge@ACM MM

  • 2018, 2nd Place in the AVEC Challenge@ACM MM

    • Academic Services

  • Organize MER@ACM MM Grand Challenge and MRAC@ACM MM Workshop, 2023

  • Committee Member, Chinese Society of Image and Graphics, Emotional Computing and Understanding Committee（2021-）

  • Committee Member, Chinese Information Processing Society of China, Emotional Computing Committee（2021-）

  • Committee Member, Chinese Association for Artificial Intelligence, Emotional Intelligence Committee（2021-）

  • Reviewers for TAC, TNNLS, TASLP, TOMM, TCSVT, TALLIP, IEEE Signal Processing Magazine, Information Fusion, Pattern Recognition, ACM Multimedia, ICASSP, Interspeech, ISCSLP, etc

    • Projects

    1. National Natural Science Foundation of China, Youth Science Fund Project, Research on Emotion Recognition Method for Interactive Scenes (Grant No. 62201572), 2023/1/1~2025/12/31, Host

    Patents

    1. Dialogue emotion correction method based on graph neural network (US Patent No. 17472511), 2022/08/25 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Xuefei Liu

    2. Automatic lie detection method and apparatus for interactive scenarios, device and medium (US Patent No. 11238289), 2022/02/01 Link
      Jianhua Tao, Zheng Lian, Bin Liu, Licai Sun

    3. Multimodal dimensional emotion recognition method (US Patent No. 11281945), 2022/03/22 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    4. Multi-modal lie detection method and apparatus, and device (US Patent No. 11244119), 2022/02/08 Link
      Jianhua Tao, Licai Sun, Bin Liu, Zheng Lian

    5. Expression recognition method under natural scene (US Patent No. 11216652), 2022/01/04 Link
      Jianhua Tao, Mingyuan Xiao, Bin Liu, Zheng Lian