TAO Quanhui,AN Junxiu,CHEN Hongsong.Multi-modal Sentiment Analysis based on Cross-modal Fusion ERNIE[J].Journal of Chengdu University of Information Technology,2022,37(05):501-507.[doi:10.16836/j.cnki.jcuit.2022.05.003]
基于跨模态融合ERNIE的多模态情感分析研究
- Title:
- Multi-modal Sentiment Analysis based on Cross-modal Fusion ERNIE
- 文章编号:
- 2096-1618(2022)05-0501-07
- 分类号:
- TP391.1
- 文献标志码:
- A
- 摘要:
- 针对情感分析主要集中于单模态文本数据，忽略多模态数据融合问题，通过结合屏蔽多模态注意力方式，提出跨模态融合ERNIE的情感分析模型(CM-ERNIE)。首先，使用CNN和BiGRU提取音频数据特征以及词向量提取文本序列特征; 其次，通过屏蔽多模态注意力作为CM-ERNIE的核心单元动态调整文本和音频数据权重，最后，文本和音频模态的交互作用微调预训练ERNIE模型。该模型在多模态电影评论观点数据集CMU-MOSEI和CMU-MOSI上评估。实验表明，模型在多模态数据集CMU-MOSEI和CMU-MOSI上评估该模型比单模态情感分析模型准确度高，并且多模态情感分析的研究蕴含巨大的价值，可为多模态场景下的情感分析、舆情分析和意图识别等实际应用问题提供决策支持。
- Abstract:
- Aiming at the fact that sentiment analysis mainly focuses on single-modal text data and ignores the problem of multi-modal data fusion, a cross-modal fusion ERNIE sentiment analysis model(CM-ERNIE)is proposed by combining the masked multi-modal attention method. First, use CNN and BiGRU to extract audio data features and word vectors to extract text sequence features; second, dynamically adjust text and audio data weights by masking multimodal attention as the core unit of CM-ERNIE, and finally, text and audio modalities The interaction of fine-tuning the pretrained ERNIE model.The model is evaluated on the multimodal movie review opinion datasets CMU-MOSEI and CMU-MOSI.Comprehensive experiments show that the model is more accurate than the single-modal sentiment analysis model on the multi-modal datasets CMU-MOSEI and CMU-MOSI, and the research of multi-modal sentiment analysis contains great value, which can be used for multi-modal sentiment analysis. It provides decision support for practical application problems such as sentiment analysis, public opinion analysis, and intent recognition in modal scenarios.
参考文献/References:
[1] Li X,Fu X,Xu G,et al.Enhancing BERT representation with context-aware embedding for aspect-based sentiment analysis[J].IEEE Access,2020,8:46868-46876.
[2] Sun C,Qiu X,Xu Y,et al.How to fine-tune bert for text classification?[C].China national conference on Chinese computational linguistics.Springer,Cham,2019:194-206.
[3] BORTH D,JI R,CHEN T,et al.Large-scale visual sentiment ontology and detectors using adjective noun pairs[C].Proceedings of the 2013 21st ACM International Conference on Multimedia.New York:ACM,2013:223-232.
[4] Guillaumin M,Verbeek J,Schmid C.Multimodal semi-supervised learning for image classification[C].2010 IEEE Computer society conference on computer vision and pattern recognition.IEEE,2010:902-909.
[5] 章荪,尹春勇.基于多任务学习的时序多模态情感分析模型[J].计算机应用,2021,41(6):1631-1639.
[6] Poria Soujanya,Erik Cambria,Devamanyu Hazarika,et al.Context-dependent sentiment analysis in user-generated videos[J].In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics 2017,1:873-883.
[7] Tsai Y H H,Bai S,Liang P P,et al.Multimodal Transformer for Unaligned Multimodal Language Sequences[C].Proceedings of the conference. Association for Computational Linguistics.Meeting,2019:6558-6569.
[8] Ronan C,Jason W.A unified architecture for natural language processing:deep neural networks with multitask learning[C].In Proceedings of ICML,2008:160-167.
[9] 王梓懿,安俊秀,王鹏.基于多尺度量子谐振子算法的相空间概率聚类算法[J].计算机应用,2017,37(8):2218-2222.
[10] 杨铠成.基于深度学习的跨模态音频情感分类方法研究[D].石家庄:河北科技大学,2020.
[11] Turian J,Ratinov L,Bengio Y.Word representations: a simple and general method for semi-supervised learning[C].Proceedings of the 48th annual meeting of the association for computational linguistics.2010:384-394.
[12] Peters M E,Neumann M,Iyyer M,et al.Deep contextualized word representations.CoRR abs/1802.05365(2018)[J].arXiv preprint arXiv:2018.
[13] Lai Y,Zhang L,Han D,et al.Fine-grained emotion classification of Chinese microblogs based on graph convolution networks[J].World Wide Web,2020,23(5):2771-2787.
[14] Pal A,Karn B.Anubhuti-An annotated dataset for emotional analysis of Bengali short stories[J].arXiv e-prints,2020.
[15] Ruposh H A,Hoque M M.A computational approach of recognizing emotion from Bengali texts[C].2019 5th International Conference on Advances in Electrical Engineering(ICAEE).IEEE,2019:570-574.
[16] Manek A S,Shenoy P D,Mohan M C,et al.Aspect term extraction for sentiment analysis in large movie reviews using Gini Index feature selection method and SVM classifier[J].World Wide Web-internet & Web Information Systems,2017,20(2):135-154.
[17] Xu J,Xu Y,Xu Y,et al.A Chinese text sentiment classification algorithm framework based on a hybrid of semantic understanding and machine learning[J].Computer Science,2015,42(6):61-66.
[18] Kalchbrenner N,Grefenstette E,Blunsom P.A Convolutional Neural Network for Modelling Sentences[J].Eprint Arxiv,2014,1:35-47.
[19] 李铁飞,生龙,吴迪.BERT-TECNN模型的文本分类方法研究[J].计算机工程与应用,2021,57(18):186-193.
[20] Mihaylov T,Frank A.Knowledgeable reader:Enhancing cloze-style reading comprehension with external commonsense knowledge[J].arXiv preprint arXiv,2018.
[21] Zaremoodi P,Buntine W,Haffari G.Adaptive knowledge sharing in multi-task learning:Improving low-resource neural machine translation[C].Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Volume 2:Short Papers).2018:656-661.
[22] Chen Q,Zhu X,Ling Z H,et al.Neural natural language inference models enhanced with external knowledge[J].arXiv preprint arXiv,2017.
[23] Han X,Liu Z,Sun M.Neural knowledge acquisition via mutual attention between knowledge graph and text[C].Thirty-second AAAI conference on artificial intelligence.2018.
[24] Madotto A,Wu C S,Fung P.Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems[J].arXiv preprint arXiv,2018.
[25] Yu F,Tang J,Yin W,et al.Ernie-vil: Knowledge enhanced vision-language representations through scene graphs[C].Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):3208-3216.
相似文献/References:
[1]陈诗汉,马洪江,王 婷,等.基于多模态融合的视频情感分析技术[J].成都信息工程大学学报,2022,37(06):656.[doi:10.16836/j.cnki.jcuit.2022.06.007]
CHEN Shihan,MA Hongjiang,WANG Ting,et al.Video Sentiment Analysis Technology based on Multimodal Fusion[J].Journal of Chengdu University of Information Technology,2022,37(05):656.[doi:10.16836/j.cnki.jcuit.2022.06.007]
[2]刘子琦,胡建成,牟谷芳.基于深度学习的中文临床实验筛选标准的分类[J].成都信息工程大学学报,2024,39(02):170.[doi:10.16836/j.cnki.jcuit.2024.02.007]
LIU Ziqi,HU Jiancheng,MOU Gufang.Classification of Screening Criteria for Chinese Clinical Trials based on Deep Learning[J].Journal of Chengdu University of Information Technology,2024,39(05):170.[doi:10.16836/j.cnki.jcuit.2024.02.007]
备注/Memo
收稿日期:2021-12-29
基金项目:国家自然科学基金面上项目(71673032); 四川省社会科学研究规划项目(22XW043)