ZHU Ting,HU Jiancheng.News Text Similarity Calculation based on Keyword Clustering[J].Journal of Chengdu University of Information Technology,2024,39(02):163-169.[doi:10.16836/j.cnki.jcuit.2024.02.006]
基于关键词聚类的新闻文本相似度计算
- Title:
- News Text Similarity Calculation based on Keyword Clustering
- 文章编号:
- 2096-1618(2024)02-0163-07
- Keywords:
- news text similarity; word2vec; TF-IDF; keyword clustering
- 分类号:
- TP391.1
- 文献标志码:
- A
- 摘要:
- 针对新闻文本篇幅长、冗余信息多、文本相似度难以准确高效计算的问题,提出一种基于关键词聚类的新闻文本相似度计算方法。首先对文本数据进行预处理,挖掘出文本中的关键信息。使用以TF-IDF值为权重的加权采样方法抽取文本数据集中的关键词,基于聚类的方法光滑噪声数据。聚类形成簇后,在簇间词语相似度计算上,使用word2vec融合TF-IDF词语加权的计算方法,同时关注词语间的语义信息和词语频率。最后,基于各簇的相似度计算两篇文本的相似度。实验表明,所提新闻文本相似度计算方法在计算效果上优于传统计算方法。
- Abstract:
- Aiming at the problems of long news text, too much redundant information, and difficulty in accurately and efficiently calculating text similarity, a news text similarity calculation method based on keyword clustering is proposed. First, the text data is preprocessed to extract the key information in the text. The weighted sampling method weighted by TF-IDF values was used to extract keywords in the text dataset, and the clustering-based method was used to smooth noise data. After getting clusters from clustering, word2vec is used to calculate the word similarity between clusters, and the TF-IDF word weighting calculation method is used, and the semantic information and word frequency between words are considered. Finally, the similarity of the two texts is calculated based on the similarity of each cluster. Experiments show that the proposed news text similarity calculation method performs better than the traditional calculation method.
参考文献/References:
[1] Berant J,Chou A,Frostig R,et al.Semantic Parsing on Freebase from Question-Answer Pairs[C].Empirical Methods in Natural Language Processing.Association for Computational Linguistics,2013.
[2] WT Yih,X He,C Meek. Semantic Parsing for Single-Relation Question Answering[C].Meeting of the Association for Computational Linguistics.2014.
[3] Shen Y,He X,Gao J,et al.Learning semantic representations using convolutional neural networks for web search[C].Proceedings of the 23rd international conference on world wide web.2014.
[4] Heidari M,Zad S,Rafatirad S.Ensemble of Supervised and Unsupervised Learning Models to Predict a Profitable Business Decision[C].2021 IEEE International IOT,Electronics and Mechatronics Conference(IEMTRONICS).IEEE,2021.
[5] Vani K,Gupta D.Detection of idea plagiarism using syntax-Semantic concept extractions with genetic algorithm[J].Expert Systems with Applications,2017,73:11-26.
[6] El Mostafa H,Benabbou F.A deep learning based technique for plagiarism detection: a comparative study[J].IAES International Journal of Artificial Intelligence,2020,9(1):81.
[7] 王春柳,杨永辉,邓霏,等.文本相似度计算方法研究综述[J].情报科学,2019,37(3):158-168.
[8] Levenshtein V I.Binary codes capable of correcting deletions,insertions,and reversals[J].Soviet physics doklady,1966,10(8):707-710.
[9] Melamed I D.Automatic evaluation and uniform filter cascades for inducing n-best translation lexicons[J].arXiv preprint cmp-lg/9505044,1995.
[10] 张焕炯,王国胜,钟义信.基于汉明距离的文本相似度计算[J].计算机工程与应用,2001(19)21-22.
[11] Salton G.A vector space model for automatic indexing[J].Communications of the ACM,1975,18(11):613-620.
[12] Mikolov T,Sutskever I,Chen K,et al.Distributed representations of words and phrases and their compositionality[J].Advances in neural information processing systems,2013,26:311-319.
[13] Huang P S,He X,Gao J,et al.Learning deep structured semantic models for web search using clickthrough data[C].Proceedings of the 22nd ACM international conference on Information & Knowledge Management.2013:2333-2338.
[14] Li M,Bi X,Wang L,et al.Text Similarity Measurement Method and Application of Online Medical Community Based on Density Peak Clustering[J].Journal of Organization and End User Computing(JOEUC),2022,34(2):1-25.
[15] Xylogiannopoulos K F,Karampelas P.Identifying Social Networks of Programmers using Text Mining for Code Similarity Detection[C].2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining(ASONAM),The Hague,Netherlands,2020:643-650.
[16] Kim Y.Convolutional neural networks for sentence classification[J].arXiv preprint arXiv:1408.5882,2014.
[17] Peng W.Semantic Clustering and Convolutional Neural Network for Short Text Categorization[J].Neurocomputing,2016,174:806-814.
[18] Ahn D.The stages of event extraction[C].Proceedings of the Workshop on Annotating and Reasoning about Time and Events.2006:1-8.
[19] Goel N,Reddy R. SemEval-2022 Task 8:Multi-lingual News Article Similarity[J].arXiv preprint arXiv:2208.09715,2022.
[20] 廖运春,舒坚.基于加权Word2Vec和TextCNN的新闻文本分类[J].长江信息通信,2022,35(9):32-35.
[21] Aizawa A.An information-theoretic perspective of tf-idf measures[J].Information Processing & Management,2003,39(1):45-65.
[22] Khatua A,Cambria E.A tale of two epidemics:Contextual Word2Vec for classifying twitter streams during outbreaks[J]. Information Processing & Management,2019,56(1):247-257.
备注/Memo
收稿日期:2023-02-08