文本語料庫的精煉研究
發(fā)布時間：2018-10-11 09:07
【摘要】：文本語料庫是文本數(shù)據(jù)挖掘的基礎。很多文本語料庫來源于生產(chǎn)生活的實際工作中,通常由行業(yè)專家為其定義類別。本文的數(shù)據(jù)集來源于市長公開電話辦公室,隨著不同時期行業(yè)類別的變更,語料庫難免會有很多錯誤數(shù)據(jù),由于語料庫較大,通常不能由專家逐條校對,所以必須使用數(shù)據(jù)挖掘的方法找出錯分類數(shù)據(jù),針對這些錯分類數(shù)據(jù)再由行業(yè)專家逐一校對。本文研究的內(nèi)容就是篩選語料庫中的錯分類數(shù)據(jù),以便于行業(yè)專家矯正數(shù)據(jù)類別。本文討論了文本數(shù)據(jù)的判別分類問題。文中首先對文本分類技術和流程給出了論述,而后討論了樸素貝葉斯方法的性質(zhì),最后討論了文本語料庫的精煉研究,討論了類別判別錯誤數(shù)據(jù)的選取方法,并給出了實證分析。在大數(shù)據(jù)條件下,通過行業(yè)專家對文本數(shù)據(jù)人工標記類別的方法,由于會消耗大量的人力、物力、財力,采用行業(yè)專家人工校正的方法是不現(xiàn)實的。按照一定的規(guī)則,批量的對文本數(shù)據(jù)標記類別是另一種有效的方法,該方法能夠有效的避免直接專家標類別的缺點,但文本數(shù)據(jù)類別標記的精確度比較低。結合以上兩種方法,提出了第三種方法,首先批量對文本數(shù)據(jù)標記類別,將類別標記錯誤的文本數(shù)據(jù)交給行業(yè)專家進行人工標記,然后用行業(yè)專家標記的文本數(shù)據(jù)對文本語料庫中的文本數(shù)據(jù)進行校正。文本語料庫的精煉研究是基于第三種方法的。利用不同的方法提取文本語料庫中類別判別錯誤的文本數(shù)據(jù),在所有的方法中類別判別均為錯誤的文本數(shù)據(jù)是最可能為類別標記錯誤的文本數(shù)據(jù)。文本語料庫精煉的目的是提取文本語料庫中最可能為類別標記錯誤的文本數(shù)據(jù)。將這部分文本數(shù)據(jù)交給行業(yè)專家人工標記類別,最后基于行業(yè)專家人工標記的文本數(shù)據(jù)將文本語料庫的文本數(shù)據(jù)的類別進行校正。本文首先簡述文本數(shù)據(jù)分類的一般流程;然后介紹樸素貝葉斯分類算法;最后對文本語料庫的預處理,特征詞提取,文本語料庫精煉的目的和方法,提取類別判別錯誤的文本數(shù)據(jù)等進行研究。本文重點內(nèi)容是研究提取類別判別錯誤的文本數(shù)據(jù)的方法。
[Abstract]:Text corpus is the foundation of text data mining. Many text corpora are derived from the actual work of production and life, and are usually defined by industry experts. The data set in this paper comes from the mayor's open telephone office. With the change of industry category in different periods, there will inevitably be a lot of incorrect data in the corpus. Because of the large corpus, it is usually not able to be proofread by experts one by one. Therefore, we must use the method of data mining to find error classification data, and then proofread the error classification data one by industry experts. The purpose of this paper is to screen the data of error classification in the corpus so as to correct the classification of data by industry experts. This paper discusses the discrimination and classification of text data. This paper first discusses the technology and flow of text classification, then discusses the nature of naive Bayes method, finally discusses the refinement of text corpus, and discusses the method of selecting category discrimination error data. An empirical analysis is given. Under the condition of big data, it is not realistic to adopt the method of manual correction of text data by industry experts because it will consume a lot of manpower, material resources and financial resources. According to certain rules, batch marking of text data categories is another effective method, this method can effectively avoid the shortcomings of direct expert classification, but the accuracy of text data class marking is low. In combination with the above two methods, the third method is put forward. Firstly, the classification of text data is labeled in batches, and the text data that is wrong in category marking is handed over to industry experts for manual marking. Then the text data in the text corpus is corrected by the text data marked by industry experts. The study of text corpus refining is based on the third method. Different methods are used to extract the text data of category discrimination errors in the text corpus. In all methods, the text data which is wrong in category discrimination is the most likely text data for category marking errors. The purpose of text corpus refining is to extract the text data which is most likely to be a category tagging error in the text corpus. This part of text data is handed over to the category of manual marking of industry experts. Finally, the category of text data of text corpus is corrected based on the text data of industry experts. This paper first introduces the general process of text data classification, then introduces the naive Bayes classification algorithm; finally, the purpose and method of text corpus preprocessing, feature extraction, text corpus refining, The text data which extract the category discrimination error and so on are studied. The emphasis of this paper is to study the method of extracting text data of category discrimination error.
【學位授予單位】：東北師范大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：H08
本文編號：2263630
[Abstract]:Text corpus is the foundation of text data mining. Many text corpora are derived from the actual work of production and life, and are usually defined by industry experts. The data set in this paper comes from the mayor's open telephone office. With the change of industry category in different periods, there will inevitably be a lot of incorrect data in the corpus. Because of the large corpus, it is usually not able to be proofread by experts one by one. Therefore, we must use the method of data mining to find error classification data, and then proofread the error classification data one by industry experts. The purpose of this paper is to screen the data of error classification in the corpus so as to correct the classification of data by industry experts. This paper discusses the discrimination and classification of text data. This paper first discusses the technology and flow of text classification, then discusses the nature of naive Bayes method, finally discusses the refinement of text corpus, and discusses the method of selecting category discrimination error data. An empirical analysis is given. Under the condition of big data, it is not realistic to adopt the method of manual correction of text data by industry experts because it will consume a lot of manpower, material resources and financial resources. According to certain rules, batch marking of text data categories is another effective method, this method can effectively avoid the shortcomings of direct expert classification, but the accuracy of text data class marking is low. In combination with the above two methods, the third method is put forward. Firstly, the classification of text data is labeled in batches, and the text data that is wrong in category marking is handed over to industry experts for manual marking. Then the text data in the text corpus is corrected by the text data marked by industry experts. The study of text corpus refining is based on the third method. Different methods are used to extract the text data of category discrimination errors in the text corpus. In all methods, the text data which is wrong in category discrimination is the most likely text data for category marking errors. The purpose of text corpus refining is to extract the text data which is most likely to be a category tagging error in the text corpus. This part of text data is handed over to the category of manual marking of industry experts. Finally, the category of text data of text corpus is corrected based on the text data of industry experts. This paper first introduces the general process of text data classification, then introduces the naive Bayes classification algorithm; finally, the purpose and method of text corpus preprocessing, feature extraction, text corpus refining, The text data which extract the category discrimination error and so on are studied. The emphasis of this paper is to study the method of extracting text data of category discrimination error.
【學位授予單位】：東北師范大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：H08
【參考文獻】
相關期刊論文 前10條
1 邸鵬;段利國;;一種新型樸素貝葉斯文本分類算法[J];數(shù)據(jù)采集與處理;2014年01期
2 劉德喜;萬常選;;社會化短文本自動摘要研究綜述[J];小型微型計算機系統(tǒng);2013年12期
3 曾青華;袁家斌;張云洲;;基于Hadoop的貝葉斯過濾MapReduce模型[J];計算機工程;2013年11期
4 衛(wèi)潔;石洪波;冀素琴;;基于Hadoop的分布式樸素貝葉斯文本分類[J];計算機系統(tǒng)應用;2012年02期
5 陳朝大;梁柱勛;鄭士基;;一種利用關聯(lián)規(guī)則的改進樸素貝葉斯分類算法[J];計算機系統(tǒng)應用;2010年11期
6 鄭煒;沈文;張英鵬;;基于改進樸素貝葉斯算法的垃圾郵件過濾器的研究[J];西北工業(yè)大學學報;2010年04期
7 黃魏;高兵;劉異;楊克巍;;基于詞條組合的中文文本分詞方法[J];科學技術與工程;2010年01期
8 鄧u&;付長賀;;四種貝葉斯分類器及其比較[J];沈陽師范大學學報(自然科學版);2008年01期
9 王雙成;忻瑞嬋;;廣義樸素貝葉斯分類器[J];計算機應用與軟件;2007年11期
10 張玉芳;彭時名;呂佳;;基于文本分類TFIDF方法的改進與應用[J];計算機工程;2006年19期
相關碩士學位論文 前4條
1 吳文岫;短文本分類語料庫的構建及分類方法的研究[D];安徽大學;2015年
2 李太白;短文本分類中特征選擇算法的研究[D];重慶師范大學;2013年
3 常娟;短文本分類方法研究[D];復旦大學;2008年
4 張虎;漢語語料庫詞性標注一致性檢查及自動校對方法研究[D];山西大學;2005年
，本文編號：2263630
本文鏈接：http://www.sikaile.net/wenyilunwen/yuyanyishu/2263630.html
