【摘要】：隨著物聯(lián)網(wǎng)技術(shù)的發(fā)展,智能化傳感器的增多,交通行業(yè)收集到的數(shù)據(jù)急速增長(zhǎng)。特別是在高速公路收費(fèi)系統(tǒng)中,每天都會(huì)產(chǎn)生海量的高速公路收費(fèi)站數(shù)據(jù)。通過(guò)分析這些結(jié)構(gòu)化的數(shù)據(jù),可以得到高速公路車流量、載運(yùn)量時(shí)空分布、高速公路運(yùn)輸景氣指數(shù)、收費(fèi)報(bào)表同比環(huán)比等非常有價(jià)值的信息,為高速公路管理人員的正確決策提供數(shù)據(jù)支持。當(dāng)前,大多數(shù)交通部門所使用的管理系統(tǒng)都是使用Oracle驅(qū)動(dòng)的數(shù)據(jù)庫(kù)。面對(duì)數(shù)據(jù)體量愈發(fā)龐大的高速公路收費(fèi)站數(shù)據(jù),這些管理系統(tǒng)已經(jīng)出現(xiàn)數(shù)據(jù)整合過(guò)程復(fù)雜、時(shí)間久、依賴專業(yè)人員、數(shù)據(jù)查詢速度慢等問題。因此,本文研究基于云平臺(tái)的高速公路交通數(shù)據(jù)倉(cāng)庫(kù)設(shè)計(jì)與查詢優(yōu)化技術(shù)。首先,本文針對(duì)高速公路收費(fèi)站數(shù)據(jù)特點(diǎn),設(shè)計(jì)一種面向海量高速公路收費(fèi)站數(shù)據(jù)的數(shù)據(jù)倉(cāng)庫(kù),其構(gòu)建過(guò)程包括數(shù)據(jù)抽取、數(shù)據(jù)預(yù)處理和數(shù)據(jù)加工等三個(gè)核心操作階段。其次,本文通過(guò)比較Hive和Impala的查詢特點(diǎn),分析數(shù)據(jù)倉(cāng)庫(kù)的分區(qū)粒度和高速公路管理的業(yè)務(wù)特點(diǎn),提出了三種數(shù)據(jù)倉(cāng)庫(kù)查詢優(yōu)化方法。然后,本文基于分布式文件存儲(chǔ)系統(tǒng)HDFS、數(shù)據(jù)倉(cāng)庫(kù)工具Hive和數(shù)據(jù)查詢引擎Impala實(shí)現(xiàn)數(shù)據(jù)倉(cāng)庫(kù)構(gòu)建,設(shè)計(jì)并實(shí)現(xiàn)了面向高速公路管理的數(shù)據(jù)可視化平臺(tái),提供數(shù)據(jù)查詢及專題分析等功能。最后,本文使用實(shí)際的高速公路收費(fèi)站數(shù)據(jù)驗(yàn)證數(shù)據(jù)倉(cāng)庫(kù)的功能和性能,結(jié)果表明本文提出的數(shù)據(jù)查詢優(yōu)化方法能夠有效提高數(shù)據(jù)查詢效率,縮短查詢時(shí)間。
[Abstract]:With the development of Internet of things technology and the increase of intelligent sensors, the data collected by transportation industry is increasing rapidly. Especially in the freeway toll collection system, a large amount of highway toll collection station data are generated every day. By analyzing these structured data, we can get very valuable information such as freeway traffic flow, space-time distribution of carrying capacity, expressway transportation boom index, toll report forms, and so on. Provide data support for highway managers to make correct decisions. Currently, most management systems used by transportation departments are Oracle-driven databases. Faced with the increasingly large data volume of highway toll station data, these management systems have problems such as complex data integration process, long time, dependence on professionals, slow data query speed and so on. Therefore, this paper studies the highway traffic data warehouse design and query optimization technology based on cloud platform. Firstly, according to the characteristics of highway toll station data, this paper designs a data warehouse for mass highway toll station data. The construction process includes three core operation stages: data extraction, data preprocessing and data processing. Secondly, by comparing the query characteristics of Hive and Impala, this paper analyzes the partition granularity of data warehouse and the business characteristics of highway management, and puts forward three query optimization methods of data warehouse. Then, based on the distributed file storage system HDFS, data warehouse tool Hive and the data query engine Impala, this paper designs and implements the data visualization platform for highway management. Provides data query and project analysis functions. Finally, the function and performance of the data warehouse are verified by the actual toll station data in this paper. The results show that the data query optimization method proposed in this paper can effectively improve the efficiency of data query and shorten the query time.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP311.13;TP393.09

【參考文獻(xiàn)】

相關(guān)期刊論文 前7條

1 吳黎兵;邱鑫;葉璐瑤;王曉棟;聶雷;;基于Hadoop的SQL查詢引擎性能研究[J];華中師范大學(xué)學(xué)報(bào)(自然科學(xué)版);2016年02期

2 趙文英;;當(dāng)前大數(shù)據(jù)管理技術(shù)探究[J];信息與電腦(理論版);2015年22期

3 曾萍;韋杰;;數(shù)據(jù)倉(cāng)庫(kù)技術(shù)在高校信息化建設(shè)中的應(yīng)用研究[J];軟件;2014年05期

4 李小強(qiáng);何珊;何金明;;通過(guò)對(duì)比數(shù)據(jù)庫(kù)來(lái)理解數(shù)據(jù)倉(cāng)庫(kù)[J];考試周刊;2013年91期

5 邱衛(wèi)云;;智能交通大數(shù)據(jù)分析云平臺(tái)技術(shù)[J];中國(guó)交通信息化;2013年10期

6 黃文依;王勁松;林勝;;HDFS可視化操作研究與實(shí)現(xiàn)[J];天津理工大學(xué)學(xué)報(bào);2012年01期

7 許春玲;張廣泉;;分布式文件系統(tǒng)Hadoop HDFS與傳統(tǒng)文件系統(tǒng)Linux FS的比較與分析[J];蘇州大學(xué)學(xué)報(bào)(工科版);2010年04期

相關(guān)碩士學(xué)位論文 前5條

1 張鵬;多數(shù)據(jù)庫(kù)環(huán)境數(shù)據(jù)集成與轉(zhuǎn)換技術(shù)研究[D];北方工業(yè)大學(xué);2016年

2 費(fèi)仕憶;Hadoop大數(shù)據(jù)平臺(tái)與傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)的協(xié)作研究[D];東華大學(xué);2014年

3 王遠(yuǎn)志;基于Hadoop的全網(wǎng)絡(luò)流量異常監(jiān)測(cè)算法研究[D];鄭州大學(xué);2014年

4 韓歡;基于大數(shù)據(jù)的智能交通運(yùn)輸平臺(tái)的研究[D];成都理工大學(xué);2014年

5 常濤;改進(jìn)型MapReduce框架的研究與設(shè)計(jì)[D];北京郵電大學(xué);2011年

，

本文編號(hào)：2397894