本文選題：智能交通　切入點：大數(shù)據(jù)　出處：《杭州電子科技大學》2017年碩士論文　論文類型：學位論文

【摘要】：隨著中國城市化規(guī)模的逐步擴大以及城市居民收入的不斷增加,私家車數(shù)量也在不斷的增加,伴隨著也帶來了一系列的交通問題。為了便于城市交通的管理,智能交通系統(tǒng)應運而生,其通過引入現(xiàn)代化技術并結合各城市的具體需求,實現(xiàn)對交通信息的實時收集和處理,了解當前的交通環(huán)境并作出相應的調控。這對于保障城市交通高效運行和可持續(xù)發(fā)展具有重要的意義。數(shù)據(jù)存儲與檢索是智能交通系統(tǒng)的核心之一。在實際公安局點應用中,道路監(jiān)控每天會產生海量的數(shù)據(jù),僅浙江省一天產生的過車數(shù)據(jù)便有幾億,并且數(shù)據(jù)到達具有隨機性。傳統(tǒng)的關系型數(shù)據(jù)庫由于其嚴格的表結構約束,無法實現(xiàn)海量數(shù)據(jù)的存取操作。并且當一張數(shù)據(jù)表的數(shù)據(jù)量達到一定級別時,索引本身就過于巨大。因此數(shù)據(jù)庫的檢索功能根本無法滿足數(shù)據(jù)檢索的需求,并且極易造成系統(tǒng)的癱瘓。本文對以上問題進行深入研究,設計了千億級智能交通大數(shù)據(jù)存儲與檢索系統(tǒng)。系統(tǒng)采用分布式集群方案,以分布式框架Hadoop為基礎將系統(tǒng)集群設計為主從架構。集群使用Zookeeper進行一致性管理,使用Yarn進行資源管理和分配。為保證系統(tǒng)集群的穩(wěn)定性,通過虛擬IP和Zookeeper實現(xiàn)了負載均衡和高可用性機制,用于處理高并發(fā)連接和單點故障問題,并保證對外地址的一致性。針對海量數(shù)據(jù)存儲和檢索這個難點,引入搜索引擎Solr和非關系型數(shù)據(jù)庫HBase實現(xiàn)數(shù)據(jù)存儲和檢索方案。針對高并發(fā)數(shù)據(jù)容易引起Solr不穩(wěn)定,設計了Kafka和Spark Streaming高并發(fā)實時數(shù)據(jù)緩存和消費策略。針對海量數(shù)據(jù)檢索延遲高,設計了自稱為Solr分Core算法和時間緊縮算法,實現(xiàn)了千億數(shù)據(jù)秒級檢索,并設計了翻頁緩存功能提升客戶端翻頁體驗。最后,本文對系統(tǒng)進行測試,測試結果表明系統(tǒng)工作穩(wěn)定,能高效存儲海量多種類型數(shù)據(jù)。當數(shù)據(jù)庫中存儲一千億條過車記錄時,對此TB級別數(shù)據(jù)進行各種條件的檢索均能在1s內響應。
[Abstract]:With the gradual expansion of the scale of urbanization in China and the increasing income of urban residents, the number of private cars is also increasing, accompanied by a series of traffic problems. In order to facilitate the management of urban traffic, Intelligent transportation system emerges as the times require. It can collect and process traffic information in real time by introducing modern technology and combining with the specific needs of each city. Understand the current traffic environment and make corresponding regulation and control. This is of great significance for ensuring the efficient operation and sustainable development of urban traffic. Data storage and retrieval is one of the core of intelligent transportation system. Road monitoring produces huge amounts of data every day. In Zhejiang Province alone, hundreds of millions of traffic data are generated in one day, and the arrival of the data is random. Traditional relational databases are constrained by their strict table structure. When the data amount of a data table reaches a certain level, the index itself is too large. Therefore, the retrieval function of the database can not meet the requirements of data retrieval at all. And it is easy to cause paralysis of the system. This paper makes a thorough study on the above problems, and designs a storage and retrieval system for big data, a 100bn level intelligent transportation system. The system adopts a distributed cluster scheme. The system cluster design is based on the distributed framework Hadoop. The cluster uses Zookeeper for consistency management and Yarn for resource management and allocation. A load balancing and high availability mechanism is implemented through virtual IP and Zookeeper, which is used to deal with the problems of high concurrent connection and single point failure, and to ensure the consistency of external address. This paper introduces search engine Solr and non-relational database HBase to realize the scheme of data storage and retrieval, aiming at the instability of Solr caused by high concurrency data. The high concurrent real-time data cache and consumption strategy of Kafka and Spark Streaming are designed. In view of the high latency of massive data retrieval, the self-called Solr sub-#en4# algorithm and the time compression algorithm are designed. Finally, the system is tested in this paper. The test results show that the system works stably and can store a large number of kinds of data efficiently. When 100 billion passing records are stored in the database, The terabyte level data can be retrieved in 1 s.
【學位授予單位】：杭州電子科技大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：U495;TP311.13

【參考文獻】

相關期刊論文 前10條

1 滕學蓓;李宏海;高劍;;中國智能交通系統(tǒng):要走自己的路[J];中國公路;2016年13期

2 王文峰;袁慶祝;陸佃龍;;基于Zookeeper綜合任務調度平臺的設計與應用[J];信息技術;2016年06期

3 黃力彬;;國內外城市智能交通系統(tǒng)建設綜述[J];城市道橋與防洪;2016年05期

4 韓學仁;王青山;郭勇;;基于Solr的空間數(shù)據(jù)語義搜索研究[J];測繪與空間地理信息;2016年02期

5 葉駿宏;王勇;強保華;;基于Lucene的Web服務查詢系統(tǒng)設計與實現(xiàn)[J];桂林電子科技大學學報;2016年01期

6 王巖;王純;;一種基于Kafka的可靠的Consumer的設計方案[J];軟件;2016年01期

7 田翠珍;;基于Thrift框架RPC的研究與實現(xiàn)[J];信息與電腦(理論版);2016年01期

8 徐海榮;陳閔葉;張興媛;;基于Flume、Kafka、Storm、HDFS的航空維修大數(shù)據(jù)系統(tǒng)[J];上海工程技術大學學報;2015年04期

9 方金云;劉羽;姚曉;陳翠婷;張夢菲;肖茁建;張廣發(fā);;基于Spark的空間數(shù)據(jù)實時訪存技術的研究[J];地理信息世界;2015年06期

10 郝強;高占春;;基于SolrCloud的網(wǎng)絡百科檢索服務的實現(xiàn)[J];軟件;2015年12期

相關碩士學位論文 前2條

1 王睿;基于主從服務器的分布式存儲系統(tǒng)的設計與實現(xiàn)[D];浙江大學;2016年

2 莊共波;智能交通系統(tǒng)中的地圖路況服務研究與實現(xiàn)[D];北京郵電大學;2013年

，

本文編號：1582429