本文關鍵詞：基于分布式框架的網絡事件實時感知系統(tǒng)　出處：《浙江大學》2017年碩士論文　論文類型：學位論文

  更多相關文章： 事件感知 分布式 實時 流式計算

【摘要】：隨著互聯(lián)網的發(fā)展,面對海量數(shù)據(jù)時,個人的精力無法滿足完成提取、獲得全面而精確的信息的任務的要求,從而掌握一個特定領域下的趨勢發(fā)展�；诖吮闾岢隽艘允录问阶鳛檩d體,通過從不斷處理的新的文檔中提取事件信息,之后合并到舊有信息中,呈現(xiàn)給用戶宏觀上的統(tǒng)計數(shù)據(jù)和具體分析內容,并輔助人們進行各類決策�，F(xiàn)階段較為成熟的事件感知系統(tǒng)依賴于大規(guī)模計算集群,以流式與批量式集合的方式,完成了大數(shù)據(jù)規(guī)模下應用的實現(xiàn)。本文聚焦于在小規(guī)模集群下能夠實時獲取事件結果,進行查詢的總體要求,以流式處理的形式,在增加系統(tǒng)整體處理效率與減少對算法影響的目標下,完成事件感知各項應用功能。本文基于上述目標,設計并開發(fā)了一套分布式處理平臺,滿足應用在各個環(huán)節(jié)下的應用要求。主要的工作包括:1)針對事件感知應用的輸入、輸出,用戶對象進行分析,將系統(tǒng)劃分為三個模塊,完成系統(tǒng)總體架構設計。2)在存儲模塊下設計了存儲形式,包括MongoDB內數(shù)據(jù)的表達與NAF標引格式。3)在處理模塊下,對事件感知傳統(tǒng)的兩種類型任務在流式數(shù)據(jù)環(huán)境下進行了分布式擴展,提出了各自的拓撲設計。同時針對系統(tǒng)運行的Storm計算框架,優(yōu)化了拓撲調度器,并針對內存計算設計了符合事件感知容錯性要求的內存數(shù)據(jù)的持久化策略。4)分析與服務模塊設計了針對不同查詢類型的響應策略,并在查詢后臺設計了在分布式內存環(huán)境下基于封閉立方體的維度統(tǒng)計方法最后以實際檢驗檢疫應用出發(fā)為導向,驗證了系統(tǒng)的可用性與性能。
[Abstract]:With the development of the Internet, in the face of massive data, the individual energy can not meet the task of extracting, obtaining comprehensive and accurate information. In order to grasp the trend of development in a specific field. Based on this, it is proposed to take the form of events as the carrier, through the continuous processing of new documents from the extraction of event information, and then merged into the old information. It presents users with macroscopic statistical data and concrete analysis content, and assists people to make all kinds of decisions. At this stage, the more mature event perception system relies on large-scale computing clusters. The implementation of big data application under the scale of big data is completed by the way of flow and batch collection. This paper focuses on the overall requirements of real-time event results and query in small scale cluster, in the form of flow processing. Under the goal of increasing the overall processing efficiency of the system and reducing the impact on the algorithm, this paper designs and develops a distributed processing platform based on the above objectives. The main work includes: 1) analyzing the input, output and user object of the event-aware application, and dividing the system into three modules. Complete the system architecture design. 2) Design the storage form under the storage module, including the data expression in MongoDB and the NAF indexing format. 3) under the processing module. Two kinds of traditional event-aware tasks are extended in the streaming data environment, and their topology design is proposed. At the same time, the topology scheduler is optimized for the Storm computing framework. The persistence strategy of memory data, which meets the requirements of event-aware fault-tolerance, is designed for memory computing. 4) Analysis and service modules are designed to respond to different query types. The dimension statistics method based on closed cube in distributed memory environment is designed in the query background. Finally, the application of practical inspection and quarantine is taken as the guide to verify the availability and performance of the system.
【學位授予單位】：浙江大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13

【參考文獻】

相關期刊論文 前4條

1 張亮;白振興;周軍;白云;;一種生成封閉數(shù)據(jù)立方體的新算法[J];彈箭與制導學報;2010年03期

2 吳飛;莊越挺;;互聯(lián)網跨媒體分析與檢索:理論與算法[J];計算機輔助設計與圖形學學報;2010年01期

3 游進國;奚建清;張平健;劉艷霞;;在PC集群上的封閉立方體計算[J];計算機科學;2009年06期

4 李盛恩,王珊;封閉數(shù)據(jù)立方體技術研究[J];軟件學報;2004年08期

，

本文編號：1407872