【摘要】：針對當前基于軟/硬件的流導出技術存在的數(shù)據(jù)失真、不易擴展等問題,本文提出一種準確、通用、易擴展的高速網(wǎng)絡流量處理框架NTCI-Flow。首先,基于PF_RING DNA實現(xiàn)了高性能的網(wǎng)絡包抓取,采用基于網(wǎng)絡包五元組的負載均衡策略對網(wǎng)絡包進行分組分發(fā),并利用批處理、無鎖隊列、多線程等技術將多個網(wǎng)絡包封裝為單條大消息并行發(fā)送,改進與優(yōu)化網(wǎng)絡包轉(zhuǎn)發(fā)性能;然后,采用Kafka消息系統(tǒng)作為中間件接收并緩存網(wǎng)絡包,從而實現(xiàn)網(wǎng)絡包的分布式導入;接著,基于Storm搭建實時流處理平臺,開發(fā)并部署分布式流重組應用,實現(xiàn)從Kafka中讀取網(wǎng)絡包,解析并抽取五元組、包大小、時間戳等信息后重組成網(wǎng)絡流;最后,增加Hive流數(shù)據(jù)導入模塊,將導出的網(wǎng)絡流數(shù)據(jù)以Parquet格式實時存入HDFS,利用Hive Metastore存儲并管理元數(shù)據(jù),同時采用基于時間的動態(tài)分區(qū)機制以減少按時間檢索時不必要的磁盤IO。實驗結(jié)果表明:網(wǎng)絡流量采集模塊可實現(xiàn)萬兆流量的準確采集與轉(zhuǎn)發(fā),即使在萬兆流量均為最小包(60字節(jié))的情況下,仍可保證僅有0.03%的丟包率;網(wǎng)絡流量導入模塊吞吐率與磁盤寫入性能相關,在使用7塊硬盤緩存數(shù)據(jù)時吞吐率可達775 MB/s;分布式流重組模塊具有良好的通用性及擴展性,通過簡單配置即可達到1.26×10~7包/s的吞吐率。目前,NTCI-Flow已用于采集與處理某機構(gòu)的出口流量,該機構(gòu)平均流量約3.5 Gbps,峰值帶寬為6 Gbps,每秒包數(shù)最高可達百萬級。在該實際應用中,NTCI-Flow運行情況良好,由其得到的流量數(shù)據(jù)比Net Stream更準確。
[Abstract]:In view of the problems existing in the current flow export technology based on software / hardware, such as data distortion and inextensibility, this paper presents an accurate, universal and extensible high-speed network traffic processing framework NTCI-Flow.. First of all, the high performance network packet grab is realized based on PF_RING DNA, and the network packet is distributed by the load balancing strategy based on the five-tuple network packet, and the batch processing is used and the unlocked queue is used. Multithreading encapsulates multiple network packets into a single large message, improves and optimizes the network packet forwarding performance, and then uses Kafka message system as middleware to receive and cache network packets, so as to realize the distributed import of network packets. Then, a real-time stream processing platform based on Storm is built to develop and deploy distributed stream recombination application, which can read network packets from Kafka, parse and extract five-tuple, packet size, timestamp and reorganize into network flow. Adding Hive stream data import module, storing the exported network stream data into HDFS, in Parquet format and using Hive Metastore to store and manage metadata, and adopting the dynamic partitioning mechanism based on time to reduce the unnecessary disk IO. when retrieving by time The experimental results show that the network traffic acquisition module can accurately collect and transmit the ten thousand megabytes, even if the ten thousand megabytes are the smallest packet (60 bytes), only 0.03% of the packet loss rate can be guaranteed. The throughput of the network traffic import module is related to the disk write performance. The throughput of 775 MB/s; distributed stream recombination module has good generality and expansibility when using 7 hard disks to cache data. The throughput of 1.26 脳 10 ~ 7 packets / s can be achieved by simple configuration. At present, NTCI-Flow has been used to collect and process the outlet flow of a certain organization. The average flow of the mechanism is about 3.5 Gbps, with a peak bandwidth of 6 Gbps, / s and the maximum number of packets per second can reach 1 million. In this practical application, NTCI-Flow is running well and the flow data obtained from it are more accurate than Net Stream.
【作者單位】： 四川大學計算機學院;
【基金】：國家自然科學基金資助項目(61272447)
【分類號】：TP393.08

【相似文獻】

相關期刊論文 前5條

1 唐磊,金連甫;大型網(wǎng)絡自動信息機設計與實現(xiàn)[J];計算機工程與設計;2004年08期

2 謝文亮;唐屹;王大星;;基于FPGA技術的網(wǎng)絡包頭分類的研究[J];廣州大學學報(自然科學版);2007年03期

3 BEAN;;我的IRC簡歷[J];軟件世界;1998年10期

4 張文波,趙海,王小英,關沫;基于ARMLinux的EWS過載性能研究[J];通信學報;2005年08期

5 ;[J];;年期

相關碩士學位論文 前3條

1 朱新宇;基于鄰居的分布式網(wǎng)絡異常節(jié)點檢測定位系統(tǒng)[D];上海交通大學;2015年

2 劉琦;網(wǎng)絡包接收擁塞問題的研究與解決[D];東北大學;2005年

3 陳金牛;嵌入式IPv6防火墻設計與實現(xiàn)[D];廈門大學;2007年

，

本文編號：2234683