本文選題：工業(yè)數(shù)據(jù)壓縮　切入點(diǎn)：磁盤調(diào)度　出處：《南京信息工程大學(xué)》2014年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著網(wǎng)格計(jì)算的高速發(fā)展,大規(guī)模實(shí)時(shí)工業(yè)數(shù)據(jù)分析在工業(yè)生產(chǎn)過(guò)程中起到了舉足輕重的作用,通過(guò)實(shí)時(shí)監(jiān)控分析工業(yè)數(shù)據(jù),能夠極大的指導(dǎo)生產(chǎn),提高生產(chǎn)力。然而在實(shí)時(shí)工業(yè)數(shù)據(jù)分析領(lǐng)域主要還存在幾大問(wèn)題：第一,實(shí)時(shí)數(shù)據(jù)生成的速度非常快,占用大量的系統(tǒng)存儲(chǔ)空間,如何在不丟失數(shù)據(jù)特征的情況下盡量減小存儲(chǔ)壓力是一個(gè)急需解決的問(wèn)題。第二,在大規(guī)模的實(shí)時(shí)數(shù)據(jù)的傳輸及處理過(guò)程中,會(huì)在采集器與中心服務(wù)器間形成千上萬(wàn)的并發(fā)數(shù)據(jù)連接,此時(shí)服務(wù)器的穩(wěn)定性、高效性至關(guān)重要。第三,傳統(tǒng)的TCP協(xié)議參數(shù)已經(jīng)不能適應(yīng)當(dāng)前的高速網(wǎng)絡(luò),如何優(yōu)化協(xié)議使得網(wǎng)絡(luò)傳輸?shù)耐掏铝扛蟆Ｔ诒疚闹形覀兎謩e從數(shù)據(jù)壓縮、網(wǎng)格服務(wù)器性能以及網(wǎng)絡(luò)傳輸性能三個(gè)方面進(jìn)行闡述我們網(wǎng)格計(jì)算相關(guān)的優(yōu)化成果。本文主要完成了以下工作： (1)工業(yè)數(shù)據(jù)實(shí)時(shí)性要求較高,會(huì)隨著時(shí)間形成一條曲線。工業(yè)產(chǎn)生的實(shí)時(shí)數(shù)據(jù)量非常大,不可以通過(guò)緩存大量數(shù)據(jù)后再分析整個(gè)數(shù)據(jù)的走向特性選擇對(duì)應(yīng)的壓縮方案,整個(gè)壓縮算法的處理過(guò)程必須具有高壓縮比和低資源消耗。本文提出了針對(duì)工業(yè)實(shí)時(shí)數(shù)據(jù)壓縮的曲線面積映射壓縮法。根據(jù)測(cè)試結(jié)果,算法壓縮比是傳統(tǒng)SDT壓縮法的2.16倍。 (2)在數(shù)據(jù)網(wǎng)格環(huán)境中,服務(wù)器的性能一直是整體網(wǎng)格性能的關(guān)鍵要素之一。在本文中我們分析了網(wǎng)格服務(wù)器常見(jiàn)的性能瓶頸并在我們的網(wǎng)格環(huán)境中引入了用戶空間I/O調(diào)度、零拷貝、事件驅(qū)動(dòng)架構(gòu)技術(shù)來(lái)提高網(wǎng)格服務(wù)器的性能。在大量小文件讀取的情況下引入用戶空間I/O調(diào)度能夠節(jié)省近50%的磁盤I/O時(shí)間。通過(guò)零拷貝,網(wǎng)格服務(wù)器可以減少CPU在大量?jī)?nèi)核與用戶空間切換之間引起的消耗,節(jié)省63%無(wú)用的上下文切換。事件驅(qū)動(dòng)架構(gòu)可以減少30%的CPU利用率并且達(dá)到線程驅(qū)動(dòng)的最佳吞吐量。通過(guò)結(jié)合上面提到的這三種優(yōu)化方法應(yīng)用到我們的網(wǎng)格服務(wù)器中,新的解決方案可以在只消耗傳統(tǒng)解決方案所占用CPU使用率的70%的情況下,系統(tǒng)飽和吞吐量比傳統(tǒng)方案提高30%。 (3)網(wǎng)絡(luò)流量測(cè)試表明,當(dāng)前網(wǎng)絡(luò)95%的數(shù)據(jù)流為TCP流,其它的為UDP或其它形式的數(shù)據(jù)流,因此TCP協(xié)議傳輸性能成為了制約整個(gè)網(wǎng)絡(luò)數(shù)據(jù)傳輸性能的關(guān)鍵。通過(guò)分析TCP協(xié)議及其多個(gè)變種,縱向比較它們的擁塞控制算法,著重討論了影響TCP協(xié)議性能的網(wǎng)絡(luò)因素以及在不同網(wǎng)絡(luò)環(huán)境下TCP協(xié)議的網(wǎng)絡(luò)性能表現(xiàn)。實(shí)現(xiàn)了當(dāng)前網(wǎng)絡(luò)傳輸常見(jiàn)場(chǎng)景下廣泛采用的相關(guān)TCP優(yōu)化方案,并提出根據(jù)神經(jīng)網(wǎng)絡(luò)來(lái)預(yù)測(cè)丟包在保持鏈路公平的前提下來(lái)提高TCP吞吐量的方案,通過(guò)網(wǎng)絡(luò)模擬實(shí)驗(yàn)顯示,預(yù)測(cè)的一級(jí)命中率最高可以達(dá)到74%。
[Abstract]:With the rapid development of grid computing, large-scale real-time industrial data analysis plays an important role in the industrial production process, through real-time monitoring and analysis of industrial data, can greatly guide production, However, there are still several problems in the field of real-time industrial data analysis. First, the speed of real-time data generation is very fast, and it takes up a lot of system storage space. How to minimize the storage pressure without losing data features is an urgent problem. Second, in the process of large-scale real-time data transmission and processing, There will be thousands of concurrent data connections between the collector and the central server, so the stability and efficiency of the server is crucial. Third, the traditional TCP protocol parameters can no longer adapt to the current high-speed network. How to optimize the protocol to increase the throughput of network transport. Three aspects of grid server performance and network transmission performance are described in this paper. The main work of this paper is as follows:. (1) Industrial data requires high real-time performance and will form a curve with time. The amount of real time data produced by industry is very large, so it is not possible to choose the corresponding compression scheme by caching a large amount of data and then analyzing the trend characteristics of the whole data. The processing process of the whole compression algorithm must have high compression ratio and low resource consumption. In this paper, a curve area mapping compression method for industrial real-time data compression is proposed. According to the test results, the compression ratio of the algorithm is 2.16 times that of the traditional SDT compression method. In the data grid environment, the performance of the server is always one of the key elements of the overall grid performance. In this paper, we analyze the common performance bottlenecks of the grid server and introduce the user-space I / O scheduling into our grid environment. Zero copy, event driven architecture technology to improve the performance of grid servers. In the case of a large number of small files read, the introduction of user space I / O scheduling can save nearly 50% disk I / O time. Grid servers can reduce the cost of CPU between a large number of kernel and user space switching, Save 63% useless context switching. Event-driven architecture can reduce the CPU utilization of 30% and achieve the best thread-driven throughput. The new solution can increase the system saturation throughput by 30% over the traditional solution when only 70% of the CPU usage is consumed by the traditional solution. Network traffic tests show that the current network 95% data flow is TCP flow, others are UDP or other forms of data flow, Therefore, the transmission performance of TCP protocol has become the key to restrict the data transmission performance of the whole network. By analyzing the TCP protocol and its variety, the congestion control algorithm of TCP protocol is compared longitudinally. The network factors that affect the performance of TCP protocol and the performance of TCP protocol in different network environments are discussed emphatically, and the related TCP optimization schemes widely used in current network transmission scenarios are realized. A scheme is proposed to improve the throughput of TCP based on neural network to predict packet loss under the premise of maintaining the fairness of the link. The network simulation results show that the highest hit rate of the first class can reach 74%.
【學(xué)位授予單位】：南京信息工程大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2014
【分類號(hào)】：TP333;TP393.05

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 劉紅霞;牛富麗;;實(shí)時(shí)數(shù)據(jù)庫(kù)數(shù)據(jù)壓縮算法探討與改進(jìn)[J];化工自動(dòng)化及儀表;2010年06期

2 錢笑宇,張彥武;工業(yè)實(shí)時(shí)數(shù)據(jù)庫(kù)的研究和設(shè)計(jì)[J];計(jì)算機(jī)工程;2005年01期

3 趙利強(qiáng);于濤;王建林;;基于SQL數(shù)據(jù)庫(kù)的過(guò)程數(shù)據(jù)壓縮方法[J];計(jì)算機(jī)工程;2008年14期

4 熊永華;吳敏;賈維嘉;;基于延時(shí)預(yù)測(cè)的TCP實(shí)時(shí)視頻傳輸方法[J];中南大學(xué)學(xué)報(bào)(自然科學(xué)版);2010年04期

，

本文編號(hào)：1580931