【摘要】：截止目前企業(yè)面向的零售戶量已經(jīng)突破900萬,隨著信息化和數(shù)據(jù)化的加深,采集零售戶數(shù)據(jù)和信息也是勢在必行的。面對900萬的用戶量,隨之引起問題的是服務(wù)器的訪問量急劇上升,數(shù)據(jù)量呈爆炸式增長。以企業(yè)目前的服務(wù)器等硬件設(shè)備已經(jīng)無法滿足高并發(fā)、大數(shù)據(jù)的需求。如果只是增加多臺服務(wù)器組成服務(wù)器集群,提高硬件、軟件的性能,來解決大量用戶并發(fā)請求和大數(shù)據(jù)處理的問題,這顯然并不是很好的辦法。本文通過分析和對比傳統(tǒng)的數(shù)據(jù)采集模式和企業(yè)業(yè)務(wù)類型,設(shè)計(jì)了一種基于移動端的數(shù)據(jù)采集模式。并詳細(xì)的分析了該數(shù)據(jù)采集模型,為了滿足企業(yè)的要求和用戶的需求,針對新型采集模式面對的高并發(fā)、大數(shù)據(jù)處理的問題,進(jìn)一步分析和研究了負(fù)載均衡調(diào)度策略,并根據(jù)基于高并發(fā)的特征和調(diào)度算法的特點(diǎn),在多時(shí)間片輪詢調(diào)度機(jī)制的基礎(chǔ)上提出了基于任務(wù)請求預(yù)測的動態(tài)調(diào)整負(fù)載均衡算法。并通過分析多維Markov鏈和排隊(duì)論對算法進(jìn)行進(jìn)一步的調(diào)整和改進(jìn),經(jīng)過實(shí)驗(yàn)驗(yàn)證,該算法較好地提高了整體系統(tǒng)的性能和負(fù)載。具體研究內(nèi)容如下:1)本文通過對比傳統(tǒng)數(shù)據(jù)采集方法,根據(jù)目前企業(yè)的業(yè)務(wù)類型和數(shù)據(jù)請求特征,設(shè)計(jì)了一種基于分布式策略的新型數(shù)據(jù)采集模型,該模型將APP和微信企業(yè)號作為數(shù)據(jù)采集的入口,并且組建服務(wù)器集群對高并發(fā)業(yè)務(wù)進(jìn)行調(diào)度處理。使用基于Hadoop平臺的策略創(chuàng)建了數(shù)據(jù)中心,并對數(shù)據(jù)進(jìn)行了分析和處理。其中對全國零售戶請求的業(yè)務(wù)類型和提交的數(shù)據(jù)性質(zhì),對該模型進(jìn)行了初步優(yōu)化處理。并通過分析模型系統(tǒng)中的層次結(jié)構(gòu)和技術(shù)要點(diǎn),提出了在高并發(fā)環(huán)境下,如何高效地調(diào)度任務(wù)請求和調(diào)整服務(wù)器節(jié)點(diǎn)的負(fù)載的問題,并在后文進(jìn)行了分析和解決。2)深入研究了基于CPU和MEM的調(diào)度算法,根據(jù)實(shí)際任務(wù)請求的特征,改進(jìn)并提出了基于預(yù)測機(jī)制的負(fù)載均衡調(diào)度算法。各個(gè)服務(wù)器節(jié)點(diǎn)會收集其它節(jié)點(diǎn)的負(fù)載情況,并預(yù)測網(wǎng)絡(luò)請求的業(yè)務(wù)類型和到達(dá)率,動態(tài)的調(diào)整請求的分發(fā),減少請求的等待時(shí)間,并縮短服務(wù)器的閑置時(shí)間,達(dá)到資源的有效利用,最終使系統(tǒng)的整體負(fù)載達(dá)到均衡狀態(tài)。通過實(shí)驗(yàn)驗(yàn)證,該算法在縮短響應(yīng)時(shí)間方面具有良好的性能,并且比基于CPU和MEM的調(diào)度算法性能更好。3)針對基于預(yù)測機(jī)制的負(fù)載均衡算法本身存在的不足,利用排隊(duì)論的知識,對網(wǎng)絡(luò)服務(wù)器方面的負(fù)載進(jìn)行了優(yōu)化,合理的安排任務(wù)請求的等待、處理和掛起。通過分析多維Markov的機(jī)制,對后續(xù)網(wǎng)絡(luò)請求的特征和聯(lián)系進(jìn)行預(yù)測。通過分析多時(shí)間片輪詢策略,提出了基于預(yù)測的Markov排隊(duì)模型。經(jīng)過實(shí)驗(yàn)的驗(yàn)證和分析,該模型較好的協(xié)調(diào)各服務(wù)器節(jié)點(diǎn)的負(fù)載狀況,合理的分發(fā)后續(xù)到達(dá)的網(wǎng)絡(luò)請求,減小響應(yīng)時(shí)間。4)根據(jù)數(shù)據(jù)采集模型和負(fù)載均衡技術(shù),建立了一套集移動終端數(shù)據(jù)采集平臺、業(yè)務(wù)處理平臺、數(shù)據(jù)處理平臺的三位一體的數(shù)據(jù)采集系統(tǒng)。并將負(fù)載均衡技術(shù)應(yīng)用到數(shù)據(jù)處理中,加快了服務(wù)器處理數(shù)據(jù)的時(shí)間,并展示了數(shù)據(jù)采集系統(tǒng)設(shè)計(jì)實(shí)現(xiàn)的成果。
[Abstract]:Up to now, the number of retail customers facing the enterprise has exceeded 9 million. With the deepening of information and data, it is imperative to collect data and information of retail customers. Facing 9 million users, the problem is that the amount of access to the server is rising rapidly and the amount of data is exploding. It is obviously not a good way to solve the problem of a large number of concurrent requests and large data processing, which is obviously not a very good solution. This article analyzes and compares the traditional data collection mode and enterprise business by analyzing and comparing the problems of large number of concurrent requests and large data processing. Type, a data acquisition model based on mobile terminal is designed, and the data acquisition model is analyzed in detail. In order to meet the requirements of the enterprise and the needs of the users, the load balancing scheduling strategy is further analyzed and studied for the problems of high concurrency and large data processing in the new collection mode. On the basis of the multi time chip polling scheduling mechanism, the dynamic adjustment load balancing algorithm based on the task request prediction is proposed. The algorithm is further adjusted and improved by analyzing the multidimensional Markov chain and queuing theory. The algorithm improves the performance and negative effect of the whole system. The specific research contents are as follows: 1) in this paper, a new data acquisition model based on the distributed strategy is designed by comparing the traditional data acquisition methods, according to the business types and data request features of the current enterprise. This model takes APP and WeChat enterprise number as the entrance of data collection, and sets up a server cluster for high concurrency services. The data center is created using the strategy based on the Hadoop platform, and the data are analyzed and processed. The model is optimized for the type of business and the nature of the data submitted by the national retail customers, and the high level structure and technical points in the model system are put forward. In the concurrent environment, how to efficiently schedule task requests and adjust the load of server nodes, and analyze and solve.2 in the later text, the scheduling algorithm based on CPU and MEM is deeply studied. According to the characteristics of the actual task request, the load balancing scheduling algorithm based on the prediction machine is improved and proposed. Collect the load of other nodes, predict the type of service and the rate of arrival of the network request, dynamically adjust the request distribution, reduce the waiting time of the request, shorten the idle time of the server, achieve the effective utilization of the resource, and finally make the whole load of the system reach the balance state. The inter aspect has good performance and is better than the scheduling algorithm based on CPU and MEM). In view of the shortcomings of the load balancing algorithm based on the prediction mechanism, using the knowledge of queuing theory, it optimizes the load of the network server and arranges the waiting, processing and hanging up. By analyzing multidimensional Ma. The mechanism of rkov predicts the characteristics and connections of the subsequent network requests. Through the analysis of the multi time slice polling strategy, a prediction based Markov queuing model is proposed. Through the experimental verification and analysis, the model is better to coordinate the load status of each server node, to distribute the following network requests reasonably, and to reduce the response time.4). According to the data collection model and the load balancing technology, a set of data collection system is set up, which sets the data collection platform of the mobile terminal, the business processing platform and the data processing platform. The load balancing technology is applied to the data processing, and the time for the server to deal with the data is speeded up, and the design of the data acquisition system is demonstrated. The present results.
【學(xué)位授予單位】：浙江理工大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：F724.2

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 王小戲;吳剛;王灝;;高并發(fā)高可用零售O2O交易系統(tǒng)的架構(gòu)設(shè)計(jì)與業(yè)務(wù)實(shí)現(xiàn)[J];計(jì)算機(jī)與現(xiàn)代化;2016年04期

2 林偉;丁志剛;;基于Agent的微信平臺自適應(yīng)負(fù)載均衡算法[J];實(shí)驗(yàn)技術(shù)與管理;2015年12期

3 喜藝;;Hadoop大數(shù)據(jù)平臺與傳統(tǒng)數(shù)據(jù)倉庫的協(xié)作探究[J];通訊世界;2015年17期

4 本刊編輯部;石菲;邢帆;孫杰賢;姜紅德;;“兩會”關(guān)鍵熱點(diǎn):信息化新階段[J];中國信息化;2015年03期

5 王輝;李晉光;;異構(gòu)網(wǎng)絡(luò)負(fù)載均衡算法[J];工業(yè)儀表與自動化裝置;2014年05期

6 蘇金樹;郭文忠;余朝龍;陳國龍;;負(fù)載均衡感知的無線傳感器網(wǎng)絡(luò)容錯(cuò)分簇算法[J];計(jì)算機(jī)學(xué)報(bào);2014年02期

7 黃素萍;葛萌;;Hadoop平臺在大數(shù)據(jù)處理中的應(yīng)用研究[J];現(xiàn)代計(jì)算機(jī)(專業(yè)版);2013年29期

8 亓開元;韓燕波;趙卓峰;房俊;;支持高并發(fā)數(shù)據(jù)流處理的MapReduce中間結(jié)果緩存[J];計(jì)算機(jī)研究與發(fā)展;2013年01期

9 尚明華;秦磊磊;王風(fēng)云;劉淑云;張曉艷;;基于Android智能手機(jī)的小麥生產(chǎn)風(fēng)險(xiǎn)信息采集系統(tǒng)[J];農(nóng)業(yè)工程學(xué)報(bào);2011年05期

10 張濤;李建;康永佳;;多任務(wù)高并發(fā)數(shù)據(jù)處理平臺的技術(shù)研究[J];網(wǎng)絡(luò)安全技術(shù)與應(yīng)用;2010年03期

相關(guān)博士學(xué)位論文 前2條

1 劉曉茜;云計(jì)算數(shù)據(jù)中心結(jié)構(gòu)及其調(diào)度機(jī)制研究[D];中國科學(xué)技術(shù)大學(xué);2011年

2 王永炎;實(shí)時(shí)事務(wù)并發(fā)控制算法優(yōu)化[D];中國科學(xué)院研究生院（軟件研究所）;2004年

相關(guān)碩士學(xué)位論文 前10條

1 鄧玉林;基于hadoop大數(shù)據(jù)框架的個(gè)性化推薦系統(tǒng)研究與實(shí)現(xiàn)[D];電子科技大學(xué);2016年

2 張慧芳;基于動態(tài)反饋的加權(quán)最小連接數(shù)服務(wù)器負(fù)載均衡算法研究[D];華東理工大學(xué);2013年

3 王延武;基于LVS的集群動態(tài)負(fù)載均衡算法研究[D];北京化工大學(xué);2012年

4 馮秀玲;云計(jì)算環(huán)境下的負(fù)載均衡算法的研究與設(shè)計(jì)[D];北京郵電大學(xué);2012年

5 胡俊;構(gòu)建分布式系統(tǒng)的關(guān)鍵技術(shù)研究與實(shí)現(xiàn)[D];南京郵電大學(xué);2012年

6 施楊斌;云計(jì)算環(huán)境下一種基于虛擬機(jī)動態(tài)遷移的負(fù)載均衡算法[D];復(fù)旦大學(xué);2011年

7 楊錦;異構(gòu)分布式系統(tǒng)中的負(fù)載均衡調(diào)度算法研究[D];湖南大學(xué);2011年

8 萬勇;基于LVS的負(fù)載均衡策略算法的研究與改進(jìn)[D];西南交通大學(xué);2010年

9 劉漢玲;基于Web Service的分布式系統(tǒng)中動態(tài)負(fù)載均衡策略的研究與實(shí)現(xiàn)[D];哈爾濱工程大學(xué);2010年

10 陳亮;集群負(fù)載均衡關(guān)鍵技術(shù)研究[D];中南大學(xué);2009年

，

本文編號：2157441