本文選題：云計(jì)算 + Hadoop��； 參考：《北京郵電大學(xué)》2014年碩士論文

【摘要】：如今我們已經(jīng)步入了一個(gè)全新的時(shí)代——大數(shù)據(jù)時(shí)代。我們每天的生活、學(xué)習(xí)和工作都要產(chǎn)生上T級(jí)別的網(wǎng)絡(luò)數(shù)據(jù)量。因此,隨著互聯(lián)網(wǎng)技術(shù)的飛速發(fā)展而產(chǎn)生的一種新型的分布式計(jì)算模式云計(jì)算應(yīng)運(yùn)而生。它的出現(xiàn)解決了海量數(shù)據(jù)存儲(chǔ)和實(shí)時(shí)備份等一系列和大數(shù)據(jù)息息相關(guān)的問(wèn)題。 近年來(lái),國(guó)內(nèi)外各大互聯(lián)網(wǎng)公司和學(xué)者專(zhuān)家們都積極投身云計(jì)算技術(shù)的研究,資源調(diào)度則是云計(jì)算研究的關(guān)鍵技術(shù)之一。云計(jì)算環(huán)境中資源調(diào)度策略的優(yōu)劣直接影響整個(gè)集群系統(tǒng)的性能。因此對(duì)云計(jì)算環(huán)境中資源調(diào)度策略的研究具有十分重大的意義。 本文主要的研究?jī)?nèi)容如下所示： (1)研究分析了云計(jì)算環(huán)境下的資源調(diào)度策略。分析了Hadoop平臺(tái)的系統(tǒng)結(jié)構(gòu)、MapReduce框架和分布式文件系統(tǒng)HDFS。 (2)深入研究了開(kāi)源分布式Hadoop平臺(tái)的資源調(diào)度策略。并對(duì)Hadoop平臺(tái)常用的三種資源調(diào)度策略：FIFO、計(jì)算能力和公平分配資源調(diào)度策略進(jìn)行了仿真實(shí)驗(yàn)。通過(guò)實(shí)驗(yàn)結(jié)果來(lái)對(duì)比分析三種資源調(diào)度策略在系統(tǒng)吞吐率和系統(tǒng)響應(yīng)時(shí)間的性能區(qū)別。 (3)針對(duì)Hadoop平臺(tái)常用資源調(diào)度策略沒(méi)有考慮作業(yè)任務(wù)有時(shí)間限制的問(wèn)題,本文提出了一種基于LLF的Deadline資源調(diào)度策略——LLFD。設(shè)計(jì)了數(shù)據(jù)本地性管理算法、Master的全局調(diào)度算法和系統(tǒng)節(jié)點(diǎn)本地調(diào)度算法。 (4)在CloudSim仿真平臺(tái)上驗(yàn)證LLFD資源調(diào)度策略的性能。通過(guò)實(shí)驗(yàn)仿真結(jié)果可以證實(shí),該資源調(diào)度策略可以在云計(jì)算大規(guī)模的基礎(chǔ)設(shè)施環(huán)境中盡可能使每個(gè)作業(yè)任務(wù)在截止日期之前得到執(zhí)行,同時(shí)能夠使集群中滿(mǎn)足時(shí)間期限的節(jié)點(diǎn)數(shù)目達(dá)到最小化,在可接受的超時(shí)時(shí)間范圍內(nèi)實(shí)現(xiàn)了最小化由于超時(shí)帶來(lái)的懲罰率。從而可以提升整個(gè)集群系統(tǒng)的性能。
[Abstract]:Now we have entered a new era-big data era. Our daily life, study and work must produce T-level network data. Therefore, with the rapid development of Internet technology, a new distributed computing model cloud computing came into being. It solves a series of problems related to big data, such as massive data storage and real-time backup. In recent years, domestic and foreign major Internet companies and scholars and experts are actively engaged in the research of cloud computing technology, resource scheduling is one of the key technologies in cloud computing research. The resource scheduling strategy in cloud computing environment directly affects the performance of the whole cluster system. Therefore, the research of resource scheduling strategy in cloud computing environment is of great significance. The main contents of this paper are as follows: 1) the resource scheduling strategy in cloud computing environment is studied and analyzed. The system structure of Hadoop platform and the distributed file system HDFS are analyzed. The resource scheduling strategy of open source distributed Hadoop platform is studied in detail. Three kinds of resource scheduling strategies: FIFO, computational power and fair resource allocation are simulated on Hadoop platform. The performance differences of the three resource scheduling strategies in the throughput and response time of the system are compared and analyzed by the experimental results. 3) aiming at the problem that the resource scheduling policy in common use in Hadoop platform does not consider the time limit of job tasks, this paper proposes a Deadline resource scheduling strategy based on LLF, LLFD-LLFD. The global scheduling algorithm of Master and the local scheduling algorithm of system node are designed. 4) verify the performance of LLFD resource scheduling strategy on CloudSim simulation platform. The simulation results show that the resource scheduling strategy can make every task execute before the deadline in the large-scale infrastructure environment of cloud computing. At the same time, it can minimize the number of nodes meeting the time limit in the cluster, and minimize the penalty rate due to the timeout in the acceptable timeout range. Thus, the performance of the whole cluster system can be improved.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2014
【分類(lèi)號(hào)】：TP393.01

【參考文獻(xiàn)】

相關(guān)期刊論文 前8條

1 葉志強(qiáng);劉晨鳴;王一梅;;基于Hadoop開(kāi)源框架的視頻內(nèi)容分發(fā)平臺(tái)研究[J];廣播電視信息;2013年03期

2 王峰;;Hadoop集群作業(yè)的調(diào)度算法[J];程序員;2009年12期

3 冼進(jìn);余桂城;;基于云計(jì)算的作業(yè)調(diào)度算法研究[J];計(jì)算機(jī)與數(shù)字工程;2011年07期

4 張建勛;古志民;鄭超;;云計(jì)算研究進(jìn)展綜述[J];計(jì)算機(jī)應(yīng)用研究;2010年02期

5 辛軍;陳康;鄭緯民;;虛擬化的集群資源管理技術(shù)研究[J];計(jì)算機(jī)科學(xué)與探索;2010年04期

6 高宏卿;邢穎;;基于經(jīng)濟(jì)學(xué)的云資源管理模型研究[J];計(jì)算機(jī)工程與設(shè)計(jì);2010年19期

7 金嘉暉;羅軍舟;宋愛(ài)波;東方;;基于數(shù)據(jù)中心負(fù)載分析的自適應(yīng)延遲調(diào)度算法[J];通信學(xué)報(bào);2011年07期

8 黃義強(qiáng);;基于Hadoop的中醫(yī)數(shù)據(jù)存儲(chǔ)平臺(tái)設(shè)計(jì)與開(kāi)發(fā)[J];江西中醫(yī)學(xué)院學(xué)報(bào);2011年06期

，

本文編號(hào)：1925592