本文關(guān)鍵詞：Hadoop框架的擴(kuò)展和性能調(diào)優(yōu)　出處：《西安建筑科技大學(xué)》2012年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 云計(jì)算 網(wǎng)格計(jì)算 LSF Hadoop MapReduce

【摘要】：云計(jì)算作為一種新的概念在2007年成為了人們熱議的話題，在隨后的幾年內(nèi)云計(jì)算得到了快速的發(fā)展。從計(jì)算模式來看，云計(jì)算、分布式計(jì)算和網(wǎng)格計(jì)算有很多相似之處，深入研究云計(jì)算產(chǎn)生的背景就可以看出，云計(jì)算是在分布式計(jì)算和網(wǎng)格計(jì)算的基礎(chǔ)之上發(fā)展起來的。以前的分布式計(jì)算和網(wǎng)格計(jì)算主要用于科學(xué)研究方面，隨著互聯(lián)網(wǎng)的迅速發(fā)展，分布式計(jì)算和網(wǎng)格計(jì)算的思想逐漸演化為一種更適合商用的計(jì)算模式-云計(jì)算。 論文首先介紹了云計(jì)算與網(wǎng)格計(jì)算的相關(guān)背景知識，并分析了兩者之間的區(qū)別，然后對云計(jì)算平臺Hadoop核心組成MapReduce、HDFS(Hadoop Distributed FileSystem)和Hbase等的關(guān)鍵技術(shù)進(jìn)行詳細(xì)的分析與研究[1]。接著詳細(xì)介紹了LSF(Load Sharing Facility)系統(tǒng)的架構(gòu)組成，包括LSF base和LSF batch兩部分，并對LSF的作業(yè)執(zhí)行流程和系統(tǒng)負(fù)載均衡進(jìn)行了深入細(xì)致的分析。 論文在對Hadoop系統(tǒng)深入研究分析之后，發(fā)現(xiàn)Hadoop在面對企業(yè)級應(yīng)用時有3大不足，分別是單點(diǎn)故障、調(diào)度算法單一、異構(gòu)平臺兼容性差[2]。針對這幾點(diǎn)不足，論文對Hadoop系統(tǒng)與LSF系統(tǒng)進(jìn)行了關(guān)聯(lián)性整合，形成一個新的系統(tǒng)LSH(LoadShare Hadoop)。系統(tǒng)整合主要有兩大結(jié)合點(diǎn)，第一，，將LSF的作業(yè)控制機(jī)制LIM(Load Information Manager)、RES (Remote Execution Server)和SBD(sbatch,一個守護(hù)進(jìn)程)加入到Hadoop系統(tǒng)的HDFS層與MapReduce層之間；第二，LSF的master節(jié)點(diǎn)與HDFS的NameNode之間通過開放接口共享信息。整合后的系統(tǒng)LSH有效地防止了Hadoop系統(tǒng)的單點(diǎn)故障問題，也解決了Hadoop調(diào)度算法單一的問題和Hadoop對異構(gòu)平臺的兼容性問題。 論文最后針對整合后的系統(tǒng)LSH和原生態(tài)的Hadoop系統(tǒng)設(shè)計(jì)了不同的實(shí)驗(yàn)，分別來驗(yàn)證兩系統(tǒng)對單點(diǎn)故障的處理、差異性作業(yè)的性能和異構(gòu)平臺的適應(yīng)性方面的表現(xiàn)，結(jié)果證明LSH系統(tǒng)完全彌補(bǔ)了原生Hadoop的不足，LSH是能夠適應(yīng)企業(yè)級的應(yīng)用。
[Abstract]:Cloud computing as a new concept in 2007 has become a hot topic, cloud computing has been rapid development in the following years. From the perspective of computing mode, cloud computing. There are many similarities between distributed computing and grid computing. Cloud computing is developed on the basis of distributed computing and grid computing. The former distributed computing and grid computing are mainly used in scientific research, with the rapid development of the Internet. The idea of distributed computing and grid computing has evolved into a more commercial computing model-cloud computing. Firstly, this paper introduces the background knowledge of cloud computing and grid computing, and analyzes the difference between them. Then, the Hadoop core of cloud computing platform is composed of MapReduce. The key technologies of HDFS(Hadoop Distributed File system and Hbase are analyzed and studied in detail. [1. Then the architecture of the LSF(Load Sharing availability) system is introduced in detail, including two parts: LSF base and LSF batch. The job execution flow and system load balance of LSF are analyzed in detail. After deeply studying and analyzing Hadoop system, it is found that Hadoop has three shortcomings in facing enterprise application, namely, single point fault, single scheduling algorithm and poor compatibility of heterogeneous platform. [2]. Aiming at these shortcomings, this paper integrates the Hadoop system and the LSF system. To form a new system LSH(LoadShare Hadoop. System integration has two main points of convergence, first. LSF's job control mechanism, LIM(Load Information Manager. RES remote Execution Server) and SBD(sbatch. A daemon is added between the HDFS layer and the MapReduce layer of the Hadoop system; Number two. The master node of LSF and the NameNode of HDFS share information through open interface. The integrated system LSH effectively prevents the single point of failure of Hadoop system. It also solves the problem of single Hadoop scheduling algorithm and compatibility of Hadoop to heterogeneous platforms. At the end of the paper, different experiments are designed for the integrated system LSH and the original Hadoop system, respectively, to verify the two systems to deal with the single point of failure. The performance of different jobs and the adaptability of heterogeneous platforms show that the LSH system can fully compensate for the deficiency of native Hadoop and is able to adapt to enterprise-level applications.
【學(xué)位授予單位】：西安建筑科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2012
【分類號】：TP338.8

【參考文獻(xiàn)】

相關(guān)期刊論文 前3條

1 李鑫;張鵬;;Hadoop集群公平調(diào)度算法的改進(jìn)與實(shí)現(xiàn)[J];電腦知識與技術(shù);2012年01期

2 李成華;張新訪;金海;向文;;MapReduce:新型的分布式并行計(jì)算編程模型[J];計(jì)算機(jī)工程與科學(xué);2011年03期

3 孫廣中;肖鋒;熊曦;;MapReduce模型的調(diào)度及容錯機(jī)制研究[J];微電子學(xué)與計(jì)算機(jī);2007年09期

相關(guān)碩士學(xué)位論文 前6條

1 陳艷金;MapReduce模型在Hadoop平臺下實(shí)現(xiàn)作業(yè)調(diào)度算法的研究和改進(jìn)[D];華南理工大學(xué);2011年

2 徐文強(qiáng);基于HDFS的云存儲系統(tǒng)研究[D];上海交通大學(xué);2011年

3 張文峰;基于MapReduce模型的分布式計(jì)算平臺的原理與設(shè)計(jì)[D];華中科技大學(xué);2010年

4 杜志源;基于OGSA的教育資源共享研究[D];西安電子科技大學(xué);2007年

5 夏yN;Hadoop平臺下的作業(yè)調(diào)度算法研究與改進(jìn)[D];華南理工大學(xué);2010年

6 張密密;MapReduce模型在Hadoop實(shí)現(xiàn)中的性能分析及改進(jìn)優(yōu)化[D];電子科技大學(xué);2010年

本文編號：1422823