本文關(guān)鍵詞： Hadoop MapReduce模型 圖像并行化　出處：《西安科技大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

【摘要】：近年來,伴隨著云計算和大數(shù)據(jù)的興起,網(wǎng)絡(luò)中各應(yīng)用領(lǐng)域所產(chǎn)生的數(shù)據(jù)量快速增長,已經(jīng)達到PB級別甚至更高。在這些數(shù)據(jù)中,圖像大數(shù)據(jù)的處理與存儲已經(jīng)成為各界研究的熱點。MapReduce技術(shù)是一種高可靠的并行編程框架,常用于進行大數(shù)據(jù)量的并行計算,對復(fù)雜的集群環(huán)境問題有著很好的解決方案。Hadoop平臺的核心之一MapReduce模型就是利用該項技術(shù)處理海量數(shù)據(jù),目前已取得了較好成效,但不足的是對于圖像文件的處理,尤其是海量小圖像文件的研究還不成熟。本文針對于此問題,探討了在MapReduce模型下圖像的并行化處理,給出了一種可以作為處理海量小圖像文件的新的數(shù)據(jù)平臺基礎(chǔ)架構(gòu),主要研究的內(nèi)容及研究成果如下:首先,本文綜述了在大數(shù)據(jù)背景下,海量圖像數(shù)據(jù)處理的研究背景、研究現(xiàn)狀及意義,并介紹了 Hadoop生態(tài)系統(tǒng),包括其核心HDFS系統(tǒng)和MapReduce框架技術(shù)。其次,詳細(xì)設(shè)計了如何改進Hadoop系統(tǒng),提出組合分片方法,提高了海量小圖片的處理效率,擴展了 MapReduce軟件框架,使之能夠很好的支持并處理圖像文件。最后,本文設(shè)計并實現(xiàn)了在MapReduce模型下,圖像的并行K-means聚類算法分析、圖像的并行Sobel邊緣檢測算法、擴展了圖像在MapReduce模型下的并行化直方圖提取等操作。通過實驗驗證,擴展后的MapReduce模型的可行性及處理圖像文件的高效性。通過指標(biāo)性能分析,驗證了 MapReduce模型下進行圖像并行化處理的有效性,為在Hadoop平臺下處理海量大數(shù)據(jù)圖像文件的應(yīng)用提供了一種可行的解決方案。
[Abstract]:In recent years, with the rise of cloud computing and big data, the amount of data generated in various application areas of the network has grown rapidly, reaching the PB level or higher. The processing and storage of image big data has become a hot topic of research. MapReduce technology is a high reliable parallel programming framework, which is often used for parallel computing of large amount of data. There is a good solution to the complex cluster environment problem. One of the core of the Hadoop platform MapReduce model is to use this technology to process massive data. At present, it has achieved good results, but the lack of the image file processing, Especially, the research of massive small image files is not mature. In this paper, we discuss the parallelization of images under MapReduce model, and give a new data platform infrastructure which can be used to deal with large amount of small image files. The main research contents and results are as follows: firstly, this paper summarizes the research background, research status and significance of massive image data processing under the background of big data, and introduces the Hadoop ecosystem. It includes its core HDFS system and MapReduce framework technology. Secondly, how to improve the Hadoop system is designed in detail, and the combined slicing method is put forward, which improves the processing efficiency of massive small images and extends the MapReduce software framework. Finally, this paper designs and implements the parallel K-means clustering algorithm under the MapReduce model, and the parallel Sobel edge detection algorithm of the image. The parallel histogram extraction of image under MapReduce model is extended. The feasibility of the extended MapReduce model and the efficiency of processing image files are verified by experiments. The validity of parallel image processing based on MapReduce model is verified, which provides a feasible solution for processing massive big data image files on Hadoop platform.
【學(xué)位授予單位】：西安科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP391.41

【參考文獻】

相關(guān)期刊論文 前10條

1 謝雪蓮;李蘭友;;基于云計算的并行K-means聚類算法研究[J];計算機測量與控制;2014年05期

2 趙慶;;基于Hadoop平臺下的Canopy-Kmeans高效算法[J];電子科技;2014年02期

3 朱長明;張新;駱劍承;李萬慶;楊紀(jì)偉;;基于樣本自動選擇與SVM結(jié)合的海岸線遙感自動提取[J];國土資源遙感;2013年02期

4 黃斌;許舒人;蒲衛(wèi);;基于MapReduce的數(shù)據(jù)挖掘平臺設(shè)計與實現(xiàn)[J];計算機工程與設(shè)計;2013年02期

5 冀素琴;石洪波;;面向海量數(shù)據(jù)的K-means聚類優(yōu)化算法[J];計算機工程與應(yīng)用;2014年14期

6 崔朝國;劉志明;李婧;陳曉凡;;一種基于Hadoop的Scool云存儲平臺[J];電腦知識與技術(shù);2013年02期

7 阮濤;那彥;王澍;;基于壓縮感知的遙感圖像融合方法[J];電子科技;2012年04期

8 仇李寅;邱衛(wèi)東;蘇芊;廖凌;;基于Hadoop的分布式哈希算法實現(xiàn)[J];信息安全與通信保密;2011年11期

9 多雪松;張晶;高強;;基于Hadoop的海量數(shù)據(jù)管理系統(tǒng)[J];微計算機信息;2010年13期

10 袁春蘭;熊宗龍;周雪花;彭小輝;;基于Sobel算子的圖像邊緣檢測研究[J];激光與紅外;2009年01期

相關(guān)碩士學(xué)位論文 前1條

1 霍樹民;基于Hadoop的海量影像數(shù)據(jù)管理關(guān)鍵技術(shù)研究[D];國防科學(xué)技術(shù)大學(xué);2010年

，

本文編號：1497555