本文選題：分布式文件系統(tǒng) + 小文件　； 參考：《微電子學與計算機》2014年07期

【摘要】：提出一種改善Hadoop文件處理效率的方法,在Hadoop中添加一個小文件處理模塊SFPM,根據文件名為海量小文件建立二級索引,同時采用預加載技術將索引提前存入緩存,可提高文件查找訪問效率;在合并文件時,采取舍棄多余空間的策略,避免將一個文件拆分存儲在兩個block上,減少了文件訪問時間開銷.實驗結果表明該方法能有效減輕NameNode的負荷,提高小文件讀寫效率.
[Abstract]:A method to improve the efficiency of Hadoop file processing is put forward. A small file processing module, SFPM, is added to the Hadoop. According to the name of the file, a two level index is set up for a large number of small files. At the same time, the preload technology is used to save the index into the cache in advance. It can improve the efficiency of the file search access. It is avoided to save a file on two block and reduce the time cost of file access. Experimental results show that the method can effectively reduce the load of NameNode and improve the efficiency of reading and writing of small files.
【作者單位】： 廣東工業(yè)大學計算機學院;
【基金】：廣東省戰(zhàn)略性新興產業(yè)核心技術攻關項目(2012A010701004) 廣東省自然科學基金重點項目(S2012020011071) 廣東省教育部產學研合作項目(2012B091000037,2012B091000041)
【分類號】：TP316.4;TP333

【參考文獻】

相關期刊論文 前6條

1 欒亞建;黃爛，

本文編號：1978055