【摘要】：隨著互聯(lián)網(wǎng)的快速發(fā)展,應(yīng)用所產(chǎn)生的數(shù)據(jù)量越來(lái)越大。分布式數(shù)據(jù)庫(kù)HBase對(duì)海量數(shù)據(jù)的管理得到了廣泛應(yīng)用。許多企業(yè)希望將原本存放在關(guān)系型數(shù)據(jù)庫(kù)中的數(shù)據(jù)遷移到分布式數(shù)據(jù)庫(kù)HBase中,并在HBase上對(duì)數(shù)據(jù)進(jìn)行管理。因此研究在HBase上提供一個(gè)數(shù)據(jù)管理系統(tǒng)具有重要意義。在分析基于HBase的數(shù)據(jù)管理系統(tǒng)設(shè)計(jì)目標(biāo)基礎(chǔ)上,給出了系統(tǒng)的整體設(shè)計(jì)方案,包括兩大功能:將關(guān)系型數(shù)據(jù)庫(kù)中的模式和數(shù)據(jù)遷移到HBase中,使用SQL語(yǔ)句管理HBase中的數(shù)據(jù)。關(guān)系型數(shù)據(jù)庫(kù)模式和數(shù)據(jù)遷移功能將關(guān)系型數(shù)據(jù)庫(kù)表的列信息、索引信息、主外鍵信息存儲(chǔ)到HBase的元數(shù)據(jù)表中。表數(shù)據(jù)遷移任務(wù)分割為多個(gè)小任務(wù)并盡可能地平分給集群中機(jī)器來(lái)執(zhí)行,同時(shí)根據(jù)主外鍵信息對(duì)數(shù)據(jù)進(jìn)行冗余存儲(chǔ),根據(jù)索引信息在HBase中創(chuàng)建索引表并記錄索引數(shù)據(jù)。使用SQL語(yǔ)句管理HBase中數(shù)據(jù),重點(diǎn)優(yōu)化多表連接查詢。將多表連接查詢?nèi)蝿?wù)根據(jù)HBase的特點(diǎn)分解為若干子多表連接查詢,使用HBase的協(xié)處理器并發(fā)的執(zhí)行子多表連接查詢。子多表連接查詢中根據(jù)連接條件的特點(diǎn)優(yōu)化表的連接順序,利用遷移流程產(chǎn)生的冗余數(shù)據(jù)、索引數(shù)據(jù)提高連接查詢效率。子多表連接查詢的中間數(shù)據(jù)利用哈希表和多叉樹(shù)存儲(chǔ)來(lái)降低內(nèi)存開(kāi)銷。在客戶端合并多個(gè)子多表連接查詢返回的結(jié)果。通過(guò)對(duì)基于HBase數(shù)據(jù)管理系統(tǒng)的實(shí)驗(yàn)測(cè)試,表明該系統(tǒng)能高效的遷移表模式和數(shù)據(jù),對(duì)遷移后的數(shù)據(jù)能正確的管理,且在多表連接查詢時(shí)相比與Hive具有較好性能。
[Abstract]:With the rapid development of the Internet, the amount of data generated by the application is increasing. Distributed database HBase has been widely used in the management of massive data. Many enterprises want to migrate the data stored in the relational database to the distributed database HBase and manage the data on the HBase. Therefore, it is of great significance to provide a data management system on HBase. Based on the analysis of the design objectives of the data management system based on HBase, the overall design scheme of the system is presented, which includes two functions: migrating the schema and data from the relational database to the HBase, and managing the data in the HBase by using the SQL statement. Relational database schema and data migration function store the column information, index information and primary foreign key information of relational database table in HBase metadata table. The task of table data migration is divided into several small tasks and distributed equally to the machines in the cluster as far as possible. At the same time, the data is stored redundant according to the primary foreign key information, and the index table is created in HBase according to the index information and the index data is recorded. Use SQL statements to manage data in HBase, focusing on optimizing multi-table join queries. According to the characteristics of HBase, the task of multi-table join query is decomposed into several sub-multi-table join queries, and the co-processor of HBase is used to execute the sub-multi-table join query. In order to optimize the join order of subtable join query according to the characteristics of join condition, the redundant data generated by migration process is used to improve the efficiency of join query. The intermediate data of sub-table join query uses hash table and multi-tree storage to reduce memory overhead. Results returned by merging multiple child multiple table join queries on the client side. The experimental results based on HBase data management system show that the system can efficiently migrate the table pattern and data, manage the migrated data correctly, and have better performance than Hive in multi-table join query.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2016
【分類號(hào)】：TP311.13

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 楊武,文守遜;匯總型多表連接查詢的一種優(yōu)化方法[J];計(jì)算機(jī)系統(tǒng)應(yīng)用;2000年01期

2 徐帆;匯總型多表連接查詢的一種優(yōu)化方法[J];計(jì)算機(jī)工程與設(shè)計(jì);2002年10期

3 張雷;唐桂芬;蘇冉冉;;基于通用空間連接圖的適應(yīng)性多元空間連接查詢[J];計(jì)算機(jī)光盤軟件與應(yīng)用;2013年13期

4 彭建平,王變琴;再探多連接查詢優(yōu)化方法[J];中山大學(xué)學(xué)報(bào)(自然科學(xué)版);2001年02期

5 劉宇,孫莉,田永青;并行空間連接查詢處理[J];上海交通大學(xué)學(xué)報(bào);2002年04期

6 王果,徐仁佐;結(jié)合哈希過(guò)濾的一種改進(jìn)多連接查詢優(yōu)化算法[J];計(jì)算機(jī)工程;2004年07期

7 陳恕勝;劉衛(wèi)東;;基于圖的適應(yīng)性多連接查詢優(yōu)化算法[J];計(jì)算機(jī)工程;2009年10期

8 郭聰莉;朱莉;李向;;基于蟻群算法的多連接查詢優(yōu)化方法[J];計(jì)算機(jī)工程;2009年10期

9 王，

本文編號(hào)：2275687