傳統(tǒng)關系型數(shù)據(jù)庫向非關系型數(shù)據(jù)庫遷移算法研究
發(fā)布時間：2018-10-11 12:52
【摘要】：近年來,隨著物聯(lián)網(wǎng)技術的應用和“互聯(lián)網(wǎng)+”概念的提出,互聯(lián)網(wǎng)中的數(shù)據(jù)量呈出指數(shù)式的增長從而由信息時代進入大數(shù)據(jù)時代。大數(shù)據(jù)時代的數(shù)據(jù)不僅在數(shù)據(jù)量上劇增還在結構、分布、價值密度及維度等方面有所改變,這些數(shù)據(jù)在并行化快速處理上的要求需要具有半結構化及非結構化數(shù)據(jù)存儲、高擴展性和高讀寫行能的非關系型數(shù)據(jù)庫及分布式計算模式來滿足。傳統(tǒng)關系型數(shù)據(jù)庫無法滿足處理大數(shù)據(jù)的要求,但是由于其廣泛應用而在歷史上積累了大量的數(shù)據(jù),為了充分利用這些數(shù)據(jù)繼續(xù)為用戶提供服務、為企業(yè)創(chuàng)造更多的價值需要設計一種自動化或半自動化的遷移工具來完成傳統(tǒng)關系型數(shù)據(jù)庫向非關系型數(shù)據(jù)庫的轉換。另外,由于非關系型數(shù)據(jù)庫與傳統(tǒng)關系型數(shù)據(jù)庫的使用接口有著本質上的區(qū)別,為了使遷移后原有業(yè)務應用能夠適應新的數(shù)據(jù)存儲模型推動向非關系型數(shù)據(jù)庫的平滑過渡需要將非關系型數(shù)據(jù)庫的原生操作接口進行封裝并通過設計一種SQL操作中間件來實現(xiàn)這個轉變。本文通過對數(shù)據(jù)庫存儲結構、大數(shù)據(jù)相關技術和HBase的操作接口的研究,設計實現(xiàn)了數(shù)據(jù)庫遷移工具并對HBase的操作接口進行了封裝。首先,對數(shù)據(jù)存儲和遷移的相關技術進行了研究,深入地分析了傳統(tǒng)關系型數(shù)據(jù)庫的存儲原理和設計規(guī)則、HBase數(shù)據(jù)庫的架構和存儲模式、Hadoop生態(tài)系統(tǒng)以及數(shù)據(jù)遷移設計的相關模型和技術。其次,針對傳統(tǒng)關系型數(shù)據(jù)庫的元數(shù)據(jù)進行深入分析提出并設計實現(xiàn)了表模式和數(shù)據(jù)由MySQL向HBase遷移的算法。然后,對HBase的原生API進行深入分析,利用SQL解析和Filter條件過濾并結合傳統(tǒng)關系型數(shù)據(jù)庫的操作習慣對其進行了封裝。最后,通過設計實驗進行性能測試,分析結果表明本文所實現(xiàn)的遷移工具和操作中間件能夠快速有效地完成歷史數(shù)據(jù)的遷移工作并在以后的使用過程中滿足查詢需求,為數(shù)據(jù)存儲從關系型數(shù)據(jù)庫向非關系型數(shù)據(jù)庫的過渡提供了保障。
[Abstract]:In recent years, with the application of the Internet of things technology and the concept of "Internet", the amount of data in the Internet has increased exponentially, thus from the information age to big data era. The data of big data era not only increased sharply in data volume, but also changed in structure, distribution, value density and dimension, etc. These data need to be stored in semi-structured and unstructured data in parallelization and fast processing. Non-relational databases and distributed computing models with high scalability and high read-write performance are satisfied. The traditional relational database can not meet the requirement of processing big data, but because of its extensive application, it has accumulated a large amount of data in history. In order to make full use of this data, it continues to provide services to users. To create more value for enterprises, we need to design an automatic or semi-automatic migration tool to complete the transformation from traditional relational database to non-relational database. In addition, because the interface between non-relational database and traditional relational database is essentially different, In order to adapt the new data storage model to the smooth transition to non-relational database, the native operation interface of non-relational database is encapsulated and a SQL operation is designed. Middleware to achieve this transformation. Through the research of database storage structure, big data technology and HBase operation interface, this paper designs and implements the database migration tool and encapsulates the operation interface of HBase. Firstly, the related technologies of data storage and migration are studied. The storage principle and design rules of traditional relational database, the structure and storage mode of HBase database, the model and technology of Hadoop ecosystem and data migration design are analyzed. Secondly, an algorithm of table schema and data migration from MySQL to HBase is proposed and implemented by analyzing the metadata of traditional relational database. Then, the native API of HBase is deeply analyzed and encapsulated by using SQL parsing and Filter conditional filtering and combining with the operation habits of traditional relational database. Finally, through designing experiments to test the performance, the analysis results show that the migration tools and operation middleware implemented in this paper can quickly and effectively complete the migration of historical data and meet the query requirements in the later use process. It provides a guarantee for the transition of data storage from relational database to non-relational database.
【學位授予單位】：華僑大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13
本文編號：2264262
[Abstract]:In recent years, with the application of the Internet of things technology and the concept of "Internet", the amount of data in the Internet has increased exponentially, thus from the information age to big data era. The data of big data era not only increased sharply in data volume, but also changed in structure, distribution, value density and dimension, etc. These data need to be stored in semi-structured and unstructured data in parallelization and fast processing. Non-relational databases and distributed computing models with high scalability and high read-write performance are satisfied. The traditional relational database can not meet the requirement of processing big data, but because of its extensive application, it has accumulated a large amount of data in history. In order to make full use of this data, it continues to provide services to users. To create more value for enterprises, we need to design an automatic or semi-automatic migration tool to complete the transformation from traditional relational database to non-relational database. In addition, because the interface between non-relational database and traditional relational database is essentially different, In order to adapt the new data storage model to the smooth transition to non-relational database, the native operation interface of non-relational database is encapsulated and a SQL operation is designed. Middleware to achieve this transformation. Through the research of database storage structure, big data technology and HBase operation interface, this paper designs and implements the database migration tool and encapsulates the operation interface of HBase. Firstly, the related technologies of data storage and migration are studied. The storage principle and design rules of traditional relational database, the structure and storage mode of HBase database, the model and technology of Hadoop ecosystem and data migration design are analyzed. Secondly, an algorithm of table schema and data migration from MySQL to HBase is proposed and implemented by analyzing the metadata of traditional relational database. Then, the native API of HBase is deeply analyzed and encapsulated by using SQL parsing and Filter conditional filtering and combining with the operation habits of traditional relational database. Finally, through designing experiments to test the performance, the analysis results show that the migration tools and operation middleware implemented in this paper can quickly and effectively complete the migration of historical data and meet the query requirements in the later use process. It provides a guarantee for the transition of data storage from relational database to non-relational database.
【學位授予單位】：華僑大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13
【參考文獻】
相關期刊論文 前10條
1 宋春紅;王佳斌;鄭力新;;一種MySQL到HBase的遷移策略的研究與實現(xiàn)[J];微型機與應用;2016年13期
2 宋春紅;王佳斌;鄭力新;;結構型數(shù)據(jù)庫向HBase遷移的方法對比分析[J];福建電腦;2016年05期
3 李青云;余文;;關系型數(shù)據(jù)庫到H Base的轉換設計[J];信息網(wǎng)絡安全;2015年01期
4 涂新莉;劉波;林偉偉;;大數(shù)據(jù)研究綜述[J];計算機應用研究;2014年06期
5 李雪曉;葉云;田苗苗;黃劉生;;基于格的大數(shù)據(jù)動態(tài)存儲完整性驗證方案[J];信息網(wǎng)絡安全;2014年04期
6 韋海清;;淺談Java通過JDBC連接Oracle數(shù)據(jù)庫技術[J];計算機光盤軟件與應用;2014年07期
7 高源;;NoSQL非關系型數(shù)據(jù)庫的發(fā)展和應用研究[J];計算機光盤軟件與應用;2014年05期
8 邱勝海;高成沖;王云霞;陸云;王志亮;馬銀忠;;大數(shù)據(jù)時代非關系型數(shù)據(jù)庫教學與實驗改革探索[J];電腦知識與技術;2013年31期
9 唐長城;楊峰;代棟;孫明明;周學海;;一種基于HBase的數(shù)據(jù)持久性和可用性研究[J];計算機系統(tǒng)應用;2013年10期
10 陶雪嬌;胡曉峰;劉洋;;大數(shù)據(jù)研究綜述[J];系統(tǒng)仿真學報;2013年S1期
相關碩士學位論文 前4條
1 劉珂;基于Hadoop平臺的大數(shù)據(jù)遷移與查詢方法研究及應用[D];武漢理工大學;2014年
2 唐韜;基于HBase的SQL轉化引擎研究[D];電子科技大學;2014年
3 黎明;昆明學院圖書館信息系統(tǒng)分析與設計[D];云南大學;2013年
4 劉欣然;基于異構表的數(shù)據(jù)遷移算法研究[D];哈爾濱工程大學;2013年
，本文編號：2264262
本文鏈接：http://www.sikaile.net/kejilunwen/ruanjiangongchenglunwen/2264262.html
最近更新
教材專著
