本文選題：分布式系統(tǒng)　切入點：廣告計費　出處：《哈爾濱工業(yè)大學(xué)》2016年碩士論文　論文類型：學(xué)位論文

【摘要】：近年來,網(wǎng)絡(luò)廣告市場規(guī)模發(fā)展迅速,各大互聯(lián)網(wǎng)公司都在布局自己的網(wǎng)絡(luò)廣告交易平臺。計費系統(tǒng)是整個網(wǎng)絡(luò)廣告交易流程中重要的、不可或缺的一環(huán)。本文根據(jù)實際的業(yè)務(wù)需求,設(shè)計并實現(xiàn)了網(wǎng)絡(luò)廣告交易計費系統(tǒng),來支持廣告交易平臺的計費需求。本文使用java、scala語言開發(fā)了廣告計費系統(tǒng),主要研究內(nèi)容分為廣告反作弊和廣告計費。廣告反作弊部分用來判定作弊的廣告,系統(tǒng)對作弊的廣告不扣費,保護(hù)廣告主的利益。本文提出了基于統(tǒng)計方法的作弊判定規(guī)則,來過濾作弊的廣告;為避免單純的統(tǒng)計方法判定結(jié)果過于武斷,提出了打分算法來計算廣告的作弊可能性,從而實現(xiàn)對作弊廣告的平滑過濾。廣告計費部分基于spark實現(xiàn)。spark是一個基于內(nèi)存的、可擴展、可容錯的分布式計算框架。它在分布式的環(huán)境下處理廣告數(shù)據(jù),過濾作弊的廣告,計算扣費金額,生成扣費日志,充分利用分布式系統(tǒng)高效,容錯等特點,提供可擴展、高可用的計費服務(wù)。為了避免分布式系統(tǒng)內(nèi),單個結(jié)點壓力過大而導(dǎo)致整個任務(wù)變慢的情況,提出對大規(guī)模數(shù)據(jù)進(jìn)行分片的解決方案,使得每個分片內(nèi)的數(shù)據(jù)量都在一個合理范圍內(nèi),數(shù)據(jù)可以平均分布到各個結(jié)點上。為了解決網(wǎng)絡(luò)訪問中的性能瓶頸,通過異步接口提升系統(tǒng)性能。由于系統(tǒng)中處理的數(shù)據(jù)都跟錢有關(guān),系統(tǒng)出現(xiàn)故障將直接導(dǎo)致計費的損失。為了盡量減少損失、規(guī)避風(fēng)險,系統(tǒng)內(nèi)進(jìn)行了多項指標(biāo)的監(jiān)控,出現(xiàn)異�？梢约皶r告警。經(jīng)過測試和實際的線上運行,證明本系統(tǒng)可以對作弊廣告進(jìn)行有效過濾,每天處理億級的廣告數(shù)據(jù),而且系統(tǒng)的設(shè)計性能高于線上的平均負(fù)載流量,可以應(yīng)對短時間的數(shù)據(jù)尖峰。整個處理過程中,重要的數(shù)據(jù)指標(biāo)有監(jiān)控,關(guān)鍵操作有日志記錄,萬一出現(xiàn)異常方便排查問題。系統(tǒng)具有可擴展、可容錯、高可用的特點,很好地支持了廣告計費的需求,具有較高的實用價值。
[Abstract]:In recent years, the scale of the online advertising market has developed rapidly, and all the major Internet companies are laying out their own online advertising trading platforms. The billing system is important in the entire network advertising transaction process. According to the actual business requirements, this paper designs and implements a network advertising transaction billing system to support the billing requirements of advertising trading platform. This paper uses Java Scala language to develop an advertising billing system. The main content of the study is divided into anti-cheating and advertising billing. The anti-cheating part of advertising is used to determine the cheating ads, the system does not charge the cheating ads, so as to protect the interests of advertisers. In order to avoid the simple statistical method to judge the results too arbitrary, a scoring algorithm is proposed to calculate the likelihood of cheating. Advertising billing part based on spark implementation. Spark is a memory-based, extensible, fault-tolerant distributed computing framework. It processes advertising data in a distributed environment and filters cheating ads. Calculate deduction amount, generate deduction log, make full use of the characteristics of distributed system, such as high efficiency, fault tolerance, provide scalable and highly available billing services. When the pressure on a single node is too great to slow down the whole task, a solution is proposed to divide the large scale data into pieces, so that the amount of data in each slice is within a reasonable range. Data can be distributed evenly among nodes. In order to solve the performance bottleneck in network access, the asynchronous interface is used to improve the performance of the system. Because the data processed in the system is related to money, The failure of the system will directly lead to the loss of accounting. In order to minimize the loss and avoid the risk, the system has carried on the monitoring of many indexes, and the abnormal can be alerted in time. It is proved that the system can filter the cheating advertisement effectively, deal with the ad data of 100 million level every day, and the design performance of the system is higher than the average load flow on the line, which can deal with the data spike of short time. The important data index has the monitoring, the key operation has the log record, in case of the unusual convenient checking problem. The system has the characteristics of expandability, fault-tolerance, high availability, which supports the demand of advertisement charging well, and has high practical value.
【學(xué)位授予單位】：哈爾濱工業(yè)大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP311.52
，

本文編號：1647227