應(yīng)對(duì)CC攻擊的自動(dòng)防御系統(tǒng)的原理與實(shí)現(xiàn)
0x00 系統(tǒng)效果
此DDOS應(yīng)用層防御系統(tǒng)已經(jīng)部署在了http://www.yfdc.org網(wǎng)站上(如果訪問(wèn)失敗,請(qǐng)直接訪問(wèn)位于國(guó)內(nèi)的服務(wù)器http://121.42.45.55進(jìn)行在線測(cè)試)。
此防御系統(tǒng)位于應(yīng)用層,可以有效防止非法用戶對(duì)服務(wù)器資源的濫用:
只要是發(fā)送高頻率地、應(yīng)用層請(qǐng)求以實(shí)現(xiàn)大量消耗系統(tǒng)資源的攻擊方式,皆可有效防御。
其實(shí)現(xiàn)的基本思想是:
定期分析所有訪問(wèn)用戶在過(guò)去各個(gè)時(shí)間段內(nèi)的請(qǐng)求頻率,將頻率高于指定閾值的用戶判定為資源濫用者,將其封殺一段時(shí)間,時(shí)效過(guò)后,防御系統(tǒng)自動(dòng)將其解封。
在線效果測(cè)試:
進(jìn)入http://www.yfdc.org -> 點(diǎn)擊右上側(cè)在線查詢,此時(shí)將會(huì)進(jìn)入/shell/yf域內(nèi),/shell/yf是一個(gè)用bash scripts寫(xiě)的動(dòng)態(tài)web-cgi程序,用戶每一次提交信息,此程序?qū)?huì)執(zhí)行一些服務(wù)器端的查詢操作,然后將數(shù)據(jù)處理后返回到客戶端。
為了防止非法用戶高頻率地訪問(wèn)這個(gè)程序,而影響到其他正常用戶的訪問(wèn),故需要進(jìn)行一些保護(hù)措施。
最終效果:

被封信息頁(yè)面
在/shell/yf域內(nèi),按住F5不放,一直刷新,幾秒后松開(kāi),就能看到被封信息和解封時(shí)間。
只要某個(gè)用戶對(duì)/shell/yf的訪問(wèn)超過(guò)了正常的頻率,服務(wù)將會(huì)對(duì)這個(gè)用戶關(guān)閉一段時(shí)間,期滿后自動(dòng)解封。
0x01 系統(tǒng)原理
操作系統(tǒng): CentOS 6.5 x86_64 開(kāi)發(fā)語(yǔ)言: Bash Shell Scripts Web服務(wù)器: Apache Httpd

(此圖為系統(tǒng)結(jié)構(gòu)的鳥(niǎo)瞰圖 可存至本地后放大查看)
2.1 自定義日志:/etc/httpd/logs/yfddos_log

(自定義日志文件的格式)
在httpd.conf的日志參數(shù)中,加入如下兩行:
LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddos
CustomLog logs/yfddos_log yfddos
我們接下來(lái)重點(diǎn)分析日志文件/etc/httpd/logs/yfddos_log.
LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddos
解釋:
%a -> 用戶的IP %U -> 請(qǐng)求的URL地址,但并不包含query string(The URL path requested, not including any query string.) %{local}p -> 用戶請(qǐng)求的服務(wù)器端口(一般為80) %D -> 這個(gè)請(qǐng)求共消耗了服務(wù)器多少微秒(The time taken to serve the request, in microseconds.) %{%s}t -> 服務(wù)器收到這個(gè)請(qǐng)求時(shí),時(shí)間戳的值(seconds since 1970-01-01 00:00:00 UTC)
例子:
192.168.31.1 "/shell/yf" 80 118231 1417164313
譯為:IP為192.168.31.1的主機(jī),在時(shí)間戳為1417164313的時(shí)候,訪問(wèn)了/shell/yf,并由服務(wù)器的80端口向其提供服務(wù),共耗時(shí)118231微秒
或?yàn)?IP為192.168.31.1的主機(jī),在2014-11-28 16:45:13的時(shí)候,訪問(wèn)了/shell/yf,并由服務(wù)器的80端口向其提供服務(wù),共耗時(shí)0.118231秒
至于為什么不使用httpd.conf中官方定義的日志,原因如下:
- 用戶訪問(wèn)日志的一條記錄可大約控制在60Bytes以內(nèi),數(shù)據(jù)量小,便于后期分析,官方定義的日志太過(guò)臃腫,影響分析速度 - 使用時(shí)間戳標(biāo)志時(shí)間,便于后期分析,官方定義的日志時(shí)間參數(shù)為常規(guī)的表達(dá)方式,不便于直接進(jìn)行處理 - httpd的日志系統(tǒng)本身就是從舊到新進(jìn)行排序記錄的,所以/etc/httpd/logs/yfddos_log日志條目的時(shí)間戳,亦為從小到大進(jìn)行排序的,數(shù)據(jù)記錄更加鮮明
2.2 yfddosd黑名單文件格式

黑名單文件格式
yfddosd黑名單文件/etc/yfddos/web-yf-search.b格式如下:
# ip add-stamp rmv-stamp 1.2.3.4 1416046335 1416046395 1.2.3.5 1416046336 1416046396 1.2.3.6 1416046339 1416046399
每一行為一個(gè)黑名單條目,上面第一個(gè)條目的意義為:
IP地址 :1.2.3.4
開(kāi)始時(shí)間:時(shí)間戳1416046335,即 2014-11-15 18:12:15
終止時(shí)間:時(shí)間戳1416046395,即 2014-11-15 18:13:15
直觀意義為:
IP地址:1.2.3.4,從2014-11-15 18:12:15開(kāi)始,被封殺1分鐘,在2014-11-15 18:13:15時(shí)自動(dòng)解封。
這個(gè)文件將由駐留在系統(tǒng)中的daemon守護(hù)進(jìn)程yfddosd進(jìn)行維護(hù)更新。
2.3 守護(hù)進(jìn)程yfddosd:防御系統(tǒng)的邏輯核心

守護(hù)進(jìn)程的原理圖
守護(hù)進(jìn)程yfddosd是整個(gè)CC防御系統(tǒng)的核心,而function analyze_and_insert_black()則是yfddosd的核心。
yfddosd的配置參數(shù):
yfddos_blackfilePath='/etc/yfddos/web-yf-search.b'
yfddos_accesslogPath='/etc/httpd/logs/yfddos_log'function analyze_and_insert_black() {
# analyze_and_insert_black() :
# $1:max frequency(seems as abuse if above that) $2:blackip-ttl,time to live,unit is seconds (s)
# $3:the access log ${3} seconds before will be analyzed to generate the abuse ip lists that we will block
# example : analyze_and_insert_black "limit" "ttl" "time"
# example : analyze_and_insert_black "4" "10" "5"
# 分析在過(guò)去5s內(nèi)的用戶訪問(wèn)日志 如果有人在這5s內(nèi)訪問(wèn)量>=4 系統(tǒng)將視其為資源濫用者 將其加入服務(wù)黑名單
# 一條黑名單的作用時(shí)間為10s 即在10s之后 系統(tǒng)自動(dòng)刪除此黑名單條目 服務(wù)則繼續(xù)向其開(kāi)放
# global vars:
# stamp logtmpfile yfddos_blackfilePath
# ......
}
函數(shù)analyze_and_insert_black有三個(gè)輸入?yún)?shù):
例子: analyze_and_insert_black "4" "10" "5"
解釋: 分析日志文件/etc/httpd/logs/yfddos_log中,在過(guò)去5s內(nèi)的用戶訪問(wèn)日志,如果有IP在這5s內(nèi)訪問(wèn)量>=4,守護(hù)進(jìn)程yfddosd將視其為資源濫用者,然后將這個(gè)IP加入到黑名單文件/etc/yfddos/web-yf-search.b中,此條黑名單的作用時(shí)間為10s,在10s之后,守護(hù)進(jìn)程yfddosd將刪除此黑名單條目。 |
例子: analyze_and_insert_black "150" "2700" "905"
解釋: 分析日志文件/etc/httpd/logs/yfddos_log中,在過(guò)去905s內(nèi)的用戶訪問(wèn)日志,如果有IP在這905s內(nèi)訪問(wèn)量>=150,守護(hù)進(jìn)程yfddosd將視其為資源濫用者,然后將這個(gè)IP加入到黑名單文件/etc/yfddos/web-yf-search.b中,此條黑名單的作用時(shí)間為2700s,在2700s之后,守護(hù)進(jìn)程yfddosd將刪除此黑名單條目。 |
簡(jiǎn)記為: analyze_and_insert_black "limit" "ttl" "time"
解釋: 分析日志文件/etc/httpd/logs/yfddos_log中,在過(guò)去(time)s內(nèi)的用戶訪問(wèn)日志,如果有IP在這(time)s內(nèi)訪問(wèn)量>=limit,守護(hù)進(jìn)程yfddosd將視其為資源濫用者,然后此IP將會(huì)被加入到黑名單文件/etc/yfddos/web-yf-search.b中,作用時(shí)間為(ttl)s,在(ttl)s之后,守護(hù)進(jìn)程yfddosd將自動(dòng)刪除此條目。 |
從上述中可看出,守護(hù)進(jìn)程yfddosd至少需要完成如下三個(gè)任務(wù):
◆分析日志文件/etc/httpd/logs/yfddos_log中指定時(shí)間內(nèi)的用戶訪問(wèn)記錄
◆將資源濫用者的IP加入文件/etc/yfddos/web-yf-search.b,并設(shè)置封殺TTL參數(shù)值
◆將/etc/yfddos/web-yf-search.b中已經(jīng)過(guò)期的條目全部及時(shí)刪除
守護(hù)進(jìn)程yfddosd是如何實(shí)現(xiàn)上面三個(gè)邏輯的:
◆分析日志文件/etc/httpd/logs/yfddos_log中指定時(shí)間內(nèi)的用戶訪問(wèn)記錄:
(1) 取出/etc/httpd/logs/yfddos_log中過(guò)去time秒的訪問(wèn)日志數(shù)據(jù),使用二分法將這一操作的時(shí)間復(fù)雜度壓縮到K*log2(N)以內(nèi),其中N為/etc/httpd/logs/yfddos_log中日志總行數(shù),K為一次測(cè)試的耗時(shí)量,一般為1ms以內(nèi),即如有1048576條訪問(wèn)記錄,這一操作將僅需要20*1ms。
(2) 使用正則RE對(duì)這些數(shù)據(jù)進(jìn)行二次處理,過(guò)濾出所有訪問(wèn)指定URL的用戶IP(這個(gè)URL為想要防御的http服務(wù)url,例如在http://www.yfdc.org系統(tǒng)中,所防御的就是/shell/yf,這個(gè)服務(wù)向訪問(wèn)者提供信息的search與get服務(wù)),再次使用sort與uniq對(duì)這些IP進(jìn)行處理,以統(tǒng)計(jì)出每個(gè)IP的訪問(wèn)次數(shù)并進(jìn)行高低排序。
◆將資源濫用者的IP加入文件/etc/yfddos/web-yf-search.b,并設(shè)置封殺TTL參數(shù)值
將所有訪問(wèn)次數(shù)超過(guò)閾值limit的IP更新到黑名單文件/etc/yfddos/web-yf-search.b中,每個(gè)黑名單條目的封殺時(shí)間為ttl秒
◆將/etc/yfddos/web-yf-search.b中已經(jīng)過(guò)期的條目全部及時(shí)刪除
遍歷/etc/yfddos/web-yf-search.b中所有黑名單條目,結(jié)合當(dāng)前時(shí)間戳,將所有已經(jīng)過(guò)期的條目一一刪去
下面是守護(hù)進(jìn)程yfddosd狀態(tài)機(jī)的偽代碼:(略去了一些處理細(xì)節(jié))
#init and FSM start work...
counter=0
while true
do
sleep 5
counter=counter+1
delete obsolete items #將/etc/yfddos/web-yf-search.b中已經(jīng)過(guò)期的條目全部刪除
if # every 5 seconds : 5s
then
analyze_and_insert_black "6" "10" "5"
# 分析在過(guò)去5s內(nèi)訪問(wèn)的用戶 如果有人其訪問(wèn)量大于等于6 系統(tǒng)將視其為資源濫用者
# 遂將其加入服務(wù)黑名單 其作用時(shí)間為10s 在10s之后 daemon進(jìn)程自動(dòng)刪除這個(gè)ip黑名單條目
fi
if #every 5*3 seconds : 15s
then
analyze_and_insert_black "14" "45" "15"
fi
if #every 5*3*4+5 seconds : 65s
then
analyze_and_insert_black "40" "840" "65"
fi
if #every 5*3*4*3*5+5 seconds : 905s : 15min
then
analyze_and_insert_black "150" "2700" "905"
fi
if #every 5*3*4*3*5*4+5 seconds : 3605s : 1h
then
analyze_and_insert_black "300" "7200" "3605"
fi
if #every 5*3*4*3*5*4*3+5 seconds : 10805s : 3h
then
analyze_and_insert_black "400" "21600" "10805"
if #在每天的00:01-04:59時(shí)間區(qū)間 一天僅執(zhí)行一次
then
#備份日志
fi
fi
done
防御者應(yīng)斟酌調(diào)整每個(gè)檢測(cè)時(shí)間點(diǎn)的參數(shù)值(封殺時(shí)間ttl與判定閾值limit),以調(diào)節(jié)系統(tǒng)應(yīng)對(duì)CC攻擊到來(lái)時(shí)的反應(yīng)時(shí)間。
0x02 源代碼
- ##################################### vim /usr/local/bin/yfddosd.sh :
- ##################################### nohup bash /usr/local/bin/yfddosd.sh &>"/etc/yfddos/""yfddosd-log-`date +%Y-%m-%d`" &
- ##################################### yfddos daemon
- mkdir /etc/yfddos
- yfddos_blackfilePath='/etc/yfddos/web-yf-search.b'
- yfddos_accesslogPath='/etc/httpd/logs/yfddos_log'
- ### refresh tll
- logtmpfile=`mktemp`
- stamp=`date +%s`
- touch "$yfddos_blackfilePath"
- if grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- cat "$yfddos_blackfilePath" | while read i
- do
- deadstamp=`echo "$i" | grep -Po '[0-9]+$'`
- if [ "$stamp" -le "$deadstamp" ]
- then
- echo "$i" >>"$logtmpfile"
- fi
- done
- fi
- chmod o+r "$logtmpfile"
- mv -f "$logtmpfile" "$yfddos_blackfilePath"
- if ! grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- echo '255.255.255.255 0 0' >> "$yfddos_blackfilePath"
- fi
- function analyze_and_insert_black() {
- # analyze_and_insert_black() :
- # $1:max frequency(seems as abuse if above that) $2:blackip-ttl,time to live,unit is seconds (s)
- # $3:the access log ${3} seconds before will be analyzed to generate the abuse ip lists that we will block
- # example : analyze_and_insert_black "limit" "ttl" "time"
- # example : analyze_and_insert_black "4" "10" "5"
- # 分析在過(guò)去5s內(nèi)的用戶訪問(wèn)日志 如果有人在這5s內(nèi)訪問(wèn)量>=4 系統(tǒng)將視其為資源濫用者 將其加入服務(wù)黑名單
- # 一條黑名單的作用時(shí)間為10s 即在10s之后 系統(tǒng)自動(dòng)刪除此黑名單條目 服務(wù)則繼續(xù)向其開(kāi)放
- # global vars:
- # stamp logtmpfile yfddos_blackfilePath
- local threshold="$1"
- local ttl="$2"
- local stamp_pre="$3"
- local i=0
- local num=""
- local fre=0
- local ip=0
- local localbuf=0
- local linenum=0
- local deadstamp=0
- stamp_pre="$((stamp-stamp_pre))"
- #二分查找初始化
- local temp=0
- local yf_x='1'
- local yf_y=`cat "$logtmpfile" | wc -l`
- if [ "$yf_y" -le "1" ]
- then
- yf_y=1
- fi
- local yf_I=$(((yf_x+yf_y)/2))
- temp=`cat "$logtmpfile" | wc -l`
- if [ "$temp" -gt "0" ]
- then
- temp=`sed -n '$p' "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -lt "$stamp_pre" ]
- then
- num=""
- else
- while true #使用二分查找的方法 快速地分析訪問(wèn)日志
- do
- temp=`sed -n "${yf_x}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -ge "$stamp_pre" ]
- then
- break
- fi
- if [ "$((yf_y-yf_x))" -le "1" ]
- then
- yf_x="$yf_y"
- break
- fi
- temp=`sed -n "${yf_I}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -lt "$stamp_pre" ]
- then
- yf_x="$yf_I"
- yf_y="$yf_y"
- yf_I="$(((yf_x+yf_y)/2))"
- continue
- fi
- yf_x="$yf_x"
- yf_y="$yf_I"
- yf_I="$(((yf_x+yf_y)/2))"
- continue
- done
- temp=`sed -n "${yf_x}p" "$logtmpfile" | grep -Po '[0-9]+ $'`
- if [ "$temp" -ge "$stamp_pre" ]
- then
- num="$yf_x"
- else
- num=""
- fi
- fi
- if [ -n "$num" ]
- then
- sed -n "${num},\$p" "$logtmpfile" | grep -Po '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort -n | uniq -c | sort -rn | while read i
- do
- fre=`echo "$i" | grep -Po '[0-9]+' | head -1`
- ip=`echo "$i" | grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' `
- if [ "$fre" -ge "$threshold" ]
- then #insert illegal ips : cat "$yfddos_blackfilePath"
- # ip add-stamp rmv-stamp
- #1.2.3.4 1416046335 1416046395
- temp=`grep -Pn "${ip//./\\.} " "$yfddos_blackfilePath"`
- if [ -n "$temp" ]
- then
- linenum=`echo "$temp" | grep -Po '^[0-9]+' | head -1`
- deadstamp=`echo "$temp" | grep -Po '[0-9]+$' | sort -rn | head -1 `
- if [ "$((stamp+ttl))" -gt "$deadstamp" ]
- then
- sed -i "${linenum}s/.*/${ip} ${stamp} $((stamp+ttl))/g" "$yfddos_blackfilePath"
- fi
- else
- sed -i "\$a ${ip} ${stamp} $((stamp+ttl))" "$yfddos_blackfilePath"
- fi
- else
- break
- fi
- done
- fi
- fi
- }
- #init and yfddosd's FSM start work...
- counter=0
- while true
- do
- sleep 5
- counter=$((counter+1))
- echo -n `date +%Y-%m-%d\ %H:%M:%S`" ""counter ${counter}:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- echo -n "refresh tll:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- ### refresh tll
- #refresh ttl: analyze file: "$yfddos_blackfilePath" if some items'ttl has been reach the date , we will remove it and open service to the ip had been banned before.
- #insert illegal ips : cat "$yfddos_blackfilePath"
- # ip add-stamp rmv-stamp
- #1.2.3.4 1416046335 1416046395
- #sed -i "/^.* $((stamp-5))$/d;/^.* $((stamp-4))$/d;/^.* $((stamp-3))$/d;/^.* $((stamp-2))$/d;/^.* $((stamp-1))$/d;/^.* $((stamp))$/d;/^$/d" "$yfddos_blackfilePath"
- logtmpfile=`mktemp`
- stamp=`date +%s`
- touch "$yfddos_blackfilePath"
- if grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- cat "$yfddos_blackfilePath" | while read i
- do
- deadstamp=`echo "$i" | grep -Po '[0-9]+$'`
- if [ "$stamp" -le "$deadstamp" ]
- then
- echo "$i" >>"$logtmpfile"
- fi
- done
- fi
- chmod o+r "$logtmpfile"
- mv -f "$logtmpfile" "$yfddos_blackfilePath"
- if ! grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' "$yfddos_blackfilePath" &>/dev/null
- then
- echo '255.255.255.255 0 0' >> "$yfddos_blackfilePath"
- fi
- logtmpfile=`mktemp`
- stamp=`date +%s`
- cat "$yfddos_accesslogPath" | grep -P ' "/shell/yf" ' >"$logtmpfile"
- if true # every 5 seconds : 5s
- then
- echo -n "analyze_and_insert_black 6 10 5:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- #analyze yfddos log : analyze_and_insert_black() $1:max frequency(seems as abuse if above that) $2:blackip-ttl $3:the access log ${3} seconds before will be analyzed to generate the abuse ips that we will block
- analyze_and_insert_black "6" "10" "5"
- # 分析在過(guò)去5s內(nèi)訪問(wèn)的用戶 如果有人其訪問(wèn)量大于等于6 系統(tǒng)將視其為資源濫用者 遂將其加入服務(wù)黑名單 其作用時(shí)間為10s 在10s之后 daemon進(jìn)程自動(dòng)刪除這個(gè)ip黑名單條目
- fi
- if [ "$((counter%(3)))" -eq "0" ] #every 5*3 seconds : 15s
- then
- echo -n "analyze_and_insert_black 14 45 15:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "10" "45" "15"
- fi
- if [ "$((counter%(3*4+1)))" -eq "0" ] #every 5*3*4+5 seconds : 65s
- then
- echo -n "analyze_and_insert_black 40 840 65:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "25" "840" "65"
- fi
- if [ "$((counter%(3*4*3*5+1)))" -eq "0" ] #every 5*3*4*3*5+5 seconds : 905s : 15min
- then
- echo -n "analyze_and_insert_black 150 2700 905:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "150" "2700" "905"
- fi
- if [ "$((counter%(3*4*3*5*4+1)))" -eq "0" ] #every 5*3*4*3*5*4+5 seconds : 3605s : 1h
- then
- echo -n "analyze_and_insert_black 300 7200 3605:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "300" "7200" "3605"
- fi
- if [ "$((counter%(3*4*3*5*4*3+1)))" -eq "0" ] #every 5*3*4*3*5*4*3+5 seconds : 10805s : 3h
- then
- echo -n "analyze_and_insert_black 400 21600 10805:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- # example : analyze_and_insert_black "limit" "ttl" "time"
- analyze_and_insert_black "400" "21600" "10805"
- #### "${yfddos_accesslogPath}" backup : 在每天的00:01-04:59時(shí)間區(qū)間內(nèi) 備份日志一次
- if [ "`date +%H`" -le "5" ] && ! [ -f "${yfddos_accesslogPath}-`date +%Y-%m-%d`" ]
- then
- service httpd stop
- mv "${yfddos_accesslogPath}" "${yfddos_accesslogPath}-`date +%Y-%m-%d`"
- service httpd start
- fi
- fi
- rm -fr "$logtmpfile"
- echo "sleep:"`cat /proc/uptime | grep -Po '[0-9\.]+' | head -1`" "
- done