一次lvs10萬+并發的優化實例
作者：BootGo
在一次實際項目中，lvs 只能承載量很低，需要對lvs和Linux內核參數進行優化。
1 緣起
在一次實際項目中，lvs 只能承載量很低，需要對lvs和Linux內核參數進行優化。
2 為什么使用lvs+keepalived架構
（1）LVS可以實現負載均衡，但是不能夠進行健康檢查。比如一個RS出現故障，LVS 仍然會把請求轉發給故障的RS服務器，這樣就會導致請求的無效性；keepalived 軟件可以進行健康檢查。
（2）使用keepalived能同時實現 LVS 的高可用性，解決 LVS 單點故障的問題。
3 lvs+keepalived部署
3.1 部署圖
注意：
（1）lvs+keepalived至少需要2臺服務器。
（2）需要一個VIP。
（3）RS服務器不能和LVS以及Keepalived復用。
4.2 lvs+keepalived部署（2臺DS服務器）
- 安裝ipvs和keepalived
- yum install ipvsadm keepalived -y
- keepalived配置
注意：
（1）keepalived是否需要爭搶主IP，如果不需要，需要把state都修改為BACKUP，并配置nopreempt。
（2）persistence_timeout的作用是：在一定時間內使來自于同一個Client的所有TCP請求被負載到同一個RealServer上，查看ipvsadm -S -n
主節點
- #主節點( MASTER )配置文件
- cat > /etc/keepalived/keepalived.conf <<'EOF'
- ! Configuration File for keepalived
- global_defs {
- router_id LVS_DEVEL
- }
- vrrp_instance VI_1 {
- state BACKUP ! 主為master，不爭搶模式改為BACKUP
- nopreempt !不爭搶模式添加
- interface eth0
- virtual_router_id 51
- priority 100
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass 1111
- }
- virtual_ipaddress {
- 192.168.112.10
- }
- }
- virtual_server 192.168.112.10 80 {
- delay_loop 6
- lb_algo rr
- lb_kind DR
- ! persistence_timeout 0
- protocol TCP
- real_server 192.168.112.13 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 192.168.112.14 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 192.168.112.15 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- }
- EOF
從節點
- cat > /etc/keepalived/keepalived.conf <<'EOF'
- ! Configuration File for keepalived
- global_defs {
- router_id LVS_DEVEL
- }
- vrrp_instance VI_1 {
- state BACKUP
- nopreempt ! 不爭搶模式添加
- interface eth0
- virtual_router_id 51
- priority 90
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass 1111
- }
- virtual_ipaddress {
- 192.168.112.10
- }
- }
- virtual_server 192.168.112.10 80 {
- delay_loop 6
- lb_algo rr
- lb_kind DR
- ! persistence_timeout 0
- protocol TCP
- real_server 192.168.112.13 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 192.168.112.14 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 192.168.112.15 80 {
- weight 1
- TCP_CHECK {
- connect_timeout 10
- retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- }
- EOF
- 內核參數
- echo 1 > /proc/sys/net/ipv4/ip_forward
- sysctl -w net.ipv4.ip_forward=1
- 啟動
- systemctl enable keepalived
- systemctl start keepalived
4.3 真實服務器配置
- 配置腳本
不需要在lvs+keepalived的服務器上配置，需要在所有的真實服務器上配置。注意vip必須與前面keepalived上的vip相同。
- # vim lvs_dr_rs.sh
- SNS_VIP=192.168.112.10
- /etc/rc.d/init.d/functions
- case "$1" in
- start)
- ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
- /sbin/route add -host $SNS_VIP dev lo:0
- echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
- echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
- echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
- echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
- sysctl -p >/dev/null 2>&1
- echo "RealServer Start OK"
- ;;
- stop)
- ifconfig lo:0 down
- route del $SNS_VIP >/dev/null 2>&1
- echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
- echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
- echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
- echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
- echo "RealServer Stoped"
- ;;
- *)
- echo "Usage: $0 {start|stop}"
- exit 1
- esac
- exit 0
- 執行生效
- # 所有RS節點上分別執行腳本：
- chmod +x lvs_dr_rs.sh
- ./lvs_dr_rs.sh start
5 參數優化
5.1 LVS參數
- 增大ipvs模塊hash table的大小
ipvs模塊hash table默認值為2^12=4096，改為2^20=1048576。可以用ipvsadm -l命令查詢當前hash table的大小。
- IP Virtual Server version 1.2.1 (size=4096)
修改方法：
在/etc/modprobe.d/目錄下添加文件ip_vs.conf，內容為：
- options ip_vs conn_tab_bits=20
重新加載ipvs模塊。
- IP Virtual Server version 1.2.1 (size=1048576)
- 修改 LVS 表中的 timeout
- ipvsadm --set 900 60 300
- ipvsadm -ln --timeout
- Timeout (tcp tcpfin udp): 900 60 300
5.2 文件句柄及進程數
- * soft nofile 1024000
- * hard nofile 1024000
- * soft nproc 1024000
- * hard nproc 1024000
5.3 內核參數
- fs.file-max = 1048576
- net.ipv4.ip_forward = 1
- net.core.wmem_default = 8388608
- net.core.wmem_max = 16777216
- net.core.rmem_default = 8388608
- net.core.rmem_max = 16777216
- net.core.somaxconn = 65535
- net.core.optmem_max = 81920
- net.core.netdev_max_backlog = 262144
- net.ipv4.route.gc_timeout = 20
- net.ipv4.tcp_syncookies = 1
- net.ipv4.tcp_abort_on_overflow = 1
- net.ipv4.tcp_max_tw_buckets = 6000
- net.ipv4.tcp_sack = 1
- net.ipv4.tcp_window_scaling = 1
- net.ipv4.tcp_no_metrics_save = 1
- net.ipv4.tcp_rmem = 32768 131072 16777216
- net.ipv4.tcp_wmem = 8192 131072 16777216
- net.ipv4.tcp_mem = 94500000 915000000 927000000
- net.ipv4.tcp_max_syn_backlog = 262144
- net.ipv4.tcp_max_orphans = 3276800
- net.ipv4.tcp_timestamps = 0
- net.ipv4.tcp_synack_retries = 1
- net.ipv4.tcp_syn_retries = 1
- net.ipv4.tcp_tw_recycle = 1
- net.ipv4.tcp_tw_reuse = 1
- net.ipv4.tcp_fin_timeout = 10
- net.ipv4.tcp_keepalive_time = 120
- net.ipv4.tcp_keepalive_probes = 3
- net.ipv4.tcp_keepalive_intvl = 15
- net.ipv4.tcp_retries2 = 5
- net.ipv4.ip_local_port_range = 1024 65000
- net.ipv4.conf.default.rp_filter = 1
- net.ipv4.conf.default.accept_source_route = 0
- net.ipv4.conf.all.arp_ignore = 1
- net.ipv4.conf.all.arp_announce = 2
- #modprobe ip_conntrack
- net.netfilter.nf_conntrack_tcp_timeout_established = 180
- net.netfilter.nf_conntrack_max = 1048576
- net.nf_conntrack_max = 1048576
- kernel.sysrq = 0
- kernel.core_uses_pid = 1
- kernel.msgmnb = 65536
- kernel.msgmax = 65536
- kernel.shmmax = 68719476736
- kernel.shmall = 4294967296
注意：
net.ipv4.tcp_tw_recycle = 1有坑，在nat環境下慎用。越是大并發，越要注意net.ipv4.tcp_max_tw_buckets的值不能太大。
責任編輯：張燕妮
來源：
今日頭條
