Using SNMP for connection metrics

[Joseph T. Duncan]

hi,

Presently I collect CPU, Memory in USE, and Network traffic statics from 
my windows terminal server "real servers" via snmp. I toss this 
information into an rrd database for makeing pretty graphs along with 
usage parsed from lvs stats. Finaly I take the CPU and Memory stats and 
use them to adjust my weight tables. My script for doing this is still in 
its infancy as I am getting stuff ready for this fall term, but it should 
be fun to see how it all works out.

the script is in its infancy. I still need to finish up lots in it, and 
more refined decision makeing. Add more output. Add in keepalive stuff 
before snmp stuff so snmp stuff doesnt barf. etc etc etc its just a rough 
example ;)

-joseph
_________________________________________________________________________
Info:                           Email:
Joseph T. Duncan                duncan at engr.orst.edu
Student Computing Facilities
109 Kidder Hall
Corvalis Or 97333


On Mon, 21 Aug 2006, Mike Green wrote:

> Hello -
>
> I've seen many projects in which SNMP metrics have been used as a heartbeat,
> but I was wondering if there was any implementation in which information
> such as CPU/memory usage on the realserver was used as a decision factor for
> load balancing.
>
> I know this may be overreaching for something kernel-integrated, but wanted
> to know if there were any LVS-associated projects that have tried this.
>
> Thanks much in advance!
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users at LinuxVirtualServer.org
> Send requests to lvs-users-request at LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>

-------------- next part --------------
#!/usr/bin/perl -w strict

#skelitons
# rrd for loadbalancer
	#rrdtool create loadblancer.rrd --start N --step 60 DS:Proc_0:GAUGE:120:-1:101 DS:Proc_1:GAUGE:120:-1:101 DS:avg_cpu_load:GAUGE:120:-1:101 DS:num_users:GAUGE:120:-1:256 DS:free_mem:GAUGE:120:-1:8193 DS:inet_recieved:GAUGE:120:-1:1000000000 DS:inet_sent:GAUGE:120:0:1000000000 RRA:AVERAGE:0.5:1:262800 RRA:AVERAGE:0.5:6:43800 RRA:AVERAGE:0.5:24:109050 RRA:AVERAGE:0.5:288:913
# rrd for ts boxes
	#rrdtool create realserverX.rrd --start N --step 60 DS:Proc_0:GAUGE:120:-1:101 DS:Proc_1:GAUGE:120:-1:101 DS:Proc_2:GAUGE:120:-1:101 DS:Proc_3:GAUGE:120:-1:101  DS:avg_cpu_load:GAUGE:120:-1:101 DS:num_users:GAUGE:120:-1:256 DS:free_mem:GAUGE:120:-1:8193 DS:inet_recieved:GAUGE:120:-1:1000000000  DS:inet_sent:GAUGE:120:0:1000000000 RRA:AVERAGE:0.5:1:262800 RRA:AVERAGE:0.5:6:43800 RRA:AVERAGE:0.5:24:109050 RRA:AVERAGE:0.5:288:913

#state veriables used for weight calcs
my $load_balancer = "hostname";
my $load_balancer_ip = "111.222.333.1";
my $num_realservers = 3; #number of active real servers
#ts1,ts2,ts3 (currently XXX,YYY,ZZZ)
my @host = ("XXX.foo.com", "YYY.foo.com", "ZZZ.foo.com"); #hosts domain name
my @host_ip = ("111.222.333.10", "111.222.333.11", "111.222.333.12"); #ip of each host
my @new_weight = (0, 0, 0); #new weight values 
my @avg_cpu = (0, 0, 0);
my @avg_free_mem = (0, 0, 0); 
my @total_con = (0, 0, 0);
my $user_total = 0;
my $community = "mycomunity";
my $rrd_dir = "/usr/local/stats/";

#collect Loadbalancer stats (local stats)
#cpu stats
my @cpustat = split(/\n/, `/usr/bin/mpstat -P ALL`);
my @cpu_avg = split(/\s+/, $cpustat[3]);
my @cpu_one = split(/\s+/, $cpustat[4]);
my @cpu_two = split(/\s+/, $cpustat[5]);

#collect memory stats:
my @memstat = split(/\n/, `/bin/cat /proc/meminfo`);
my @mem_real_total = split(/\s+/, $memstat[0]);
my @mem_real_free  = split(/\s+/, $memstat[1]);
my @mem_swap_total = split(/\s+/, $memstat[11]);
my @mem_swap_free  = split(/\s+/, $memstat[12]);

#calculate users thru system and total, grab wweights for adjustment
my @ip_vs_output = split(/ |\n/, `cat /proc/net/ip_vs`);
my @current_weight = ($ip_vs_output[37],$ip_vs_output[75],$ip_vs_output[113]);
my @active_con = ($ip_vs_output[43],$ip_vs_output[81],$ip_vs_output[119]);
my @inactive_con = ($ip_vs_output[53],$ip_vs_output[91],$ip_vs_output[129]);
for($index = 0; $index < $num_realservers; $index++){
	$total_con[$index] = $active_con[$index] + $inactive_con[$index];
	$user_total += $active_con[$index] + $inactive_con[$index];
	}

#add code to collect loadbalancer network traffic to be added later ;p

my $cpu_a = $cpu_one[2] + $cpu_one[3];
my $cpu_b = $cpu_two[2] + $cpu_two[3];
my $cpu_av = $cpu_avg[2] + $cpu_avg[3];
my $mem = $mem_real_total[1] - $mem_real_free[1];

`rrdtool update $rrd_dir$load_balancer.rrd "N:$cpu_a:$cpu_b:$cpu_av:$mem:$user_total:0:0"`;

#collect terminal server N stats (remote stats)
for($index = 0; $index < 2; $index++){
	#add code to test if server is up: if up flag up continue with tests, if down flag incrment? if down 5min+ page joseph, skip rest
	#get cpu stats from server
	my @cpu_0 = split(/ |\n/, `snmpget -c $community -v 2c $host_ip[$index] HOST-RESOURCES-MIB::hrProcessorLoad.5`); 
	my @cpu_1 = split(/ |\n/, `snmpget -c $community -v 2c $host_ip[$index] HOST-RESOURCES-MIB::hrProcessorLoad.2`); 
	my @cpu_2 = split(/ |\n/, `snmpget -c $community -v 2c $host_ip[$index] HOST-RESOURCES-MIB::hrProcessorLoad.3`); 
	my @cpu_3 = split(/ |\n/, `snmpget -c $community -v 2c $host_ip[$index] HOST-RESOURCES-MIB::hrProcessorLoad.4`); 
	#calculate avg
	$avg_cpu[$index] = ($cpu_0[3] + $cpu_1[3] + $cpu_2[3] + $cpu_3[3])*(1/4);
	#get free memory
	my @free_mem = split(/ |\n/, `snmpget -c $community -v 2c $host_ip[$index] 1.3.6.1.4.1.9600.1.1.2.3.0`);
	$avg_free_mem[$index] = $free_mem[3];
	#get network traffic in/out
	my @inet_recieved = split(/ |\n/, `snmpwalk -c $community -v 2c $host_ip[$index] 1.3.6.1.4.1.9600.1.1.3.1.2`);
	my @inet_sent = split(/ |\n/, `snmpwalk -c $community -v 2c $host_ip[$index] 1.3.6.1.4.1.9600.1.1.3.1.3`);
	`rrdtool update $rrd_dir$host[$index].rrd "N:$cpu_0[3]:$cpu_1[3]:$cpu_2[3]:$cpu_3[3]:$avg_cpu[$index]:$free_mem[3]:$total_con[$index]:$inet_recieved[11]:$inet_sent[11]"`;
}

#IPVSADM weight table adjustments!
for($index = 0; $index < 2; $index++){
	#calculate potential weight change
	if($avg_cpu[$index] >= 75 && $avg_free_mem[$index] <= 512){
		$new_weight[$index] = 1;
	}
	elsif($avg_cpu[$index] >= 75 && $avg_free_mem[$index] > 512){
		$new_weight[$index] = 9;
	}
	elsif($avg_cpu[$index] < 75 && $avg_free_mem[$index] <= 512){
		$new_weight[$index] = 9;
	}
	else{
		$new_weight[$index] = 27;
	}

	#make weight change
	if($current_weight[$index] != 0 || $current_weight[$index] != $new_weight[$index]){
		`/sbin/ipvsadm -e -t 111.222.333.1:3389 -r $host_ip[$index]:3389 -w $new_weight[$index]`;		
	}
}

#add output to /var/www/htdocs for state information on which picture set to display