First the problem:
Some Linux users
like to run servers and or services. You could be running Samba,
MySQL, Web Servers and maybe even a Chat Server. You could be
running these all on your desktop, but that wastes electricity and
uses up your desktop when you may want to use it for other purposes.
A primary desktop would need to be up all the time, and many desktops
have a high end video card which uses as much electricity as the rest
of the computer combined. Why not run them all on one or more
headless servers? This could be one big processor running multiple
virtual machines or on single less powerful computers (maybe even
“Raspberry Pi”s) each supporting a specific service. A headless
i3 could come in between 60 -70 watts, while a gaming rig can use 400
watts or more.
But how do you
monitor them? You can download and install Nagios or something like
that, but it could be more fun to write your own. You can design the
GUI app to fit your needs, not what someone says you need.
Maybe you want the app to sit in a little corner and not use up much
space, or maybe you want it large on one computer all by itself.
Second, what
language (Compiler or Interpreter) are you going to use.
There are many
choices and in reality you can make almost any of them work, so it is
going to come down to your personal choice. Okay, I like Lazarus,
which is a GUI front end to the Free Pascal Compiler. It is a true
compiler and generates relatively fast code. Lazarus will run
circles around Python performance-wise, but in truth c++ will
outperform Lazarus. Pascal is easier to write then c++ and you can
have a working app in much less time. In performance tests I have
seen, Free Pascal generated programs that run on average 2.5 times
longer then an equal program written in c++, but Python will run
about 6 times longer then an equivalent Free Pascal program. Your
results may vary, in the below charts, Python took 50 times longer to
run then Free Pascal, and the differences between Free Pascal and c
was marginal. On the learning curve, Python is the easiest to learn
(but not by much), followed by Pascal, and the hardest to learn is
c++.
If you think that
this charts is irrelevant to computing today. Sorting is a fairly
good way to compare performance of compilers or interpreters and the
source code is freely available on the internet. And lastly, I have
an application that had a Procedure that updated a MySQL Table with a
sort value that would change on a daily basis, and then sort the
output. It had a performance issue due to the fact that MySQL
updates data slowly, but can select it very quickly. I rewrote the
application to load the table into RAM, update it there and then run
an internal sort. This Procedure went from running minutes to
milliseconds. There may come a point where it will become necessary
to run a sort on data inside your program.
The Algorithm.
I knew that I wanted
to start ssh processes into each server and then process the returned
data. When you put the command right after the address to connect
to, the session will terminate after the command is run. I have
written previous programs that have started background processes that
just returned the output, which could then be processed by the main
application. This is relatively easy to do; you need to add Process
to the uses clause and declare a TProcess and a TStringList Variable.
uses
Classes,
SysUtils, Process;
Var
MyProcess: TProcesss;
MyStrList: TStringList;
And then the
following needs added at the point where you want to start the
process:
MyStrList
:= TStringList.Create;
MyProcess
:= TProcess.Create(nil);
MyProcess.CommandLine
:= 'ValidLinuxCommand';
MyProcess.Options
:= MyProcess.Options + [poWaitOnExit, poUsePipes];
MyProcess.Execute;
MyStrList.LoadFromStream(MyProcess.Output);
{
Do something with the String List Here }
MyStrList.Free;
MyProcess.Free;
That isn't really
all that hard, and it will work as long as the output is smaller then
about 2k. If it is going to be larger, we need to add more
complexity. This is explained in detail at:
http://wiki.lazarus.freepascal.org/Executing_External_Programs
Initially I was
thinking it would need to be a multi-threaded app, and I started
researching that option. I was afraid one stuck process could freeze
the whole application. The more I looked at that option, however,
the more complex it sounded. Each thread could not update graphical
components within the main app (that can crash the Window Manager),
so they would have to update basic objects (Integers, Strings, floats
and such) within the main thread and then have the main thread update
the component objects and then there is added complexity of needing
to use Critical Sections (this would prevent more than one thread
from accessing shared object at the same time).
I then thought, well
maybe I can start each subprocess and not have to wait on it
finishing, and then just create a loop that tests each subprocess's
state. If the subprocess has finished, process its output and then
just start a replacement subprocess. No messing around with all of
the multi-thread stuff.
So first of all I
declared a simple record:
TProcessRec
= Record
Process:
TProcess;
Addr,
User,
Password:
String;
Pct:
Double;
StrList:
TStringList;
End;
And then declared an
Array of that Record:
ProcessTable:
Array[1..KKArraySize]
of TProcessRec;
Now we want to set
up the main loop, but first we want to prime the pump and start all
the ssh processes needed:
For
i := 1 to KKArraySize Do
Begin
If
Length(ProcessTable[i].Addr) > 1 Then
Begin
ProcessTable[i].Process :=
TProcess.Create(nil);
ProcessTable[i].StrList:=TStringList.Create;
ProcessTable[i].Process.CommandLine
:= CmdLine(i);
ProcessTable[i].Process.Options := ProcessTable[1].Process.Options +
[poUsePipes];
ProcessTable[i].Process.Execute;
end;
end;
I am just looping
through my Array and checking to see if there is an address defined,
if so start a ssh. Note, I removed the poWaitOnExit from
ProcessOptions because we don't want to stop and wait for each
process to finish. Instead, we will test the subprocess in the next
step. Also note that I am calling a function CmdLine to build the
Linux command I want to run. Below is that function:
Function
CmdLine(i: Integer): String;
Begin
CmdLine
:= 'sshpass -p "' + ProcessTable[i].Password +
'" ssh ' +
ProcessTable[i].User + '@' +
ProcessTable[i].Addr + ' sar -u 1
1;free';
end;
This is pretty
simple, it just uses the Array to fill in user, Password and Address.
As you can also note, I am using sshpass to pass the password to
ssh and am also using the sar utility. sshpass can be found
in most repositories and sar is part of the sysstat monitoring
utilities which also can be found in most repositories. I found sar
gave cpu utilization numbers very close to what top gives, so
that is why I ended up using it. sar or whatever utility you
end up using will have to be installed on all of the servers you are
monitoring.
Now finally the main
loop:
EndTime
:= Now + EncodeTime(0, 0, 5, 0); // 5 Seconds
While
Now < EndTime do
Begin
Application.ProcessMessages;
Sleep(10);
End;
For
i:=1 to KKArraySize Do
Begin
If
Length(ProcessTable[i].Addr) > 1 Then
Begin
If
ProcessTable[i].Process.Running = False then
Begin
ProcessTable[i].StrList.LoadFromStream
(ProcessTable[i].Process.Output);
For
j:=0 to ProcessTable[i].StrList.Count-1 Do
Begin
If
copy(ProcessTable[i].StrList[j],1,7)
= 'Average'
then
Begin
Sum :=
SumProcessorStats
(ProcessTable[i].StrList[j]);
PctStr := Format('%5.2f', [Sum]);
ProcessTable[i].Pct := Sum;
End;
End;
BladeSG.Cells[i,2] :=
Format('%5.2f',
[ProcessTable[i].Pct]);
ProcessTable[i].StrList.Clear;
ProcessTable[i].Process.CommandLine
:= CmdLine(i);
ProcessTable[i].Process.Options :=
ProcessTable[1].Process.Options +
[poUsePipes];
ProcessTable[i].Process.Execute;
End;
End;
End;
Well, that is a
mouthful! This is all in a Repeat ... Forever Loop, well, almost
a forever loop because I do have a way to break it. The first While
Loop, is just a means to get the App to wait 5 seconds before
checking on the status of all of our background processes. The first
two If statements seem silly and seem like they should be combined,
but ... Since I only initialized a Process in the array where the
Addr field had actual data in it, if we ran the “Process.Running”
statement for an uninitialized Process variable, the app would have
had a memory error and would have crashed.
ProcessTable[i].StrList.LoadFromStream
(ProcessTable[i].Process.Output);
―— just
loads the output from our ssh into our TStringList, which was
declared in our Array. A TStringList is actually just an array of
Strings and the output of the sar utility has the word Average
in it. So the following block of code just sums up the output from
the sar utility. I summed up user, nice, and system values
listed from the sar utility. You may decide on a different strategy.
For j:=0 to ProcessTable[i].StrList.Count-1 Do
Begin
If
copy(ProcessTable[i].StrList[j],1,7) = 'Average'
then
Begin
Sum := SumProcessorStats
(ProcessTable[i].StrList[j]);
PctStr := Format('%5.2f', [Sum]);
ProcessTable[i].Pct := Sum;
End;
End;
SumProcessorStats is
a function and it does the adding of the sar outputed fields
together.
Function
SumProcessorStats(s: String): Double;
Var
UsrStr,
NiceStr, SysStr: String;
Sum,
Dbl: Double;
begin
UsrStr
:= copy(s,24,6);
NiceStr
:= copy(s,34,6);
SysStr
:= copy(s,44,6);
Sum
:= 0.0;
Dbl
:= StrToFloat(UsrStr);
Sum
:= Sum + Dbl;
Dbl
:= StrToFloat(NiceStr);
Sum
:= Sum + Dbl;
Dbl
:= StrToFloat(SysStr);
Sum
:= Sum + Dbl;
SumProcessorStats := Sum;
end;
Finally, since the
previous subprocess had finished, this last piece starts a new
subprocess:
ProcessTable[i].StrList.Clear;
ProcessTable[i].Process.CommandLine
:= CmdLine(i);
ProcessTable[i].Process.Options
:= ProcessTable[1].Process.Options +
[poUsePipes];
ProcessTable[i].Process.Execute;
Security is an issue
here. The addresses, users and passwords could be hard-coded into
your App. If somebody broke into your system though, they could get
all of your passwords by looking at your source code. An improvement
would be to have all that info in MySQL Table and have the Operator
prompted for the password to access that table. Even better would be
to have the data all encrypted and have the Operator prompted for a
decryption key.
At some point, we
may want to start sending e-mails out when certain conditions occur.
Like if one of the servers is maxed out for several minutes, or if a
server becomes unresponsive. Sending out e-mails isn't that hard to
add. There is a library that can be downloaded for free called
Synapse (http://wiki.freepascal.org/Synapse). You would need to
download the library and add the following libraries to uses clause:
synautil, synacode,
blcksock, pop3send, smtpsend
I copied the whole
Synapse Library into the directory containing my app, and then
whatever Lazarus needed, it would compile. The content of the e-mail
would need to be added to a tstringlist and then to send the e-mail,
use something similar to the following:
If
Not SendTo('ToEmailAddress', 'FromEmailAddress', 'Subject',
'EmailServer', MsgStringList) Then
ShowMessage('E-Mail Didn''t Work');
ToEmailAddress could
be your gmail address on your phone, FromEmailAddress would be your
regular e-mail address, EmailServer will be your e-mail (Probably
something like mail.myservername.com). There is more information on
the Wiki Site listed above.
Performance
On an Intel i5, the
main thread uses about 6% of the CPU on one core. Very acceptable.
