Niko's Project Corner

There al­ready ex­ists many server mon­itor­ing and log­ging sys­tems, but I was in­ter­ested to de­velop and de­ploy my own. It was also a good chance to learn about Elas­tic­Search's ag­gre­ga­tion queries (new in v1.0.0). Orig­inally Elas­tic­Search was de­signed to provide scal­able doc­ument based stor­age and ef­fi­cient search, but now it is gain­ing more ca­pa­bil­ities. The pro­ject con­sists of a cron job which pushes new met­rics to Elas­tic­Search, a REST­ful JSON API to query statis­tics on recorded num­bers and plot the re­sults in a browser (based on High­Charts).

The cron job is sched­uled to pro­duce a new dump of com­puter's sen­sor data, disk S.M.A.R.T. data, tem­per­atures, disk us­age and so forth into a text file on the disk. Then it is parsed by PHP, con­verted into a "doc­ument" and stored into Elas­tic­Search. The Elas­tic­Search's "schema" is de­signed to sup­port log items from mul­ti­ple ma­chi­nes, mul­ti­ple disks / ma­chine and so forth.

Figure 1: Ag­gre­gated hard disk data (tem­per­ature, start-stop count and power on hours) at 5 min, 6 hours and 7 days in­ter­vals. For each data the min, av­er­age and max are cal­cu­lated.

Ex­am­ple graphs are shown in fig­ures 1 and 2. These are gen­er­ated by a static HTML5/JS file, and the UI gen­er­ation is fully driven by API's re­sponses. The ex­posed API con­sists of REST­ful URLs such as /api/smart/6h to get S.M.A.R.T. ag­gre­gated at 6 hour in­ter­vals.

Figure 2: This graph vi­su­al­izes that how big frac­tion of the time disk has been spin­ning (blue bar) or idle (green bar). Not hav­ing the disk al­ways spin­ning can save en­ergy and in­crease disk's life span, but the num­ber of spin-downs and spin-ups should also be min­imized.

HTTP re­quests are han­dled by a PHP script which con­structs the cor­re­spond­ing Elas­tic­Search ag­gre­ga­tion query, trans­forms the re­sponse into the de­sired struc­ture struc­ture and gen­er­ates the JSON re­sponse. Elas­tic­Search is able to ag­gre­gate 500k doc­uments in 150 mil­lisec­onds, thanks to good caching and an ef­fi­cient im­ple­men­ta­tion. It will be in­ter­est­ing to use it in fu­ture pro­jects as well. This li­brary is avail­able at GitHub.com/nikonyrh/elas­ticaggre­ga­tor-php.

Related blog posts:

Benchmarking Elasticsearch and MS SQL on NYC Taxis, 2017 May (Matching: Databases, Elasticsearch)
Analyzing NYC Taxi dataset with Elasticsearch and Kibana, 2017 Mar (Matching: Databases, Elasticsearch)
An efficient schema for hierarchical data on Elasticsearch, 2016 Nov (Matching: Databases, Elasticsearch)
Very fuzzy searching with Elasticsearch, 2015 Oct (Matching: Databases, Elasticsearch)
JGit blame for fun and profit(?), 2018 Apr (Matching: Elasticsearch)