Niko's Project Corner

Cheap off-site backup at Amazon Glacier

In ad­di­tion to a mir­rored and check-summed ZFS based backup server, I wanted to have back­ups out­side by premises to be safer against haz­ards such as bur­glary, fire and wa­ter dam­age. ZFS can al­ready re­sist sin­gle disk fail­ure and can re­pair silent data cor­rup­tion, but for im­por­tant mem­ories that isn't suf­fi­cient level of pro­tec­tion. My ever-grow­ing data set is cur­rently 150k files, hav­ing a to­tal size of 520 Gb. Ama­zon's Glacier seems to be the most cost ef­fi­cient so­lu­tion with so­phis­ti­cated APIs and SDKs.

Real-time car tracking and counting

From my of­fice win­dow I've got an un­blocked size-view to the Ring Road I (Kehä I) in Es­poo, Fin­land. It is one of the bus­iest roads in Fin­land, hav­ing up-to 100.000 cars / day. I wanted to cre­ate a pro­gram which would re­ceive a video feed from a we­bcam and would pro­cess im­ages in real time on com­mon hard­ware.

Server monitoring and analytics

There al­ready ex­ists many server mon­itor­ing and log­ging sys­tems, but I was in­ter­ested to de­velop and de­ploy my own. It was also a good chance to learn about Elas­tic­Search's ag­gre­ga­tion queries (new in v1.0.0). Orig­inally Elas­tic­Search was de­signed to provide scal­able doc­ument based stor­age and ef­fi­cient search, but now it is gain­ing more ca­pa­bil­ities. The pro­ject con­sists of a cron job which pushes new met­rics to Elas­tic­Search, a REST­ful JSON API to query statis­tics on recorded num­bers and plot the re­sults in a browser (based on High­Charts).

Efficient in-memory analytical database

Tra­di­tional databases such as MySQL are not de­signed to per­form well in an­alyt­ical queries, which re­quires ac­cess to pos­si­bly all of the rows on se­lected columns. This re­sults in a full table scan and it can­not ben­efit from any in­dexes. Column-ori­ented en­gi­nes try to cir­cum­vent this is­sue, but I went one step deeper and made the stor­age column value ori­ented, sim­ilar to an in­verted in­dex. This re­sults in 2 — 10× speedup from op­ti­mized colum­nar so­lu­tions and 80× the speed of MySQL.

Real-time Heat Map Generator

I wanted a real-time map gen­er­ator to vi­su­al­ize re­gional prop­erty price changes based on cho­sen time in­ter­val. I didn't want to re­sort to pre-gen­er­ated tiles, be­cause this would pre­vent user-cus­tomized out­put and limit con­fig­ura­tion op­tions. To get the best per­for­mance, I im­ple­mented a FastCGI pro­cess in C++ with a REST­ful in­ter­face to gen­er­ate the re­quired tiles in par­al­lel. The re­sult­ing pro­gram can gen­er­ate a cus­tomized 1280 × 720 res­olu­tion JPG in 30 mil­lisec­onds and equiv­alent PNG in 60 mil­lisec­onds.

HTTP API load tester

When de­vel­op­ing REST­ful APIs, it is im­por­tant to know how many re­quests per min­ute the end point is able to serve. Be­cause of my in­ter­est in Ng­inx, FastCGI and multi-threaded C+++, I de­cided to de­velop my own in-broser HTTP load tester which sup­ports easy con­fig­ura­tion, any num­ber of par­al­lel load-gen­er­at­ing worker threads and real-time graph­ing based on jQuery pow­ered High­Charts li­brary.

Real-time interest point tracking

As men­tioned in an other ar­ti­cle about om­ni­di­rec­tional cam­eras, my Mas­ter's The­sis' main topic was real-time in­ter­est point ex­trac­tion and track­ing on an om­ni­di­rec­tional im­age in a chal­leng­ing forest en­vi­ron­ment. I found OpenCV's rou­ti­nes mostly rather slow and run­ning in a sin­gle thread, so I ended up im­ple­ment­ing ev­ery­thing my­self to gain more con­trol on the data flow and threads' de­pen­den­cies. The im­ple­mented code would si­mul­ta­ne­ously use 4 threads on CPU and a few hun­dred on the GPU, ex­ecut­ing in­ter­est point ex­trac­tion and match­ing at 27 fps (37 ms/frame) for 1800 × 360 pix­els (≈0.65 Mpix) panoramic im­age.

Automatic file maneger in PHP

I got tired of man­ually or­ga­niz­ing my down­loads folder, and af­ter not be­ing happy with ex­ist­ing so­lu­tions I de­cided to write a PHP script which would en­able me to eas­ily con­fig­ure new rules for file and folder man­age­ment. This is achieved by hav­ing a flex­ible class hi­er­ar­chy, so that you don't end up re­peat­ing your­self when con­fig­ur­ing new rules and ac­tions.

Global illumination

I im­ple­mented a sim­ple global il­lu­mi­na­tion al­go­rithm, which con­structs and solves the sparse ma­trix which de­scribes how much tiles re­flect light to each other. It sup­ported only gray-scale ren­der­ing and wouldn't scale up to big­ger sce­nes, but it was nev­er­the­less an in­ter­est­ing pro­ject and I learnt a lot about 3D ge­om­etry, lin­ear al­ge­bra and com­puter graph­ics. Graph­ics was drawn by a soft­ware ren­derer which uses only stan­dard SDL prim­itive draw calls.

Robot localization and path planning

At spring 2012 I did a course in robotics, which in­volved pro­gram­ming a semi-au­to­matic robot which could fetch items from pre-de­ter­mined lo­ca­tions and re­turn them back to cor­rect de­posit bins. I had a team of four peo­ple, and I solved the prob­lems of con­tin­uous ro­bust robot lo­cal­iza­tion, task plan­ning and path plan­ning. Oth­ers fo­cused on the over­all source code ar­chi­tec­ture, state-ma­chine logic, closed loop driv­ing con­trol and other gen­eral top­ics. The used robot can be seen in Fig­ure 1.

English hyphenation algorithm in PHP

A good pre­sen­ta­tion about hy­phen­ation in HTML doc­uments can be seen here, but it is client side (JavaScript) ori­ented. Ba­si­cally you shouldn't use jus­ti­fied text un­less it is hy­phen­ated, be­cause long words will cause huge spaces be­tween words to make the line stretch out the whole width of the el­ement. I found a few PHP scripts such as ph­pHy­phen­ator 1.5, but typ­ically they weren't im­ple­mented as a sin­gle stand-alone PHP class. Since the un­der­ly­ing al­go­rithm is fairly sim­ple, I de­cided to write it from scratch.

CUDA realtime rendering engine

So far I've writ­ten a ba­sic ren­der­ing en­gine which uses Nvidia's CUDA (Com­pute Uni­fied De­vice Ar­chi­tec­ture) which can ren­der re­flec­tive sur­faces with en­vi­ron­men­tal map­ping and anti-alias­ing and mo­tion blur at 200 fps with min­imal us­age of 3^rd party li­braries such as OpenGL. This let me fully im­ple­ment the cross-plat­form ren­der­ing pipeline from data trans­fer to pixel-level RGB cal­cu­la­tions, all in C-like syn­tax.

Visualizing laser scanned geography

Dur­ing the sum­mer of 2012 when I was mainly work­ing on my Mas­ter's The­sis, I also had a look at Na­tional Land Sur­vey of Fin­land's open data down­load ser­vice. There I down­loaded a point cloud dataset which had typ­ically 4 - 5 mea­sured points / square me­ter. This means that to vi­su­al­ize a re­gion of 2.5 × 2 km, I had to work with a point cloud con­sist­ing of 5 × 2500 × 2000 → 25 mil­lion points. I chose to con­cen­trate on my cam­pus area, be­cause I know it well and it has many in­ter­est­ing land­marks. For ex­am­ple the iconic main build­ing can be seen in Fig­ure 1.

Rendering omnidirectional images

As I men­tioned in the pre­vi­ous ar­ti­cle about om­ni­di­rec­tional cam­eras, my Mas­ters of Sci­ence The­sis in­volved the us­age of this spe­cial kind of imag­ing sys­tem which con­sists of a tra­di­tional cam­era lens and a con­cave mir­ror, which pro­vided 360° × 90° Field of View. It was or­dered from Japan and there was some de­lay in the de­liv­ery, so mean­while I wrote an all-Mat­lab script to sim­ulate this sys­tem's prop­er­ties, cal­ibra­tion and panorama gen­er­ation in prac­tice.

Omnidirectional cameras

My Mas­ters of Sci­ence The­sis in­volved the us­age of a so-called "om­ni­di­rec­tional cam­era". There are vari­ous ways of achiev­ing 180° or even 360° view, with their dis­tinct pros and cons. The gen­eral ben­efit of these al­ter­na­tive cam­era sys­tems is that ob­jects don't need to be tracked, be­cause gen­er­ally they stay with­ing the ex­tremely broad Field of View (FoV) of the cam­era. This is also very ben­efi­cial in vi­sual odom­etry tasks, be­cause land­marks can be tracked for longer pe­ri­ods of time.