Niko's Project Corner

Service discovery with Docker, Consul and Registrator

Tra­di­tion­ally com­put­ers were named and not eas­ily re­placed in the event it broke down. Server soft­ware was lis­ten­ing on a hard-coded port, and to link pieces to­gether these ma­chine names and ser­vice ports were hard-coded into other soft­ware's con­fig­ura­tion files. Now in the era of cloud com­put­ing and ser­vice ori­ented ar­chi­tec­ture this is no longer an ad­equate so­lu­tion, thus elas­tic scal­ing and ser­vice dis­cov­ery are be­com­ing the norm. One easy so­lu­tion is to com­bine the pow­ers of Docker, Con­sul and Reg­is­tra­tor.

English hyphenation algorithm in Clojure

This is noth­ing that spec­tac­ular (as if any­thing on my blog is), but I still wanted to de­scribe the out­line of the pro­ject of port­ing the hy­phen­ation al­go­rithm from PHP to Clo­jure. The im­ple­men­ta­tion is only about 80 lines of code + com­ments + 20 lines of unit tests. For com­par­ison the orig­inal PHP abom­ina­tion is about is about 160 LoCs, al­though it is a bit bloated by im­ple­ment­ing the pat­terns search via a trie data struc­ture in­stead of us­ing the str­pos func­tion.

Finnish Invoice Template

Af­ter fin­ish­ing a paid pro­ject, ide­ally a for­mal look­ing in­voice would be sent to the client. I know there are many com­mer­cial prod­ucts avail­able, but I couldn't find a good open source al­ter­na­tive es­pe­cially with the stan­dard Finnish for­mat­ting. I was happy to find jheusala/finnish-in­voice-tem­plate from GitHub which had all of the tricky L^aT_eX stuff done.

Hash-based commitment schemes without a computer

An in­ter­est­ing ques­tion was posted to crypto.stack­ex­change.com: "Is there a sim­ple hash func­tion that one can com­pute with­out a com­puter?" Here are three pro­posed al­go­rithms based on Zo­brist hash­ing, RC4 and A5/1. These should be reasonably secure even against attacks with a calculator, except the one based on Zobrist hashing (but I don't know how to prove or dis-prove this claim). These constructs are especially well suited for com­mit­ment schemes.

Simulating gravitational field near a torus

There are many games with a strong em­pha­sis on grav­ity, and at times even multi-body tra­jec­tory sim­ula­tions. How­ever they all seem to be based on spher­ical ge­om­etry (as plan­ets are shaped by grav­ity), but other shapes should cre­ate in­ter­est­ing tra­jec­to­ries. As torus has ro­ta­tional sym­me­try its grav­ity field can be mod­elled on a 2D cross-sec­tion. In this pro­ject torus' field is es­ti­mated in 3D, pro­jected to 2D and in­ter­po­la­tion func­tions are fit­ted. The space- and time-ef­fi­cient model could be used in a game to do physics sim­ula­tion in real time.

Nginx docker image for easy file access via HTTP

Of­ten I find my­self hav­ing a SSH con­nec­tion to a re­mote server, and I'd like to re­trieve some files to my own ma­chine. Com­mon meth­ods for this in­clude Win­dows/Samba share, SSHFS and up­load to cloud (which isn't triv­ial to do via plain cURL). Here an easy-to-use al­ter­na­tive is de­scribed: a sin­gle line com­mand to load and run a docker im­age which con­tains a pre-con­fig­ured Ng­inx in­stance. Then files can be ac­cessed via plain HTTP at the user-as­signed port (as­sum­ing fire­wall isn't block­ing it).

Scalable analytics with Docker, Spark and Python

Tra­di­tion­ally data sci­en­tists in­stalled soft­ware pack­ages di­rectly to their ma­chi­nes, wrote code, trained mod­els, saved re­sults to lo­cal files and ap­plied mod­els to new data in batch pro­cess­ing style. New data-driven prod­ucts re­quire rapid de­vel­op­ment of new mod­els, scal­able train­ing and easy in­te­gra­tion to other as­pects of the busi­ness. Here I am propos­ing one (per­haps al­ready well-known) cloud-ready ar­chi­tec­ture to meet these re­quire­ments.

Very fuzzy searching with CUDA

This is an al­ter­na­tive an­swer to the ques­tion I en­coun­tered at Stack Over­flow about fuzzy search­ing of hashes on Elas­tic­search. My orig­inal an­swer used lo­cal­ity-sen­si­tive hash­ing. Su­pe­rior speed and sim­ple im­ple­men­ta­tion were gained by us­ing nVidia's CUDA via Thrust li­brary.

Very fuzzy searching with Elasticsearch

I en­coun­tered an in­ter­est­ing ques­tion at Stack Over­flow about fuzzy search­ing of hashes on Elas­tic­search and de­cided to give it a go. It has na­tive sup­port for fuzzy text searches but due to per­for­mance rea­sons it only sup­ports an edit dis­tance up-to 2. In this con­text the max­imum al­lowed dis­tance was eight so an al­ter­na­tive so­lu­tion was needed. A so­lu­tion was found from lo­cal­ity-sen­si­tive hash­ing.

Anonymous and secure information storing and sharing

Nowa­days en­cryp­tion is stan­dard prac­tice on web when data is in tran­si­tion, and there are even a few ser­vices which of­fer client-side en­cryp­tion and thus are truly end-to-end. Nev­er­the­less for some rea­son they all re­quire you to cre­ate and ac­count by pro­vid­ing your email and pass­word, al­though this is not strictly nec­es­sary for stor­ing and shar­ing data. In this sys­tem the doc­ument id, en­cryp­tion key and HMAC key are gen­er­ated ad-hoc on the client and only min­imal nec­es­sary in­for­ma­tion is re­vealed to the server. A live demo should be avail­able at no­knowl­edgenotes.nikonyrh.org.

Automated image capturing + API

Out of in­ter­est on na­ture ob­ser­va­tion, com­puter vi­sion, im­age pro­cess­ing and so forth I de­vel­oped an au­to­mated sys­tem to cap­ture one photo / min­ute and store it on a disk. The pro­ject also has Bash and PHP scripts co­or­di­nat­ing ex­ter­nal tools such as mon­tage for im­age stitch­ing and men­coder for video gen­er­ation. PHP also pro­vides an HTTP API for im­age gen­er­ation and file size statis­tics.

Approximating planets' orbits in closed-form

I wanted to find or cre­ate a for­mula which would ac­cept an epoch times­tamp, lat­itude and lon­gi­tude and it would pro­duce the Sun's ob­served az­imuth and al­ti­tude in ra­di­ans. It needs to take into ac­count de­tails earth's ax­ial tilt and its po­si­tion on its or­bit around the sun. To my sur­prise I wasn't able to find such for­mula, so I had to de­velop it from scratch. Luck­ily earth's or­bit (and or­bits in gen­eral) is a well stud­ied and doc­umented prob­lem, so I could take some short­cuts.

Automatic map stitching

Nowa­days there are many HTML5-based map ser­vices, but typ­ically they don't of­fer any ex­port func­tion­al­ity. To cre­ate a full view of the de­sired re­gion, one can ei­ther zoom out (and lose map de­tails) or take many screen­shots of dif­fer­ent lo­ca­tions and man­ually stitch them to­gether. This pro­ject can au­to­mat­ically load all stored screen­shots, de­tect the map, crop rel­evant re­gions, de­ter­mine im­ages rel­ative off­sets and gen­er­ate the high-res out­put with zero con­fig­ura­tion from any map ser­vice.

Publishing internal services behind a NAT

Even in desk­top ap­pli­ca­tions it is be­com­ing more and more com­mon to provide a HTTP based APIs or full user in­ter­faces. For ex­am­ple Bit­Tor­rent's μTor­rent and Bit­Tor­rent Sync don't have any built-in UI, and in­stead users just head with their pre­ferred in­ter­net browser to http://lo­cal­host:8080 or http://lo­cal­host:8888. How­ever they typ­ically lack HTTPS en­cryp­tion and each port needs to be con­fig­ured to the NAT router in­di­vid­ually. This so­lu­tion uses a Ng­inx in­stance on a vir­tual ma­chine to provide a HTTPS re­verse proxy to all these ser­vices in a sin­gle port un­der dif­fer­ent sub-do­mains.

Image distortion estimation and compensation

This pro­ject's goal was to au­to­mat­ically and ro­bustly es­ti­mate and com­pen­sate dis­tor­tion from any re­ceipt pho­tos. The user is able to just snap the photo and OCR could ac­cu­rately iden­tify bought prod­ucts and their prices. How­ever this task is some­what chal­leng­ing be­cause typ­ically re­ceipts tend to get crum­bled and bent. Thus they won't lie nicely flat on a sur­face for easy anal­ysis. This set of al­go­rithms solves that prob­lem and pro­duces dis­tor­tion-free thresh­olded im­ages for the next OCR step.