Niko's Project Corner

Service discovery with Docker, Consul and Registrator

Tra­di­tion­ally com­put­ers were named and not eas­ily re­placed in the event it broke down. Server soft­ware was lis­ten­ing on a hard-coded port, and to link pieces to­gether these ma­chine names and ser­vice ports were hard-coded into other soft­ware's con­fig­ura­tion files. Now in the era of cloud com­put­ing and ser­vice ori­ented ar­chi­tec­ture this is no longer an ad­equate so­lu­tion, thus elas­tic scal­ing and ser­vice dis­cov­ery are be­com­ing the norm. One easy so­lu­tion is to com­bine the pow­ers of Docker, Con­sul and Reg­is­tra­tor.

Nginx docker image for easy file access via HTTP

Of­ten I find my­self hav­ing a SSH con­nec­tion to a re­mote server, and I'd like to re­trieve some files to my own ma­chine. Com­mon meth­ods for this in­clude Win­dows/Samba share, SSHFS and up­load to cloud (which isn't triv­ial to do via plain cURL). Here an easy-to-use al­ter­na­tive is de­scribed: a sin­gle line com­mand to load and run a docker im­age which con­tains a pre-con­fig­ured Ng­inx in­stance. Then files can be ac­cessed via plain HTTP at the user-as­signed port (as­sum­ing fire­wall isn't block­ing it).

Scalable analytics with Docker, Spark and Python

Tra­di­tion­ally data sci­en­tists in­stalled soft­ware pack­ages di­rectly to their ma­chi­nes, wrote code, trained mod­els, saved re­sults to lo­cal files and ap­plied mod­els to new data in batch pro­cess­ing style. New data-driven prod­ucts re­quire rapid de­vel­op­ment of new mod­els, scal­able train­ing and easy in­te­gra­tion to other as­pects of the busi­ness. Here I am propos­ing one (per­haps al­ready well-known) cloud-ready ar­chi­tec­ture to meet these re­quire­ments.

Automated image capturing + API

Out of in­ter­est on na­ture ob­ser­va­tion, com­puter vi­sion, im­age pro­cess­ing and so forth I de­vel­oped an au­to­mated sys­tem to cap­ture one photo / min­ute and store it on a disk. The pro­ject also has Bash and PHP scripts co­or­di­nat­ing ex­ter­nal tools such as mon­tage for im­age stitch­ing and men­coder for video gen­er­ation. PHP also pro­vides an HTTP API for im­age gen­er­ation and file size statis­tics.

Publishing internal services behind a NAT

Even in desk­top ap­pli­ca­tions it is be­com­ing more and more com­mon to provide a HTTP based APIs or full user in­ter­faces. For ex­am­ple Bit­Tor­rent's μTor­rent and Bit­Tor­rent Sync don't have any built-in UI, and in­stead users just head with their pre­ferred in­ter­net browser to http://lo­cal­host:8080 or http://lo­cal­host:8888. How­ever they typ­ically lack HTTPS en­cryp­tion and each port needs to be con­fig­ured to the NAT router in­di­vid­ually. This so­lu­tion uses a Ng­inx in­stance on a vir­tual ma­chine to provide a HTTPS re­verse proxy to all these ser­vices in a sin­gle port un­der dif­fer­ent sub-do­mains.

Cheap off-site backup at Amazon Glacier

In ad­di­tion to a mir­rored and check-summed ZFS based backup server, I wanted to have back­ups out­side by premises to be safer against haz­ards such as bur­glary, fire and wa­ter dam­age. ZFS can al­ready re­sist sin­gle disk fail­ure and can re­pair silent data cor­rup­tion, but for im­por­tant mem­ories that isn't suf­fi­cient level of pro­tec­tion. My ever-grow­ing data set is cur­rently 150k files, hav­ing a to­tal size of 520 Gb. Ama­zon's Glacier seems to be the most cost ef­fi­cient so­lu­tion with so­phis­ti­cated APIs and SDKs.