Nodeflake is a distributed computing framework for Node. It aims to solve the following problems:
- Writing separate logic for masters and slaves creates more engineering effort in defining how tasks are distributed. Nodeflake presents a functional programming paradigm that can be seamlessly distributed across a tree topology.
- Distributed computing should enable the generation of rich visualizations, but existing methodologies involve separate technology for processing and visualizing data. Nodeflake abstracts the entire analysis workflow with powerful functions for capturing raw data, processing it for analysis and visualization, and exporting the generated visualizations to browser-ready widgets.
- Distributed computing should be seamless across physical devices and the cloud.
This write-up is a work in progress. I built a fully-functional cluster but the code became a patchwork of network services. The bottlenecks are quite obvious with particular workloads and I might be going back to the drawing board for this project when I pick it back up.