Fast, asynchronous, functional reactive stream programming framework with multi‑threading capability based on ES6+.

Introduction to Scramjet

Scramjet is a fast, simple, free and open source functional reactive stream programming framework written on top of node.js streams with multi-threadding support. The code is written by chaining functions that transform data easily with ES7 async/await syntax. It is built upon the logic behind three well known javascript array operations: map, filter and reduce. Scramjet transforms are so standard and natural that we're sure you can start writing your code straight away.

StringStream
  .from(async function* () => {
    let current_url = url;
    while (current_url) {
      const res = await fetch(url);
      if (!res.ok) throw new Error("Aaaaargh!")
      yield* res.body;
      current_url = res.headers['x-next-page'];
    }
  })
  .split(/\r?\n/)
  .parse(str => str.split("\t"))
  .map(
    async ([name, ...items]) => insertToDB({name, items})
  )
  .reduce(
    async (acc, {score}) => acc + await dedupe(inserted),
    0
  )
  .then(() => console.log(`Import completed`))
  .catch(e => console.error(`Import failed "${e.message}"`))