Get unlimited access to the best of Medium for less than $1/week.

Become a member

Handling workflows in Node.js

using the builder design pattern: A
Comprehensive Guide
Omar MOKHFI · Follow
8 min read · May 19, 2024

109 2

A workflow is basically a list of tasks that run in order, and optionally, each
task has the ability to provide its output as an input to the next task. You can
find different use cases for these workflows, of which we will mention few
examples:
 The workflow of user registration

• Registering a user which involves uploading a profile image to storage,

then creating the user in the database with the image’s link, and then
sending the user a verification email as the last step.

• When a user sends a message via a website’s contact form, we first add
them to the CRM as a new lead, push their message to the sales channel
on Slack , and then determine whether they are a new contact and send
out marketing emails accordingly.
In this step-by-step guide, we will walk through the process of making our
own workflow handler using the builder design pattern, and to make things
interesting, we will use the registration example.

Prerequisites: Before diving into the implementation, make sure you have
the following:

• Node.js and npm

• Basic knowledge on TypeScript

Project Setup
Before jumping to the real thing, let’s try to see how we can achieve the
workflow with the usual basic approach:

1- Create a new Node.js app

mkdir workflow-sample
cd workflow-sample

npm init -y
2- Add TypeScript
First, we will install the packages we need

npm install -D typescript ts-node @types/node

Next, we will create a new tsconfig.json file in our root folder and paste this
code:

{
"compilerOptions": {
"target": "es2022",
"module": "commonjs",
"outDir": "./dist",
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"strict": true,
"skipLibCheck": true
}
}

To be able to run the code with the command “npm run dev”, we will add
this script to our package.json.

{
...
"scripts": {
"dev": "ts-node index.ts"
}
}

Finally, we will create a new index.ts file and start coding

3- Create the registration function

Since the goal of this guide is handling the workflow rather than going in
details on how each step of the workflow works, we will use a method called
— Separation of Concern

There are multiple concerns inside the registration process, and for that, we
will separate the main function into multiple sub-functions and only care
about the input and output of each one.

So, our main function in the index.ts file will look like this:
 // The registration takes the email, password and image file of the user
// This can be acheived by creating an API and using a library like multer
const register= async (email:string, password: string, image: Blob) => {
try {
// 1. this function uploads the image file and returns the link of that image
let imageLink = await uploadImage(image);
// 2. this function saves the user to the database
let savedUser = await saveUser({ email, password, imageLink });
// 3. this function sends a verification email to the user
let result = await sendVerificationEmail(email);
} catch (error) {
console.log("Process failed");
}
};

Now, we will try to simulate the input and output of each function. And since
each sub-function takes some time to run, we will emulate that with a
custom timeout function.

To achieve this let’s add this code to the index.ts file

function timeout(ms: number) {

return new Promise((resolve) => setTimeout(resolve, ms));
}
 // 1. this function uploads the image file and returns the link of that image
const uploadImage = async (image: Blob) => {
await timeout(1000);
return { imageLink: "image link" };
};

// 2. this function saves the user to the database

const saveUser = async ({
email,
password,
imageLink,
}: {
email: string;
password: string;
imageLink: string;
}) => {
await timeout(1000);
return {
id: "1",
email,
password,
image: imageLink,
};
};

// 3. this function sends a verification email to the user

const sendVerificationEmail = async ({ email }: { email: string }) => {
await timeout(1000);
console.log("Email sent successfully to " + email);
};

Issues with this approach

Even if this process does the work successfully, there are still some issues
that push us to create a workflow which handles them:

The first issue is that there is no retrying mechanism. So, if for example, the
storage service doesn’t respond because of a connectivity issue, the whole
process will fail. At this point, you will say — well, we can just try re-running
the process when it fails, we can yes but although this solution works, it still
has a drawback, which is an infinite loop if the error is not a temporary
connectivity issue.

The second issue is that, for example, when the second step fails, we will
need to re-run all steps from the beginning. So, if the first step is complex
and takes 10 seconds to run then we will have to re-run it every time the next
step fails.

In the next part, we will create a class that handles all the mentioned issues
using a simplified useful design pattern which is the builder.

Creating our Workflow Class

From the issues we saw, our class should be able to fit few conditions:

• We can create a pipeline of functions

• Each function’s output can be passed to the next function as an input

 • If a function throws an error, the re-run should start from that order

• the workflow can be configured with a number of retries limit (it means
if the retries limit is 3 then we can only re-run a function 3 times, if it still
persists the workflow will be aborted)

Process of running different steps of the workflow

Let’s create a new /core/workflow.ts and export a class from it

export class Workflow {

 private retryLimit: number;

constructor(retryLimit: number = 3) {
this.retryLimit = retryLimit;
}

static createWorkflow(
retryLimit: number,
callback: (workflow: Workflow) => void
): Workflow {
const workflow = new Workflow(retryLimit);
callback(workflow);
return workflow;
}

async run(): Promise<void> {}

}

Our class contains a “createWorkflow” method, it can be configured with a

limit number of retries and will pass the created workflow instance to a
callback function. Calling “run” method would trigger the start of the
workflow.

Now let’s start fitting our conditions, one by one.

1- We can create a pipeline of functions

We want our class to create as many steps (functions) as we want and also, to
create a final step. For that we will add 2 methods, “create” method that will
build a new StepFunction, and “finally” method that will set the
FinallyFunction. Let’s modify our class:

type StepFunction = (input: any) => Promise<any>;

type FinallyFunction = (input: any) => void;

export class Workflow {

private steps: StepFunction[] = [];
private retryLimit: number;
private finallyCallback?: FinallyFunction;

constructor(retryLimit: number = 3) {
this.retryLimit = retryLimit;
}

static createWorkflow(
retryLimit: number,
callback: (workflow: Workflow) => void
): Workflow {
const workflow = new Workflow(retryLimit);
callback(workflow);
return workflow;
}

create(stepFunction: StepFunction): this {

this.steps.push(stepFunction);
return this;
}

finally(callback: FinallyFunction) {
this.finallyCallback = callback;
}
 async run(): Promise<void> {}
}

You might have noticed that the “create” method returns “this”, and the reason
behind that is to create chaining (you will see how it works when we use the class)

For the other conditions, we will modify our “run” method:

2- Each function’s output can be passed to the next function as an input

// core/workflow.ts

async run(initialInput?: any): Promise<void> {

// providing an initial input to the first step
let input = initialInput;

for (let i = 0; i < this.steps.length; i++) {

const step = this.steps[i];
try {
// passing step's input to the next step
input = await step(input);
} catch (error) {
console.error("Workflow aborted due to step failure.");
break;
}
}

if (this.finallyCallback) {
try {
 this.finallyCallback(input);
} catch (error) {
console.error("Error in final step:", error);
}
}
}

3- Handling errors and managing retries

To handle retries, instead of throwing an error and aborting the workflow,
we will re-run the step till we reach the number of retires’ limit. To achieve
that we will modify our method as follow:

// core/workflow.ts

async run(initialInput?: any): Promise<void> {

// providing an initial input to the first step
let input = initialInput;
let attempts = 0;
let success = false;

for (let i = 0; i < this.steps.length; i++) {

const step = this.steps[i];

// retrying the step as long as it is not successful and retries's limit is not reached yet
// this also makes sure we re-run the workflow starting from the failed step
while (attempts < this.retryLimit && !success) {
try {
// passing step's input to the next step
input = await step(input);
success = true;
 } catch (error) {
attempts++;
if (attempts === this.retryLimit) {
console.error(
`Step ${i + 1} failed after ${attempts} attempts:`,
error
);
}
}
}

// if the step is still not successful after all attempts, the workflow will be aborted
if (!success) {
console.error("Workflow aborted due to step failure.");
break;
}
}

if (this.finallyCallback) {
try {
this.finallyCallback(input);
} catch (error) {
console.error("Error in final step:", error);
}
}
}

With that, our Workflow class fits all conditions and handles all issues.

Using our Workflow Class

In this part, we will use our class to handle the registration process.
 import { Workflow } from "./core/workflow";

const register = (email: string, password: string, image: Blob) => {

// Creating a workflow with 3 maximum retries
Workflow.createWorkflow(3, (workflow) => {
workflow
.create(uploadImage)
// imageLink is the output of the first step passed to the second one
.create(({ imageLink }) =>
saveUser({
email,
password,
imageLink,
})
)
.finally(sendVerificationEmail);
// Passing initial input
}).run(image);
};

register("[email protected]", "password", new Blob());

Although this way works just fine, it is recommended to keep the inputs and
outputs of all steps visible. We can modify our registration function this way:

const register = (email: string, password: string, image: Blob) => {

 Workflow.createWorkflow(3, (workflow) => {
workflow
.create(async (myImage) => {
let imageLink = await uploadImage(myImage);
return { imageLink };
})
.create(async ({ imageLink }) => {
let user = await saveUser({
email,
password,
imageLink,
});
return user; // { id, email, password, image }
})
.finally(async ({ email }) => {
await sendVerificationEmail({ email });
// If you are using this workflow in an API, you can respond here
// res.status(200).send("User has been created successfully")
});
}).run(image);
};

What’s next ?
I know! A boring guide with too many details. Of course you can just take the
code of workflow.ts and check the last part on how to use it, but the goal of
this guide was that you can understand each part of the code so you can
modify it yourself.

For example, you can add another condition to this workflow so you can test
your comprehension.

• If any step fails after all attempts, the workflow should run a rollback

Try to test it on the same registration process, by throwing an error when

saving the user and deleting the uploaded image from the storage through a
Rollback.

Thanks for reading! Please feel free to give your feedback on

what can be enhanced🙏 🙏

Nodejs Workflow JavaScript

Written by Omar MOKHFI Follow

65 Followers

Software Engineer and Bootcamp Instructor at Code Labs Academy

More from Omar MOKHFI

Omar MOKHFI Omar MOKHFI

Building Microservices Building a Full Stack App with

Architecture with CQRS Pattern… NextJS 14, Supabase and ShadcnUI
In this step-by-step guide, we will walk At the end of this article, you will understand
through the process of building a sample… and create a complete full stack app using…

Mar 18 180 1 Feb 17 103 1

 Omar MOKHFI

Web Audit & Generating Hot Leads

(Eg: Trustme.work)
Once you know the weaknesses, you can use
it to generate opportunites. Website Auditin…

Apr 15 53

See all from Omar MOKHFI

Recommended from Medium

Brian Jenney Dipanshu in AWS in Plain English

3 Lessons from the Smartest Why Developers Are Ditching

Developers I’ve Worked With PostgreSQL, MySQL and MongoDB
I have a confession. The Database Revolution Your Project Is
Missing Out On

Oct 11 2.8K 45 Oct 1 1.2K 32

Lists
 Stories to Help You Grow as a General Coding Knowledge
Software Developer 20 stories · 1663 saves
19 stories · 1436 saves

data science and AI Generative AI Recommended

40 stories · 269 saves Reading
52 stories · 1446 saves

Games24_7 Blogs Probir Sarkar

Node.js v20 upgrade guide: Best Hono.js Benchmark: Node.js vs.

Practices and Performance… Deno 2.0 vs. Bun—Which Is the…
As of April 2023, Node.js versions less than Compare the performance of Node.js, Deno
v16 have reached their end of life, marking a… 2.0, and Bun using Hono.js. Discover which…

Sep 6 50 1 Oct 13 34
 Vitaliy Korzhenko Nidhi Jain 👩💻 in Code Like A Girl

JavaScript Array Interview 7 Productivity Hacks I Stole From a

Questions Principal Software Engineer
1. How Do You Find the Largest Number in an Golden tips and tricks that can make you
Array? unstoppable

Sep 18 129 6d ago 1.7K 37

See more recommendations

Help Status About Careers Press Blog Privacy Terms Text to speech Teams