Try : Insurtech, Application Development

AgriTech(1)

Augmented Reality(20)

Clean Tech(8)

Customer Journey(17)

Design(44)

Solar Industry(8)

User Experience(67)

Edtech(10)

Events(34)

HR Tech(3)

Interviews(10)

Life@mantra(11)

Logistics(5)

Strategy(18)

Testing(9)

Android(48)

Backend(32)

Dev Ops(11)

Enterprise Solution(29)

Technology Modernization(8)

Frontend(29)

iOS(43)

Javascript(15)

AI in Insurance(38)

Insurtech(66)

Product Innovation(57)

Solutions(22)

E-health(12)

HealthTech(24)

mHealth(5)

Telehealth Care(4)

Telemedicine(5)

Artificial Intelligence(146)

Bitcoin(8)

Blockchain(19)

Cognitive Computing(7)

Computer Vision(8)

Data Science(21)

FinTech(51)

Banking(7)

Intelligent Automation(27)

Machine Learning(47)

Natural Language Processing(14)

expand Menu Filters

Implementing a Clean Architecture with Nest.JS (Part 2)

7 minutes read

It’s been a while since my last article Implementing Clean architecture with NestJS was rolled out where we went through some high-level concepts of clean architecture and the underlying structure of Nest.js. This article is the next part of the same series wherein I will try to break down different layers of Clean architecture and share some of the useful Nest.js tools/concepts that can be used in our implementation. Although, we will not be doing “Real” coding in this part.

So without any further delay. Let’s dive in…

Clean architecture aligns with the objective that the system should be independent of any external agency. It can be a framework, Database, or any third-party tools that are being used by the system. It also focuses on ‘Testability’ which in the modern era of development is a major consideration.

If we dissect the above diagram, what we find is

  • Layers: Each circle represents an independent layer in the system.
  • Dependency: The direction is from out to in. Basically, it means that the outer layers are dependent on the inner layer and the Entity layer is independent.
  • Entities: It will comprise all the entities that will construct your application. e.g. User Entity, Product Entity, Etc.
  • Use cases: To every entity, there are going to be specific use cases that will actually comprise all your core logic. E.g. Generating a password for the user, Adding a product, Etc.
  • Controllers/Presenters: These are basically gateways to your system. You can think of them as entry/exit points to the use cases.
  • Frameworks: It contains all the specific implementations e.g. web framework, database, loggers, Exception Handling.

What I find really interesting and intriguing about this type of system is that it focuses on business logic instead of the frameworks and tools used to run the logic. 

That means it hardly matters which database you choose, and which framework you are using. It can change and evolve over time but what will remain constant and intact is your business logic and the entities of the application.

Let’s try to slowly stack up the layers and try to understand through an example…

Basic application with Users and Products

In our example, we will represent a simple application that actually is responsible for CRUD operations on User and Product Entities.

I will be taking Nest.JS as a reference wherever I will explain the implementation of any functionality. Will touch base on concepts/tools like Dependency Injection, Repository, Class-Validator and DTO.

Entities and Use Cases: Core of our business

At its core, our business logic is defined by two layers of clean architecture: 

  1. Entity layer: Comprises all the business entities that define our application.
  2. Use-case layer: Comprises all the business scenarios that our entities support.
  1. Entities: The building block of our application

Entities are the only independent layer in our system and will comprise information that is not meant to change over a period of time and will shape our application functionality. This also means that these layers will not be affected by any change in the external environment e.g. Services, controllers, and routing. 

There will be only two entities in our application:

a) Product: ID, Name, Category, Cost & Quantity

b) User: ID, Name & Email

  1. Use-Cases: Fundamental part of our business logic.

Use cases are only dependent on Entities and orchestrate all the scenarios that comprehend their counterpart Entity. For two of our entities, we will have the following use cases:

Product

  • Add a product
  • Get a product by Id
  • Get All products
  • Get Products by cost, category, or quantity
  • Update a product
  • Delete a product

User

  • Add user
  • Get User by Id
  • Get All Users

Now, we need to declare these entities and use cases in our codebase and we need to have some form of database service. Database service can be an ORM/SDK which will connect with our database Postgres/MongoDB etc.

  • One way to do it is to use this SDK/ORM as a direct implementation in our use cases and hence making our use cases dependent on the SDK/ORM we use. Any change in the SDK/ORM will directly impact our logic hence defying the main motive of clean architecture.
  • Other way or I would say, the best way, is to take advantage of abstraction and dependency injection. With help of something called a repository(Abstract service/layer) which sits between our use case and the DB implementation. We can create an abstract layer of service injected into our use cases where we will define methods that will be independent of the type of database and the implementation of those methods will depend on the database used. 

This will give us the flexibility to change the DB at our will. All we need to do is the DB implementation of the repository methods and we do not touch our logic at all.

Repository: As mentioned above, the Repository will contain the implementation of the abstract functions which will be used in our use case. In our example, those functions can be InsertItem, updateItemById, getItemById, getItems, fiterItems. This can be a generic repository. We can also have specific repositories for use cases as well which will contain more specific functions like addUser, getUserById, getUsers, addProduct. getProducts, etc.

Our use cases will always call these abstract methods without actually knowing about the DB used.

Controllers and Presenters: Data carriers for our business

Now that the core is set, we have defined our entities and use cases. Something has to act upon our use cases in order for the system to work. Data needs to be passed to the use cases to be stored and updated and similarly, processed data has to be sent out for the end users to be able to use the information. Well, That’s what Controllers and Presenters are there for.

One can think of them as adapters that bind our use cases to the outer world. In basic terminology, Controllers are used to responding to user input e.g. validation, transformation, etc. while the presenters are used to format and prepare data to be sent out.

In clean architecture, the controller’s job is:

  • Receive the user input — some kind of DTO 
  • Sanitization: validate user input
  • Transformation: convert user inputs to the desired type required by entities/use cases
  • Last but not least, call the use case

The controller will only have the formatting of the data. No business logic. On the other hand, the job of the presenter is to get data from the application repository and then build a formatted response for the client. Its main responsibilities include

  • Formatting: converting strings and dates.
  • If needed, add presentation data, like flags.
  • Prepare the data to be displayed in the UI.

In NestJS, the functionality of the controller and presenter is implemented under controllers only, there is no different presentation component in nestjs. With the help NestJS, we will validate our user input using validation pipes and transform our DTOs to business objects using transformation pipes.

Under the hood, nestJs uses a class-validator library for all validations. All we need to do is wrap our DTO with a class-validator decorator and specify our validation properties and we are good to go.

External Interfaces: Our Frameworks

Now that our mission mangal is set to launch. All we need is the right platform. What I mean is that now we have our core ready, and gateways and channels are ready to take inputs. What we need is to set it up with the right frameworks eg. Database, logging, error handling, etc.

For example, for DB we can use TypeORM for our data services and database modelling. We can use Winston for logging. Web application frameworks can either be Express or Fastify.

Summary

This article summarizes all the layers of clean architecture using a real-world example with help of Nest.JS. We tried to demonstrate how we can build a robust structure of layer-by-layer services that decouple our core business logic from frameworks. 

It gives us the superpower to change our frameworks without even worrying about breaking the code. We can easily move from MySQL to PostgreSQL, Express to Fastify without bothering our business logic. This will help us reduce the cost of transition and ease of testing.

I believe that this article would help a lot of you who strive to write clean and maintainable code. Will be back with more content soon. Till then, sayonara!

About the Author:

Junaid Bhat is currently working as a Tech Lead in Mantra Labs. He is a tech enthusiast striving to become a better engineer every day by following industry standards and aligned towards a more structured approach to problem-solving. 

Cancel

Knowledge thats worth delivered in your inbox

Lake, Lakehouse, or Warehouse? Picking the Perfect Data Playground

By :

In 1997, the world watched in awe as IBM’s Deep Blue, a machine designed to play chess, defeated world champion Garry Kasparov. This moment wasn’t just a milestone for technology; it was a profound demonstration of data’s potential. Deep Blue analyzed millions of structured moves to anticipate outcomes. But imagine if it had access to unstructured data—Kasparov’s interviews, emotions, and instinctive reactions. Would the game have unfolded differently?

This historic clash mirrors today’s challenge in data architectures: leveraging structured, unstructured, and hybrid data systems to stay ahead. Let’s explore the nuances between Data Warehouses, Data Lakes, and Data Lakehouses—and uncover how they empower organizations to make game-changing decisions.

Deep Blue’s triumph was rooted in its ability to process structured data—moves on the chessboard, sequences of play, and pre-defined rules. Similarly, in the business world, structured data forms the backbone of decision-making. Customer transaction histories, financial ledgers, and inventory records are the “chess moves” of enterprises, neatly organized into rows and columns, ready for analysis. But as businesses grew, so did their need for a system that could not only store this structured data but also transform it into actionable insights efficiently. This need birthed the data warehouse.

Why was Data Warehouse the Best Move on the Board?

Data warehouses act as the strategic command centers for enterprises. By employing a schema-on-write approach, they ensure data is cleaned, validated, and formatted before storage. This guarantees high accuracy and consistency, making them indispensable for industries like finance and healthcare. For instance, global banks rely on data warehouses to calculate real-time risk assessments or detect fraud—a necessity when billions of transactions are processed daily, tools like Amazon Redshift, Snowflake Data Warehouse, and Azure Data Warehouse are vital. Similarly, hospitals use them to streamline patient care by integrating records, billing, and treatment plans into unified dashboards.

The impact is evident: according to a report by Global Market Insights, the global data warehouse market is projected to reach $30.4 billion by 2025, driven by the growing demand for business intelligence and real-time analytics. Yet, much like Deep Blue’s limitations in analyzing Kasparov’s emotional state, data warehouses face challenges when encountering data that doesn’t fit neatly into predefined schemas.

The question remains—what happens when businesses need to explore data outside these structured confines? The next evolution takes us to the flexible and expansive realm of data lakes, designed to embrace unstructured chaos.

The True Depth of Data Lakes 

While structured data lays the foundation for traditional analytics, the modern business environment is far more complex, organizations today recognize the untapped potential in unstructured and semi-structured data. Social media conversations, customer reviews, IoT sensor feeds, audio recordings, and video content—these are the modern equivalents of Kasparov’s instinctive reactions and emotional expressions. They hold valuable insights but exist in forms that defy the rigid schemas of data warehouses.

Data lake is the system designed to embrace this chaos. Unlike warehouses, which demand structure upfront, data lakes operate on a schema-on-read approach, storing raw data in its native format until it’s needed for analysis. This flexibility makes data lakes ideal for capturing unstructured and semi-structured information. For example, Netflix uses data lakes to ingest billions of daily streaming logs, combining semi-structured metadata with unstructured viewing behaviors to deliver hyper-personalized recommendations. Similarly, Tesla stores vast amounts of raw sensor data from its autonomous vehicles in data lakes to train machine learning models.

However, this openness comes with challenges. Without proper governance, data lakes risk devolving into “data swamps,” where valuable insights are buried under poorly cataloged, duplicated, or irrelevant information. Forrester analysts estimate that 60%-73% of enterprise data goes unused for analytics, highlighting the governance gap in traditional lake implementations.

Is the Data Lakehouse the Best of Both Worlds?

This gap gave rise to the data lakehouse, a hybrid approach that marries the flexibility of data lakes with the structure and governance of warehouses. The lakehouse supports both structured and unstructured data, enabling real-time querying for business intelligence (BI) while also accommodating AI/ML workloads. Tools like Databricks Lakehouse and Snowflake Lakehouse integrate features like ACID transactions and unified metadata layers, ensuring data remains clean, compliant, and accessible.

Retailers, for instance, use lakehouses to analyze customer behavior in real time while simultaneously training AI models for predictive recommendations. Streaming services like Disney+ integrate structured subscriber data with unstructured viewing habits, enhancing personalization and engagement. In manufacturing, lakehouses process vast IoT sensor data alongside operational records, predicting maintenance needs and reducing downtime. According to a report by Databricks, organizations implementing lakehouse architectures have achieved up to 40% cost reductions and accelerated insights, proving their value as a future-ready data solution.

As businesses navigate this evolving data ecosystem, the choice between these architectures depends on their unique needs. Below is a comparison table highlighting the key attributes of data warehouses, data lakes, and data lakehouses:

FeatureData WarehouseData LakeData Lakehouse
Data TypeStructuredStructured, Semi-Structured, UnstructuredBoth
Schema ApproachSchema-on-WriteSchema-on-ReadBoth
Query PerformanceOptimized for BISlower; requires specialized toolsHigh performance for both BI and AI
AccessibilityEasy for analysts with SQL toolsRequires technical expertiseAccessible to both analysts and data scientists
Cost EfficiencyHighLowModerate
ScalabilityLimitedHighHigh
GovernanceStrongWeakStrong
Use CasesBI, ComplianceAI/ML, Data ExplorationReal-Time Analytics, Unified Workloads
Best Fit ForFinance, HealthcareMedia, IoT, ResearchRetail, E-commerce, Multi-Industry
Conclusion

The interplay between data warehouses, data lakes, and data lakehouses is a tale of adaptation and convergence. Just as IBM’s Deep Blue showcased the power of structured data but left questions about unstructured insights, businesses today must decide how to harness the vast potential of their data. From tools like Azure Data Lake, Amazon Redshift, and Snowflake Data Warehouse to advanced platforms like Databricks Lakehouse, the possibilities are limitless.

Ultimately, the path forward depends on an organization’s specific goals—whether optimizing BI, exploring AI/ML, or achieving unified analytics. The synergy of data engineering, data analytics, and database activity monitoring ensures that insights are not just generated but are actionable. To accelerate AI transformation journeys for evolving organizations, leveraging cutting-edge platforms like Snowflake combined with deep expertise is crucial.

At Mantra Labs, we specialize in crafting tailored data science and engineering solutions that empower businesses to achieve their analytics goals. Our experience with platforms like Snowflake and our deep domain expertise makes us the ideal partner for driving data-driven innovation and unlocking the next wave of growth for your enterprise.

Cancel

Knowledge thats worth delivered in your inbox

Loading More Posts ...
Go Top
ml floating chatbot