CT Wu

Explained with a code sample

This article is the last of this series. We have already described the problem encountered in data-oriented design. In this article, we will introduce a better way to tackle a feature requirement.

We continue the previous example, a sign-in mission, and try a different design flow. Before we start, let’s review the onion architecture again.

In order to make it easier to understand the process to be introduced later, let’s first define several important legends of this diagram.

• Entity: In clean architecture, entity means the business logic. Different from the entity in domain-driven design, the entity here can be realized as the domain in domain-driven design.
• Use cases: With the domain, the outer layer is use cases, which refers to clients who use domain knowledge to fulfill specific needs. In domain-driven design, it is also known as the domain service.
• Controller: The Controller is quite simple. It is responsible for managing the ingress and egress of the entire domain, including input checking, and converting domain knowledge into a data structure that is presented on the client side.
• DB: The outermost layer is the external dependencies of the system, including the database.
• Arrows: The arrow pointing from the outside to the inside is a reference. The outer module can reference the inner module, but it cannot be referenced from the inside to the outside.

According to this description, we can know that the order of design should be from the inside to the outside. After the inner layer is established, and then it is able to be referenced by the outer layer. In other words, to complete a design in a clean architecture way, the domain behavior must be defined first, and the database design should be the last. This is the exact opposite of data-oriented design.

Domain-driven Design

Before starting the actual design, let me explain my usual design process, which also echos the onion architecture.

1. Discover user stories (entities)
2. Design use cases
3. Model domain objects
4. Implement unit tests
5. Code

In later sections, I will also design with this process. The problem we want to solve is to build a sign-in mission mentioned earlier.

Discover user stories

To start a design, we must be able to understand the whole picture of the entire requirement, and user stories are a language that can describe the requirements. In our needs this time, the stories are similar to the following.

1. Get corresponding rewards when you sign in consecutively.
2. Display the sign-in status and received rewards for this cycle.
3. Get 100 diamonds when opening the gift box.

We convert the descriptions in the requirements document into semantics that developers can understand through an ubiquitous language. With any requirement, there must be a story behind it, and the designer’s job is to discover those stories. On the other hand, for the developers, they implement those stories in coding.

Design use cases

With the story, then we need design the use cases that the story faces. Unlike a story, a use case refers to the outcome of a given user scenario. For example:

1. Sign in: When a user signs in for four consecutive days, the first sign-in on the fifth day can get 30 diamonds and a gift box. But the second sign-in got nothing.
2. Open the gift box: When opening the gift box, you can get 100 diamonds, but it cannot be opened again.

From the above description, use cases are actually an extension of user stories and describe details that are not defined in the story. Therefore, from the use cases, we can draw a flowchart to explain the whole user scenario in detail. Let’s take sign-in as an example with a flowchart.

Starting from the top starting point, it is the moment when the sign-in action occurs, so it is represented by SignIn: now. Next, we need to know how long is the difference between this sign-in and the “last sign-in” in days. If it is 0 days, it means that you have already signed in, and there is no reward to get. Or the difference is greater than 1, indicating that the sign-in is not continuous this time, and the entire cycle needs to be reset. In case of 1 exactly, it is continuous sign-in, thus the continuous date is incremented, and the current time is recorded.

Finally, check the table according to the number of consecutive days to know how many rewards you will get.

It is also easy to display how many consecutive days you have signed in. Suppose we use list to represent the signed-in records.

• Only sign in for one day: [1, 0, 0, 0, 0, 0, 0]
• Sign in for three consecutive days: [1, 1, 1, 0, 0, 0, 0]

Therefore, we can know how many 1 to insert to the list from counter.

The flow of opening the gift box is similar, so I won’t explain too much here. The final code will include opening the gift box.

Model domain objects

From the use cases we can know that we will need two very important variables: counter and last. In fact, the rest of the state is determined by these two variables, so we can start modeling.

In order to describe the entire sign-in mission, I believe that each user will have its own state, so we encapsulate the user state into a domain object called SignInRepo. The Repository in DDD is used here. Then with the user state, we can describe the whole story. There are two actions in the story, signIn and getTimeline, which represent story 1 and story 2 respectively.

Because SignInRepo is defined on the basis of use cases, it is part of the entity in the onion architecture. According to the flowchart, it has two private variables and two public methods. The reason why update has a parameter is that we can see from the flowchart that we only have one operation counter++, set last=now, and now must be passed in from the outside. As for SignInService, it can be known from the name that it belongs to the domain service.

Once we have domain objects, we can start developing in test-driven development, TDD.

Implement unit tests

In the development process of TDD, we write the corresponding tests according to our user stories at first, and then the actual coding is carried out. Hence, in this section, we’ll explain how to write unit tests with our defined stories and models. Let’s take a regular story as an example, suppose we’ve signed in for six days continuously, and on the seventh day, we’ll get 100 diamonds and a gift box.

First, write a test based on our story.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

describe("step1", () => {  
  it("continuous 6d and signin 7th day", () => {  
    const user = "User A";  
    const now = "2022-01-07 1:11:11";  
    const service = new SignInService(user);  
      
    const timeline1 = service.getTimeline();  
    expect(timeline1).to.deep.equal([1, 1, 1, 1, 1, 1, 0]);  
      
    const result = service.signIn(now);  
    expect(result).to.be.equal(100);  
      
    const timeline2 = service.getTimeline();  
    expect(timeline2).to.deep.equal([1, 1, 1, 1, 1, 1, 1]);  
        
    const result = service.signIn(now);  
    expect(result).to.be.equal(0);  
  });  
});

One of the stories is briefly described above, there is a user, A, who has signed in for six consecutive days, and when he signs in at 2022-01-07 1:11:11, it is the seventh day to sign in. He gets 100 diamonds as our expectation.

But such a story is not complete, because six consecutive sign-ins have not been defined. So let’s modify the test a bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

describe("step2", () => {  
  it("continuous 6d and signin 7th day", () => {  
    const user = "User A";  
    const now = "2022-01-07 1:11:11";  
    const repo = new SignInRepo(user);  
    repo.restoreSingInRecord(6, "2022-01-06 5:55:55");  
    const service = new SignInService(repo);  
  
    const timeline1 = service.getTimeline();  
    expect(timeline1).to.deep.equal([1, 1, 1, 1, 1, 1, 0]);  
      
    const result = service.signIn(now);  
    expect(result).to.be.equal(100);  
      
    const timeline2 = service.getTimeline();  
    expect(timeline2).to.deep.equal([1, 1, 1, 1, 1, 1, 1]);  
        
    const result = service.signIn(now);  
    expect(result).to.be.equal(0);  
  });  
});

In order to restore the entire use cases, we newly defined a repo and added an auxiliary method: restoreSingInRecord. This helper can also be used as an interface to retrieve values from the database in future implementations. Subsequently, such a story is complete and can begin to go into the production code.

Code

In the previous section, we have a complete unit test, and then start implementing SignInRepo and SignInService.

SignInRepo is easy to implement when there is no database, just follow the flowchart to finish update and reset. SignInService is totally implemented in accordance with the use cases, and the flowchart is converted into the actual code.

In this way, this requirement is half completed, and the remaining process of opening the gift box is basically the same, so I will just post the final result. The full implementation can be seen as follows.

Summary of Domain-driven Design

In fact, the above implementation just borrows some DDD terminologies, and does not fully implement DDD’s “prescriptions”. From my point of view, DDD provides a concept that enables people to know the importance of the domain and has the ability to abstract the domain.

That is to say, it is up to you whether to follow the textbook to implement Entity, Value Object, Aggregate, and Repository or not. It needn’t implement in DDD by following the textbook approach. The implementation depends on the proficiency and understanding of needs.

In this article, a standard design process is provided, so that everyone can disassemble the original requirements and convert them into models with domain knowledge by following this process.

The process of implementing the model, it begins with the corresponding tests to achieve test-driven development.

Of course, in the real world, it is not as simple as the example in this article. But the design process is the same, starting from the story, defining the use cases through the story, then modeling according to the use cases, writing tests according to the stories, and finally implementing it.

By the way, I explained some design details a while ago, such as:

• Q1: Why do we need to define a repo? Dependency Injection
• Q2: Why do we need layers? Layered Architecture
• Q3: How to evolve a system? Shift from Monolith to CQRS

If you encounter software design problems, you are also welcome to discuss them with me.

Originally published on Medium

A look at the data-oriented design approach

• Part 1, Data-oriented Design
• Part 2, Domain-driven Design

There are many articles and books talking about Clean Architecture, but the focus is on why and what, such as why to use an onion architecture, what each layer represents, and what is the SOLID principle.

However, developers actually want to pay more attention to how, that is, how to write a good program architecture according to those design principles.

In this article, we try to solve a common architectural problem in a clean architectural way.

In many mobile games or commercial websites, in order to improve user retention, a sign-in mission is designed. If you sign in, you will get some rewards. When you sign in continuously for a certain number of days, there will be additional gift boxes that can be opened, and you will get bonuses after opening. Let’s try to fulfill this requirement together.

Problem Domain

Users who log in continuously will get diamonds, as shown in the following table:

To simplify the problem, let’s get a fixed amount of 100 diamonds the gift box is opened each time. For example, if a user signs in for four consecutive days, the user can get 30 and a gift box on the fifth day of sign-in.

Every time the user enters the homepage, he will see a sign-in map, telling the user how many consecutive sign-ins have been completed in this cycle and how many rewards have been received.

Design Phase

There are two different design approaches, data-oriented design and domain-driven design. We will start talking about data-oriented design and tell you why it’s not a mainstream design solution.

Data-oriented Design (Can’t Say Good Approach)

The data-oriented design means that when we see a problem, we first think about how to store the data, and then manipulate the defined data format to try to solve the original problem.

Therefore, after getting the question, the first step is to think about what database to use, what schema it will have, and what format the data will exist in.

Let’s take the sign-in mission as an example. Suppose we design on a small monolith, usually with a relational database, so we first define a table to realize our problem. Define a sign-in table to store the time of each sign-in, so that you can know how many rewards you can get based on the previous sign-in records.

From the above table, if today is 2022-01-04, then John can get 15 diamonds when he signs in, and Mary will start a new cycle because she has not signed in consecutively and only get 10 diamonds.

The logic of the mission will be to filter the sign-in date of a specific user and sort them by date, then retrieve the latest N records. By calculating the total days of the consecutive sign-ins, and then you can know what reward to get this time.

There is a problem with such design. When the user’s sign-in is not interrupted at all, i.e., the number of consecutive sign-in days is greater than N, then the amount of data is not enough to determine the current reward. Nevertheless, we do not want to expand N to avoid pulling too much data makes the database a bottleneck. Thus, we should try making some changes to the original schema, such as adding a new field, diamond.

For John, signing in on 2022-01-09 will get 10 diamonds because we know the end of the previous cycle was 2022-01-07.

The story is not over yet. When the number of users grow, the performance of RDBMS will be tough. Every time you enter the homepage, you have to pull data from the database, run a complex calculation, and finally see the result. This process is very inefficient.

In order to improve the efficiency of the homepage, most projects will introduce a cache to record the status of the entire sign-in. However, how do you know if the cache is invalid? Or the cache is actually not invalid, but distorted. I have covered this problem in a previous article Resilient Caching. By adding integrity, we can distinguish whether the data is reliable. Once the data is corrupted, we will rebuild the data from the RDBMS.

Summary

From the above introduction, we found that even the cache has many aspects to consider, the whole data-oriented design is very complicated, and the process is very struggled.

Hence, the domain-driven design was born.

In part 2, we will start designing from Entity, covering use cases, then implementing the unit tests like TDD, and finally completing our mission. I will provide a better design process, and you can follow the steps to simplify every user stories you will encounter. Let’s call it a day. Until next time

Originally published on Medium

I often get a consultation.

Hey, architect. I want to use a microservice to refactor the system, could you help reviewing my architecture design?

This situation is very common, but instead of saying yes right away, I will ask a question.

What problem is your microservice trying to solve?

Guess what, most of the responses were “No, I think this is where microservices should be used.”

And, most of the time, I would just say no.

I know that developers like to try new technologies, whether it is new languages, new frameworks or even new architectures, it is easy to try those brand new technologies through a microservice. You can do these new experiments on new business requirements, and I’m open to that. However, when it comes to refactoring, it’s not that simple.

As I said in my previous article, the evolution of any system must solve some problems that are currently unsolvable. Please ask yourself a few questions before refactoring your legacy system with microservices.

1. Do you reuse the same code base?
2. Does the microservice share the existed databases?
3. Do you just change the function call in the monolith to an API call through a microservice?
4. Is the same release cycle as the original system?

If the answer to any question is yes, please stop and think carefully, do you really need microservices? Microservices built with these thoughts are just for doing, not refactoring, or even worse.

There are several principles when refactoring a legacy system with microservices.

1. Independent deployments
2. Data isolation
3. Non-overlapping domains

For a huge legacy system, data isolation may not be achieved immediately. Nevertheless, the original system must be strangled by the new microservice in implementing an anticorruption layer.

From the above diagram, in the process of continuous evolution, Legacy should become smaller and smaller, while Modern will become larger, and finally achieve the purpose of replacing Legacy with Modern.

In addition, there is another problem with the data isolation of microservices. How to do when microservices and original systems must maintain data consistency? I’ve covered distributed transaction before, but are you ready for eventual consistency? This means that has your client identified and handled the eventual consistency correctly?

Changes in software architecture come with pros and cons. From a monolith to a microservice, there will be many benefits.

1. Reduce the problem domain.
2. De-couple the release cycle.
3. Productivity increases by using new techniques without the original big ball of mud.

However, it will also create new problems.

1. Data consistency.
2. Maintain effort increases, especially when the team size remains the same.
3. Performance drops due to more coexistence between the monolith and microservice.

There is no one-size-fit-all solution, we can only make the least worse decision. And with microservices, are you really ready?

Originally published on Medium

Resilient Caching in Redis

Before the introduction, I will describe what is the resilience.

Distributed systems will fail, a resilient software system will not try to avoid failure but expect it and respond gracefully.

We all know data in cache will sooner or later be lost. The reason comes from many situations even the writing operation is successful. Let’s take Redis as an example.

1. Redis server crashed: of course, data was lost, because all data in Redis was stored in the memory.
2. Redis replica went through the partition failure and recovered again.
3. Even without any hardware/software failure, Redis spent all memory spaces and proceeded to evict the existing cache.
4. If using ElastiCache of AWS, the upgrade process will also drop all entries.
5. And so on.

As far as we know, the cache is not reliable and hard to ensure the data consistency. However, there are indeed some tasks which leveraged cache to improve the performance requiring the reliability.

We have talked about the cardinality counting in my previous article, and that is a great example to show the importance of data persistence. If the collections in Redis is no longer trusted, how can we design a large system? The solution is making your cache more resilient.

Resilient Collections

Following my example of health status, we want to make sure our sensor networks work properly; therefore, we record the activity of API calls in Redis. In order to simplify the explanations, I will choose bitmap with recording in days in this section.

The data in Redis bitmap must answer a question, how many days does the sensor work in this month?

We can leverage my previous statements as follows.

1
2
3

SETBIT sensorA_jan 1 1  
SETBIT sensorA_jan 2 1  
SETBIT sensorA_jan 3 1

The statements show that A worked at 1/1, 1/2 and 1/3. And we can get the answer from: BITCOUNT sensorA_jan. What if the Redis crashed at 1/2? The bitocunt would become 2 without the record of 1/1. In such case, the bitcount is distorted.

To tackle this issue, we can add an integrity into the collection, let’s say SETBIT sensorA_jan 0 1. When we want to retrieve bitcount, we check the integrity first, GETBIT sensorA_jan 0. If the result is 1, which means the bitcount is under controlled, otherwise, we have to rebuild the data from a durable storage like a database.

Conclusion

Since we have to rebuilt the result from a database, why not do it at the first? The reason is quite simple, we want to keep the system high performance. We should ensure the reporting throughput by using cache and expect there should be failed someday. Nevertheless, we still can get the correct result by some heavy aggregating operations on the database.

This is how resilience does, and this article provides a straightforward example to demonstrate how to accomplish a cardinality counting resiliently.

Originally published on Medium

Shift from Monolith to CQRS

Software design is an evolving process. Every large system starts from a tiny system. When a problem is encountered in the existing architecture but cannot be solved, the system will begin to evolve. Every evolution is accompanied by some technical selections. What problems should be solved? What price will it pay? As an architect or a senior engineer, there must find a reasonable way to evolve, regardless of the development schedule, technical stack, and team level, it is necessary to be able to meet these criteria before a feasible solution can be made.

This article will introduce the spirit of CQRS (Command Query Responsibility Segmentation) and the problems to be solved. We will start from a small monolith and evolve it like the evolution of every software system, and this article will introduce the reasons and approaches behind each evolution.

Traditional Monolith

This is the most common system design. There is an API server, usually a restful API, and a database. Client negotiates the transmission format with the backend in advance. Both reading and writing are done through DTO, data transfer object. However, when the backend processes business logic, it converts DTO into a domain object with domain knowledge and uses the domain object as the storage unit of the database.

In order to achieve Read/Write Splitting, in the write path on the left, the client sends up DTO to the backend to perform CUD (create/update/delete) operation on the database, and the backend responses to the client with Ack for success and Nak for failure after processing. In restful API, usually 2xx represents success and 4xx represents failure. The read path on the right simply obtains the corresponding DTO through a read request.

I further explain the meaning of DTO for the client. DTO on the client usually contains all the data to render on the screen. For example, when you look at your profile on a social medium, it will include your name, account, and other personal information, as well as your own recent activity, and even the activity you followed. DTO contains all the information that needs to be presented on this page.

Why do we need to emphasize Read/Write Splitting? Can’t we use the same procedure on both the read and write path? Because we want to better optimize our system in the future. The write path has a particular optimization method, and so is the read path. For instance, to make a cache, read aside caching can be used on the read path to reduce the response time. And, the write path can be improved by write through caching. Secondly, the writing may also be performed asynchronously. All the DTOs are written into the message queue and processed by the worker to handle the huge amount of written data. Moreover, each appropriate database may be used for writing and reading.

Therefore, Read/Write Splitting is essential. And it should be taken into consideration in the early stages of system design. The write path is to concentrate on data persistence; while the read path is to concentrate on data query.

Nevertheless, there are two main problems in this system design model.

1. Anemic model. It is also known as CRUD model. When backend focuses on data conversion, it is difficult to have space to handle business logic, which will cause business logic to be scattered everywhere. Domain knowledge will also disappear, e.g., to an ecommerse website, we will say “purchase” instead of “inserting an order record”.
2. Insufficient scalability. From the perspective of system architecture, the database can easily become a bottleneck of the entire system. Both reading and writing must be onto it. The problem of RDBMS is even more serious due to no horizontal scaling.

Task-based Monolith

In order to solve the problems encountered by the above traditional monolith, here we try to introduce the concept of domains.

This diagram is basically the same as the above one. The only difference is replacing DTO with messages on the write path. Messages contain actions and data, not just the data itself like DTO. Thus, we can carry domain-specific actions in the message to make it easier for the backend to recognize each action, and have a corresponding domain implementation.

At this stage, C in CQRS has appeared, and message is a kind of commands. However, the problem of scalability is still unresolved.

In addition, although we have simplified DTO and changed to use messages to communicate, we still need DTO on the read path. Let’s take the social medium as an example again. When modifying the nickname, the format of a message may be {"rename": "LazyDr"}. But when rendering the profile, we still need additional information such as activities. This information gap makes it necessary to do a lot of processing on the read path to retrieve DTO.

CQS (Command Query Segmentation)

The emergence of CQS is to solve the above pain points of Read/Write Splitting.

When reading, the client needs DTO, so the backend can do some optimizations dedicated to reading on the read path, such as pre-generating DTO from the original domain object, and storing DTO in a dedicated database for reading.

In this way, on the read path, the implementation of the application service becomes simpler. The application service can become a thin read layer, which only needs to be responsible for paging, sorting, etc. After requesting, the client can easily retrieve DTO from the database.

So the question is, who is going to generate those pre-built DTOs? It is the responsibility of the write path.

Although the diagram is similar with the examples seen before, in fact, in addition to persist the domain object, the application service must also persist DTO. In other words, most of the business logic will be pressed on the write path, and various read views need to be prepared.

At this stage, we have solved most of the problems encountered by the domain, but scaling still has no solution. Now, we further define scaling. Scaling has two different aspects.

1. Traffic: increase in write volume.
2. Extension: functional requirements increase, such as the need for a variety of different read views. Continuing to take the social medium as an example, there is one presentation on the profile, but there may be another presentation on the timeline.

CQRS

Why is the write path responsible for preparing the read view? Writing should focus on persistence, and those various read views should not be processed on the write path. But there is only reading on the read path, who should prepare those read views?

Therefore, the total solution is as follows.

The write path on the left and the read path on the right have been introduced in the CQS section. The only difference is that an eventually block is added, which is responsible for converting the database on the write path into the database used on the read path. Once data synchronization is involved, it is possible to encounter data consistency issues, so here is a list of several approaches for implementing eventually consistency, sorted by time consuming from short to long:

1. Background thread: The typical representative is Redis. After the data is written to the primary, Redis will immediately send the data to the replicas in the background.
2. Message queue plus workers: This is a common practice for asynchronous data replication. When writing to the database, an event is initiated into the message queue and processed by the workers.
3. Extract-Transform-Load: This time interval is the longest, ranging from a few minutes to a few hours. Use map-reduce or other methods to write the results on the other side.

No matter which approach, the single source of truth is mandatory. That is to say if there is any failure occurred on converting, the system must be able to recover the unfinished jobs. Therefore, the data has to be unique and reliable.

Data usually laid on two types,

1. state: State refers to what you see at the moment, such as the balance written on the bank passbook.
2. event: An event is an action to modify each state, such as every transaction record on the bank passbook

Actually, we already have messages that can be stored as events. For the write path, it is very efficient to store messages in order. Through each different message, you can easily build a different read view according to your needs. This approach is also called event sourcing.

But only events are difficult to use efficiently. In order to obtain the final result, every conversion must be run from the beginning to the end to rebuild the read view. As a result, the hybrid method would be ideal. On the write path, both the state and the event are kept, and the conversion process can choose the data source based on the actual situation.

To sum up the whole life cycle of data in CQRS.

The data starts from the client and then enters to the backend in the command format. According to the business logic, it is converted into a domain object and stored in the database. These domain objects are converted into various read views and stored in different read special databases according to requirements. Finally, the client takes these read views back in the form of DTO.

Conclusion

There are many books and articles describing DDD and CQRS with many patterns. From my point of view, those patterns lead to the limitation on imagination of DDD like Entity, Value Object, Aggregate, etc. It results in most developers feel DDD is far from themselves and hard to realize as well as implementations. In fact, the concepts of DDD is not so complicated; instead, DDD is proposed to encapsulate the business logic and then facilitate expending the functional requirements.

CQRS is even simpler. In this article, we start from the process of system evolution, understand the entire system design process and the problems to be solved, and finally derive the conclusion of CQRS naturally.

There is no silver bullet in the system design. Each evolution is made for solving some specific problems, however, it may come out a new problem. Take the design processes of this article as an example, CQRS seems to resolve all mentioned problems, anemic model, and insufficient scalability, but actually CQRS also brings new problems, such as data consistency. Each technical selection has its trade-off, as long as understanding all threats behind every option, you can choose relatively acceptable approaches.

Even if you choose CQRS, in practice, there are still three choices on implementing eventual consistency. System design is the result of continuous selection.

The purpose of this article is to tell you that DDD is not that scary, and CQRS is not that complicated, it is just a decision.

Originally published on Medium

Array Operation in MongoDB

MongoDB is a document-based database, and each document is in JSON-like format. Therefore, MongoDB can store various data structures. Moreover, in order to manage those documents, MongoDB provides a powerful ubiquitous language to operate documents.

Here comes a question. We usually perform CRUD on documents, however, do you know how to update the element in a list from certain documents? This article is my experiment of array operations.

The original collections is given as follows:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Update a key of elements in an array

We have an array, prop, with elements, and those elements contains a and b are integer and boolean respectively. What we want to do is update b to false if a is equal to 1.

The command is:

db.test.updateMany({"prop.a": 1}, {$set: {"prop.$.b": false}}).

There is a dollar sign representing the result set from filtering. So, prop.$.b means all b with a is 1.

After executing the command, the collection would be:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1, b: false }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1, b: false }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Remove a key of elements in an array

We can $set a key; on the other hand, we can remove a key either. The command is like the $set one.

db.test.updateMany({"prop.a": 1}, {$unset: {"prop.$.b": ""}})

When we want to $unset a key, the syntax is {key: ""}, thus, {$unset: {"prop.$.b": ""} is as expected.

Finally, the collection would be:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1 }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1 }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1 } ] }

Remove a key of certain elements in an array

Furthermore, if we want to remove b when b is true rather than all b. How can we do?

The solution is:

db.test.updateMany({"prop.a": 1}, {$unset: {"prop.$[elem].b": ""}}, {arrayFilters: [{"elem.b": true}]}).

We are using arrayFilters to match the specific elements and indicating those elements to the $unset criteria. Then, the collection would become:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1 }, { a: 2 } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1 }, { a: 2 } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Although we can use filter like {a: 1, b: true} to leverage the dollar sign operations, arrayFilters provides more possibilities to achieve more complex user scenarios. You can try to skill it.

Originally published on Medium

Are Design Pattern and Clean Code Useful?

I overheard our junior developers discussing the value of design pattern and clean code. I vaguely remember their conclusion is, “those techniques make simple things become complex, and that is the favorite thing of those technical authorities who are far from the reality”. Their discussion ends up a negative agreement.

There are some examples from their conversation. For instance, in clean code, the length of a function is limited in a regular size, which results in to overlook a whole function’s functionality, it has to dive into several nested sub-functions. Besides, design pattern ask us encapsulate a simple task into an object, and the objects are everywhere. Coding with objects will make the source code much more longer.

Well, I can understand their pain points. Those pain points come from some reasons such as

1. Their duty is quite simple without too many complicated design or further expandability needs.
2. In a small organization, they don’t need to consider the cooperation. All features are handled by a few people.
3. They don’t realize the concepts of those paradigms, instead, they just see the “tricks”.

If I were in their position, I perhaps agreed with them. Nevertheless, I had not joined their discussion. After all, I am more close to an authority for them. But, I have to say in their mentioned examples, those methodologies have its meaning.

Firstly, restricting the function length must be combined with a well-defined naming convention and a good parameter design to let everyone understand what’s the purpose of this function without traversing the detail implementations. Secondly, encapsulating objects is to accomplish a better abstraction and become easier to achieve Open-Closed Principle. Whether encapsulate logic in Strategy Pattern or encapsulate the creation in Factory Method, they make developers be able to use the existing code without reinventing the wheel. In addition, developers can extend the functionality more easily.

Design pattern and clean code provide some tricks and principles, however, the most important thing is the spirit and purpose behind them. For example,

1
2
3
4
5
6
7
8

if prop == 1:  
    do_a()  
elif prop == 2:  
    do_b()  
elif prop == 3:  
    do_c()  
else:  
    do_default()

This simple example can be considered that those do_XX are some flatted source code. Those junior developers would like to see those behaviors clearly on the same code level like the above python example. In order to understand the implementation at a glance, actually, there are many a way to do better.

1
2
3
4
5

func = {  
    1: do_a,  
    2: do_b,  
    3: do_c,  
}

1
2
3
4

if prop in func:  
    func[prop]()  
else:  
    do_default()

Through simple packaging, it is easier for people who want to read the code to understand the purpose. They can inquire about the part of details they want to see by themselves. In addition to add a new feature, developers just need a handler in func. Furthermore,

1

factory.create(prop).run()

By adopting the factory method, the function become more slim. The cognitive load for readers is low as well. In fact, this piece of code encapsulate the logic in polymorphism to run, aka, Strategy Pattern. It is unnecessary to talk about these details implementation of design pattern or clean code. Once you understand his spirit, you can easily have your own implementation. In the above example, it is indeed not a “standard” factory method implementation.

After skilled design pattern, those pattern names just provide a common language between developers, so that they can communicate more smoothly. For instance, “I have done an observer pattern over here”. It is no need to explain further that you can realize there are a subject and an observer with attach and notify. Even no documents, you still can understand the prototype of the code in your mind.

The purpose of coding is not only making features but also be a communication tool. Clean Code said, “Don’t document your code. Code your documentation”. But these can only be experienced after those junior developers have written about larger functions, cooperated with more people, and met more changing requirements. Everyone chooses a different career path, and not everyone has the opportunity to enter the world where the design pattern and clean code are facing in.

Originally published on Medium

Cardinality Counting in Redis

Cardinality counting is used to calculate the amount of elements without any duplication. In Redis, there are many data structures able to accomplish this job. However, what is the most applicable way for your use cases? This article will show the consideration behind the technical selection.

User Scenario

Suppose we need to get the failure rate in a sensor network to investigate the reporting qualities. Therefore, we have to record the health status in hours by the incoming requests.

The key point is to simplify the process, we don’t want to get the value first, determine whether it is existing, and then insert the record like:

Instead, we should insert the record every time, and the storage can de-duplicate for us. Or, we can limited pre-process data to make the storage do faster.

Assume that we have a sensor A, and the sensor requested to the server at 1/2 1:11, 1/3 2:22, and 1/8 3:00.

Alright, let’s see how Redis did cardinality counting.

Set

The basic idea is using set. Before adding to the set, we have to pre-process the date. Due to our requirement, we only keep the hour without minutes and seconds.

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.toISOString();

Then, we can add d1 to the set via SADD.

1
2
3

SADD sensorA "2021-01-02T01:00:00.000Z"  
SADD sensorA "2021-01-03T02:00:00.000Z"  
SADD sensorA "2021-01-08T03:00:00.000Z"

In order to get the health status, we can use SCARD.

1
2

SCARD sensorA  
> 3

The implementation of using set is simple; nevertheless, if we want to count the health status during the specific period like in 2021, set cannot handle this request.

Sorted Set

Therefore, if we would like to meet the needs of specific time period and whole time, we can leverage sorted set. The implementation is similar to the set. Firstly, pre-process the date.

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.getTime();

It is unlike using ISO string, we are using epoch here to find the specific time range easily. Now, we can add to the sorted set via ZADD.

1
2
3

ZADD sensorA 1609520400000 1609520400000  
ZADD sensorA 1609610400000 1609610400000  
ZADD sensorA 1610046000000 1610046000000

To find whole count in it:

1
2

ZCARD sensorA  
> 3

On the other hand, in order to search the specific time range, we assign the start and end in ZCOUNT.

1
2

ZCOUNT sensorA 1609520400000 1610040000000  
> 2

Bitmap

We walked through two approaches, but neither set nor sorted set are space efficiency. The detail implementation in Redis takes a huge space to indicate the data structure. When the amount of sensors become larger or the duration of records become longer, the space in Redis will grow quickly.

How to reduce the space? We can leverage the extended function of string, bitmap. Bitmap is very space efficiency, each bit takes 1 bit as its meaning.

But the pre-process is a little complicated, we have to get an offset to operate bits. For example, we can calculate the diff hours between the service launch time and the current time, e.g. 2021/1/2 1:11.

1
2
3
4

const base = new Date(2021, 0, 1, 0, 0);  
const date1 = new Date(2021, 0, 2, 1, 11);  
const diffTime = Math.abs(date1 - base);  
const diffHours = Math.ceil(diffTime / (1000 * 60 * 60));

After that, make the diffHours be 26.

1
2
3

SETBIT sensorA 26 1  
SETBIT sensorA 51 1  
SETBIT sensorA 171 1

Hence we can get the total health status by BITCOUNT.

1
2

BITCOUNT sensorA  
> 3

BITCOUNT also provides the range match, so we can assign start and end to search a specific time range like sorted set. It is noteworthy that the start and end here represents the bytes offset. We have to convert the start time and end time to the diff hours bytes, the calculation is complex, so I am not going to provide an example in this article to prevent losing focus.

HyperLogLog

The final approach is called hyperloglog. This is an algorithm for big data statistics. Redis provides it as the built-in method.

Whether it is set, sorted set or bitmap, the space usage will larger and larger as time goes by. For instance, if we keep the health status for 10 years, even bitmap will take huge space, 365 * 10 * 24 / 1024 ~ 85.5 KB.

However, in hyperloglog, the space usage is constant. No matter how long the retention you need, hyperloglog takes 12 KB constantly. And the pre-process is like set,

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.toISOString();

Then, we can add the date to hyperloglog via PFADD.

1
2
3

PFADD sensorA "2021-01-02T01:00:00.000Z"  
PFADD sensorA "2021-01-03T02:00:00.000Z"  
PFADD sensorA "2021-01-08T03:00:00.000Z"

It is simple to get the total count.

1
2

PFOCUNT sensorA  
> 3

Hyperloglog is not quite precise, the result of PFCOUNT may contain some deviations when the data set is huge, but the performance is pretty well.

Conclusion

Let’s summarize those 4 approaches.

The example in this article is trivial, however, I believe you can realize the concepts of those approaches. The most important thing is each approach has its own strength and drawback, using them in a smart way is developer’s responsibility.

Originally published on Medium

The exciting 2021 is almost over, let me review my architect journey. Actually, I had been an architect in embedded system before, then I decided to change my career to the cloud service and backend development. After working and studying for years, I gradually changed from an engineer to a principle engineer. At the beginning of this year, I made my mind to be a backend architect. And finally, the dream comes true.

In retrospect, I feel the change of professional position went smoothly. Therefore, I will review what I have done and will be done for your reference.

Why changing to architect

People often ask me why I want to be an architect instead of being a developer. There are two main reasons,

1. Programming language can never be learned. New languages will appear every once in a while and become popular, e.g., golang. Even the existed languages are still enhanced and become unfamiliar like C++11, C++14, C++17, then C++20. On the other hand, the software engineering is much more constant. The core design of data stores, architectures, distributed systems are classic. In other words, mastering these technologies has a relatively high return on investment.
2. When I was a developer, every time a clean architecture is written, it becomes a mess soon. Cross referencing, circular referencing, etc. are everywhere, I cannot stop it even being a staff engineer. It is better to become an authority and make rules instead of actual development.

What is R&R of architect

“Managers are not confronted with problems that are independent of each other; but with dynamic situations that consist of complex systems of changing problems that interact with each other. I call such situations messes…Managers do not solve problems, they manage messes.”

Some f-Laws from Ackoff, R. L., H. J. Addison and S. Bibb. 2007. Management f-Laws. Triarchy Press Ltd.

The same applies to architects.

1. Find out the root cause of the intractable online problem and hand it over to an engineer to solve it.
2. Monitor and clarify system bottlenecks, find ways to optimize and improve.
3. Educate and guide the development team to improve everyone’s skills, such as the low-level details of the database, SOLID principles for the system design, etc.
4. Participate in the requirement meeting on the product side and propose a reasonable estimated schedule and change scope.
5. Improve development process like CI/CD pipelines, design review meetings, trunk-based development, etc.
6. Participate in seminars and survey new technologies, techniques, and methodologies.

From above description, we can find an architect does not solve issues completely; instead, the core value of an architect is simplifying the process whether it is an issue or a product requirement.

How to be an architect

In order to be an architect, there are some properties helping the position change, and I find those properties really work.

1. A lot of reading, you must have the habit of reading. Work process books, professional technical books, and management books are all essential. You have to be greedy.
2. A lot of thinking, there is no code or architecture that cannot be optimized, and finding a reasonable improvement way is the responsibility of an architect.
3. A lot of communication, the architect is not the person who actually does things, and (s)he must be able to guide developers in a timely manner while establishing authority.
4. Stay motivated and optimistic. The work of the architect is very heavy. You have to deal with a large number of people and issues every day. If you don’t manage your mood well, you can easily fall into frustration.

These four properties are useful, even though you don’t want to be an architect, you still can grow yourself up on the career path of a software engineer. At least, thanks to these properties, I have been able to change from an embedded system engineer to a backend architect successfully in 10 years.

Afterword

Being a software architect is pretty interesting. I have no assigned or dedicated jobs in my organization, what I have to do is finding an issue and trying to fix it. All my tasks are assigned by myself. I can schedule my tasks in a proper way and be free to study from any resource. This is my journey in past 10 years, and you can write your own story on your career. It would be great if you could share with me.

Originally published on Medium

Layered architecture is a common pattern of an architecture design, and it is very suitable for a small scale monolith. There are many advantages such as,

1. very easy to design and implement
2. very in line with human nature
3. cooperate with each other more easily

When we are talking about the layered architecture, we often have a myth; in order to replace the dependency, we should make those dependencies as a layer.

In fact, that’s not the purpose of layered architecture. I believe we seldom need to change a database, a framework, even a programming language. Therefore, what is the benefit of using layers?

Before we discuss further, we should see an example first.

The above diagram is a classic layered architecture for a web server. We separate a request flow into 4 components, route, controller, service and repository. Each component has its responsibility to handle a part of a request flow. This kind of implementation is also known as Single-Responsibility Principle.

In many articles, we had been taught that route is made for the web framework changed, and the repo is to nail the database. However, as I mentioned earlier, we seldom need change those hard things. Route and repo here are just completing their own duties.

Single-Responsibility Principle

I will describe those 4 components’ responsibilities to let you realize them more confidently.

• Route is a the wrapper of a web framework. To interact with a framework and hide the implementation, we build a layer to enclose those details. In route layer, we usually define HTTP routes right here including HTTP methods and URI.
• Controller is the first line to handle a request. The main responsibility is validating the input request and assembling the response. This layer often implements some error handlers to reflect like “400 Bad Request”, “500 Internal Server Error”, etc.
• Service is where the business logic in. The core values of a web product are all here. If it is a E-commerce site, service will calculate the recommendation, orchestrate the render content, and all other logic.
• Repo is an abstraction onto a database. In order to isolate the business logic, we don’t want the database operation to pollute the code in services, thus, repo plays this role in accessing persisted data.

The text description is a bit abstract, so I provide a more concrete example: https://github.com/wirelessr/Clean-Architect-in-Golang

The GitHub repository is quite simple due to the screaming architecture, hence, we can know this is a classic web service.

• Route is a mux wrapper in http. It will do nothing and route HTTP request to the controller.
• Controller defines API interfaces. In AddPost, it also validates the request format and assembles the response.
• Service own all business logic including how to access durable data, what the post format is, and how to generate the post identifier.
• Repo hides all database details, so you can choose what data store you want. But, again, we don’t change our data store decision easily. Therefore, repo here is to encapsulate the data store access to make service be independent from firestore.

What’s Benefit

Do we just separate our web service into several layers for SRP? No. Actually, this comes some benefits. Let me show two most important advantages,

1. Layers make testing become more easily.
2. If those layers are modules, we can decouple the deployment of layer modules.

The second one is straightforward. Thus, I will dive into the test. According to my previous article, we can leverage the dependency injection to accomplish a high testing coverage. For instance, in my golang example, in order to verify the service, we can create a mocked repo to ignore the data store access.

Conclusion

There are many articles trying to compare layered architecture and microservice architecture. However, layered architecture is not only an architecture pattern but also the coding technique for a SOLID software. This article tells you still can use layered architecture in your microservice like my post example. They are not two mutually exclusive things; moreover, they are more close to each other.

Originally published on Medium