CT Wu

I often get a consultation.

Hey, architect. I want to use a microservice to refactor the system, could you help reviewing my architecture design?

This situation is very common, but instead of saying yes right away, I will ask a question.

What problem is your microservice trying to solve?

Guess what, most of the responses were “No, I think this is where microservices should be used.”

And, most of the time, I would just say no.

I know that developers like to try new technologies, whether it is new languages, new frameworks or even new architectures, it is easy to try those brand new technologies through a microservice. You can do these new experiments on new business requirements, and I’m open to that. However, when it comes to refactoring, it’s not that simple.

As I said in my previous article, the evolution of any system must solve some problems that are currently unsolvable. Please ask yourself a few questions before refactoring your legacy system with microservices.

1. Do you reuse the same code base?
2. Does the microservice share the existed databases?
3. Do you just change the function call in the monolith to an API call through a microservice?
4. Is the same release cycle as the original system?

If the answer to any question is yes, please stop and think carefully, do you really need microservices? Microservices built with these thoughts are just for doing, not refactoring, or even worse.

There are several principles when refactoring a legacy system with microservices.

1. Independent deployments
2. Data isolation
3. Non-overlapping domains

For a huge legacy system, data isolation may not be achieved immediately. Nevertheless, the original system must be strangled by the new microservice in implementing an anticorruption layer.

From the above diagram, in the process of continuous evolution, Legacy should become smaller and smaller, while Modern will become larger, and finally achieve the purpose of replacing Legacy with Modern.

In addition, there is another problem with the data isolation of microservices. How to do when microservices and original systems must maintain data consistency? I’ve covered distributed transaction before, but are you ready for eventual consistency? This means that has your client identified and handled the eventual consistency correctly?

Changes in software architecture come with pros and cons. From a monolith to a microservice, there will be many benefits.

1. Reduce the problem domain.
2. De-couple the release cycle.
3. Productivity increases by using new techniques without the original big ball of mud.

However, it will also create new problems.

1. Data consistency.
2. Maintain effort increases, especially when the team size remains the same.
3. Performance drops due to more coexistence between the monolith and microservice.

There is no one-size-fit-all solution, we can only make the least worse decision. And with microservices, are you really ready?

Originally published on Medium

Resilient Caching in Redis

Before the introduction, I will describe what is the resilience.

Distributed systems will fail, a resilient software system will not try to avoid failure but expect it and respond gracefully.

We all know data in cache will sooner or later be lost. The reason comes from many situations even the writing operation is successful. Let’s take Redis as an example.

1. Redis server crashed: of course, data was lost, because all data in Redis was stored in the memory.
2. Redis replica went through the partition failure and recovered again.
3. Even without any hardware/software failure, Redis spent all memory spaces and proceeded to evict the existing cache.
4. If using ElastiCache of AWS, the upgrade process will also drop all entries.
5. And so on.

As far as we know, the cache is not reliable and hard to ensure the data consistency. However, there are indeed some tasks which leveraged cache to improve the performance requiring the reliability.

We have talked about the cardinality counting in my previous article, and that is a great example to show the importance of data persistence. If the collections in Redis is no longer trusted, how can we design a large system? The solution is making your cache more resilient.

Resilient Collections

Following my example of health status, we want to make sure our sensor networks work properly; therefore, we record the activity of API calls in Redis. In order to simplify the explanations, I will choose bitmap with recording in days in this section.

The data in Redis bitmap must answer a question, how many days does the sensor work in this month?

We can leverage my previous statements as follows.

1
2
3

SETBIT sensorA_jan 1 1  
SETBIT sensorA_jan 2 1  
SETBIT sensorA_jan 3 1

The statements show that A worked at 1/1, 1/2 and 1/3. And we can get the answer from: BITCOUNT sensorA_jan. What if the Redis crashed at 1/2? The bitocunt would become 2 without the record of 1/1. In such case, the bitcount is distorted.

To tackle this issue, we can add an integrity into the collection, let’s say SETBIT sensorA_jan 0 1. When we want to retrieve bitcount, we check the integrity first, GETBIT sensorA_jan 0. If the result is 1, which means the bitcount is under controlled, otherwise, we have to rebuild the data from a durable storage like a database.

Conclusion

Since we have to rebuilt the result from a database, why not do it at the first? The reason is quite simple, we want to keep the system high performance. We should ensure the reporting throughput by using cache and expect there should be failed someday. Nevertheless, we still can get the correct result by some heavy aggregating operations on the database.

This is how resilience does, and this article provides a straightforward example to demonstrate how to accomplish a cardinality counting resiliently.

Originally published on Medium

Shift from Monolith to CQRS

Software design is an evolving process. Every large system starts from a tiny system. When a problem is encountered in the existing architecture but cannot be solved, the system will begin to evolve. Every evolution is accompanied by some technical selections. What problems should be solved? What price will it pay? As an architect or a senior engineer, there must find a reasonable way to evolve, regardless of the development schedule, technical stack, and team level, it is necessary to be able to meet these criteria before a feasible solution can be made.

This article will introduce the spirit of CQRS (Command Query Responsibility Segmentation) and the problems to be solved. We will start from a small monolith and evolve it like the evolution of every software system, and this article will introduce the reasons and approaches behind each evolution.

Traditional Monolith

This is the most common system design. There is an API server, usually a restful API, and a database. Client negotiates the transmission format with the backend in advance. Both reading and writing are done through DTO, data transfer object. However, when the backend processes business logic, it converts DTO into a domain object with domain knowledge and uses the domain object as the storage unit of the database.

In order to achieve Read/Write Splitting, in the write path on the left, the client sends up DTO to the backend to perform CUD (create/update/delete) operation on the database, and the backend responses to the client with Ack for success and Nak for failure after processing. In restful API, usually 2xx represents success and 4xx represents failure. The read path on the right simply obtains the corresponding DTO through a read request.

I further explain the meaning of DTO for the client. DTO on the client usually contains all the data to render on the screen. For example, when you look at your profile on a social medium, it will include your name, account, and other personal information, as well as your own recent activity, and even the activity you followed. DTO contains all the information that needs to be presented on this page.

Why do we need to emphasize Read/Write Splitting? Can’t we use the same procedure on both the read and write path? Because we want to better optimize our system in the future. The write path has a particular optimization method, and so is the read path. For instance, to make a cache, read aside caching can be used on the read path to reduce the response time. And, the write path can be improved by write through caching. Secondly, the writing may also be performed asynchronously. All the DTOs are written into the message queue and processed by the worker to handle the huge amount of written data. Moreover, each appropriate database may be used for writing and reading.

Therefore, Read/Write Splitting is essential. And it should be taken into consideration in the early stages of system design. The write path is to concentrate on data persistence; while the read path is to concentrate on data query.

Nevertheless, there are two main problems in this system design model.

1. Anemic model. It is also known as CRUD model. When backend focuses on data conversion, it is difficult to have space to handle business logic, which will cause business logic to be scattered everywhere. Domain knowledge will also disappear, e.g., to an ecommerse website, we will say “purchase” instead of “inserting an order record”.
2. Insufficient scalability. From the perspective of system architecture, the database can easily become a bottleneck of the entire system. Both reading and writing must be onto it. The problem of RDBMS is even more serious due to no horizontal scaling.

Task-based Monolith

In order to solve the problems encountered by the above traditional monolith, here we try to introduce the concept of domains.

This diagram is basically the same as the above one. The only difference is replacing DTO with messages on the write path. Messages contain actions and data, not just the data itself like DTO. Thus, we can carry domain-specific actions in the message to make it easier for the backend to recognize each action, and have a corresponding domain implementation.

At this stage, C in CQRS has appeared, and message is a kind of commands. However, the problem of scalability is still unresolved.

In addition, although we have simplified DTO and changed to use messages to communicate, we still need DTO on the read path. Let’s take the social medium as an example again. When modifying the nickname, the format of a message may be {"rename": "LazyDr"}. But when rendering the profile, we still need additional information such as activities. This information gap makes it necessary to do a lot of processing on the read path to retrieve DTO.

CQS (Command Query Segmentation)

The emergence of CQS is to solve the above pain points of Read/Write Splitting.

When reading, the client needs DTO, so the backend can do some optimizations dedicated to reading on the read path, such as pre-generating DTO from the original domain object, and storing DTO in a dedicated database for reading.

In this way, on the read path, the implementation of the application service becomes simpler. The application service can become a thin read layer, which only needs to be responsible for paging, sorting, etc. After requesting, the client can easily retrieve DTO from the database.

So the question is, who is going to generate those pre-built DTOs? It is the responsibility of the write path.

Although the diagram is similar with the examples seen before, in fact, in addition to persist the domain object, the application service must also persist DTO. In other words, most of the business logic will be pressed on the write path, and various read views need to be prepared.

At this stage, we have solved most of the problems encountered by the domain, but scaling still has no solution. Now, we further define scaling. Scaling has two different aspects.

1. Traffic: increase in write volume.
2. Extension: functional requirements increase, such as the need for a variety of different read views. Continuing to take the social medium as an example, there is one presentation on the profile, but there may be another presentation on the timeline.

CQRS

Why is the write path responsible for preparing the read view? Writing should focus on persistence, and those various read views should not be processed on the write path. But there is only reading on the read path, who should prepare those read views?

Therefore, the total solution is as follows.

The write path on the left and the read path on the right have been introduced in the CQS section. The only difference is that an eventually block is added, which is responsible for converting the database on the write path into the database used on the read path. Once data synchronization is involved, it is possible to encounter data consistency issues, so here is a list of several approaches for implementing eventually consistency, sorted by time consuming from short to long:

1. Background thread: The typical representative is Redis. After the data is written to the primary, Redis will immediately send the data to the replicas in the background.
2. Message queue plus workers: This is a common practice for asynchronous data replication. When writing to the database, an event is initiated into the message queue and processed by the workers.
3. Extract-Transform-Load: This time interval is the longest, ranging from a few minutes to a few hours. Use map-reduce or other methods to write the results on the other side.

No matter which approach, the single source of truth is mandatory. That is to say if there is any failure occurred on converting, the system must be able to recover the unfinished jobs. Therefore, the data has to be unique and reliable.

Data usually laid on two types,

1. state: State refers to what you see at the moment, such as the balance written on the bank passbook.
2. event: An event is an action to modify each state, such as every transaction record on the bank passbook

Actually, we already have messages that can be stored as events. For the write path, it is very efficient to store messages in order. Through each different message, you can easily build a different read view according to your needs. This approach is also called event sourcing.

But only events are difficult to use efficiently. In order to obtain the final result, every conversion must be run from the beginning to the end to rebuild the read view. As a result, the hybrid method would be ideal. On the write path, both the state and the event are kept, and the conversion process can choose the data source based on the actual situation.

To sum up the whole life cycle of data in CQRS.

The data starts from the client and then enters to the backend in the command format. According to the business logic, it is converted into a domain object and stored in the database. These domain objects are converted into various read views and stored in different read special databases according to requirements. Finally, the client takes these read views back in the form of DTO.

Conclusion

There are many books and articles describing DDD and CQRS with many patterns. From my point of view, those patterns lead to the limitation on imagination of DDD like Entity, Value Object, Aggregate, etc. It results in most developers feel DDD is far from themselves and hard to realize as well as implementations. In fact, the concepts of DDD is not so complicated; instead, DDD is proposed to encapsulate the business logic and then facilitate expending the functional requirements.

CQRS is even simpler. In this article, we start from the process of system evolution, understand the entire system design process and the problems to be solved, and finally derive the conclusion of CQRS naturally.

There is no silver bullet in the system design. Each evolution is made for solving some specific problems, however, it may come out a new problem. Take the design processes of this article as an example, CQRS seems to resolve all mentioned problems, anemic model, and insufficient scalability, but actually CQRS also brings new problems, such as data consistency. Each technical selection has its trade-off, as long as understanding all threats behind every option, you can choose relatively acceptable approaches.

Even if you choose CQRS, in practice, there are still three choices on implementing eventual consistency. System design is the result of continuous selection.

The purpose of this article is to tell you that DDD is not that scary, and CQRS is not that complicated, it is just a decision.

Originally published on Medium

Array Operation in MongoDB

MongoDB is a document-based database, and each document is in JSON-like format. Therefore, MongoDB can store various data structures. Moreover, in order to manage those documents, MongoDB provides a powerful ubiquitous language to operate documents.

Here comes a question. We usually perform CRUD on documents, however, do you know how to update the element in a list from certain documents? This article is my experiment of array operations.

The original collections is given as follows:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Update a key of elements in an array

We have an array, prop, with elements, and those elements contains a and b are integer and boolean respectively. What we want to do is update b to false if a is equal to 1.

The command is:

db.test.updateMany({"prop.a": 1}, {$set: {"prop.$.b": false}}).

There is a dollar sign representing the result set from filtering. So, prop.$.b means all b with a is 1.

After executing the command, the collection would be:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1, b: false }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1, b: false }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Remove a key of elements in an array

We can $set a key; on the other hand, we can remove a key either. The command is like the $set one.

db.test.updateMany({"prop.a": 1}, {$unset: {"prop.$.b": ""}})

When we want to $unset a key, the syntax is {key: ""}, thus, {$unset: {"prop.$.b": ""} is as expected.

Finally, the collection would be:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1 }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1 }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1 } ] }

Remove a key of certain elements in an array

Furthermore, if we want to remove b when b is true rather than all b. How can we do?

The solution is:

db.test.updateMany({"prop.a": 1}, {$unset: {"prop.$[elem].b": ""}}, {arrayFilters: [{"elem.b": true}]}).

We are using arrayFilters to match the specific elements and indicating those elements to the $unset criteria. Then, the collection would become:

1
2
3
4
5
6
7
8

{ _id: ObjectId("61bb0d7cf08a56308c62110b"),  
  prop: [ { a: 1 }, { a: 2 } ] }  
{ _id: ObjectId("61bb0dbaf08a56308c62110c"),  
  prop: [ { a: 1 }, { a: 2 } ] }  
{ _id: ObjectId("61bb0dcaf08a56308c62110d"),  
  prop: [ { a: 3, b: true }, { a: 2, b: true } ] }  
{ _id: ObjectId("61bb0fcef08a56308c62110e"),  
  prop: [ { a: 1, b: false } ] }

Although we can use filter like {a: 1, b: true} to leverage the dollar sign operations, arrayFilters provides more possibilities to achieve more complex user scenarios. You can try to skill it.

Originally published on Medium

Are Design Pattern and Clean Code Useful?

I overheard our junior developers discussing the value of design pattern and clean code. I vaguely remember their conclusion is, “those techniques make simple things become complex, and that is the favorite thing of those technical authorities who are far from the reality”. Their discussion ends up a negative agreement.

There are some examples from their conversation. For instance, in clean code, the length of a function is limited in a regular size, which results in to overlook a whole function’s functionality, it has to dive into several nested sub-functions. Besides, design pattern ask us encapsulate a simple task into an object, and the objects are everywhere. Coding with objects will make the source code much more longer.

Well, I can understand their pain points. Those pain points come from some reasons such as

1. Their duty is quite simple without too many complicated design or further expandability needs.
2. In a small organization, they don’t need to consider the cooperation. All features are handled by a few people.
3. They don’t realize the concepts of those paradigms, instead, they just see the “tricks”.

If I were in their position, I perhaps agreed with them. Nevertheless, I had not joined their discussion. After all, I am more close to an authority for them. But, I have to say in their mentioned examples, those methodologies have its meaning.

Firstly, restricting the function length must be combined with a well-defined naming convention and a good parameter design to let everyone understand what’s the purpose of this function without traversing the detail implementations. Secondly, encapsulating objects is to accomplish a better abstraction and become easier to achieve Open-Closed Principle. Whether encapsulate logic in Strategy Pattern or encapsulate the creation in Factory Method, they make developers be able to use the existing code without reinventing the wheel. In addition, developers can extend the functionality more easily.

Design pattern and clean code provide some tricks and principles, however, the most important thing is the spirit and purpose behind them. For example,

1
2
3
4
5
6
7
8

if prop == 1:  
    do_a()  
elif prop == 2:  
    do_b()  
elif prop == 3:  
    do_c()  
else:  
    do_default()

This simple example can be considered that those do_XX are some flatted source code. Those junior developers would like to see those behaviors clearly on the same code level like the above python example. In order to understand the implementation at a glance, actually, there are many a way to do better.

1
2
3
4
5

func = {  
    1: do_a,  
    2: do_b,  
    3: do_c,  
}

1
2
3
4

if prop in func:  
    func[prop]()  
else:  
    do_default()

Through simple packaging, it is easier for people who want to read the code to understand the purpose. They can inquire about the part of details they want to see by themselves. In addition to add a new feature, developers just need a handler in func. Furthermore,

1

factory.create(prop).run()

By adopting the factory method, the function become more slim. The cognitive load for readers is low as well. In fact, this piece of code encapsulate the logic in polymorphism to run, aka, Strategy Pattern. It is unnecessary to talk about these details implementation of design pattern or clean code. Once you understand his spirit, you can easily have your own implementation. In the above example, it is indeed not a “standard” factory method implementation.

After skilled design pattern, those pattern names just provide a common language between developers, so that they can communicate more smoothly. For instance, “I have done an observer pattern over here”. It is no need to explain further that you can realize there are a subject and an observer with attach and notify. Even no documents, you still can understand the prototype of the code in your mind.

The purpose of coding is not only making features but also be a communication tool. Clean Code said, “Don’t document your code. Code your documentation”. But these can only be experienced after those junior developers have written about larger functions, cooperated with more people, and met more changing requirements. Everyone chooses a different career path, and not everyone has the opportunity to enter the world where the design pattern and clean code are facing in.

Originally published on Medium

Cardinality Counting in Redis

Cardinality counting is used to calculate the amount of elements without any duplication. In Redis, there are many data structures able to accomplish this job. However, what is the most applicable way for your use cases? This article will show the consideration behind the technical selection.

User Scenario

Suppose we need to get the failure rate in a sensor network to investigate the reporting qualities. Therefore, we have to record the health status in hours by the incoming requests.

The key point is to simplify the process, we don’t want to get the value first, determine whether it is existing, and then insert the record like:

Instead, we should insert the record every time, and the storage can de-duplicate for us. Or, we can limited pre-process data to make the storage do faster.

Assume that we have a sensor A, and the sensor requested to the server at 1/2 1:11, 1/3 2:22, and 1/8 3:00.

Alright, let’s see how Redis did cardinality counting.

Set

The basic idea is using set. Before adding to the set, we have to pre-process the date. Due to our requirement, we only keep the hour without minutes and seconds.

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.toISOString();

Then, we can add d1 to the set via SADD.

1
2
3

SADD sensorA "2021-01-02T01:00:00.000Z"  
SADD sensorA "2021-01-03T02:00:00.000Z"  
SADD sensorA "2021-01-08T03:00:00.000Z"

In order to get the health status, we can use SCARD.

1
2

SCARD sensorA  
> 3

The implementation of using set is simple; nevertheless, if we want to count the health status during the specific period like in 2021, set cannot handle this request.

Sorted Set

Therefore, if we would like to meet the needs of specific time period and whole time, we can leverage sorted set. The implementation is similar to the set. Firstly, pre-process the date.

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.getTime();

It is unlike using ISO string, we are using epoch here to find the specific time range easily. Now, we can add to the sorted set via ZADD.

1
2
3

ZADD sensorA 1609520400000 1609520400000  
ZADD sensorA 1609610400000 1609610400000  
ZADD sensorA 1610046000000 1610046000000

To find whole count in it:

1
2

ZCARD sensorA  
> 3

On the other hand, in order to search the specific time range, we assign the start and end in ZCOUNT.

1
2

ZCOUNT sensorA 1609520400000 1610040000000  
> 2

Bitmap

We walked through two approaches, but neither set nor sorted set are space efficiency. The detail implementation in Redis takes a huge space to indicate the data structure. When the amount of sensors become larger or the duration of records become longer, the space in Redis will grow quickly.

How to reduce the space? We can leverage the extended function of string, bitmap. Bitmap is very space efficiency, each bit takes 1 bit as its meaning.

But the pre-process is a little complicated, we have to get an offset to operate bits. For example, we can calculate the diff hours between the service launch time and the current time, e.g. 2021/1/2 1:11.

1
2
3
4

const base = new Date(2021, 0, 1, 0, 0);  
const date1 = new Date(2021, 0, 2, 1, 11);  
const diffTime = Math.abs(date1 - base);  
const diffHours = Math.ceil(diffTime / (1000 * 60 * 60));

After that, make the diffHours be 26.

1
2
3

SETBIT sensorA 26 1  
SETBIT sensorA 51 1  
SETBIT sensorA 171 1

Hence we can get the total health status by BITCOUNT.

1
2

BITCOUNT sensorA  
> 3

BITCOUNT also provides the range match, so we can assign start and end to search a specific time range like sorted set. It is noteworthy that the start and end here represents the bytes offset. We have to convert the start time and end time to the diff hours bytes, the calculation is complex, so I am not going to provide an example in this article to prevent losing focus.

HyperLogLog

The final approach is called hyperloglog. This is an algorithm for big data statistics. Redis provides it as the built-in method.

Whether it is set, sorted set or bitmap, the space usage will larger and larger as time goes by. For instance, if we keep the health status for 10 years, even bitmap will take huge space, 365 * 10 * 24 / 1024 ~ 85.5 KB.

However, in hyperloglog, the space usage is constant. No matter how long the retention you need, hyperloglog takes 12 KB constantly. And the pre-process is like set,

1
2

const date1 = new Date(2021, 0, 2, 1, 0);  
const d1 = date1.toISOString();

Then, we can add the date to hyperloglog via PFADD.

1
2
3

PFADD sensorA "2021-01-02T01:00:00.000Z"  
PFADD sensorA "2021-01-03T02:00:00.000Z"  
PFADD sensorA "2021-01-08T03:00:00.000Z"

It is simple to get the total count.

1
2

PFOCUNT sensorA  
> 3

Hyperloglog is not quite precise, the result of PFCOUNT may contain some deviations when the data set is huge, but the performance is pretty well.

Conclusion

Let’s summarize those 4 approaches.

The example in this article is trivial, however, I believe you can realize the concepts of those approaches. The most important thing is each approach has its own strength and drawback, using them in a smart way is developer’s responsibility.

Originally published on Medium

The exciting 2021 is almost over, let me review my architect journey. Actually, I had been an architect in embedded system before, then I decided to change my career to the cloud service and backend development. After working and studying for years, I gradually changed from an engineer to a principle engineer. At the beginning of this year, I made my mind to be a backend architect. And finally, the dream comes true.

In retrospect, I feel the change of professional position went smoothly. Therefore, I will review what I have done and will be done for your reference.

Why changing to architect

People often ask me why I want to be an architect instead of being a developer. There are two main reasons,

1. Programming language can never be learned. New languages will appear every once in a while and become popular, e.g., golang. Even the existed languages are still enhanced and become unfamiliar like C++11, C++14, C++17, then C++20. On the other hand, the software engineering is much more constant. The core design of data stores, architectures, distributed systems are classic. In other words, mastering these technologies has a relatively high return on investment.
2. When I was a developer, every time a clean architecture is written, it becomes a mess soon. Cross referencing, circular referencing, etc. are everywhere, I cannot stop it even being a staff engineer. It is better to become an authority and make rules instead of actual development.

What is R&R of architect

“Managers are not confronted with problems that are independent of each other; but with dynamic situations that consist of complex systems of changing problems that interact with each other. I call such situations messes…Managers do not solve problems, they manage messes.”

Some f-Laws from Ackoff, R. L., H. J. Addison and S. Bibb. 2007. Management f-Laws. Triarchy Press Ltd.

The same applies to architects.

1. Find out the root cause of the intractable online problem and hand it over to an engineer to solve it.
2. Monitor and clarify system bottlenecks, find ways to optimize and improve.
3. Educate and guide the development team to improve everyone’s skills, such as the low-level details of the database, SOLID principles for the system design, etc.
4. Participate in the requirement meeting on the product side and propose a reasonable estimated schedule and change scope.
5. Improve development process like CI/CD pipelines, design review meetings, trunk-based development, etc.
6. Participate in seminars and survey new technologies, techniques, and methodologies.

From above description, we can find an architect does not solve issues completely; instead, the core value of an architect is simplifying the process whether it is an issue or a product requirement.

How to be an architect

In order to be an architect, there are some properties helping the position change, and I find those properties really work.

1. A lot of reading, you must have the habit of reading. Work process books, professional technical books, and management books are all essential. You have to be greedy.
2. A lot of thinking, there is no code or architecture that cannot be optimized, and finding a reasonable improvement way is the responsibility of an architect.
3. A lot of communication, the architect is not the person who actually does things, and (s)he must be able to guide developers in a timely manner while establishing authority.
4. Stay motivated and optimistic. The work of the architect is very heavy. You have to deal with a large number of people and issues every day. If you don’t manage your mood well, you can easily fall into frustration.

These four properties are useful, even though you don’t want to be an architect, you still can grow yourself up on the career path of a software engineer. At least, thanks to these properties, I have been able to change from an embedded system engineer to a backend architect successfully in 10 years.

Afterword

Being a software architect is pretty interesting. I have no assigned or dedicated jobs in my organization, what I have to do is finding an issue and trying to fix it. All my tasks are assigned by myself. I can schedule my tasks in a proper way and be free to study from any resource. This is my journey in past 10 years, and you can write your own story on your career. It would be great if you could share with me.

Originally published on Medium

Layered architecture is a common pattern of an architecture design, and it is very suitable for a small scale monolith. There are many advantages such as,

1. very easy to design and implement
2. very in line with human nature
3. cooperate with each other more easily

When we are talking about the layered architecture, we often have a myth; in order to replace the dependency, we should make those dependencies as a layer.

In fact, that’s not the purpose of layered architecture. I believe we seldom need to change a database, a framework, even a programming language. Therefore, what is the benefit of using layers?

Before we discuss further, we should see an example first.

The above diagram is a classic layered architecture for a web server. We separate a request flow into 4 components, route, controller, service and repository. Each component has its responsibility to handle a part of a request flow. This kind of implementation is also known as Single-Responsibility Principle.

In many articles, we had been taught that route is made for the web framework changed, and the repo is to nail the database. However, as I mentioned earlier, we seldom need change those hard things. Route and repo here are just completing their own duties.

Single-Responsibility Principle

I will describe those 4 components’ responsibilities to let you realize them more confidently.

• Route is a the wrapper of a web framework. To interact with a framework and hide the implementation, we build a layer to enclose those details. In route layer, we usually define HTTP routes right here including HTTP methods and URI.
• Controller is the first line to handle a request. The main responsibility is validating the input request and assembling the response. This layer often implements some error handlers to reflect like “400 Bad Request”, “500 Internal Server Error”, etc.
• Service is where the business logic in. The core values of a web product are all here. If it is a E-commerce site, service will calculate the recommendation, orchestrate the render content, and all other logic.
• Repo is an abstraction onto a database. In order to isolate the business logic, we don’t want the database operation to pollute the code in services, thus, repo plays this role in accessing persisted data.

The text description is a bit abstract, so I provide a more concrete example: https://github.com/wirelessr/Clean-Architect-in-Golang

The GitHub repository is quite simple due to the screaming architecture, hence, we can know this is a classic web service.

• Route is a mux wrapper in http. It will do nothing and route HTTP request to the controller.
• Controller defines API interfaces. In AddPost, it also validates the request format and assembles the response.
• Service own all business logic including how to access durable data, what the post format is, and how to generate the post identifier.
• Repo hides all database details, so you can choose what data store you want. But, again, we don’t change our data store decision easily. Therefore, repo here is to encapsulate the data store access to make service be independent from firestore.

What’s Benefit

Do we just separate our web service into several layers for SRP? No. Actually, this comes some benefits. Let me show two most important advantages,

1. Layers make testing become more easily.
2. If those layers are modules, we can decouple the deployment of layer modules.

The second one is straightforward. Thus, I will dive into the test. According to my previous article, we can leverage the dependency injection to accomplish a high testing coverage. For instance, in my golang example, in order to verify the service, we can create a mocked repo to ignore the data store access.

Conclusion

There are many articles trying to compare layered architecture and microservice architecture. However, layered architecture is not only an architecture pattern but also the coding technique for a SOLID software. This article tells you still can use layered architecture in your microservice like my post example. They are not two mutually exclusive things; moreover, they are more close to each other.

Originally published on Medium

Unique Array in MongoDB Document

In order to explain the solution in detail, I have to introduce the user scenario first. We need a collection to record the group information including name, members, etc. However, we cannot allow a group with the same members in another group. For example, if A and B are in group 1, then they cannot be in group 2. Thus, we need maintain a members array which has an unique constraint.

There is an assumption in our requirement.

The amount of group members will be always 2. In other words, a group only has members ["A", "B"] but not ["A", "B", "C"].

Here comes an example collection:

1
2
3
4
5
6
7
8
9

[{  
    "_id": 1,  
    "name": "group1",  
    "members": ["A", "B"]  
}, {  
    "_id": 2,  
    "name": "group2",  
    "members": ["A", "C"]  
}]

We want to query more faster, so we add an index on members, however, MongoDB Multikey Index is very different from what our thought.

• Can we add an unique index on members?
• If the index type is regular, the answer is no.
• Can db.group.find({members: ["B", "A"]}) find anything？
• No, the array should be exactly matched.

The root cause is creating an index on array, the index will become multikey index automatically. For the document with id 1, there will be two indexes Indicating to this document, namely "A" and "B". In the same way, there will be two indexes, "A" and "C", on document with id 2. That is to say, unique constraint doesn’t work while they all have index "A".

Although, there are two indexes "A" and "B" on document 1, it just makes MongoDB retrieve data faster. Instead of the full-table scan, MongoDB can reduce the search scope, especially, the linear search with O(n) is very slow on the array match. However, it has to exactly match the array including the order.

How to solve these problems? There is a trick can handle, that is, use text index. The figure comes from MongoDB Compass as follows.

Although the number of uses is 0, this is actually caused by MongoDB’s implementation. The search for the text index is to compare the text first, and then use _id to locate the real content.

Nevertheless, this approach comes other problems when implementing applications:

1. To find the content, we must know the order of members; otherwise, data cannot be retrieved.
2. Similar to the problem one, even though this approach can make sure ["A", "B"] will not be duplicated, it cannot reject the insertion with ["B", "A"].
3. The search pattern, regarding whether a single target exists in members, is inefficient. We have to use $in without the bound from an index to reduce the search scope.

There is a comprise solution to the problem 1 and 2. Before inserting or fetching data in the database, we sort the members in the application. However, this approach ties the data model and application logic together, which is hard to maintain and use. Hence, it is highly not recommended. As for the problem 3, there is no alternative solution.

Solution

The correct (*see the last section) approach is still using the multikey and text index. But we don’t store scalar string in members; instead, we store objects. Thus, the original example will be modified slightly:

1
2
3
4
5
6
7
8
9

[{  
    "_id": 1,  
    "name": "group1",  
    "members": [{"name": "A"}, {"name": "B"}]  
}, {  
    "_id": 2,  
    "name": "group2",  
    "members": [{"name": "A"}, {"name": "C"}]  
}]

In addition, the index on members should be on members.name with the text type and unique constraint.

From the above example we can see the original members is changed from ["A", "B"] to [{"name": "A"}, {"name": "B"}]. So what are the benefits of doing this? Yes, in fact, the three problems mentioned above have all been solved.

1. We don’t care the order of members when searching a group. We can just use db.group.find({$and: [{"members.name": "A"}, {"members.name": "B"}]}).
2. We don’t care whether "A" is the former or the letter, it can be rejected correctly if there is indeed a group with "A" and "B".
3. To find a group belongs to a specific member, we can use db.group.find({"members.name": "A"}). This can find out all groups "A" belongs.

It must be emphasized again that this solution is only applicable when all groups are composed of two members. Take the above example, if you want to create a group 3 with members "A", "B", "C", it will be denied.

Why we have a such requirement? We want to simplify the know-how of database operations on the client side. In theory, the client does not need to have any knowledge to manipulate the data of the database, and the database can always respond under inappropriate circumstances. Therefore, we don’t have to embed the database logic into every client. Nonetheless, the approach has its disadvantages. Firstly, the design has been fixed to only support groups of 2 members. Secondly, there is only one text index in a collection, i.e., we sacrifice the possibility of other text indexes. Finally, text index takes lots of spaces, it is considered a high price index.

What should be the correct way? There are many possibilities, but the most feasible one should be findAndModify with upsert. By doing this, we can avoid creating a group with duplicated
members under the race condition. If there is a need to add more members, we can leverage $addToSet to operate on members to keep the uniqueness in members. From this we can know that to maintain the correctness of the data, there are many implementation requirements on the client side, so in the end we decided to write the integrity to the database and let the database take care of part of the correctness of the data.

Updated

We have found the multikey index in text type does not work in some conditions. Therefore, we are using $addToSet to be the temporary solution and still trying to find out a better approach.

Originally published on Medium

Message Queue in Redis

My organization has done the technical selection recently, and we want to build an event-driven system. However, the budget is limited. We cannot choose a classic queuing service like RabbitMQ or a streaming process like Kafka. We have to find an affordable solution that can meet our needs. Currently, all we have is Redis. Therefore, we will build a message queue in Redis.

In this article, I will introduce some properties of a message queue and describe how Redis can be used to build a message queue.

Message Queue

There are many aspects to consider when choosing a message queue, such as propagation, delivery, persistence, and consumer groups. In this section, I will explain them briefly.

Propagation

The propagation means how messages are transferred by the message queue. There are 2 types of propagation,

• 1-to-1
• 1-to-many (fan-out)

One-to-one is quite simple. The producer sends a message to the queue, and this message is received by only one consumer. On the other hand, one-to-many is a message that can be delivered to multiple consumers. It’s worth mentioning that the producer just sent a message, but the message can be transferred to many receivers. Such behavior is also called fan-out.

Delivery

Delivery is interesting. Most queuing systems have their delivery guarantees. There are three common guarantees,

• At-most-once
• At-least-once
• Exactly-once

At-most-once is relatively easy to achieve. It can be said that all queuing systems have this guarantee. The consumer can receive a sent message or nothing. This may happen in several situations. Firstly, the message is lost whereas a networking problem occurred. Secondly, although the consumer received it, he did not handle it well like crashed. The message disappears if it is gone, and it is impossible to retrieve the message again.

At-least-once is a guarantee often used by some well-known systems such as RabbitMQ, Kafka, etc. Compared to at-most-once, at-least-once has a stronger guarantee. It can make sure the message must be processed. However, the message may be processed many times. For instance, a consumer does not acknowledge the queue that the message is handled, thus the queue sends a message to that consumer again.

Exactly-once is the strictest guarantee. It ensures the message must be handled once. Even the popular systems can’t do this well, e.g. RabbitMQ. Nevertheless, the correct use and configuration of Kafka can still be achieved. The price is to sacrifice some performance.

Persistence

Persistence means whether the message will disappear after it is sent to the system. There are also three types of persistence,

• In-memory
• In-disk
• Hybrid

We all know what they means. But the interesting thing is, is it slower to persist messages in disk? No, not really. It depends on how persistence is implemented. Kafka uses LSM-tree to achieve a lot of throughput; in addition, it is better than RabbitMQ who uses memory. There is another example in Cassandra, Cassandra has very fast writing speed and uses LSM-tree as well.

Hybrid is a special case combined with in-memory and in-disk. In order to improve the writing performance, the queuing system writes to the memory first, and then flush into the disk. RabbitMQ is a typical example in hybrid. However, RabbitMQ is also able to be configured as in-disk.

Consumer Group

In my opinion, consumer group is the most important feature in a queuing system. Processing a message usually takes time, so that we have to use more consumers to deal with messages, aka scale-out. In consumer group scenes, both the target of one-to-one and one-to-many become a group of consumers instead of a single consumer.

Redis Queue

After talking about the properties in a queuing system, let’s talk about how Redis be a message queue. There are 3 ways to do it,

• Pub/Sub
• List
• Stream

We will introduce one by one, and then give a comprehensive summary.

Pub/Sub

Pub/Sub is a widely known solution for notifying, this feature was born almost at the same time as Redis. The consumer SUBSCRIBE a topic, aka a key, and then receive the data after a client PUBLISH messages to the same topic. As a traditional Pub/Sub feature, it also can fan-out a message to multiple consumers. Moreover, A certain degree of messaging routing can also be achieved through PSUBSCRIBE.

But Pub/Sub in Redis are not popular for most use cases. The biggest problem is the message will delivery at most once. When a message is published, if the consumer doesn’t receive it right now, the message disappears. Furthermore, Redis doesn’t persist messages. All messages are gone away if Redis is shutdown.

Let’s summarize Pub/Sub:

• 1-to-1 and 1-to-many are fine
• at-most-once
• no persistence
• no consumer group

List

List is a useful data structure in Redis, and we can accomplish a FIFO queue easily by using it. The trick is we can use BLPOP to wait for a message in blocking mode. However, adding a timeout is recommended.

According to the figure, we can see if there are multiple consumers wait for the same list, they are becoming a consumer group. Without configuring anything, the consumer group can be spontaneously formed by consumers. On the other hand, list cannot fan-out a message. If a message is BLPOP by a consumer, others can not be retrieved this message anymore, even the message is lost in that consumer.

Nevertheless, Redis list can persist messages in memory. In addition, if you are enabling AOF or RDB, messages can be backed up into the disk. I have to say, following my previous article, this approach is not entirely data persistence.

To sum up,

• 1-to-1 is okay, but no 1-to-many
• at-most-once
• persist in-memory, and backup in-disk
• consumer group works

Stream

After introducing the Pub/Sub and List, we notice that neither of these two methods is very good. They have their own drawbacks. Therefore, Stream has come to solve these issues since Redis 5.0.

Because Stream is much more complicated, let’s first look at what benefits Stream brings.

• 1-to-1 and 1-to-many are fine
• at-least-once
• persist in-memory, and backup in-disk
• consumer group works

As a result, Stream solves all issues in Pub/Sub and List and enhances a lot, for e.g., at-least-once delivery.

The diagram is like Pub/Sub, but the workflow is closer to List. The producer can generate messages at any time, and then XADD to Redis Stream. You can consider Stream as a list maintains all incoming messages. Consumers can also retrieve messages at any time via XREAD. The identifier in XREAD command represents where you want to read the message from.

• $: No matter what messages are in Stream before, only retrieve from now on.
• 0-0: Always read from the head.
• <id>: Start from the specific message id.

Apart from supporting one-to-one mapping, Stream supports consumer groups as follows:

To achieve at-least-once guarantee, like most queuing systems, the consumer must acknowledge Stream after processing a message by using XACK.

The use of the special identifier, <, here is to start reading from a position that no one has read in the group.

After the above explanation, I provide a real example to show a consumer’s bootstrap in Node.js.

Please note that every consumer has his own name, ConsumerName. First, the consumer read from the beginning to determine its last position. The response will be a empty array with no length, so that the consumer can get the correct lastid. Then, the consumer reads from the lastid and processes those messages. Finally, acknowledge Stream with finished id.

Stream Consumer Failover

In the distributed system, we cannot name a consumer easily. For example, the consumer is run in a container within K8s: How do I maintain names to every pod? Even if we lock everyone’s name, how do we face the scale-out and scale-in scene?

Therefore, keeping the name in the distributed system is impractical.

In spite of this, we cannot name a consumer in uuid and forget the name after the consumer is down. Redis Stream maintains a table of names against last positions. So, if we generate a random name every time, the mapping table will become larger and larger. Worst of all, those messages that have been received but not acknowledged will never be processed.

Fortunately, Redis Stream provides a method to claim those pending messages. The workflow is like this:

1. Find out all pending message ids.
2. Claim those ids to transfer the ownership.

Therefore, the completed workflow in a consumer bootstrap is:

1. XPENDING StreamName GroupName
2. XCLAIM StreamName GroupName <ConsumerName in uuid> <min-idle-time> <ID-1> <ID-2> ... <ID-N>
3. The above script

The min-idle-time is a very useful approach. By using min-idle-time, we can avoid multiple consumers claim the same messages at the same time. The first consumer claims some messages, so such messages will no longer be idle. Hence, other consumers cannot claim those messages again.

Redis Stream Persistence

Redis does not guarantee that the data will not be lost at all, even if the strictest setting is turned on. If we use Redis as a message queue, we must take additional measures to ensure persistence. The most common way is event-sourcing. Before publishing a message, we write this message into a durable storage like MySQL. Our consumers can work generally. However, if an error occurs, we can still leverage the durable messages in MySQL to recover our work.

Besides, if Stream persists more and more messages, the memory usage of Redis would be a disaster. If we are looking at the Redis manual, we can find a command, XDEL. However, XDEL does not delete the messages, it only marks those messages as unused, and the messages are still there.

How can we prevent memory leakage in Redis Stream? We can use MAXLEN whereas XADD is invoking. The command line is:

XADD StreamName MAXLEN 1000 * foo bar

But there is one thing you have to know, MAXLEN affects performance of Redis very much. It blocks the main process for a while, and no command can be executed during that period. If there are many incoming messages and the amount of queued messages reaches maximum, then Stream will be very busy to maintain MAXLEN.

An alternative approach can be adopted. Instead of fixing the hard limit, we can give Redis the right to choose a comfortable length at its free time. Hence, the command will be:

XADD StreamName MAXLEN ~ 1000 * foo bar

The ~ sign means the maximum length is about 1000, it might be 900 or even 1300. Redis will pick a good time to strip a good size for it.

Conclusion

Let me summarize these three approaches.

There is an unfamiliar property complexity, which refers to the complexity of a technology but also the complexity of implementing a consumer.

From my point of view, these three approaches have their pros and cons and also have their own applicable scenarios.

• Pub/Sub: Best-effort notification.
• List: Tolerate message queues with some data loss.
• Stream: Loose streaming process.

So, why is Redis Stream a loose streaming process? Because the consumer group in Redis Stream is not like Kafka. It cannot preserve the message ordering. In a high-volume traffic environment, consumers within the same group cannot be scaled out successfully.

In the end, we chose List as the message queue. Our use cases are simple. We just want to throttle the notifications in the broadcast scene. The broadcast notification can tolerate the message loss. It’s good enough that most users can receive the message. In addition, the implementation effort is very low in Node.js, so we can finish it as soon as possible. Although it is not the best solution, it is good enough for our organization.

The world’s fastest cloud data warehouse:

When designing analytics experiences which are consumed by customers in production, even the smallest delays in query response times become critical. Learn how to achieve sub-second performance over TBs of data with Firebolt.

Originally published on Medium