CT Wu

Unique Array in MongoDB Document

In order to explain the solution in detail, I have to introduce the user scenario first. We need a collection to record the group information including name, members, etc. However, we cannot allow a group with the same members in another group. For example, if A and B are in group 1, then they cannot be in group 2. Thus, we need maintain a members array which has an unique constraint.

There is an assumption in our requirement.

The amount of group members will be always 2. In other words, a group only has members ["A", "B"] but not ["A", "B", "C"].

Here comes an example collection:

1
2
3
4
5
6
7
8
9

[{  
    "_id": 1,  
    "name": "group1",  
    "members": ["A", "B"]  
}, {  
    "_id": 2,  
    "name": "group2",  
    "members": ["A", "C"]  
}]

We want to query more faster, so we add an index on members, however, MongoDB Multikey Index is very different from what our thought.

• Can we add an unique index on members?
• If the index type is regular, the answer is no.
• Can db.group.find({members: ["B", "A"]}) find anything？
• No, the array should be exactly matched.

The root cause is creating an index on array, the index will become multikey index automatically. For the document with id 1, there will be two indexes Indicating to this document, namely "A" and "B". In the same way, there will be two indexes, "A" and "C", on document with id 2. That is to say, unique constraint doesn’t work while they all have index "A".

Although, there are two indexes "A" and "B" on document 1, it just makes MongoDB retrieve data faster. Instead of the full-table scan, MongoDB can reduce the search scope, especially, the linear search with O(n) is very slow on the array match. However, it has to exactly match the array including the order.

How to solve these problems? There is a trick can handle, that is, use text index. The figure comes from MongoDB Compass as follows.

Although the number of uses is 0, this is actually caused by MongoDB’s implementation. The search for the text index is to compare the text first, and then use _id to locate the real content.

Nevertheless, this approach comes other problems when implementing applications:

1. To find the content, we must know the order of members; otherwise, data cannot be retrieved.
2. Similar to the problem one, even though this approach can make sure ["A", "B"] will not be duplicated, it cannot reject the insertion with ["B", "A"].
3. The search pattern, regarding whether a single target exists in members, is inefficient. We have to use $in without the bound from an index to reduce the search scope.

There is a comprise solution to the problem 1 and 2. Before inserting or fetching data in the database, we sort the members in the application. However, this approach ties the data model and application logic together, which is hard to maintain and use. Hence, it is highly not recommended. As for the problem 3, there is no alternative solution.

Solution

The correct (*see the last section) approach is still using the multikey and text index. But we don’t store scalar string in members; instead, we store objects. Thus, the original example will be modified slightly:

1
2
3
4
5
6
7
8
9

[{  
    "_id": 1,  
    "name": "group1",  
    "members": [{"name": "A"}, {"name": "B"}]  
}, {  
    "_id": 2,  
    "name": "group2",  
    "members": [{"name": "A"}, {"name": "C"}]  
}]

In addition, the index on members should be on members.name with the text type and unique constraint.

From the above example we can see the original members is changed from ["A", "B"] to [{"name": "A"}, {"name": "B"}]. So what are the benefits of doing this? Yes, in fact, the three problems mentioned above have all been solved.

1. We don’t care the order of members when searching a group. We can just use db.group.find({$and: [{"members.name": "A"}, {"members.name": "B"}]}).
2. We don’t care whether "A" is the former or the letter, it can be rejected correctly if there is indeed a group with "A" and "B".
3. To find a group belongs to a specific member, we can use db.group.find({"members.name": "A"}). This can find out all groups "A" belongs.

It must be emphasized again that this solution is only applicable when all groups are composed of two members. Take the above example, if you want to create a group 3 with members "A", "B", "C", it will be denied.

Why we have a such requirement? We want to simplify the know-how of database operations on the client side. In theory, the client does not need to have any knowledge to manipulate the data of the database, and the database can always respond under inappropriate circumstances. Therefore, we don’t have to embed the database logic into every client. Nonetheless, the approach has its disadvantages. Firstly, the design has been fixed to only support groups of 2 members. Secondly, there is only one text index in a collection, i.e., we sacrifice the possibility of other text indexes. Finally, text index takes lots of spaces, it is considered a high price index.

What should be the correct way? There are many possibilities, but the most feasible one should be findAndModify with upsert. By doing this, we can avoid creating a group with duplicated
members under the race condition. If there is a need to add more members, we can leverage $addToSet to operate on members to keep the uniqueness in members. From this we can know that to maintain the correctness of the data, there are many implementation requirements on the client side, so in the end we decided to write the integrity to the database and let the database take care of part of the correctness of the data.

Updated

We have found the multikey index in text type does not work in some conditions. Therefore, we are using $addToSet to be the temporary solution and still trying to find out a better approach.

Originally published on Medium

Message Queue in Redis

My organization has done the technical selection recently, and we want to build an event-driven system. However, the budget is limited. We cannot choose a classic queuing service like RabbitMQ or a streaming process like Kafka. We have to find an affordable solution that can meet our needs. Currently, all we have is Redis. Therefore, we will build a message queue in Redis.

In this article, I will introduce some properties of a message queue and describe how Redis can be used to build a message queue.

Message Queue

There are many aspects to consider when choosing a message queue, such as propagation, delivery, persistence, and consumer groups. In this section, I will explain them briefly.

Propagation

The propagation means how messages are transferred by the message queue. There are 2 types of propagation,

• 1-to-1
• 1-to-many (fan-out)

One-to-one is quite simple. The producer sends a message to the queue, and this message is received by only one consumer. On the other hand, one-to-many is a message that can be delivered to multiple consumers. It’s worth mentioning that the producer just sent a message, but the message can be transferred to many receivers. Such behavior is also called fan-out.

Delivery

Delivery is interesting. Most queuing systems have their delivery guarantees. There are three common guarantees,

• At-most-once
• At-least-once
• Exactly-once

At-most-once is relatively easy to achieve. It can be said that all queuing systems have this guarantee. The consumer can receive a sent message or nothing. This may happen in several situations. Firstly, the message is lost whereas a networking problem occurred. Secondly, although the consumer received it, he did not handle it well like crashed. The message disappears if it is gone, and it is impossible to retrieve the message again.

At-least-once is a guarantee often used by some well-known systems such as RabbitMQ, Kafka, etc. Compared to at-most-once, at-least-once has a stronger guarantee. It can make sure the message must be processed. However, the message may be processed many times. For instance, a consumer does not acknowledge the queue that the message is handled, thus the queue sends a message to that consumer again.

Exactly-once is the strictest guarantee. It ensures the message must be handled once. Even the popular systems can’t do this well, e.g. RabbitMQ. Nevertheless, the correct use and configuration of Kafka can still be achieved. The price is to sacrifice some performance.

Persistence

Persistence means whether the message will disappear after it is sent to the system. There are also three types of persistence,

• In-memory
• In-disk
• Hybrid

We all know what they means. But the interesting thing is, is it slower to persist messages in disk? No, not really. It depends on how persistence is implemented. Kafka uses LSM-tree to achieve a lot of throughput; in addition, it is better than RabbitMQ who uses memory. There is another example in Cassandra, Cassandra has very fast writing speed and uses LSM-tree as well.

Hybrid is a special case combined with in-memory and in-disk. In order to improve the writing performance, the queuing system writes to the memory first, and then flush into the disk. RabbitMQ is a typical example in hybrid. However, RabbitMQ is also able to be configured as in-disk.

Consumer Group

In my opinion, consumer group is the most important feature in a queuing system. Processing a message usually takes time, so that we have to use more consumers to deal with messages, aka scale-out. In consumer group scenes, both the target of one-to-one and one-to-many become a group of consumers instead of a single consumer.

Redis Queue

After talking about the properties in a queuing system, let’s talk about how Redis be a message queue. There are 3 ways to do it,

• Pub/Sub
• List
• Stream

We will introduce one by one, and then give a comprehensive summary.

Pub/Sub

Pub/Sub is a widely known solution for notifying, this feature was born almost at the same time as Redis. The consumer SUBSCRIBE a topic, aka a key, and then receive the data after a client PUBLISH messages to the same topic. As a traditional Pub/Sub feature, it also can fan-out a message to multiple consumers. Moreover, A certain degree of messaging routing can also be achieved through PSUBSCRIBE.

But Pub/Sub in Redis are not popular for most use cases. The biggest problem is the message will delivery at most once. When a message is published, if the consumer doesn’t receive it right now, the message disappears. Furthermore, Redis doesn’t persist messages. All messages are gone away if Redis is shutdown.

Let’s summarize Pub/Sub:

• 1-to-1 and 1-to-many are fine
• at-most-once
• no persistence
• no consumer group

List

List is a useful data structure in Redis, and we can accomplish a FIFO queue easily by using it. The trick is we can use BLPOP to wait for a message in blocking mode. However, adding a timeout is recommended.

According to the figure, we can see if there are multiple consumers wait for the same list, they are becoming a consumer group. Without configuring anything, the consumer group can be spontaneously formed by consumers. On the other hand, list cannot fan-out a message. If a message is BLPOP by a consumer, others can not be retrieved this message anymore, even the message is lost in that consumer.

Nevertheless, Redis list can persist messages in memory. In addition, if you are enabling AOF or RDB, messages can be backed up into the disk. I have to say, following my previous article, this approach is not entirely data persistence.

To sum up,

• 1-to-1 is okay, but no 1-to-many
• at-most-once
• persist in-memory, and backup in-disk
• consumer group works

Stream

After introducing the Pub/Sub and List, we notice that neither of these two methods is very good. They have their own drawbacks. Therefore, Stream has come to solve these issues since Redis 5.0.

Because Stream is much more complicated, let’s first look at what benefits Stream brings.

• 1-to-1 and 1-to-many are fine
• at-least-once
• persist in-memory, and backup in-disk
• consumer group works

As a result, Stream solves all issues in Pub/Sub and List and enhances a lot, for e.g., at-least-once delivery.

The diagram is like Pub/Sub, but the workflow is closer to List. The producer can generate messages at any time, and then XADD to Redis Stream. You can consider Stream as a list maintains all incoming messages. Consumers can also retrieve messages at any time via XREAD. The identifier in XREAD command represents where you want to read the message from.

• $: No matter what messages are in Stream before, only retrieve from now on.
• 0-0: Always read from the head.
• <id>: Start from the specific message id.

Apart from supporting one-to-one mapping, Stream supports consumer groups as follows:

To achieve at-least-once guarantee, like most queuing systems, the consumer must acknowledge Stream after processing a message by using XACK.

The use of the special identifier, <, here is to start reading from a position that no one has read in the group.

After the above explanation, I provide a real example to show a consumer’s bootstrap in Node.js.

Please note that every consumer has his own name, ConsumerName. First, the consumer read from the beginning to determine its last position. The response will be a empty array with no length, so that the consumer can get the correct lastid. Then, the consumer reads from the lastid and processes those messages. Finally, acknowledge Stream with finished id.

Stream Consumer Failover

In the distributed system, we cannot name a consumer easily. For example, the consumer is run in a container within K8s: How do I maintain names to every pod? Even if we lock everyone’s name, how do we face the scale-out and scale-in scene?

Therefore, keeping the name in the distributed system is impractical.

In spite of this, we cannot name a consumer in uuid and forget the name after the consumer is down. Redis Stream maintains a table of names against last positions. So, if we generate a random name every time, the mapping table will become larger and larger. Worst of all, those messages that have been received but not acknowledged will never be processed.

Fortunately, Redis Stream provides a method to claim those pending messages. The workflow is like this:

1. Find out all pending message ids.
2. Claim those ids to transfer the ownership.

Therefore, the completed workflow in a consumer bootstrap is:

1. XPENDING StreamName GroupName
2. XCLAIM StreamName GroupName <ConsumerName in uuid> <min-idle-time> <ID-1> <ID-2> ... <ID-N>
3. The above script

The min-idle-time is a very useful approach. By using min-idle-time, we can avoid multiple consumers claim the same messages at the same time. The first consumer claims some messages, so such messages will no longer be idle. Hence, other consumers cannot claim those messages again.

Redis Stream Persistence

Redis does not guarantee that the data will not be lost at all, even if the strictest setting is turned on. If we use Redis as a message queue, we must take additional measures to ensure persistence. The most common way is event-sourcing. Before publishing a message, we write this message into a durable storage like MySQL. Our consumers can work generally. However, if an error occurs, we can still leverage the durable messages in MySQL to recover our work.

Besides, if Stream persists more and more messages, the memory usage of Redis would be a disaster. If we are looking at the Redis manual, we can find a command, XDEL. However, XDEL does not delete the messages, it only marks those messages as unused, and the messages are still there.

How can we prevent memory leakage in Redis Stream? We can use MAXLEN whereas XADD is invoking. The command line is:

XADD StreamName MAXLEN 1000 * foo bar

But there is one thing you have to know, MAXLEN affects performance of Redis very much. It blocks the main process for a while, and no command can be executed during that period. If there are many incoming messages and the amount of queued messages reaches maximum, then Stream will be very busy to maintain MAXLEN.

An alternative approach can be adopted. Instead of fixing the hard limit, we can give Redis the right to choose a comfortable length at its free time. Hence, the command will be:

XADD StreamName MAXLEN ~ 1000 * foo bar

The ~ sign means the maximum length is about 1000, it might be 900 or even 1300. Redis will pick a good time to strip a good size for it.

Conclusion

Let me summarize these three approaches.

There is an unfamiliar property complexity, which refers to the complexity of a technology but also the complexity of implementing a consumer.

From my point of view, these three approaches have their pros and cons and also have their own applicable scenarios.

• Pub/Sub: Best-effort notification.
• List: Tolerate message queues with some data loss.
• Stream: Loose streaming process.

So, why is Redis Stream a loose streaming process? Because the consumer group in Redis Stream is not like Kafka. It cannot preserve the message ordering. In a high-volume traffic environment, consumers within the same group cannot be scaled out successfully.

In the end, we chose List as the message queue. Our use cases are simple. We just want to throttle the notifications in the broadcast scene. The broadcast notification can tolerate the message loss. It’s good enough that most users can receive the message. In addition, the implementation effort is very low in Node.js, so we can finish it as soon as possible. Although it is not the best solution, it is good enough for our organization.

The world’s fastest cloud data warehouse:

When designing analytics experiences which are consumed by customers in production, even the smallest delays in query response times become critical. Learn how to achieve sub-second performance over TBs of data with Firebolt.

Originally published on Medium

Read-after-write Consistency in MongoDB

In this article, I will describe how to achieve read-after-write consistency in MongoDB. The problem what we want to solve is:

• A collection maintains profiles.
• After we insert a new profile, we want to retrieve the top 20 ordered by some criteria in the same function.
• We find the newest profile is always not been found.

Our MongoDB client’s configuration is almost default with a connection string, readPreference=secondaryPreferred.

Changing Read Preference (incorrect)

We want to make the query always comes from the primary, so we try to assign a session-level read preference to primary. The official manual shows the collection-level preference can overwrite the client-level one.

However, it does not work.

Changing Read and Write Concern (solution)

After consulting the MongoDB solution architect, we find the default read concern is local which doesn’t guarantee the operation order. Thus, the solution is not only changing the read concern to majority but also changing the write concern to majority.

Then, the problem is resolved.

By the way, the solution is a little complicated than MySQL does. If we use MySQL as one primary and two replicas, we can switch the read operation from a replica to the primary to ensure read-after-write consistency. On the other hand, MongoDB is designed to face the scene of large-volume data, he has to do more to balance the efficiency and functionality. Thus, in addition to configuring the read preference, using read concern and write concern are also essential.

Originally published on Medium

Trunk-based Development Can Help

The DevOps Handbook tells using trunk-based development can improve the efficiency of developing software. Thus, what is the magic of trunk-based development make us to abandon our familiar git-related workflow?

Git workflow

It should start from the git-related workflow. Whether it is git workflow, github workflow or even gitlab workflow, in general, the entire development process is completed by using feature branches, development branches, and main branches. However, from this point of view, we find that to complete a feature development, at least three branches are involved. Each branch have to be merged into another branch. In addition to resolving conflicts during the merge, when to merge is also important.

Engineers in the same team must be fully aware of the status of each branch, which not only brings cognitive load, but also brings management troubles. Besides, when many major functions are developed, the online code base will be very different from the code under development. Therefore, these several workflows tell us to divide a feature into some small changes and continue to integrate them.

The above figure is a typical gitlab workflow. It is not difficult to see that it is very complicated. There are many interactions between various branches, and there are many rules to be followed. Of course, with these rules, comes many restrictions.

Trunk-based workflow

Extreme Programming Explained advocates getting feedback as soon as possible. Instead of testing slowly in the test environment and staging environment, it is better to go directly to the production to receive faster results. Since git-related workflows have already suggested small changes and continuous integration, why not just take away those branches with many management burdens, leaving only one branch: the main branch?

This is the spirit of trunk-based development. There is only one trunk branch. All features are branched from this branch and continuously merged back into the trunk. The online environment is based on this trunk branch and is the continuous deployment.

In fact, there are many problems to be executed in this most ideal way. After all, no one wants to destroy the formal environment because of a small change, so there are still various test environments. At this time, it is still necessary to have a mechanism to correspond the trunk to each test environment. In practice, the most commonly used method that will not violate the spirit of trunk-based is to use tags on the trunk branch to identify the version. If there are urgent bugs that must be dealt with, a hotfix branch must still be generated from the tag. Nevertheless, unlike git workflow, this hotfix branch that belongs to a specific tag has its life cycle. When the next tag is released in the environment that originally used this tag, this hotfix branch should be eliminated.

Therefore, although there will be release branches or hotfix branches that branch from the main trunk, these two types of branches will not return to the main trunk. In other words, all fixes first enter the main branch and then enter the hotfix branch through cherry-pick. This produces a benefit, the environment that is released after the hotfix can immediately receive the effects of the hotfix.

The workflow is as follows.

Conclusion

In order to drive this development process well, we must have some “good” practices, and the most frequently mentioned is the feature toggle. Because all submitted pieces have to enter the main trunk and will be released at any time, it is necessary to be able to isolate and conditionally control the testing scope for unprepared code. Feature toggle benefits this need.

On the other hand, if it is a very efficient development team, there is no need to have a release branch or a hotfix branch, i.e., online problems can be solved in a forward fix. That is to say the entire software system must have complete telemetries and monitoring systems to find out the online defects faster.

All these good practices are described in The DevOps Handbook, so I really recommend this book.

We can regard trunk-based development as the holy grail of the software development. You must have many good practices in order to be able to use well. Once applying the trunk-based development, it will not only reduce the complexities of managing branches and environments but also reduce the overhead of every engineer, e.g., merging.

Recently, I guide my team to introduce the feature toggle, and my team members begin to experience the benefits of trunk-based development. For a two-pizza team, it can indeed greatly increase the productivity.

Originally published on Medium

I have noticed there are many engineers who cannot distinguish between the unit test and integration test. Even though these two tests are as different as their names, we still cannot use them in the right place. This article helps you understand them and use them correctly.

In order to explain them better, I recommend this article, which introduces the dependency injection to the unit tests to achieve more testing coverage rate. I will leverage the example in it and dive into the integration test further. Before talking about the integration test, let’s take a quick look at the relationship between the unit test and the dependency injection.

Unit Test and Dependency Injection

Here comes a function, aka a unit, saveData.

As we have seen, we need to verify both the success case and failure case to achieve complete test coverage within the unit tests.

Therefore, we can leverage the dependency injection to prune the external dependency from a database.

Like the above examples, we fake the database objects and make sure our business logic is correct. The keyword is “business logic”. We verify the whole business logic in unit tests no matter what the database is. By using the dependency injection, we can easily verify the business logic and reach a high coverage rate.

Integration Test

Alright, we have already ensured the unit works without the database. Things are not likely to go so smoothly after the database is involved. Thus, we have to make some integration tests to verify the database works as our expectations.

We have already verified the units, therefore, we can only verify the database part, i.e "insert into mydatabase.mytable (data) value ('" + data +"')" as follows.

This example is not structured well, because we can apply the layered architecture to build an abstraction upon SQL query, called DAL (data access layer). Hence, we can have a cleaner interface to test the database instead of using raw SQL in a test case. Moreover, in Domain-Driven Development, there is a similar pattern, Repository, and it provides encapsulation for the database access. Those methods are able to provide convenience for writing integration tests.

Of course you can replace the dependency injection with other techniques like mocking. However, in my opinion, mocking will introduce much more implement efforts on writing the integration tests. By using the dependency injection, we will have an independent module/object/interface for the integration.

Conclusion

Why should we distinguish between the unit test and integration test? The reason is doing integration tests will take a lot of time, most of the time from the database access. Suppose an integration test case takes 100 ms, which is very fast for the database access, then it is hard for us to write thousands of test cases. In order to fully test a system, we always try to cover every decision in every function from every file, thus, controlling total time consumption is essential.

That is why Test Triangle shows the unit test at the bottom, and the integration test is up on it.

Let me summarize what is the main difference between unit tests and integration tests.

Unit tests are made for testing business logic; on the other hand, integration tests are for verifying the external dependencies.

With messing up the scenes, it will end up spending more effort and getting less results.

Originally published on Medium

Data Persistence in MongoDB

We have talked about the data persistence in Redis in my previous article, and we had come out a conclusion.

There is no way to ensure no data loss in the single instance Redis even the strictest setting is turned on.

Because Redis is designed as a in-memory data storage handling the high-throughput user scenarios, aka a cache not a database. Hence, the persistence only needs to be usable not reliable.

On the other hand, MongoDB as a NoSQL database is designed to deal with high-volume data and can be scaled out as needed. How about the data persistence in MongoDB?

WiredTiger

We are going to talk about MongoDB with the most popular storage engine, WiredTiger. First, I have to tell that MongoDB is able to persist data. However, if you don’t use MongoDB correctly, data would still be lost.

The story starts from the journaling. By default, MongoDB stores its data into the memory to improve the performance. After reaching the criteria, MongoDB flushes data to the disk. Until then, the data is finally persisted. The criteria is:

1. At every 100 milliseconds (can be adjusted by storage.journal.commitIntervalMs)
2. Every 100 MB of data

The journaling is like Redis AOF, it uses WAL to write data to the disk.

Write Concern

If the behavior of MongoDB is storing data in the memory first, how should I make sure the data is durable? Fortunately, MongoDB provides the write concern to accomplish the data persistence. You can set the write concern to “{j: true}” to make MongoDB acknowledge clients after the data is stored on the disk. Or, you can use “{w: majority}”, this implies “{j: true}” as well as replicates data to most slaves.

The connect string also supports the journaling, journal=true. This provides some convenience if you don’t want to assign the write concern every time.

Conclusion

The data persistence plays a very important role in the system design. In fact, many NoSQL databases have some tricks in them. When you are using them, you should not only use them carefully but also understand the implementations behind them.

MongoDB provides the data persistence as long as the write concern is used properly. In my opinion, “{w: majority}” is the best choice when you are considering the write concern. Because, we usually would like our data can be persisted in addition to persist to replicas.

Originally published on Medium

Data Persistence in Redis

I had written an article about the distributed transactions, which mentioned that Redis does not provide data persistence, with only a brief description. Today, I will deep dive into this topic to see how Redis AOF does.

TL;DR

Redis does not guarantee that the data will not be lost at all, even if the strictest setting is turned on.

The strictest setting here is AOF with fsync at every query. Before we are talking about the reason, I will introduce the AOF further.

Redis AOF

AOF (Append Only File) is a mechanism like LSM Tree. After Redis server finishes processing the commands from clients, it will write a command log appending to the end of the log file. There are three settings to determine when these logs are actually written to the hard disk.

1. No fsync at all
2. fsync every second
3. fsync at every query

The best performance is the first one, and then it gets worse and worse. According to the experiment, we can know the last option works even worse than LSM tree.

From the official manual, it looks like the data is durable when the AOF is enabled. Actually, no, not at all. The trick is the launch time of AOF; If you look closely you will find AOF is starting after commands are processed. It is not like WAL (Write Ahead Log) of other databases. Therefore, if the command is finished processing, and then the system is crashed, you will obviously lost this operation.

fsync every second

fsync at every query makes lots of performance impact, hence we usually adopt fsync every second. This is also the default value in enabling Redis AOF. There comes another question:

Does this mean I will only lose one second of data?

The answer is no. There are two steps in AOF writes logs into the disk.

• WRITE: Write data to file
• SAVE: fsync, i.e., write file to disk

From the implementation of Redis, the flow is as follows:

• Scenario 1: Return without WRITE and SAVE
• Scenario 2: WRITE, but no SAVE
• Scenario 3: WRITE and SAVE

To sum up, if the system crashed, you will lost data within 2 seconds. On the other hand, the description in official manual — appendfsync everysec: fsync every second. Fast enough (in 2.4 likely to be as fast as snapshotting), and you can lose 1 second of data if there is a disaster. — is incorrect.

Let me add, in parenthesis, I had attended EuropeCloud Summit 2021, and there is a session, Accelerating Application Modernization and Cloud Migration with Redis, in day 2; one of a slide shows: Zero Data Loss around 9+ years in production. I am curious how he did it, so I thus ask my question; I have got no answer until today.

Here is my conclusion, using Redis AOF is much more durable, however, it cannot make sure zero data loss. If you want to persist the data as much as possible, you have to not only use AOF but also the replica even as well as RDB at the same time.

Originally published on Medium

Design Patterns of Event-driven Architecture Part 2

In my previous article, I have described some design patterns to build a scalable, robust, efficient, and fault-tolerant system. Today we will talk about some other methods to solve the problems when designing a event-driven architecture.

Request-response Model

Wait, we are designing an event-driven architecture which is asynchronous definitely. Why we are talking about the synchronous model here? In my opinion, the request-response models is very classic way to start up a project. We used to request a response through the restful APIs. This implementation can reduce the development effort and give a straightforward view. However, we don’t want to lose the flexibility of event-driven architectures. The solution is we implement a request-response models upon an event system.

The sequential diagram is shown as the following figure 1.

The server is a traditional HTTP server with some synchronous Restful APIs, and the client requests the server as usual. Nevertheless, the server doesn’t handle this request directly; instead, he delegate this job to the event worker and wait for the response at the agreed place which is another queue. After using this pattern, we can enjoy not only the benefits of the event-driven architecture but also the simplicity between clients and servers.

Ambulance Pattern

The ambulance pattern is used to handle the message priority correctly. The simple design for an emergency event is always put this event to the head of a queue, so that workers can process those high-priority events Immediately. Well, this implementation has some drawbacks, especially, the starvation of the low-priority events. Those normal events might not be processed at all, because there are always emergency events come in.

In order to handle events evenly, we can separate emergency events from ordinary events and submit to a new queue like the follows:

The workers can pick events from those queues to avoid the starvation. There can be a weight to determine the ratio of the two sides, or simply use round-robin to handle those events sequentially.

Furthermore, you can dedicate a worker focus on the emergency events.

Conclusion

We have described two patterns to face some design decisions. In order to make the client more easily, we can leverage the request-response model. If we encounter a scenario is to distinguish the event priority, we can use the ambulance pattern.

The trend of using event-driven architecture has become apparent. However, until now, there is no one-size-fits-all solution can design a well architecture for events. We have to find the corresponding solution according to various situations. Hope these two articles are helpful to you.

Originally published on Medium

How to choose a MongoDB shard key

In this article, I will show you what is the ideal pattern of a MongoDB shard key. Although there is a good page on the MongoDB official manual, it still does not provide a formula to choose a shard key.

TL;DR

The formula is

{coarselyAscending : 1, searchPattern : 1}

I will explain the reason in the following sections.

User Scenario

In order to well-describe the formula, I will use an example to illustrate the scenario. There is a collection within application logs, and the format is like:

1
2
3
4
5
6
7

{  
    "id": "4df16cf0-2699-410f-a07e-ca0bc3d3e153",  
    "type": "app",  
    "level": "high",  
    "ts": 1635132899,  
    "msg": "Database crash"  
}

Each log has the same template, id is a UUID, ts is an epoch, and both type and level are a finite enumeration. I will leverage the terminologies in the official manual to explain some incorrect designs.

Low Cardinality Shard Key

From the mentioned example, we usually choose type at first sight. Because we always use type to identify the logging scope. However, if we choose the type as the shard key, it must encounter a hot-spot problem. Hot-spot problem means there is a shard size much larger than others. For example, there are 3 shards corresponding to 3 types of logs, app, web, and admin, the most popular user is on app. Therefore, the shard size with app log will be very large. Furthermore, due to the low-cardinality shard key, the shards cannot be rebalanced anymore.

Ascending Shard Key

Alright, if type cannot be the shard key, how about ts? We always search for the most recently logs, and ts are fully uniform distributed, it should be a proper choice. Actually, no. When the shard key is an ascending data, it works at the very first time. Nevertheless, it will result in a performance impact soon. The reason is ts is always ascending, so the data will always insert into the last shard. The last shard will be rebalanced frequently. Worst of all, the query pattern used to search from the last shard as well, i.e. the search will often be the rebalance period.

Random Shard Key

Based on the previous sections, we know type, level and ts all are not good shard key candidates. Thus, we can use id as the shard key, so that we can spread the data evenly without frequent changes. This approach will work fine when the data set is limited. After the data set becomes huge, the overhead of rebalance will be very high. Because the data is random, MongoDB has to random access the data while rebalancing. On the other hand, if the data is ascending, MongoDB can retrieve the data chunks via the sequential access.

Solution

A good MongoDB shard key should be like this:

{coarselyAscending : 1, searchPattern : 1}

In order to prevent the random access, we choose the coarsely ascending data be the former. This pick also won’t meet the problem of frequently rebalancing. And we put a search pattern on the latter to ensure the related data can be located at the same shard as much as possible. In our example, I will not only choose the shard key but also redesign our search pattern. The ts is fine to address the log at the specific time; however, it is a bit inefficient for a time range query like from 3 month ago til now. Hence, I will add one more key, month, in the document, so we therefore can leverage the MongoDB date type and make a proper shard key. The collection will be:

1
2
3
4
5
6
7
8

{  
    "id": "4df16cf0-2699-410f-a07e-ca0bc3d3e153",  
    "type": "app",  
    "level": "high",  
    "ts": 1635132899,  
    "msg": "Database crash",  
    "month": new Date(2021, 10) // only month  
}

And, the shard key is {month: 1, type: 1}.

The key point here is we use month instead of ts to avoid frequently rebalancing. The month is not made just for the shard key; on the contrary, we also use it for our search pattern. Instead of calculating the relationship between timestamp and the date, we can use getMonth to find results faster. For instance,

1
2
3

var d = new Date();  
d.setMonth(d.getMonth() - 1); //1 month ago  
db.data.find({month:{$gte: d}});

To sum up, this article provides the concepts of designing MongoDB shard key. You might not have a coarsely ascending data so, but you can refer to the concepts and find out a proper key design for your applications.

Originally published on Medium

Kafka vs. RabbitMQ

I’ve written this article for explaining what’s the main difference between Kafka and RabbitMQ. I know there are a lot of articles that try to compare them on their capabilities like message routing, performance, persistence, etc. I like this post the most, which gives a precise and fair summary to both of them.

Nevertheless, those articles all stand on the server-side to consider how to choose Kafka or RabbitMQ. They rarely consider from the client-side, i.e the story on workers. In my opinion, both Kafka and RabbitMQ itself work very well on every use case. The most common thing we encounter is not that the server cannot handle messages, but the workers are too slow to process messages. If the workers indeed cannot afford the workload, we usually adopt horizontal scaling to increase the throughput.

RabbitMQ

I demonstrate this scenario as follows:

The MQ represents either RabbitMQ or other message queue systems. In this case, we can extend the number of workers to handle more messages at the same time. However, the order of messages cannot be preserved. RabbitMQ treats every message as a standalone entity and can be dispatched to any one of the unified workers. In fact, you can set up a complex exchange and routing rule to ensure the message ordering manually. If you do so, you will introduce the complexity in a system and significantly make a tightly coupling between producers and consumers.

Kafka

How to deal with this problem? I mean how does Kafka preserve the message order. The illustration is:

The magic is a producer still publishes a message to a topic, but Kafka pushes the message into the corresponding partition based on the key. Consumers can be a consumer group, and the consumers in the same group will be dispatched to the certain partition(s). In other words, the same kind of messages are processed on the same consumer to make sure the correctness of orderings. Furthermore, the participants in a consumer group can be dynamically adjusted to sort out the run-time workload.

Conclusion

From my point of view, if the user scenario is the producer generates lots of messages, and you have to add more consumers to digest them, you should use Kafka. Otherwise, Kafka and RabbitMQ both are great tools, you can choose according to the technology stack in your organization.

Originally published on Medium