CT Wu

Read-after-write Consistency in MongoDB

In this article, I will describe how to achieve read-after-write consistency in MongoDB. The problem what we want to solve is:

• A collection maintains profiles.
• After we insert a new profile, we want to retrieve the top 20 ordered by some criteria in the same function.
• We find the newest profile is always not been found.

Our MongoDB client’s configuration is almost default with a connection string, readPreference=secondaryPreferred.

Changing Read Preference (incorrect)

We want to make the query always comes from the primary, so we try to assign a session-level read preference to primary. The official manual shows the collection-level preference can overwrite the client-level one.

However, it does not work.

Changing Read and Write Concern (solution)

After consulting the MongoDB solution architect, we find the default read concern is local which doesn’t guarantee the operation order. Thus, the solution is not only changing the read concern to majority but also changing the write concern to majority.

Then, the problem is resolved.

By the way, the solution is a little complicated than MySQL does. If we use MySQL as one primary and two replicas, we can switch the read operation from a replica to the primary to ensure read-after-write consistency. On the other hand, MongoDB is designed to face the scene of large-volume data, he has to do more to balance the efficiency and functionality. Thus, in addition to configuring the read preference, using read concern and write concern are also essential.

Originally published on Medium

Trunk-based Development Can Help

The DevOps Handbook tells using trunk-based development can improve the efficiency of developing software. Thus, what is the magic of trunk-based development make us to abandon our familiar git-related workflow?

Git workflow

It should start from the git-related workflow. Whether it is git workflow, github workflow or even gitlab workflow, in general, the entire development process is completed by using feature branches, development branches, and main branches. However, from this point of view, we find that to complete a feature development, at least three branches are involved. Each branch have to be merged into another branch. In addition to resolving conflicts during the merge, when to merge is also important.

Engineers in the same team must be fully aware of the status of each branch, which not only brings cognitive load, but also brings management troubles. Besides, when many major functions are developed, the online code base will be very different from the code under development. Therefore, these several workflows tell us to divide a feature into some small changes and continue to integrate them.

The above figure is a typical gitlab workflow. It is not difficult to see that it is very complicated. There are many interactions between various branches, and there are many rules to be followed. Of course, with these rules, comes many restrictions.

Trunk-based workflow

Extreme Programming Explained advocates getting feedback as soon as possible. Instead of testing slowly in the test environment and staging environment, it is better to go directly to the production to receive faster results. Since git-related workflows have already suggested small changes and continuous integration, why not just take away those branches with many management burdens, leaving only one branch: the main branch?

This is the spirit of trunk-based development. There is only one trunk branch. All features are branched from this branch and continuously merged back into the trunk. The online environment is based on this trunk branch and is the continuous deployment.

In fact, there are many problems to be executed in this most ideal way. After all, no one wants to destroy the formal environment because of a small change, so there are still various test environments. At this time, it is still necessary to have a mechanism to correspond the trunk to each test environment. In practice, the most commonly used method that will not violate the spirit of trunk-based is to use tags on the trunk branch to identify the version. If there are urgent bugs that must be dealt with, a hotfix branch must still be generated from the tag. Nevertheless, unlike git workflow, this hotfix branch that belongs to a specific tag has its life cycle. When the next tag is released in the environment that originally used this tag, this hotfix branch should be eliminated.

Therefore, although there will be release branches or hotfix branches that branch from the main trunk, these two types of branches will not return to the main trunk. In other words, all fixes first enter the main branch and then enter the hotfix branch through cherry-pick. This produces a benefit, the environment that is released after the hotfix can immediately receive the effects of the hotfix.

The workflow is as follows.

Conclusion

In order to drive this development process well, we must have some “good” practices, and the most frequently mentioned is the feature toggle. Because all submitted pieces have to enter the main trunk and will be released at any time, it is necessary to be able to isolate and conditionally control the testing scope for unprepared code. Feature toggle benefits this need.

On the other hand, if it is a very efficient development team, there is no need to have a release branch or a hotfix branch, i.e., online problems can be solved in a forward fix. That is to say the entire software system must have complete telemetries and monitoring systems to find out the online defects faster.

All these good practices are described in The DevOps Handbook, so I really recommend this book.

We can regard trunk-based development as the holy grail of the software development. You must have many good practices in order to be able to use well. Once applying the trunk-based development, it will not only reduce the complexities of managing branches and environments but also reduce the overhead of every engineer, e.g., merging.

Recently, I guide my team to introduce the feature toggle, and my team members begin to experience the benefits of trunk-based development. For a two-pizza team, it can indeed greatly increase the productivity.

Originally published on Medium

I have noticed there are many engineers who cannot distinguish between the unit test and integration test. Even though these two tests are as different as their names, we still cannot use them in the right place. This article helps you understand them and use them correctly.

In order to explain them better, I recommend this article, which introduces the dependency injection to the unit tests to achieve more testing coverage rate. I will leverage the example in it and dive into the integration test further. Before talking about the integration test, let’s take a quick look at the relationship between the unit test and the dependency injection.

Unit Test and Dependency Injection

Here comes a function, aka a unit, saveData.

As we have seen, we need to verify both the success case and failure case to achieve complete test coverage within the unit tests.

Therefore, we can leverage the dependency injection to prune the external dependency from a database.

Like the above examples, we fake the database objects and make sure our business logic is correct. The keyword is “business logic”. We verify the whole business logic in unit tests no matter what the database is. By using the dependency injection, we can easily verify the business logic and reach a high coverage rate.

Integration Test

Alright, we have already ensured the unit works without the database. Things are not likely to go so smoothly after the database is involved. Thus, we have to make some integration tests to verify the database works as our expectations.

We have already verified the units, therefore, we can only verify the database part, i.e "insert into mydatabase.mytable (data) value ('" + data +"')" as follows.

This example is not structured well, because we can apply the layered architecture to build an abstraction upon SQL query, called DAL (data access layer). Hence, we can have a cleaner interface to test the database instead of using raw SQL in a test case. Moreover, in Domain-Driven Development, there is a similar pattern, Repository, and it provides encapsulation for the database access. Those methods are able to provide convenience for writing integration tests.

Of course you can replace the dependency injection with other techniques like mocking. However, in my opinion, mocking will introduce much more implement efforts on writing the integration tests. By using the dependency injection, we will have an independent module/object/interface for the integration.

Conclusion

Why should we distinguish between the unit test and integration test? The reason is doing integration tests will take a lot of time, most of the time from the database access. Suppose an integration test case takes 100 ms, which is very fast for the database access, then it is hard for us to write thousands of test cases. In order to fully test a system, we always try to cover every decision in every function from every file, thus, controlling total time consumption is essential.

That is why Test Triangle shows the unit test at the bottom, and the integration test is up on it.

Let me summarize what is the main difference between unit tests and integration tests.

Unit tests are made for testing business logic; on the other hand, integration tests are for verifying the external dependencies.

With messing up the scenes, it will end up spending more effort and getting less results.

Originally published on Medium

Data Persistence in MongoDB

We have talked about the data persistence in Redis in my previous article, and we had come out a conclusion.

There is no way to ensure no data loss in the single instance Redis even the strictest setting is turned on.

Because Redis is designed as a in-memory data storage handling the high-throughput user scenarios, aka a cache not a database. Hence, the persistence only needs to be usable not reliable.

On the other hand, MongoDB as a NoSQL database is designed to deal with high-volume data and can be scaled out as needed. How about the data persistence in MongoDB?

WiredTiger

We are going to talk about MongoDB with the most popular storage engine, WiredTiger. First, I have to tell that MongoDB is able to persist data. However, if you don’t use MongoDB correctly, data would still be lost.

The story starts from the journaling. By default, MongoDB stores its data into the memory to improve the performance. After reaching the criteria, MongoDB flushes data to the disk. Until then, the data is finally persisted. The criteria is:

1. At every 100 milliseconds (can be adjusted by storage.journal.commitIntervalMs)
2. Every 100 MB of data

The journaling is like Redis AOF, it uses WAL to write data to the disk.

Write Concern

If the behavior of MongoDB is storing data in the memory first, how should I make sure the data is durable? Fortunately, MongoDB provides the write concern to accomplish the data persistence. You can set the write concern to “{j: true}” to make MongoDB acknowledge clients after the data is stored on the disk. Or, you can use “{w: majority}”, this implies “{j: true}” as well as replicates data to most slaves.

The connect string also supports the journaling, journal=true. This provides some convenience if you don’t want to assign the write concern every time.

Conclusion

The data persistence plays a very important role in the system design. In fact, many NoSQL databases have some tricks in them. When you are using them, you should not only use them carefully but also understand the implementations behind them.

MongoDB provides the data persistence as long as the write concern is used properly. In my opinion, “{w: majority}” is the best choice when you are considering the write concern. Because, we usually would like our data can be persisted in addition to persist to replicas.

Originally published on Medium

Data Persistence in Redis

I had written an article about the distributed transactions, which mentioned that Redis does not provide data persistence, with only a brief description. Today, I will deep dive into this topic to see how Redis AOF does.

TL;DR

Redis does not guarantee that the data will not be lost at all, even if the strictest setting is turned on.

The strictest setting here is AOF with fsync at every query. Before we are talking about the reason, I will introduce the AOF further.

Redis AOF

AOF (Append Only File) is a mechanism like LSM Tree. After Redis server finishes processing the commands from clients, it will write a command log appending to the end of the log file. There are three settings to determine when these logs are actually written to the hard disk.

1. No fsync at all
2. fsync every second
3. fsync at every query

The best performance is the first one, and then it gets worse and worse. According to the experiment, we can know the last option works even worse than LSM tree.

From the official manual, it looks like the data is durable when the AOF is enabled. Actually, no, not at all. The trick is the launch time of AOF; If you look closely you will find AOF is starting after commands are processed. It is not like WAL (Write Ahead Log) of other databases. Therefore, if the command is finished processing, and then the system is crashed, you will obviously lost this operation.

fsync every second

fsync at every query makes lots of performance impact, hence we usually adopt fsync every second. This is also the default value in enabling Redis AOF. There comes another question:

Does this mean I will only lose one second of data?

The answer is no. There are two steps in AOF writes logs into the disk.

• WRITE: Write data to file
• SAVE: fsync, i.e., write file to disk

From the implementation of Redis, the flow is as follows:

• Scenario 1: Return without WRITE and SAVE
• Scenario 2: WRITE, but no SAVE
• Scenario 3: WRITE and SAVE

To sum up, if the system crashed, you will lost data within 2 seconds. On the other hand, the description in official manual — appendfsync everysec: fsync every second. Fast enough (in 2.4 likely to be as fast as snapshotting), and you can lose 1 second of data if there is a disaster. — is incorrect.

Let me add, in parenthesis, I had attended EuropeCloud Summit 2021, and there is a session, Accelerating Application Modernization and Cloud Migration with Redis, in day 2; one of a slide shows: Zero Data Loss around 9+ years in production. I am curious how he did it, so I thus ask my question; I have got no answer until today.

Here is my conclusion, using Redis AOF is much more durable, however, it cannot make sure zero data loss. If you want to persist the data as much as possible, you have to not only use AOF but also the replica even as well as RDB at the same time.

Originally published on Medium

Design Patterns of Event-driven Architecture Part 2

In my previous article, I have described some design patterns to build a scalable, robust, efficient, and fault-tolerant system. Today we will talk about some other methods to solve the problems when designing a event-driven architecture.

Request-response Model

Wait, we are designing an event-driven architecture which is asynchronous definitely. Why we are talking about the synchronous model here? In my opinion, the request-response models is very classic way to start up a project. We used to request a response through the restful APIs. This implementation can reduce the development effort and give a straightforward view. However, we don’t want to lose the flexibility of event-driven architectures. The solution is we implement a request-response models upon an event system.

The sequential diagram is shown as the following figure 1.

The server is a traditional HTTP server with some synchronous Restful APIs, and the client requests the server as usual. Nevertheless, the server doesn’t handle this request directly; instead, he delegate this job to the event worker and wait for the response at the agreed place which is another queue. After using this pattern, we can enjoy not only the benefits of the event-driven architecture but also the simplicity between clients and servers.

Ambulance Pattern

The ambulance pattern is used to handle the message priority correctly. The simple design for an emergency event is always put this event to the head of a queue, so that workers can process those high-priority events Immediately. Well, this implementation has some drawbacks, especially, the starvation of the low-priority events. Those normal events might not be processed at all, because there are always emergency events come in.

In order to handle events evenly, we can separate emergency events from ordinary events and submit to a new queue like the follows:

The workers can pick events from those queues to avoid the starvation. There can be a weight to determine the ratio of the two sides, or simply use round-robin to handle those events sequentially.

Furthermore, you can dedicate a worker focus on the emergency events.

Conclusion

We have described two patterns to face some design decisions. In order to make the client more easily, we can leverage the request-response model. If we encounter a scenario is to distinguish the event priority, we can use the ambulance pattern.

The trend of using event-driven architecture has become apparent. However, until now, there is no one-size-fits-all solution can design a well architecture for events. We have to find the corresponding solution according to various situations. Hope these two articles are helpful to you.

Originally published on Medium

How to choose a MongoDB shard key

In this article, I will show you what is the ideal pattern of a MongoDB shard key. Although there is a good page on the MongoDB official manual, it still does not provide a formula to choose a shard key.

TL;DR

The formula is

{coarselyAscending : 1, searchPattern : 1}

I will explain the reason in the following sections.

User Scenario

In order to well-describe the formula, I will use an example to illustrate the scenario. There is a collection within application logs, and the format is like:

1
2
3
4
5
6
7

{  
    "id": "4df16cf0-2699-410f-a07e-ca0bc3d3e153",  
    "type": "app",  
    "level": "high",  
    "ts": 1635132899,  
    "msg": "Database crash"  
}

Each log has the same template, id is a UUID, ts is an epoch, and both type and level are a finite enumeration. I will leverage the terminologies in the official manual to explain some incorrect designs.

Low Cardinality Shard Key

From the mentioned example, we usually choose type at first sight. Because we always use type to identify the logging scope. However, if we choose the type as the shard key, it must encounter a hot-spot problem. Hot-spot problem means there is a shard size much larger than others. For example, there are 3 shards corresponding to 3 types of logs, app, web, and admin, the most popular user is on app. Therefore, the shard size with app log will be very large. Furthermore, due to the low-cardinality shard key, the shards cannot be rebalanced anymore.

Ascending Shard Key

Alright, if type cannot be the shard key, how about ts? We always search for the most recently logs, and ts are fully uniform distributed, it should be a proper choice. Actually, no. When the shard key is an ascending data, it works at the very first time. Nevertheless, it will result in a performance impact soon. The reason is ts is always ascending, so the data will always insert into the last shard. The last shard will be rebalanced frequently. Worst of all, the query pattern used to search from the last shard as well, i.e. the search will often be the rebalance period.

Random Shard Key

Based on the previous sections, we know type, level and ts all are not good shard key candidates. Thus, we can use id as the shard key, so that we can spread the data evenly without frequent changes. This approach will work fine when the data set is limited. After the data set becomes huge, the overhead of rebalance will be very high. Because the data is random, MongoDB has to random access the data while rebalancing. On the other hand, if the data is ascending, MongoDB can retrieve the data chunks via the sequential access.

Solution

A good MongoDB shard key should be like this:

{coarselyAscending : 1, searchPattern : 1}

In order to prevent the random access, we choose the coarsely ascending data be the former. This pick also won’t meet the problem of frequently rebalancing. And we put a search pattern on the latter to ensure the related data can be located at the same shard as much as possible. In our example, I will not only choose the shard key but also redesign our search pattern. The ts is fine to address the log at the specific time; however, it is a bit inefficient for a time range query like from 3 month ago til now. Hence, I will add one more key, month, in the document, so we therefore can leverage the MongoDB date type and make a proper shard key. The collection will be:

1
2
3
4
5
6
7
8

{  
    "id": "4df16cf0-2699-410f-a07e-ca0bc3d3e153",  
    "type": "app",  
    "level": "high",  
    "ts": 1635132899,  
    "msg": "Database crash",  
    "month": new Date(2021, 10) // only month  
}

And, the shard key is {month: 1, type: 1}.

The key point here is we use month instead of ts to avoid frequently rebalancing. The month is not made just for the shard key; on the contrary, we also use it for our search pattern. Instead of calculating the relationship between timestamp and the date, we can use getMonth to find results faster. For instance,

1
2
3

var d = new Date();  
d.setMonth(d.getMonth() - 1); //1 month ago  
db.data.find({month:{$gte: d}});

To sum up, this article provides the concepts of designing MongoDB shard key. You might not have a coarsely ascending data so, but you can refer to the concepts and find out a proper key design for your applications.

Originally published on Medium

Kafka vs. RabbitMQ

I’ve written this article for explaining what’s the main difference between Kafka and RabbitMQ. I know there are a lot of articles that try to compare them on their capabilities like message routing, performance, persistence, etc. I like this post the most, which gives a precise and fair summary to both of them.

Nevertheless, those articles all stand on the server-side to consider how to choose Kafka or RabbitMQ. They rarely consider from the client-side, i.e the story on workers. In my opinion, both Kafka and RabbitMQ itself work very well on every use case. The most common thing we encounter is not that the server cannot handle messages, but the workers are too slow to process messages. If the workers indeed cannot afford the workload, we usually adopt horizontal scaling to increase the throughput.

RabbitMQ

I demonstrate this scenario as follows:

The MQ represents either RabbitMQ or other message queue systems. In this case, we can extend the number of workers to handle more messages at the same time. However, the order of messages cannot be preserved. RabbitMQ treats every message as a standalone entity and can be dispatched to any one of the unified workers. In fact, you can set up a complex exchange and routing rule to ensure the message ordering manually. If you do so, you will introduce the complexity in a system and significantly make a tightly coupling between producers and consumers.

Kafka

How to deal with this problem? I mean how does Kafka preserve the message order. The illustration is:

The magic is a producer still publishes a message to a topic, but Kafka pushes the message into the corresponding partition based on the key. Consumers can be a consumer group, and the consumers in the same group will be dispatched to the certain partition(s). In other words, the same kind of messages are processed on the same consumer to make sure the correctness of orderings. Furthermore, the participants in a consumer group can be dynamically adjusted to sort out the run-time workload.

Conclusion

From my point of view, if the user scenario is the producer generates lots of messages, and you have to add more consumers to digest them, you should use Kafka. Otherwise, Kafka and RabbitMQ both are great tools, you can choose according to the technology stack in your organization.

Originally published on Medium

According to the report, we can see the event-driven model is a hot topic for the system design. However, designing a good event-driven architecture is quite challenging. We are familiar with the synchronization models, which request and response directly, but the stories are totally different in the asynchronization schemes. Nevertheless, here are some architecture patterns for reference.

The blueprint comes from the video and shows the whole vision on a well-defined event-driven architecture.

I will briefly introduce those terminologies in the later sections.

Channel monitoring pattern

The channel monitoring pattern define a role called channel monitor to watch the usage and utilization of the main queue. It monitors some metrics like the amount of pending messages, the amount of consumers and the consuming rate.

Having those metrics helps to understand the workload and the healthy of the system. Moreover, the information can let the system be able to self-healing by adjusting the amount of consumers and throttling the producer.

Consumer supervisor pattern

This supervisor is responsible for adjusting the amount of consumers. It listens to the channel monitor and determines if the current consumers can handle the events. If the supervisor thinks the throughput of consumers is not enough, it will call more consumers to help, and vise versa.

Producer control flow pattern

In the previous section, the supervisor improves the throughput by making more consumers. However, if the cost is a concern, using more consumers obviously is infeasible.

Therefore, there is another approach, producer control flow pattern. When the consuming rate cannot align to the producing rate, a controller will send a signal to the producer to slow down the event generation. After the pending events are processed, the producer can disable the throttling to restore the normal state.

Thread delegate pattern

The thread delegate pattern is a useful pattern not only in the event-driven architecture but the other types of the system design.

The pattern has two purposes:

1. Preserve the message order
2. Track the metrics of event processing, ex. response time.

Let’s begin from the first purpose. When a huge amounts of events enter the queue, we are used to bring up more consumers to handle them, aka scale-out. This method is common in the stateless scenarios. But, some kind of events are stateful, the order of events must be addressed. For instance,

• Put some items in the cart
• Clean up the cart
• Put other items in the same cart
• Charge it

The event order must be kept; otherwise, you may buy the wrong items. Thus, how to handle the events on multiple consumers? In thread delegate pattern, it introduces a new role, event dispatcher, and the dispatcher dispatch an event to the corresponding consumer based on the event type. In other words, all events with the same type will be in the same place, and the order can be ensured.

So, the dispatcher maintains a mapping table to record which type of event belongs to who. After the consumer finishes an event, it callback to the dispatcher. The dispatcher knows all processing results of consumers, i.e., it can track the response time as well.

Workflow event pattern

When event consumer encounter an error while processing, it may not fix the error due to many reasons like the malformed event, transaction conflicts, etc. Hence, the consumer emit this event to another queue handled by workflow processor. The processor will either fix the event or store the event in the storage. In addition, the processor can display the event on the dashboard and inform the human to take over the case. The human can fix the situation manually and resend this event to the queue.

It is worth to mention that if the event is resent to the queue, the order cannot be guaranteed.

Summary

We walk through several design patterns of an event-driven architecture. We can design a self-monitoring and self-healing system leverage those patterns. The system is scalable, robust, efficient, and fault-tolerant. However, there are some noteworthy patterns that have not been covered. I will introduce them in the future.

Originally published on Medium

This article will:

1. explain what is the distributed transaction.
2. introduce how to resolve the distributed problem.
3. provide a real-world example to demonstrate how dose the proper solution look like.

To discuss the distributed transaction, we should talk about the transaction first. We will enter the world of the distributed system design step by step. So, let’s begin.

What is Transaction

The traditional transaction system provide 4 guarantees by the definition. They are known as ACID:

• Atomicity: All tasks in a transaction either all have been done or done with nothing. In this premise, we can ensure our database modifications will be all or nothing.
• Consistency: The operation in a transaction still follows the constraint. In addition, once it is committed, the changes can be read by others correctly.
• Isolation: The isolation is defined as 4 isolation levels. They are Read Uncommitted, Read Committed, Repeatable Read, and Serializable. Different isolation levels can solve different user scenarios. The Seriablizable treats all operations are in order; therefore, he will not encounter the racing conditions anymore.
• Durability: Once the data changes are committed, they will not be lost even the database crashes.

Who has Transaction

After describing ACID briefly, let’s see who can support them. I pick 3 very common targets, MySQL, Mongo and Redis.

• MySQL: Of course, yes. As a representative of OLTP(Online Transactional Processing), MySQL supports the transaction very well, and it is widely used to the transaction scheme. Although, the transaction level is usually set to Repeatable Read, MySQL is still the most appropriate solution.
• Mongo: Since v4.0 released, Mongo has supports the transaction. As long as the correct use Write Concern and Read Concern can ensure Consistency of the transaction. In my opinion, Mongo is very suitable for the de-normalized user scenario.
• Redis: Well, not really. We know Redis has the atomical operation due to the single-thread design. In addition, Redis has MULTI to execute multiple commands or EVAL to call the Lua script, so that Redis can perform lots of data changes at once. Moreover, Redis can use AOF to accomplish the data persistence. I have to say that Redis cannot support the transaction. Redis may leverage those mechanisms to solve part of the transaction stories; as a consequence, it does still not fully provide ACID.

Let me summarize in a table:

How about Distributed Transaction

Alright, we have gone through the traditional transaction. Now, we can talk about the distributed transaction. In order to ease the problem scope, I will give a simple definition to the distributed transaction, that is, provide ACID across multiple homogeneous/heterogeneous data sources. Here are few examples:

1. Insert a row in one MySQL database, and insert another row into another MySQL database.
2. Insert a row in a MySQL database, and insert a document into a Mongo database.
3. Insert a document in one of Mongo shards, and insert another document into another Mongo shards.

The third example can be solved by native Mongo’s transaction after Mongo v4.2 released; therefore, we can skip this example.

Simple Design but Incorrect

Okay, let’s quick look a basic design concept. We perform the transaction on each data source, if there is one of them failed, we rollback the previous tasks.

Based on our simple design, there is an implementation on data source A, B and C like this:

The problem here is: after we commit the result to A, we are hard to perform rollback on A if B is failed.

Even if we adopt the nested transaction, the problem is still not resolved like the following diagram:

According to the diagram below, we can see the same problem as the figure 1. We cannot rollback on C easily.

Two-phase Commit Protocol

In the field of Microservice, this kind of problems are becoming common; hence, many protocols have been proposed to figure out the solution like XA and SAGA. Both XA and SAGA are based on the same theory, 2PC, aka the two-phase commit protocol. I won’t dive into 2PC, I will provide an introduction to explain its concept.

There is a coordinator in the story, everyone want to perform the distributed transaction has to register(first commit) with it. The coordinator will poll every data source who participates in this transaction to get the permission. If there is any participant says no, the coordinator will reject the transaction request and perform rollback to every participant. If fortunately all data sources reply yes, the coordinator will response okay to the client. The client is able to perform the real commit(second commit) to the coordinator to proceed the process.

See this, we can find out three problems:

1. It is very time consuming. Client must wait for every data source agree his request, and then submit the real commit to them. Although this synchronization can ensure the strong consistency on data changes, we have to invest more time.
2. SPOF(Single point of failure) is on the coordinator. If the coordinator crashes, all the distributed transactions do not work anymore.
3. If the coordinator’s commit is failed due to the network loss or other reasons, the data will be inconsistent on those participated data sources.

Eventually Consistency

It seems that multi-phase commit protocol is not feasible, we should try to find out a light weight approach called eventually consistency. Literally we know that we give up the strong consistency to reduce the implementation overhead. In other words, we hope the data can be consistent in the end. To achieve the eventually consistency approach, there are two keywords should keep in mind.

1. Event sourcing: To make this term simple, it means we record the status changes instead of the status itself. For instance, the bank passbook records every transaction not only the balance.
2. Idempotence: The idempotence is quite intuitive. We can perform a tasks many times, and the result is the same as we performed once.

Combine these two concepts, we can design our system now. We make every transaction be an event, if a client want to perform a distributed transaction which means the client emit an event to the event processor. The event processor receives a event and start to do transaction on those multiple data sources. After all tasks finish, the event can be mark as DONE, and the event processor can handle the next one. If there is a transaction cannot succeed on one of the data sources, we don’t mark this event DONE, so that the event processor will retry later. Due to the idempotence, the event can be processed more than once. Nevertheless, putting a retry limit is recommended.

Let’s see the diagram to show the design flow:

Perfect, as a result of the design concept, we can handle the distributed transaction correctly, and the data consistency will be kept eventually.

Even though there is a queue in our design, in fact, you don’t need invoke a real queue in your system. There are many alternatives, e.g., store all events in a MySQL table having a column be the status. The core of this design is the event sourcing and the idempotence, you can accomplish them in what way you preferred.

Originally published on Medium