CT Wu

Original Sin of Microservices, Part 2

Last time, we talked about 8 fallacies of distributed computing and why our systems end up being more complex than we ever imagined. In our last conclusion, we mentioned that two nodes already have a lot of aspects to consider, let alone a microservice architecture.

Microservice architectures have many additional dilemmas to face. Therefore, in this article, we will introduce what are the challenges of designing a microservice architecture.

9 challenges of distributed architecture

When designing a microservice architecture, there are 9 things that must be carefully considered, and these 9 things are the source of the complexity of microservice architecture. In order to design a microservice architecture correctly, there are many factors that must be aware, such as data consistency, architecture scalability, system availability, etc., in addition to avoiding the fallacies mentioned in the previous article.

Service Granularity

When we want to implement a notification service, suppose that there are three types of notifications, email, sms and mobile notifications. So should the granularity of this microservice include all three types of notifications, or should the three types of notifications be separated into a standalone microservice? How to make the right choice?

There are several points to deliberate. First, will these codes need to be changed very often? If not, then it seems feasible to put all notifications into the same notification service. On the contrary, if the content of emails needs to be changed very often, then perhaps emails can be separated out into a standalone microservice.

In addition, if sms has a large number of notifications, then sms may need to be horizontally scaled independently, i.e., sms can be a separate microservice.

How do you determine the correct granularity of microservices? It’s hard, isn’t it? This is the first challenge.

Shard Functionality

Shared service vs. Shared code

When there are shared functions, how do you decide where to place them? For example, when A, B and C all have an authentication function, should there be a microservice for authentication or should the authentication code be replicated in each microservice?

There are also some aspects to think about here. If there is a shared authentication service, who will authenticate the authentication service? When the authentication service goes down, will it result in a single point of failure (SPOF)? What happens when more and more services need to be authenticated and the authentication service can’t hold up?

On the other hand, if the code is copied to each service, how can we make online improvements if the logic of authentication has changed? In fact, the same problem can happen to data. When there is data needed by three microservices, should the data be shared with each other in a separate database or duplicated to each service?

Remember the ideal microservice mentioned at the beginning? No sharing of data. So here, I prefer to use the second approach, replicating the code instead of sharing the service.

Communication Protocols

Synchronous vs. Asynchronous

Should the communication between two services be synchronous or asynchronous? There are many factors to consider.

Asynchronous has the obvious benefit of being more efficient and less coupled with each other. When there is a need to scale horizontally, each microservice can also scale according to its own needs.

But synchronization also has the advantage that Service A can respond more quickly when Service B fails. In other words, synchronization can have higher consistency.

There is no standard answer between synchronous and asynchronous, it depends on the context.

Workflow Management

Orchestration vs. Choreography

Again, these are two very different types of workflow management. Similar to synchronous and asynchronous above, when orchestration is chosen, it is easier to control the entire workflow, and when any task fails, the top Service A has a way to know and take the correct action, i.e., orchestration makes it easier to get higher data consistency.

On the other hand, choreography can have better scalability and availability because of the low coupling between each other. Even when there are functions that need to be implemented, choreography is more productive because it does not need to be initiated by Service A. Each microservice communicates with each other through messages and only needs to implement new message processors.

Monolithic Decomposition

Component-based vs. Tactical forking

When we want to decompose a monolith to microservices, there are two different strategies that can be used. Component-based decomposition is a traditional domain-driven development that first correctly recognizes the function and size of each component, then analyzes the dependencies between the components, and finally tries to decompose the components according to a domain-based manner and generate domain-sized microservices.

In contrast, tactical forking is a completely different process.

First insert a proxy between the original client and the monolith. Have you noticed? The topology is not immutable, in order to compose the microservice, the topology is changed. Then, the original monolith is replicated exactly, and the two different functions are pointed to different targets in the proxy. Finally, according to the characteristics of the two functions, the original monolith is individually reduced to produce two different microservices.

Similarly, the two different strategies also have their own advantages and disadvantages. If component-based decomposition is used, there is a longer lead time, but the whole system can be decomposed with the right domain perspective, and in fact, the process must involve refactoring. In opposite, tactical forking does not have this refactoring process, but in return it is a much faster modification process.

Contract Management

Strict vs. Loose

Contract management refers to what kind of data format should be used to communicate between two services? If you use a loose contract, such as JSON or XML, the coupling between the services will be tighter, and it will not be easy to manage the versioning and achieve forward-backward compatibility. But the advantage is that it is easier to develop, especially since JSON is already the main data format for frontend and backend communication, so JSON will also work more naturally with the service.

On the other hand, using a strict contract, such as Avro or protocol buffer, then the services only need to understand each other through the schema. The old and new versions of the service can also coexist easily.

Database Style

This is also a very popular topic. First you have to choose from PACELC, then ACID or BASE, and finally there are various options depending on the storage structure, such as K-V, columnar, row-based or graph.

The biggest drawback of choosing a relational database in the past was its lack of horizontal scalability, but recently there has been a new breakthrough with more distributed SQL options. But again, there is no perfect solution, you still need to understand what distributed SQL can do and what it lacks.

Behind each of these choices is a lot of experience and domain knowledge, and it’s really hard.

Architecture Style

Choosing the architecture type is interesting. There are many types of architectures alone, such as service-oriented architectures, microservice architectures, event-driven architectures, and so on.

In fact, a system does not have only one architecture style, but usually a hybrid system. There are many aspects to determine when choosing event-driven and when choosing microservices for a “function”.

Distributed Transaction

The last challenge is distributed transactions. I mentioned in my previous article that there are 3 factors should be considered when designing a distributed transaction.

• Consistency: Atomic vs. Eventual Consistency
• Communication: Synchronous vs. Asynchronous
• Coordination: Orchestration vs. Choreography

Therefore, there are 8 combinations, each of which has a different structure also pros and cons. According to the conclusion of distributed transaction introduction, we should choose between eventual consistency and asynchronous, but then we still have to choose between orchestration and choreography.

In the conclusion of the last article, it was stated that if you choose choreography you can get a better decouple and better scalability, but there will be a higher complexity of implementation. This is because it is very difficult to maintain the workflow of the whole transaction in choreography.

Complexity is killing software developers

After discussing so many challenges and fallacies, I’m sure you’ll agree how difficult and complex it is to design a microservice system correctly, right? So, in recent seminars, the topic has slowly shifted from microservices implementation to microservices challenges.

Complexity is the pitfall of modern software architectures. But how do we evaluate complexity? We’ve talked about a lot of complex stuff today, so I’ll just offer a simple suggestion to measure it from my own experience.

When you want to make a change but are very hesitant because you don’t know the consequences of doing so.

Then, the complexity is pretty high.

Are there any specific evaluation criteria? Yes, there are. One of the methods is to analyze the degree of coupling. I have introduced instability in a previous article. This is one of the formulas to evaluate the coupling degree.

Conclusion

We talked a lot about the challenges and problems of designing a microservice architecture, or a distributed system. Every architect has had a journey from believing in microservices to questioning them. In order to prove the change of architect’s mindset, I list the sessions of Worldwide Software Architecture Summit ’22 I cited in these two articles, and also provide you with a reference.

• Software architecture in a non-enterprise world — a practical approach
• Correctly Built Distributed Systems
• What not to do on your next cloud-native architecture
• Software observability: Visualize Code Quality at Scale
• Designing Pragmatic Observability that Works: Avoiding Pitfalls
• Complexity is the Problem
• Microservices anti-patterns and pitfalls

Originally published on Medium

Original Sin of Microservices, Part 1

• Part 1, 8 fallacies of distributed computing
• Part 2, 9 challenges of distributed architecture

This series of articles is a summary of Worldwide Software Architecture Summit'22, thus most of the contents are coming from the seminar. Of course, there are also some insights from my understanding.

Microservice architecture had been a popular topic for several years, and there were various sessions to talk about how to deploy, manage and implement microservices at every seminar about software architecture. However, such a trend in the last six months to a year of time sharply frozen, more and more people noticed that although microservices is a good approach, but not a total solution.

It is beneficial to use microservices, but it also comes at a cost, such as complexity.

What’s the ideal microservice?

Before talking about the original sin of microservices, let’s define what the ideal microservice would look like.

Ideally,

• Non-shared code
• Non-shared data
• With bounded context in a specific domain
• Deployment separately
• Single responsibility principle

It may look like as follows in an architecture diagram.

When clients communicate with backend services, backend usually routes the request to the corresponding microservice through a API gateway. And, this microservice is able to process this request correctly by its own data and response to the clients.

What’s the real world?

But the real world is often not as rosy as we think.

Do you know what kind of animal this is?

As far as I know, there are at least three possibilities, which could be a horse, a donkey or a mule.

Why is it painted like this?

At first, we designed the whole architecture according to our ideal or textbook style and started to implement it, and everything looked so perfect. But as time went on, more and more requirements were added and we were forced to make it work in a limited time frame, even if there were mistakes to be fixed. In the end, the horse-like animal became more and more sketchy.

If there was a good design and planning, we can at least tell from the back half that it might have been a horse. On the other hand, if there was no design at all, no one would probably know what you were trying to draw.

If we represent it in an architecture diagram, it would probably look something like this. Welcome to the real world.

A, B and C all have their own responsibility to handle their own bounded context, but as the requirement grows, A finds that it needs B’s data, and B needs C’s cooperation, and even C has to call A. Finally, it becomes a big ball of mud.

There are several reasons.

1. Usually, there are no architects. So there is no way to evolve the system correctly with each requirement.
2. Even because there was no architect, architecture is not clearly defined.
3. Focus on fast delivery of features, and ignore non-functional requirements.
4. Feedback cycle is very short, and we must respond to requests or incidents within a short period of time.
5. It’s all online and real-time without any downtime until the requirements are eliminated.

8 fallacies of distributed computing

After understanding why systems, in reality, are the way they are, let’s look at 8 fallacies of distributed computing. These are the most commonly overlooked pitfalls of microservice architecture.

The network is reliable

People wishfully assume that the network between two endpoints is reliable. But the truth is that packets may drop and connections may be broken. Therefore, there should be a retry mechanism between the two endpoints to improve the reliability as much as possible.

Latency is zero

There is an unconscious belief that the latency between two endpoints is zero, and this leads to many simple functions being spread out to other endpoints. Unfortunately, the latency between any endpoints is never zero, even on an intranet. So when making remote calls, it is always important to consider setting a timeout, even for connections to internal databases.

In addition, when frequent operations are required on Redis, mechanisms such as Redis pipeline are always used to reduce the round trip time (RTT) between endpoints.

Bandwidth is infinite

People usually feel that the network bandwidth between two endpoints is unlimited, so there is no restraint when transferring data. In fact, the bandwidth is much smaller than you think.

In particular, if you use MySQL’s select * often, you may run out of bandwidth without realizing it. As a MySQL table grows with requirements, there may be many more costs you don’t realize, such as BLOB or TEXT, and such data structures often store huge amounts of data to further consume bandwidth.

The network is secure

This is a very common pitfall, people always believe that the intranet is secure. But this is absolutely unrealistic, and this is why the concept of zero trust has been proposed in recent years.

Topology doesn’t change

This fallacy is a bit interesting. People feel that the topology of a network does not change, meaning that the two endpoints are always written to each other’s location, whether it is FQDN or IP, but in reality, network topology changes for many reasons, such as VPC segmentation, public cloud migration, and even system evolution.

There is one administrator

You would expect that there would be at least one administrator maintaining each service, right? Usually, in every organization, there is an operation role that is responsible for maintaining the system. Therefore, operation plays the role of an administrator, and if there is a problem with the system, operation is definitely the first to know.

Wrong, absolutely wrong.

Any system must implement its own observability, and system developers must have the ability to identify potential problems from these observabilities. There are four most common types of observability.

1. Logging: The logs left by system execution are usually related to system behavior.
2. Tracing: The execution cycle of a task. If it is a distributed system, you must be able to track the telemetries on each system, e.g., jaeger or OpenTracing.
3. Metrics: The measurable status left by the system run, such as the number of API executions, the number of failures, etc.
4. Profiling: Resource consumption behind the system like CPU and memory, etc.

Transport cost is zero

This is absolutely the most serious fallacy. You have to understand that any remote call has a cost, especially on a public cloud, which is even more pricey. Even if it’s a database access or an outbound HTTP call, these all cost money.

So, no matter what the remote call is, it must always be optimized, both in terms of frequency and data size.

The network is homogeneous

This is an easy pitfall to overlook. The network is absolutely heterogeneous, so when one endpoint calls another endpoint in series, you cannot guarantee that the order of arrival is the same as the order you sent it. In other words, any assumptions about order should not exist.

Conclusion

Alright, after this discussion, I believe we can all agree that distributed systems are really complex. There are so many factors that must be considered for just two endpoints, not to mention microservices.

Microservices are composed of countless endpoints, and each endpoint has to consider just those factors, and at the same time, microservice architectures have their own issues to face, such as data consistency, system scalability, and so on.

Since this article is already a bit overloaded, I will schedule the challenges of microservices for the next article. In that time, we will discuss how difficult it is to design a microservice correctly, and we will also recognize why the recent seminars are emphasizing the shortcomings of microservices.

Originally published on Medium

What a design review actually looks like

Last time, we have discussed, how to prepare a design review like an expert. There are three items that should be prepared:

• C4 model
• User stories and use cases
• Design decisions

In this article, I will use a practical example to show you what is a design review looks like. Some discussions with too many details will be skipped, and only demonstrate the critical designs.

User Stories

First, we are going to define the user story. Like the previous article mentioned, user story aligns with the context perspective in C4 model. Therefore, we write down the user stories clearly.

This time we are going to do the function of giving gifts. And, the whole story is as follows.

• A User can decide how many of the same gifts they want to give to others at a time.
• As long as the user has enough money, then there is no limit to the number of people giving gifts.
• After finishing giving gifts, it must inform both sender and receivers that everything has been done.

Use cases

The entire gift-giving scene is clearly described in the user story, however, there are some details that are not sufficiently described. For example,

• If the user’s balance is not enough, then all gifts will fail and cannot be partially successful.
• Finishing giving gifts represents all receivers have received the gift.
• The content of the notification must contain sender, total amount of gifts and all receivers.
• No matter how many people to give, the entire process should be finished in a few minutes.

As we can see above, in the use cases, we added details that were not in the story, and presented the whole scene in more detail.

C4 Model

Now, we can draw the C4 model of the entire design.

Context

Based on the user story, we draw a context to describe the interaction between the user and the system. From the context, we know that there are several key points that must be fully discussed.

1. The line that contains the gift
2. Behavior of the Server itself
3. The line containing the notify

You may ask how about the notification service and receivers. The answer is simple. That is a third party service, and we cannot control its behavior. Therefore, there is no need to discuss it in a design review on the topic of giving gifts. Of course, if there is any concern about the notification, we can hold another design review to dig in.

Container

Once we have the context, we start to dive into the three points listed above and expand them in the form of a container.

Finally we got this diagram, this is what it looks like when it is finished. There are many design decisions here, but in this section, I will only introduce the meaning of this diagram. As for the design decisions, we will leave the analysis in depth in the later sections.

1. Users send gift requests, including what to send and whom to send.
2. After receiving the request, Server first debits the user’s wallet and responds directly to nak if the balance is not enough, then divides the request into batches of fixed size and writes the batch information to the database. After that, the batch task is sent to the worker for execution asynchronously with the serial number of the transaction.
3. Although the process is not complete, the preparations have been done, so reply to the user ack to show that the whole process is in progress.
4. When the worker receives the command, it focuses on the batch task it is assigned, and when it finishes, it deducts the counter from the database. If the worker encounters any errors, just retry the task itself.
5. When the counter is found to be 0 after deducting the counter, which means everyone has finished the task, then the last worker will notify all the receivers.

Component and Code

From the perspective of the container, the next steps are component and code, but these are already related to some implementation details, and each system faces different problems, so I won’t go into more detail here.

Design Decisions

We found many details in the container diagram. As I did in the previous article, we will use the why do A instead of B formula to ask a lot of questions.

1. Why sender and server communicate with each other in a semi-asynchronous way (between synchronous and asynchronous)?
2. Why is the server divided into an orchestration and workers?
3. Why orchestration and workers are completely asynchronous?
4. Why is the worker notifying the receivers?
5. Maybe you are able to consider some questions I have never listed.

All of these questions should be told from the original architecture.

At the beginning, we have already the feature of giving a gift to a person. So, the simplest way is performing a for-loop to send all receivers through the one-to-one gift regardless of the receiver number.

This is fine when there are only a few receivers, but once the number of receivers starts to increase, performance will be a serious challenge. In our measurements, it takes about 100-200 ms to deliver a person, not including notifications, which means that when the number of people reaches 10, it will reach the second-magnitude. This is obviously unacceptable.

It seems that the batching is inevitable, so someone drew up the first architecture diagram.

From the diagram, we can find that the orchestration has already been appeared. However, the communication with the worker is still synchronized, and the notification will be sent only when all the tasks are done. It will not reply to the user until all the tasks are finished. This may seem to have significantly reduced the performance bottleneck, but it hasn’t gotten better at all.

Let’s do a simple math. Suppose we want to send gifts to 1000 people, how do we set the batch size and the number of workers?

To finish it in seconds, the maximum batch size is 10, so 100 workers need to be generated at the same time to handle a gift request. This is a very strict challenge for the system, and it is not an easy task to generate 100 workers in an instant. Because of this, it is difficult to keep users waiting in a fully synchronized manner. So, what happens when it is completely asynchronized?

In the second attempt, we changed the orchestration to choreography, so that the user could get the response in a very quick time and the gift could be sent smoothly. But, is it really so?

What happens if the middle worker fails? The whole chain is broken, and the user may not feel it without notification. It is fine, but for the giver, the middle successful worker has already deducted the user’s money, but the notification is not sent. Moreover, back to the first point of the use cases, partial success is not allowed.

Choreography compared to orchestration will indeed have better performance and better scalability, but will get more complex workflow control. Therefore, in this use case, orchestration would be more appropriate.

So let’s implement full asynchronization through orchestration.

This architecture encounters two problems immediately.

1. Who should send the notification?
2. How to do if the money is not enough in the half way?

The problem of sending notifications is well solved, as in the previous container view in the C4 model, has actually solved the problem of who to send notifications. Nevertheless, it is basically impossible to handle the error in the half way to a completely asynchronous architecture.

Due to this reason, we finally adopted a semi-asynchronous approach. First, the orchestration determines whether the balance is enough to send, and in order to avoid the racing condition, the money is deducted directly. So the worker only needs to handle sending the gift, not to deduct the money from the giver, not even to check the balance.

Error Handle and Disaster Recovery

Under the such architecture, there is a huge trouble.

How about the worker failed?

If it is occurred due to database congestion, it should be okay after just retrying several times. Otherwise, if there is a malfunction in the implementation, it cannot be resolved even though retry many times.

As a result, an additional monitoring system needs to exist to periodically check which asynchronous tasks have failed, and retry those that can be retried, or notify human intervention if they cannot recover by retries. The applicable solution is in a previous article I have already introduced, so I won’t explain too much here.

Conclusion

To sum up, the system breaks down gift-giving into several steps.

1. The user sends the request synchronously and the balance is deducted in advance.
2. All processes of giving gifts are performed asynchronously.
3. The process of giving gifts can be consistent eventually, and the money deducted is equal to the money issued.

In this article, we discuss challenges while designing a distributed system in a practical example.

• Synchronous vs. Asynchronous
• Orchestration vs. Choreography
• Atomic vs. Eventual consistency

In fact, those items are the trade-offs in a typical distributed transaction as well. These aspects can significantly affect the overall distributed transaction model. I have introduced distributed transactions in my previous article. Next time, I’ll take a closer look at the challenges and issues faced when designing a distributed system.

Originally published on Medium

Books that Developers Should Read

There are many posts on medium that are recommending must-read books for developers, but I find that most of the lists are biased towards specific technologies or specific languages, like Fluent Python or Microservices Patterns.

I don’t deny that these books are really good, but not everyone needs to learn a specific language or a specific technology. In the case of microservices, many seminars in the past few years have talked about how to manage, how to deploy, how to develop, but more recently, more and more seminars are talking about

Complexity is killing software developers.

by Scott Carey.

Therefore, building the right background knowledge is far more important than studying a particular technology.

In the next section, I will recommend books for the two stages of a developer’s career, first from junior developer to senior developer, and then from senior developer to the next milestone, perhaps architect or principle engineer.

Junior Developer to Senior Developer

In the junior phase, developers need to write code well and build a proper understanding of the system. So there are several types of books that you can try to read.

For example, to improve the ability to write programs.

Refactoring: Improving the Design of Existing Code

by Martin Fowler

Or, to improve knowledge of the architecture and to design applications that can be easily modified and extended.

Clean Architecture: A Craftsman’s Guide to Software Structure and Design

by Robert C. Martin

Even, in order to speed up software development, you need to know what background knowledge is available, such as telemetries, CI/CD pipelines, feature toggles, etc.

The DevOps Handbook： How to Create World-Class Agility, Reliability, and Security in Technology Organizations

by Gene Kim

Senior Developer to Architect

Then, when you have become proficient in development and understand the entire development process and background knowledge of the application, you are ready to be promoted to senior developer.

What you need at this stage is to be able to design the system architecture correctly, understand the strengths and weaknesses behind each design, and make the right trade-offs.

This requires an understanding of the different architectural paradigms and the risks behind distributed systems.

Fundamentals of Software Architecture: An Engineering Approach

by Mark Richards

Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures

by Neal Ford

Once you have a proper understanding of distributed systems, it’s time to understand the details of distributed storage.

Designing Data-Intensive Applications

by Martin Kleppmann

At this point, you should already have good technical skills and the ability to make the right decision, but also to evolve the system.

In addition, an engineer who specializes in development needs to have not only technical skills, but also a lot of soft skills.

The Pragmatic Programmer

by Andrew Hunt

The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact

by Edmond Lau

Of course, there are many other good books that should be read. But everyone has different career goals, so I’ll just list the soft/hard skills needed to move from a novice developer to a significant position.

My normal articles are a compilation of materials I’ve accumulated through leading teams. But this week I’m at Worldwide Software Architecture Summit 22, and I don’t have much time with the team. So I take this opportunity to compile a list of books that I’ve been asked to read in order to become an architect.

If anyone would like to hear about a specific topic or discuss a design problem you’ve encountered, please feel free to DM me on twitter, @LazyProMedium.

Originally published on Medium

How to Prepare a Design Review Like an Expert?

Know the three main items of a good design review

This article will take backend development as an example, but the same concepts can be applied to various fields.

Before explaining how to prepare a good design review, let’s first introduce what a design review should have. As an architect, I’m going to tell you what I want to see.

1. C4 model
2. User stories and use cases
3. Design decisions

As an architect, what I want to see the most is the above three items, not code flow nor database schema even nor RESTful API spec.

And the most important of them is the C4 model.

C4 Model

Why is the C4 model so important? Before explaining, let’s briefly introduce the C4 model.

The C4 model contains four Cs.

1. Context: This is an overview of the system that describes how users interact with whole system. It will be displayed from the user’s view, so the details of the system will not be shown.
2. Container: Containers here do not refer to container virtualization like docker. A container in the C4 model refers to an entity that can be deployed separately, such as an API server, database, message queue, etc.
3. Component: Components are the modules under a container. It contains the interaction between modules or the association between domains in the domain-driven design.
4. Code: The last is the source code under the component. But when describing the architecture, we will not show the details of the code, it is enough to be able to describe the relationship between objects.

From above description, the purpose of the C4 model is to explain interactions and relationships within the system. From the largest unit, user interaction, to the interaction between objects and objects, these interactions help architects get a complete picture of the entire system and know if such interactions are appropriate.

For example, a small application might look like this from the container’s perspective.

Some developers may think that such a diagram is too simple, however, such a simple diagram is enough for architects to discover possible problems.

1. Scalability: When the number of users increases, is there any way for the API to handle it? Even if API can satisfy the throughput, what about the database?
2. Reliability: When the API crashes, is there a risk of a SPOF (single point of failure)?
3. Efficiency: When the amount of data increases and the response time of the database becomes slower, how will users feel?

Such non-functional requirements can be analyzed from a relationship diagram. The above are just a few common non-functional requirements, see the full list in wiki.

Then, take the perspective of components as an example. A simple social media might be as follows.

This is also a very intuitive diagram. It looks like it just lists the various modules. Is such a diagram really worth it?

Yes. Absolutely.

When we assess the degree of coupling of an architecture, we usually use instability as an indicator.

Ce refers to the number of efferent couplings and Ca refers to the number of afferent couplings. The closer to 0 the more stable the module is, and vice versa. Regarding coupling, I have an article that specifically describes the problem of coupling.

For this social media, its instability of Recommendation module is 0.8. This means that the recommendation is unstable. As long as any dependency module is modified, no matter the interface or behavior, it will directly or indirectly affect the recommendation system.

The architect analyzes the entire architecture through such a simple diagram. Based on different fields and different architectures, there will be completely different decoupling methods. Without a complete view of the existing architecture, the architect can only provide some innocuous comments.

User stories and use cases

How to write good user stories and use cases I have covered in previous article.

In this section I just echo the C4 model above. In fact, the user story also corresponds to the context in the C4 model. To have a good story, there is a way to draw the correct context.

Design decisions

Design decisions are at the heart of the entire design review. The whole design review is about examining whether each design decision is the right one. The C4 model, user stories and use cases mentioned earlier are only meant to provide background information for the whole review, in order to understand the meaning and risks behind each decision.

But what is the design decision?

It feels like a simple word, but in practice it is hard to feel it. So I offer a simple formula:

Why do A (instead of B*)?*

For instance,

1. Why use MySQL instead of MongoDB?
2. Why use cache? (instead of accessing the database directly)
3. Why use the synchronous restful API instead of the asynchronous CQRS?

The questions mentioned above are a bit broad, so let’s continue to narrow them down.

1. Why do we want the recommendation system to use synchronous calls to get data from the post system?
2. Why is this function split into two sub-functions?
3. Why is this line of source code written like this?

Have you noticed? It’s all design decisions, from system orientation to the reason for each line of code, except that in a design review you may not actually be asked what the reason for each line is. Thus, what kind of decisions should be brought up for discussion?

My opinion is that the first three C’s of the C4 model are worthy targets for discussion. By writing down the C4 model and listing the purpose and reason for each line and block, architects will be able to identify potential problems and hidden risks among the many functional and non-functional requirements.

As for the last C, code, let’s leave it to the developers to find out during the code review.

Conclusion

This article explains the three main items of a good design review. The C4 model gives the architect a quicker overview of the system; then the user stories and use cases that the feature will address so that the architect understands what the functional requirements are; and finally the developers’ design decisions so that the architect knows the trade-offs behind each decision.

In this way, everyone is on the same page. The architect can make his recommendations and point out the risks behind the system, and the developers can understand the direction of the system’s evolution and the problems that need to be solved.

It’s important to remember to provide enough information so that the architect can make the right recommendations, otherwise you’ll just get hot air.

In my previous articles, I have actually explained the different design decisions, hoping that the analysis will make you aware of where decisions need to be made and how they should be made.

• Why do we need to define a repo? Dependency Injection
• Why do we need layers? Layered Architecture
• How to evolve a system? Shift from Monolith to CQRS
• How to decouple? Event-driven architecture

Originally published on Medium

Implement Event-driven Architecture With Minimal Effort

Temporal coupling is the most overlooked pitfall

Last time, we explained that temporal coupling can be effectively solved through event-driven architecture. Among them, we discussed the three approaches separately. Starting with the lowest reliability, simply using the event emitter can solve most of the cases with the least effort; secondly, in order to further improve the reliability, a message queue can be introduced to ensure that the event will be executed at least once. Finally, implement event sourcing to ensure that events are not lost at all.

Nevertheless, in organizations with some resource constraints, message queues seem to be out of reach. The resources here include human resources and organizational budget, whether there is no extra manpower to maintain a new messaging service or no extra funds to start a messaging service. Message queuing is one of the high price systems in restricted organizations.

Therefore, in this article, I will introduce how to achieve decoupling through event-driven architecture with minimal resources.

System Overview

Why this topic is born? Because one of our products belongs to this restricted organization. Therefore, in the process of system evolution, we gradually improve the reliability by making lots of design decisions.

The overall architecture above is the final look of our system. We can see from the picture, there is no message queue. Even so, we still achieve high reliability. At least, if something goes wrong, there is a way to recover.

The components, alert manager, crontab and DB are all existing in the system, and no additional components are added. We just split the unit originally executed by a function into an emitter and a handler.

In the next section, I’ll explain how we did it step by step.

System Evolution

The entire system evolution process has gone through four stages, and we have gradually improved the reliability of the entire system.

• Best effort: In the beginning, we simply split the function into emitter and handler, that’s all. This is the most basic practice, all events are fire-and-forget. Of course, if there is no accident occurred, such an implementation is actually not bad. Decoupling can be done with minimal resources. But as mentioned in the previous article, there are two main problems, event loss and emit loss.
• Integrate with Alert Manager: Then we add an alert when the handler fails to execute. By writing the necessary information into Elastic Search and presenting it on Kibana, the person in charge can take corresponding actions after receiving an alert from slack. That is to say, we resolve event loss through manual recovery.

• Event sourcing: In order to track events more completely to avoid event loss and emit loss, we use the existing database to implement a simplified version of event sourcing. Before emitting the event, the emitter writes a metadata into the database and marks the expected handler. If the writing to the database fails, it is regarded as an event emission failure, and the alert manager will be triggered. The handler also updates itself into the database when it completes the event. For the example mentioned above, the metadata of the event looks like this:

1
2
3
4
5
6
7
8

{  
    eventName: "purchased",  
    createAt: "2022/01/01 1:11:11",  
    expected: ["giveCoupon", "lottery"],  
    status: 0, // 0: emitted, 1: timeout, 2: processed  
    done: [],  
    args: ["user A", 5000]  
}

• Apply crontab plus event idempotency: At this moment, it is the same as the previous complete architecture diagram. All the above steps have a fatal flaw, and it needs to be manually recovered by humans. Although this can make sure that the problem can be solved eventually, the mean time to recovery (MTTR) will be pretty long. Therefore, the entire recovery mechanism can be further enhanced, through the workflow event pattern introduced in my previous article. Periodically check which events are not executed correctly through a crontab, and then rerun it.

There are two important points worth noting. First, the processing of each event must be idempotent, which is very important even when using message queues. Because message queues provide the guarantee of at-least-once, not exactly-once. Second, even with idempotency, there should be an upper bound on retries. If the retry fails several times, we still have to notify the person in charge to deal with follow-up matters.

Trade-off

In fact, the above architecture has a lot to discuss. For example,

• Know which handlers are expected when the emitter writes to the database. In other words, the emitter is somehow coupled to the handler. However, in my opinion, such coupling is acceptable. As long as the coupling can be reduced through appropriate coding, for instance, there is a global mapping table to have the relationship between each event and handlers, so this can be regarded as a configuration rather than a coupling.
• Retry in the handler instead of crontab. Each has its own advantages. Retrying in the handler can recover the error as soon as possible, but sometimes the main reason for the failure of the handler is database congestion, and retrying immediately will further increase the load on the database.
• The data are inconsistent. The handler executes successfully but the update fails. Using idempotency to ensure that even repeated execution will not cause problems. On the other hand, the task of the emitter is executed successfully but fails to write to the database, which has to be re-emitted manually. This can be done with a more complex mechanism to implement automatic retries, but at the cost of more complexity, I don’t feel it’s worth it for the corner case where the write fails.

Conclusion

In this article, we discuss some of the trade-offs some restricted organizations face with event-driven architecture. It has to be said that event-driven architecture itself is a highly complex architecture, and whether it is really suitable in small organizations has always been a matter of debate. However, this article provides a simple way to implement event-driven architecture on a small system. In addition to not creating new components, it does not generate too much coding complexity, and it is an easy-to-practice implementation.

Still, it’s not easy to find the right approach for each organization in this long list of design decisions. Behind every straightforward answer, there are many considerations and possible risks. When I’m designing a system, especially a distributed system, I always remind myself to be careful with FLP Theorem.

No completely asynchronous consensus protocol can tolerate even a single unannounced process death

In simple english, it is Murphy’s law.

Anything that can go wrong will go wrong.

How to be as reliable as possible with limited resources, whether in terms of time, manpower, or cost, is the most interesting part of system design.

Originally published on Medium

Understand Temporal Coupling in Code

We often talk about coupling, what exactly is coupling?

Generally, there are three types of component coupling.

1. Afferent coupling: The task of the A component must depend on the implementation of B, C, and D.

2. Efferent coupling: After the task of the A component is completed, B, C, D must be executed.

3. Temporal coupling: After the task of the A component is completed, B and C must be executed. In addition, B is earlier than C.

The components mentioned here can be source code level, module level or even service level based on the granularity.

In this article we will dive into the temporal coupling in particular, because this is the most common and most overlooked pitfall. First we describe in Node.js as follows:

At this point, we found that this is really generic. Almost all of our code looks like this. It is normal to do three things in sequence in a method, isn’t it?

Let’s take a more concrete example. Suppose we have an e-commerce with a function, purchase. Therefore, we begin to code in a simple way.

First summarize the price of all items in the cart. And then call the payment service to deal with the credit card. Simple, right?

Alright, the marketing team wants to let people who spend over 1,000 dollars get a discount coupon, so we continue to modify our purchase.

This feature is also quite common, and then the sales team found that coupon is a good promotion method, so they proposed that people who reached 5,000 dollars could get a lottery chance. This purchase keeps growing.

This is a temporal coupling. Either giveCoupon or lottery actually depend on purchase, which must be done within the life cycle of purchase. Once the feature requirement becomes larger and larger, the performance of the entire purchase will be continuously dragged down. Especially, the lottery usually requires huge calculations, and the purchase is forced to wait for the lottery success to be considered a success.

De-couple timing by domain events

From the previous section, we learned that purchase should only need to process payments, the rest of the behavior is additional, and should not be in the same life cycle as purchase. In other words, even if the giveCoupon fails, it should not affect purchase or lottery.

There is a method in domain-driven development called domain events. When a task is completed, it will issue an event, and the handler that cares about the event can take the corresponding action after receiving the event. By the way, this approach is also called the Observer Pattern in the design pattern. In domain-driven development, the “notification” contains the domain’s ubiquitous language, hence the notification is named domain events.

Therefore, let’s modify purchase a little bit in the Node’s way.

With events, we can completely decouple giveCoupon and lottery from purchase. Even if any one of the handler fails, it does not impact the original payment flow.

Whereas purchase only needs to concentrate on the payment process. When the payment is successful, emit the event and let other functions take over.

If there are more needs in the future, there is no need to change the original purchase, just add a new handler. And this is the concept of decoupling. Here we remove the code-level coupling and timing-level coupling.

How to handle event loss

In my previous article, we mentioned that whenever failures can happen, we have to expect them and handle them gracefully. This is called resilience engineering.

When we decouple the coupons and lottery through domain events, we will immediately face a problem. What if the event is lost? The payment is finish, but the coupon has not been issued, which is definitely a big problem for the customer.

In other words, how do we ensure that the emitted event will be executed. This is exactly why message queues were introduced into the system.

We discussed the message queue before, there are three different levels of guarantees in message delivery, which are:

• At most once
• At least once
• Exactly once

Most message queues have the at-least-once guarantee. That is to say, through the message queue we can make sure that all events can be executed at least once. This also ensures that messages are not lost.

Thus, to avoid event loss, we will change emitter.emit to a queue submission with like RabbitMQ or Kafka. At this stage, we have introduced decoupling at the system level, i.e., make event producers and consumers belong to different execution units.

How to handle emitting loss

The story isn’t over yet. We can already ensure that emitted events are executed. What if the event isn’t sent at all? Continue to take purchase as an example, when payByCreditCard has been successful, but it doesn’t send the event due to the system crashes for unexpected reasons. Then, even with a message queue, we still get the incorrect result.

In order to avoid this problem, we can leverage the event sourcing. In Distributed Transaction and CQRS, I have described the core concept of event sourcing.

Before the event is emitted, store the event into a storage first. After the handler finishes processing the event, then mark the event in the storage as “processed”.

There is one thing should be aware, the writing of events and the payment must be under the same transaction. In this way, as long as the payment is successful, the event will also be written successfully. Finally, we can periodically monitor for overdue events to know what went wrong.

Conclusion

This time we are still going through a step-by-step evolution of the system as we did in Shift from Monolith to CQRS to let you know how to decouple when systems become large and complex. At the beginning, we first decoupled source code and execution timing through domain events; then we introduced message queues with message producers and consumers to achieve system-level decoupling.

As I said before, a system evolves to solve a problem, but it also creates new problems. We can only choose the most acceptable solution and seek compromises in complexity, performance, productivity and other factors.

Splitting a complete action into different execution units must encounter inconsistencies. In resolving inconsistencies, there are many considerations, such as:

• Regardless of whether the event will be lost or not, just use the simplest architecture, EventEmitter. This approach is the simplest, and there may be no problem in 80% of the cases, but what should we do if there is a problem?
• Attempting to be as reliable as possible, so introduce message queues, which should be 99% sure that there will be no problems. But there is still 1%, is such a risk bearable?
• Implementing event sourcing comes at the cost of increased complexity and performance may be affected. Is this acceptable?

Just like I always say, there is no perfect solution to system design. Each organization has a different level of risk tolerance. In various indicators, we look for the most acceptable solution for ourselves and think about the risks and failures we face at any time. As a result, everyone should be able to build a resilient system.

Originally published on Medium

How Kanban Works and Why I Prefer It Over Scrum

Improve work productivity by using Agile development

We have talked about using trunk-based development to improve software development productivity. In fact, to dramatically increase productivity, you must apply multiple methods, which is why there are so many agile methodologies, two of which are well-known agile methods: Scrum and Kanban.

Scrum is currently the most used by most organizations. Nevertheless, it is not easy to use Scrum well. In the many agile methodologies, Scrum is the most variant, and it can be said that there are a hundred Scrum practices in a hundred organizations.

This is because the original Scrum has many “rules”, which cannot perfectly match the existing organizational structure of each organization. According to Conway’s law, most software architectures are closely related to organizational structures, resulting in various agile methods that can only provide the prototype and cannot be applied entirely.

This article will briefly introduce the purpose of the agile method and Scrum, as well as the problems encountered by running Scrum, and then describe the main topic: Kanban.

Why not Waterfall?

Why do we need agile methods?
What exactly needs to be agile?

First, let’s define what agile is.

The purpose of agile is to generate value “faster”. So, what is the value? Frankly speaking, it is something that can make money. In other words, it usually exists in the form of features on a product. That is to say, only when the feature or product is delivered to the customer can generate the real value, and the purpose of agile development is to accelerate the delivery of this value stream.

When we evaluate the delivery speed of the value stream, one indicator is usually used: lead time. The entire time from a feature request is generated to the requirement is actually delivered to the customer is called lead time.

Under traditional waterfall development, this lead time can take several months, which doesn’t sound agile at all. Moreover, if this requirement is implemented halfway, it turns out that this is not what the customer wants, then all the resources invested are wasted.

Therefore, modern software development focuses on dividing requirements into small pieces, and each small pieces has its own value. By continuously delivering these small values, the product iteration cycle can be faster.

Scrum

I believe most of you have heard or even experienced the Scrum development process. From the conclusion of the previous section, we can know the core spirit of Scrum is to decompose large requirements into small requirements through a preset short cycle: Sprint. And, each small requirement is mapped in the form of a user story. It is important to have deliverable outputs at the end of each Sprint.

There are three roles in Scrum.

1. Product Owner: Obviously, this role is the owner of the product . He is like the traditional PM and sales, as a gateway between the development team and the customer, to find out what the customer desires.
2. Scrum Master: The person, who is the most familiar with the Scrum process in the Scrum team, is responsible for hosting every “Scrum-defined” event, and must also act as the coordinator of product requirements and the development teams.
3. Functional Teams: This role represents the development teams, the people who implement the product requirements. There are many different divisions depending on the team size and nature of the teams, such as frontend, backend, QA, etc.

A Sprint consists of several different events.

1. Planning: This event is used to decide what requirements should be taken in this Sprint. At this stage, each small requirement is usually made into a card with its cost points. Each fixed Sprint is equipped with a fixed number of points, that is to say, in this cycle, which tasks should be completed according to the number of points and priorities.
2. Developing: After each member receives the assigned card, they start to work.
3. Retroactive: At the end of Sprint, review the status of everyone’s cards and ensure the value that this Sprint can deliver, as well as review the performance of this Sprint.

The above description briefly describes a complete Scrum development process. As far as we know, Scrum is complex, there are many dedicated roles and events, and there are many overheads for Scrum. Taking the team I used to work for as an example, Planning will take half a day, and Retroactive will also take half a day, plus there are regular meetings such as stand-up in the morning, the basic expenses alone can take two working days.

In addition, estimating the number of points is also an imprecise practice. Who should set the price? Who will assign it? How much points can a cycle handle? It takes a long time to get used to each other before we can gradually get on the right way.

Furthermore, how should the allocation of roles be configured? Each organization usually already has a basic organizational structure, and it is extremely difficult to extract roles that can adapt to Scrum.

All of the above situations are even worse if there are several parallel Scrums. Especially when someone is in multiple Scrum teams, regular meetings can take up most of the time.

Kanban

To sum up, Scrum is not efficient for small teams. Therefore, the teams I lead use another agile method: Kanban.

Compared with Scrum, Kanban is used by fewer teams. The main reason is that most teams don’t use Kanban correctly and thus don’t see the benefits of Kanban. Most people just plan the various stages of the Kanban board, and then place the cards on the lanes like Scrum, and then move them through stage by stage. In the end, they didn’t get any benefit, just using Kanban in form, and then going back to Scrum, after all, the card making method is the same.

Because of this, in this section I will first introduce the purpose of Kanban, then explain the key to using Kanban, and finally describe the development process of Kanban.

Objective

Kanban is just like its appearance, it is actually a view of the value stream, and each stage of the value stream should be presented in Kanban. Assuming that your organization’s development process is, analysis, design, implementation, testing, and release, then there should be five stages on the Kanban board.

As mentioned in the first section, the agile method is to accelerate the value stream; therefore, Kanban presents the entire value stream in a concrete way, and can also see the spending time of each task. When there is a card that cannot move to the next stage for a long time, it means that there is a problem, and the task status should be checked immediately and appropriate measures should be taken right away.

For example, If the task content covered by the card is too large, the card should be divided into two immediately. And, let someone take over the small cards that are divided out. By checking the status of each card, the delivery of value can be accelerated.

Keys

There are only three key points to using Kanban correctly without any defined events.

1. Cards can only be moved backward, not forward. Each stage of Kanban represents the stage of the value stream. Like the flow of water, water only flows down, and value only flows up to the customer. For instance, suppose one of the stages is development and the other is testing, even if a bug is found during testing, the card will not go back to development, instead, it should remain in testing.
2. Define the Definition of Done (DoD): DoD is important for Kanban, what conditions must be met to go from one stage to the next is about quality. Taking development and testing as an example again, if the condition of DoD is that the unit test coverage reaches 80%, the card cannot enter the testing phase until this condition is met. On the other hand, it is also important to define a “measurable” DoD. It is recommended to turn the DoD of each stage into a checklist, which can make sure that every card movement is compliant and qualified.
3. Determine Work-in-Progress (WIP): WIP is arguably the most important concept of using Kanban. Each stage must define its WIP, that is, the maximum number of cards that can be accepted. If the WIP of the next stage is full, even if the DoD has been completed, the card still cannot move to the next stage. The purpose of WIP is to be able to see at a glance where the value stream is stuck. As mentioned in the previous section, once a value stream gets stuck, immediate action must be taken. How do you know it’s stuck? The spending time of a certain card is an indicator, but when the number of cards is large, this indicator is easily ignored. Therefore, WIP is a more effective enforcement measure, as long as you see that the WIP is full and the card is forced to wait, then there is something blocking.

Working Flow

In this section, I will explain how my team uses Kanban. First of all, the Kanban boards we used are: open, to-do, design, develop, test, released, and closed.

• Open: All the feature requirements will be placed here after they are issued. At this time, the content of this card will be huge, and the time-consuming and so on are not estimated.
• To-do: According to the priority, the cards are taken out from the open, evaluated and split, and a large card will be split into several small cards and put into to-do. The working time per card is about 1–2 weeks. The WIP of the to-do will be set slightly larger, which represents the capacity of my team. We will queue up to do it. If the to-do is full, it means that my team has currently exceeded the load.
• Design: In the to-do stage, the due date of each card should be planned, and the design stage will further determine the spending time of the card in each stage. If it gets stuck on a stage in the future, it means that something is wrong. The DoD in the design stage is to produce the design documents included in the user story and use cases mentioned in my previous article.
• Develop: After entering the development stage, it is the time for the feature development. At this time, the focus is whether the expected time and WIP are exceeded. WIP is defined here as roughly twice the size of the team. As for DoD, the unit test is completed, and the integration test on the local environment must be passed.
• Test: Cards in this stage have actually deployed to the staging or pre-production environment. Due to trunk-based development, every code commits into the trunk, and each test environment is released from the trunk. It is tested by QA in a pre-production environment, and after completion, it is ready to the release stage.
• Released: The feature has deployed to the production environment. At this stage, we will pay attention to a period of time to clarify whether the feature is working properly, and the time is DoD. If the observation time is satisfied, the card will enter the final stage: closed.
• Closed: Features in this stage are basically working normally, and every once in a while we will review the number of deliveries as our performance and then achieve.

Why do I Prefer Kanban?

As you know from the description in previous sections, the use of Kanban is flexible and can be easily applied regardless of the organizational structure. There will be no fixed regular meetings and no dedicated roles, and everyone can still develop like before. The Kanban board is just a visual tool that provides managers with an intuitive view of the progress of the work.

The whole Kanban approach has no complicated rules, just the value stream, DoD, and WIP defined by each organization based on their existed working processes. By mastering the purpose and principles of Kanban, it is easy for organizations to implement their own customizations on Kanban, which is also the benefit of Kanban.

As a leader experienced with Scrum and Kanban, I will continue to lead my team with the Kanban method in the future. After all, meetings are such a waste of time.

Originally published on Medium

Explained with a code sample

This article is the last of this series. We have already described the problem encountered in data-oriented design. In this article, we will introduce a better way to tackle a feature requirement.

We continue the previous example, a sign-in mission, and try a different design flow. Before we start, let’s review the onion architecture again.

In order to make it easier to understand the process to be introduced later, let’s first define several important legends of this diagram.

• Entity: In clean architecture, entity means the business logic. Different from the entity in domain-driven design, the entity here can be realized as the domain in domain-driven design.
• Use cases: With the domain, the outer layer is use cases, which refers to clients who use domain knowledge to fulfill specific needs. In domain-driven design, it is also known as the domain service.
• Controller: The Controller is quite simple. It is responsible for managing the ingress and egress of the entire domain, including input checking, and converting domain knowledge into a data structure that is presented on the client side.
• DB: The outermost layer is the external dependencies of the system, including the database.
• Arrows: The arrow pointing from the outside to the inside is a reference. The outer module can reference the inner module, but it cannot be referenced from the inside to the outside.

According to this description, we can know that the order of design should be from the inside to the outside. After the inner layer is established, and then it is able to be referenced by the outer layer. In other words, to complete a design in a clean architecture way, the domain behavior must be defined first, and the database design should be the last. This is the exact opposite of data-oriented design.

Domain-driven Design

Before starting the actual design, let me explain my usual design process, which also echos the onion architecture.

1. Discover user stories (entities)
2. Design use cases
3. Model domain objects
4. Implement unit tests
5. Code

In later sections, I will also design with this process. The problem we want to solve is to build a sign-in mission mentioned earlier.

Discover user stories

To start a design, we must be able to understand the whole picture of the entire requirement, and user stories are a language that can describe the requirements. In our needs this time, the stories are similar to the following.

1. Get corresponding rewards when you sign in consecutively.
2. Display the sign-in status and received rewards for this cycle.
3. Get 100 diamonds when opening the gift box.

We convert the descriptions in the requirements document into semantics that developers can understand through an ubiquitous language. With any requirement, there must be a story behind it, and the designer’s job is to discover those stories. On the other hand, for the developers, they implement those stories in coding.

Design use cases

With the story, then we need design the use cases that the story faces. Unlike a story, a use case refers to the outcome of a given user scenario. For example:

1. Sign in: When a user signs in for four consecutive days, the first sign-in on the fifth day can get 30 diamonds and a gift box. But the second sign-in got nothing.
2. Open the gift box: When opening the gift box, you can get 100 diamonds, but it cannot be opened again.

From the above description, use cases are actually an extension of user stories and describe details that are not defined in the story. Therefore, from the use cases, we can draw a flowchart to explain the whole user scenario in detail. Let’s take sign-in as an example with a flowchart.

Starting from the top starting point, it is the moment when the sign-in action occurs, so it is represented by SignIn: now. Next, we need to know how long is the difference between this sign-in and the “last sign-in” in days. If it is 0 days, it means that you have already signed in, and there is no reward to get. Or the difference is greater than 1, indicating that the sign-in is not continuous this time, and the entire cycle needs to be reset. In case of 1 exactly, it is continuous sign-in, thus the continuous date is incremented, and the current time is recorded.

Finally, check the table according to the number of consecutive days to know how many rewards you will get.

It is also easy to display how many consecutive days you have signed in. Suppose we use list to represent the signed-in records.

• Only sign in for one day: [1, 0, 0, 0, 0, 0, 0]
• Sign in for three consecutive days: [1, 1, 1, 0, 0, 0, 0]

Therefore, we can know how many 1 to insert to the list from counter.

The flow of opening the gift box is similar, so I won’t explain too much here. The final code will include opening the gift box.

Model domain objects

From the use cases we can know that we will need two very important variables: counter and last. In fact, the rest of the state is determined by these two variables, so we can start modeling.

In order to describe the entire sign-in mission, I believe that each user will have its own state, so we encapsulate the user state into a domain object called SignInRepo. The Repository in DDD is used here. Then with the user state, we can describe the whole story. There are two actions in the story, signIn and getTimeline, which represent story 1 and story 2 respectively.

Because SignInRepo is defined on the basis of use cases, it is part of the entity in the onion architecture. According to the flowchart, it has two private variables and two public methods. The reason why update has a parameter is that we can see from the flowchart that we only have one operation counter++, set last=now, and now must be passed in from the outside. As for SignInService, it can be known from the name that it belongs to the domain service.

Once we have domain objects, we can start developing in test-driven development, TDD.

Implement unit tests

In the development process of TDD, we write the corresponding tests according to our user stories at first, and then the actual coding is carried out. Hence, in this section, we’ll explain how to write unit tests with our defined stories and models. Let’s take a regular story as an example, suppose we’ve signed in for six days continuously, and on the seventh day, we’ll get 100 diamonds and a gift box.

First, write a test based on our story.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

describe("step1", () => {  
  it("continuous 6d and signin 7th day", () => {  
    const user = "User A";  
    const now = "2022-01-07 1:11:11";  
    const service = new SignInService(user);  
      
    const timeline1 = service.getTimeline();  
    expect(timeline1).to.deep.equal([1, 1, 1, 1, 1, 1, 0]);  
      
    const result = service.signIn(now);  
    expect(result).to.be.equal(100);  
      
    const timeline2 = service.getTimeline();  
    expect(timeline2).to.deep.equal([1, 1, 1, 1, 1, 1, 1]);  
        
    const result = service.signIn(now);  
    expect(result).to.be.equal(0);  
  });  
});

One of the stories is briefly described above, there is a user, A, who has signed in for six consecutive days, and when he signs in at 2022-01-07 1:11:11, it is the seventh day to sign in. He gets 100 diamonds as our expectation.

But such a story is not complete, because six consecutive sign-ins have not been defined. So let’s modify the test a bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

describe("step2", () => {  
  it("continuous 6d and signin 7th day", () => {  
    const user = "User A";  
    const now = "2022-01-07 1:11:11";  
    const repo = new SignInRepo(user);  
    repo.restoreSingInRecord(6, "2022-01-06 5:55:55");  
    const service = new SignInService(repo);  
  
    const timeline1 = service.getTimeline();  
    expect(timeline1).to.deep.equal([1, 1, 1, 1, 1, 1, 0]);  
      
    const result = service.signIn(now);  
    expect(result).to.be.equal(100);  
      
    const timeline2 = service.getTimeline();  
    expect(timeline2).to.deep.equal([1, 1, 1, 1, 1, 1, 1]);  
        
    const result = service.signIn(now);  
    expect(result).to.be.equal(0);  
  });  
});

In order to restore the entire use cases, we newly defined a repo and added an auxiliary method: restoreSingInRecord. This helper can also be used as an interface to retrieve values from the database in future implementations. Subsequently, such a story is complete and can begin to go into the production code.

Code

In the previous section, we have a complete unit test, and then start implementing SignInRepo and SignInService.

SignInRepo is easy to implement when there is no database, just follow the flowchart to finish update and reset. SignInService is totally implemented in accordance with the use cases, and the flowchart is converted into the actual code.

In this way, this requirement is half completed, and the remaining process of opening the gift box is basically the same, so I will just post the final result. The full implementation can be seen as follows.

Summary of Domain-driven Design

In fact, the above implementation just borrows some DDD terminologies, and does not fully implement DDD’s “prescriptions”. From my point of view, DDD provides a concept that enables people to know the importance of the domain and has the ability to abstract the domain.

That is to say, it is up to you whether to follow the textbook to implement Entity, Value Object, Aggregate, and Repository or not. It needn’t implement in DDD by following the textbook approach. The implementation depends on the proficiency and understanding of needs.

In this article, a standard design process is provided, so that everyone can disassemble the original requirements and convert them into models with domain knowledge by following this process.

The process of implementing the model, it begins with the corresponding tests to achieve test-driven development.

Of course, in the real world, it is not as simple as the example in this article. But the design process is the same, starting from the story, defining the use cases through the story, then modeling according to the use cases, writing tests according to the stories, and finally implementing it.

By the way, I explained some design details a while ago, such as:

• Q1: Why do we need to define a repo? Dependency Injection
• Q2: Why do we need layers? Layered Architecture
• Q3: How to evolve a system? Shift from Monolith to CQRS

If you encounter software design problems, you are also welcome to discuss them with me.

Originally published on Medium

A look at the data-oriented design approach

• Part 1, Data-oriented Design
• Part 2, Domain-driven Design

There are many articles and books talking about Clean Architecture, but the focus is on why and what, such as why to use an onion architecture, what each layer represents, and what is the SOLID principle.

However, developers actually want to pay more attention to how, that is, how to write a good program architecture according to those design principles.

In this article, we try to solve a common architectural problem in a clean architectural way.

In many mobile games or commercial websites, in order to improve user retention, a sign-in mission is designed. If you sign in, you will get some rewards. When you sign in continuously for a certain number of days, there will be additional gift boxes that can be opened, and you will get bonuses after opening. Let’s try to fulfill this requirement together.

Problem Domain

Users who log in continuously will get diamonds, as shown in the following table:

To simplify the problem, let’s get a fixed amount of 100 diamonds the gift box is opened each time. For example, if a user signs in for four consecutive days, the user can get 30 and a gift box on the fifth day of sign-in.

Every time the user enters the homepage, he will see a sign-in map, telling the user how many consecutive sign-ins have been completed in this cycle and how many rewards have been received.

Design Phase

There are two different design approaches, data-oriented design and domain-driven design. We will start talking about data-oriented design and tell you why it’s not a mainstream design solution.

Data-oriented Design (Can’t Say Good Approach)

The data-oriented design means that when we see a problem, we first think about how to store the data, and then manipulate the defined data format to try to solve the original problem.

Therefore, after getting the question, the first step is to think about what database to use, what schema it will have, and what format the data will exist in.

Let’s take the sign-in mission as an example. Suppose we design on a small monolith, usually with a relational database, so we first define a table to realize our problem. Define a sign-in table to store the time of each sign-in, so that you can know how many rewards you can get based on the previous sign-in records.

From the above table, if today is 2022-01-04, then John can get 15 diamonds when he signs in, and Mary will start a new cycle because she has not signed in consecutively and only get 10 diamonds.

The logic of the mission will be to filter the sign-in date of a specific user and sort them by date, then retrieve the latest N records. By calculating the total days of the consecutive sign-ins, and then you can know what reward to get this time.

There is a problem with such design. When the user’s sign-in is not interrupted at all, i.e., the number of consecutive sign-in days is greater than N, then the amount of data is not enough to determine the current reward. Nevertheless, we do not want to expand N to avoid pulling too much data makes the database a bottleneck. Thus, we should try making some changes to the original schema, such as adding a new field, diamond.

For John, signing in on 2022-01-09 will get 10 diamonds because we know the end of the previous cycle was 2022-01-07.

The story is not over yet. When the number of users grow, the performance of RDBMS will be tough. Every time you enter the homepage, you have to pull data from the database, run a complex calculation, and finally see the result. This process is very inefficient.

In order to improve the efficiency of the homepage, most projects will introduce a cache to record the status of the entire sign-in. However, how do you know if the cache is invalid? Or the cache is actually not invalid, but distorted. I have covered this problem in a previous article Resilient Caching. By adding integrity, we can distinguish whether the data is reliable. Once the data is corrupted, we will rebuild the data from the RDBMS.

Summary

From the above introduction, we found that even the cache has many aspects to consider, the whole data-oriented design is very complicated, and the process is very struggled.

Hence, the domain-driven design was born.

In part 2, we will start designing from Entity, covering use cases, then implementing the unit tests like TDD, and finally completing our mission. I will provide a better design process, and you can follow the steps to simplify every user stories you will encounter. Let’s call it a day. Until next time

Originally published on Medium