Another Way to Design Short URL Service

Posted on In Views:

Another Way to Design Short URL Service

An out-of-the-box feature in AWS

Short URL service is a common feature, not only does it look better with shorter URLs, but it also has many additional benefits. For example, when a commercial website is sending out promotion SMS, shortening the URL can effectively save the cost of SMS.

In addition, short URL services involve many important concepts of system design and are a common question during technical interviews. To design the architecture of a short URL service, we usually start with the following diagram and discuss some considerations.

General Rules

  • Generation rate: Knowing the rate at the client side to generate short URLs allows us to evaluate what kind of database is needed. In addition, the generation rate also affects whether ACID, i.e. InsertIfNotExisted, is required.
  • Read frequency: If it is a system with a large number of reads, we will always need to consider caching.
  • Retention period: How long to keep the generated short URLs and the generation rate can be multiplied to know the collision probability of the algorithm and whether the database needs to be sharded.
  • Additional features: The need to provide additional features also directly affects the system design, e.g., if we want to track usage, how to update the counters in the case of caching.

The above is a list of functional requirements, but of course, there are also non-functional requirements.

  • Availability: How long does the short URL service allow for downtime, which will determine whether a load balancer needs to be used and whether the database needs to be clustered.
  • Scalability: When the usage increases, not only the service itself must be able to scale horizontally, but also the bottleneck of the database must be addressed.
  • Consistency: What should be done when two concurrently generated short addresses are the same?

In the end, in order to achieve a scalable and highly availability short URL service, the architecture diagram usually becomes as follows.

Well, it’s a bit complicated, but it’s actually putting in all the factors mentioned before.

In order to provide scalability, there is a load balancer to spread the traffic to many API services, and each API generates its own short URL and writes to the corresponding database cluster based on sharding rules. When reading, in order to reduce the database load, we first read from the cache, and then read from the database if it does not exist.

In addition, we have to consider whether the database is SQL or NoSQL, and if it is SQL, we have to design our own sharding mechanism. If it is NoSQL, how to implement ACID. Oh, we haven’t mentioned the monitoring, the monitoring of API, the monitoring of each database shard, and so on and so forth.

As for the hashing algorithm, that is not the focus of this article. After all, the system architecture alone is troublesome enough.

Another Way

In fact, there is a simpler way to achieve a highly scalable and available short URL service.

Using AWS S3 with AWS Cloudfront, there is a step-by-step tutorial in the following article.

Briefly, the S3 object setting x-amz-website-redirect-location can automatically convert the read object to HTTP 301 Redirect, and then with the built-in integration of AWS Cloudfront and S3, we can do the caching.

The whole architecture will become very simple.

Because S3 itself is highly available and scalable, and can be treated as unlimited volume, our problem becomes very simple, we just need to manage the creation of the short URL itself.

Furthermore, both S3 and Cloudfront are monitored by AWS Cloudwatch, so it saves a lot of maintenance effort.

If we want to know the performance of short URLs, we just need to enable Cloudfront’s access log and analyze it with AWS Athena. The official AWS documentation will have enough information.

One more point to note is that AWS S3 has a built-in rate limit. If short URLs are generated and written directly to the root directory, then the entire service is limited to 3500 short URLs per second.

However, this can be avoided by a little trick. The rate limit rule is based on the object prefix, so we just need to add slash appropriately after generating short URLs. For example, if the generated short URL is 10 characters, abcdefghij, then the mid slash can effectively avoid the rate limit by turning it into abcde/fghij.

Another problem is that AWS S3 itself does not have ACID, so when two objects with the same path but different metadata will occur the latter override the former. To avoid this problem is not difficult, we need only to add additional characters into the short URL, e.g. the last few digits of epoch.

Consider the ten-character short URL abcdefghij with the rate limit problem can become 12/abcdefghij.

This depends on creativity, there is no fixed answer.

Conclusion

Short URL service is a good topic to learn system design, although the function is simple, but can cover most of the system design issues.

But if we really want to implement a short URL service on our own, then those interesting topics will soon become nightmares. Therefore, I prefer to use some existing solutions, such as AWS S3, rather than building a complete system from scratch.

However, even with an existing solution, those system design issues must be carefully evaluated, such as rate limit and ACID.

System design is interesting, especially when there are many alternatives, and the process of weighing the pros and cons is always engaging.

Originally published on Medium