The growing demand to share content through the different social networks has rapidly led us to the need to shorten URLs in order to free space within a post (Twitter is a good example here) and it actually looks good.<\/p>\n\n\n\n
We didn\u2019t want to reinvent the wheel \u2013 so we have been using popular external services for this purpose.
\nHowever, by producing a somewhat heavy load on the service, we\u2019ve run into some well known problems: request limit per hour, external service timeouts, service downtime, and other nasty things.<\/p>\n\n\n\n
The shortening service was used as a part of a few business tools.
\nShortening was done both in the background, meaning it had to be highly reliable, and in the foreground, necessitating speed of process.
\nAlso, when shortening was finished, the new, truncated URLs had to be accessible for the consumers.<\/p>\n\n\n\n
From time to time, we encountered all of the issues, resulting in slow, unreliable, inaccessible shortened URLs.<\/p>\n\n\n\n
So, we had to do something.
\nThe solution should meet the following requirements:<\/p>\n\n\n\n
1) In order to maintain high accessibility, the new method would require the ability to do a full fallback, just in case the main service went down completely;<\/p>\n\n\n\n
2) There would have to be a visual consistency in the shortened URLs; and<\/p>\n\n\n\n
3) It must be easy to develop, cheap to host and take zero support efforts.<\/p>\n\n\n\n
The popular shortener service was set to use a custom domain<\/strong> xxx.aa, so that we have control over the accessibility. A very simple backup shortener service<\/strong> was to be developed and hosted on a xxx.bb domain for visual similarity. A shortener client<\/strong> was to be developed to automatically balance between services, when the url creation wouldn’t work. The shortener client<\/strong> would also contain a caching mechanism to reduce duplicated requests to the external service, thus saving limits usage.<\/p>\n\n\n\n We would monitor the external service and in case it goes down, the xxx.aa domain had to be switched to the backup shortener service<\/strong>. Backup shortener service<\/strong><\/p>\n\n\n\n To make sure that both urls are visually similar, we were running a custom domain (xxx.aa) on one of the external shortener while running our own backup shortener on xxx.bb. So when the backup model took place – the difference was not really noticeable.<\/p>\n\n\n\n Initially memcached was used for caching.<\/p>\n\n\n\n Several hardware upgrades helped us to switch to Redis quickly. On the “bad” days, or just times when we had a big volume of urls to shorten, we had near 50\/50 conversions between external service and our backup. Architecture of our solution is as simple as possible. At the time of its creation we’ve used Rails 2 and Redis as a storage of short URLs.<\/p>\n\n\n\n We have two controllers – shortener<\/em> and redirector<\/em>.<\/p>\n\n\n\n Shortener<\/em> takes incoming parameters, parses them and tries to create short URL. Redirector <\/em>runs as Rails Metal application. All it does is searching for short url hash taken from incoming parameters and sending a redirect request to the original URL. Nothing more, nothing less.<\/p>\n\n\n\n The client was created as a gem, so it could be reused in multiple projects. The get_service_short_url method implements remote requests to each service with response parsing.<\/p>\n\n\n\n For consistency purpose we would use the popular Base62 encoded string of 6 characters length.<\/p>\n\n\n\n After trying several approaches we came to the current one finally. It is based on using MD5 hashing of incoming string.<\/p>\n\n\n\n Here we’re taking string, adding salt, converting it to MD5 hash and taking the first 7 chars. Sample code to convert Base36 to Base10 and then to Base62<\/p>\n\n\n\n Great! We’ve decided to add trailing “_” sign to the original URL for the conversion purposes to produce different result. After implementing all of the above – we’ve never had any issues. By the time the external service got more stable and don’t fall down as often as they did earlier, especially with the caching solution in place.<\/p>\n\n\n\n But if it do fall, our balancing and the backup service is right in the place doing it’s job.<\/p>\n","protected":false},"excerpt":{"rendered":" The growing demand to share content through the different social networks has rapidly led us to the need to shorten URLs in order to free space within a post (Twitter is a good example here) and it actually looks good. We didn\u2019t want to reinvent the wheel \u2013 so we have been using popular external…<\/p>\n","protected":false},"author":20,"featured_media":9469,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["Igor Antonyuk"],"class_list":["post-246","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"categories_data":[{"name":"Engineering","link":"https:\/\/railsware.com\/blog?category=development"}],"post_thumbnails":"https:\/\/railsware.com\/blog\/wp-content\/themes\/railsware\/vendors\/images\/article-thumbnail-default.jpg","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/246","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/comments?post=246"}],"version-history":[{"count":31,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/246\/revisions"}],"predecessor-version":[{"id":14001,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/246\/revisions\/14001"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media\/9469"}],"wp:attachment":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media?parent=246"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/categories?post=246"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/tags?post=246"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/coauthors?post=246"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nThe xxx.aa DNS TTL would be set to 5 minutes, so we can switch it any time.<\/p>\n\n\n\n
\nIt wouldn’t contain any extra functionality: just shorten a long url and follow a shortened url.<\/p>\n\n\n\n
\nFirst, it would try the external service and if it doesn’t work – would switch to the backup service.
\nFor a quick response time, it would perform just one shortening try when called from the foreground, but would perform multiple tries, with incremental back-off, from the background, to use as much external service shortening as possible.<\/p>\n\n\n\n
\nThe backup shortener service<\/strong> would use the cached data to serve all the external services shortened urls, for the accessibility purpose.<\/p>\n\n\n\nResults caching and the external service “bad” days<\/h2>\n\n\n\n
\nThat’s because the second one has permanent storage feature while the first one “forgets” everything after server restart.
\nThis switch helps us to keep external service away from overloads produced by us in such cases.<\/p>\n\n\n\n
\nBut overall, the backup service was only used in less than 1% cases.<\/p>\n\n\n\nBackup shortener service architecture<\/h2>\n\n\n\n
\nWe used a simple hashing algorithm (see below).
\nOutput is given in JSON format including status code, message, if needed, and a bunch of other stuff which includes the shortened url itself.<\/p>\n\n\n\nShortener client<\/h2>\n\n\n\n
\nThe minor balancing was done automatically from the code using something like this:<\/p>\n\n\n\n services.each_with_index do |service, i| next_service = services[i+1] begin return try_get_service_short_url(long_url, service, options) rescue ShortenerOutOfTriesError => e # out of tries can be logged in a file here end end def try_get_service_short_url(long_url, service, options) options = {:incremental_backoff => false, :retry_count => 3, :timeout => config.timeout}.merge(options) timeout = options[:timeout].to_i retry_count = options[:retry_count] incremental_backoff = options[:incremental_backoff] tries = 0 errors = [] begin return get_service_short_url(long_url, service, timeout) rescue ShortenerValidationError => e # give long url back if validation failed raise ShortenerValidationError, e.message if e.instance_of?(ShortenerValidationError) rescue Exception => e # log raise ShortenerOutOfTriesError.new(e.message, errors) if e.instance_of?(ShortenerHourlyLimitError) raise ShortenerOutOfTriesError.new(\"Retries count exceeded (#{retry_count})\", errors) if (tries += 1) > retry_count sleep(2**tries) if tries > 1 && incremental_backoff retry end end <\/pre>\n\n\n\nHashing algorithm<\/h2>\n\n\n\n
url36 = Digest::MD5.hexdigest(long_url+\"in salt we trust\").slice(0..6) <\/pre>\n\n\n\n
\nWhy just 7, you’ll wonder? It’s simple: 7 chars of Base62 is enough for 6 chars of Base36. 36^7 > 62^6.<\/p>\n\n\n\n base62 = ['0'..'9','A'..'Z','a'..'z'].map{|a| a.to_a}.flatten base36 = {};['0'..'9','a'..'z'].map{|range| range.to_a}.flatten.each_with_index{|char, position| base36[char] = position} url10 = 0; url62 = \"\" # convert to base10 url36.reverse.chars.to_a.each_with_index { |c,i| url10 += base36[c] * (36 ** i)} # convert to base62 6.times{|i| url62 << base62[url10 % 62]; url10 = url10 \/ 62} <\/pre>\n\n\n\n
\nBut what if a produced result is already used by previous conversion? You’ll have to repeat the process.<\/p>\n\n\n\n
\nThe problem will occur when most of Base62 variants will be used, so you’ll have to be well prepared to deal with eternal conversion cycle :), but as it’s a backup service and is used only for less than 1% cases – it will take a while.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\n
\nAt first, we have also implemented some monitoring to see how often would the balancing take place, checking few cases a week if it works.
\nBut finally, it all got stable and we never look into the solution again.<\/p>\n\n\n\n