![\"queuing_time_average_and_max\"](\"\/\/railsware.com\/blog\/wp-content\/uploads\/2012\/12\/queuing_time_average_and_max-1024x264.png\")
<\/figure>\n\n\n\nTo get the maximums you have to create custom dashboard and add chart for Why is Queuing Time so huge? Maybe we need more dynos? But adding more dynos doesn’t help.<\/p>\n\n\n\n So, let\u2019s see what is Queuing Time on Heroku, how it works and what we can do.<\/p>\n\n\n\n Let’s take a look at the Heroku docs<\/a> to understand queuing time better. When a request reaches Heroku, it’s passed from a load balancer to the “routing mesh”. The routing mesh is a custom Erlang solution based on MochiWeb that routes requests to a specific dyno. Each dyno has its own request queue, and the routing mesh pushes requests to a dyno queue. When dyno is available, it picks up a request from its queue and your application processes it. The time between when routing mesh receives the request and when the dyno picks it up is the queuing time.<\/p>\n\n\n\n Each request has an The routing mesh uses a random selection algorithm for HTTP request load balancing across web processes.<\/p><\/blockquote>\n\n\n\n That means that even if a dyno is stuck processing a long-running request, it will still get more requests in its queue. If a dyno serves a 15-second request and is routed another request that ends up taking 100ms, that second request will take 15.1 seconds to complete. That’s why Heroku recommends putting all long-running actions in background jobs.<\/p>\n\n\n\n These queuing time peaks were a major pain in the neck for us. Under high load (~10-15k requests per minute), a stuck dyno could result in a hundreds of timeouts. Here are some ways to minimize request queuing:<\/p>\n\n\n\n <\/p>\n With this solution we got same results like with \u201cSplitting into multiple applications\u201c but without additional pain. We setup Haproxy balancer which have two queues, one for “slow” and another for “fast” requests. By limiting parallel processing in “slow” queue we always have free workers for “fast” queue.<\/p>\n\n\n\n If you’re running an application under high load on Heroku, you’re almost certainly facing this problem and don’t know about it. New Relic showed that our application performance was very fast (< 100ms) and average queuing time very low even when these problems were at their worst. But check your maximums for Queuing Time with custom dashboard graph!<\/p>\n\n\n\n For most applications, it probably doesn’t matter if a fraction of a percent of your requests take a long time. But sometimes we can’t afford to have any requests that take more than a couple seconds, even if our average response time is well under 100ms.<\/p>\n\n\n\n Each of possible solutions resulted in fewer timeouts. But the best solution which helps us get fast results is solution with Haproxy Balancer. Next we will describe how we implemented it for Heroku application.<\/p>\n\n\n\n Previously we discussed application with slow resources<\/a> and heroku queuing time<\/a> problems. Today we find out how to setup HAProxy as “slow-fast” request balancer for heroku application using Amazon Elastic Compute Cloud.<\/p>\n\n\n\n You will need:<\/p>\n\n\n\n Although you can manage AWS via WEB interface we prefer command line tools.<\/p>\n\n\n\n Now switch to console and setup your credentials for tools.<\/p>\n\n\n\n Check access by invoking (for example):<\/p>\n\n\n\n Tools are ready.<\/p>\n\n\n\n In order to access balancer we need ssh private key.<\/p>\n\n\n\n Choose name for your key (e.g. haproxy-balancer) and create it by:<\/p>\n\n\n\n Save output to some file e.g haproxy-balancer.pem<\/em>. Don’t forget to set right permissions:<\/p>\n\n\n\n Open to ports in order to access our instances via SSH and HTTP:<\/p>\n\n\n\n Note. From security point of view we recommend to create special access group for balancer instead of using default group.<\/p>\n\n\n\n We are going to use standard latest Ubuntu AMI. On alestic<\/a> page choose desired ubuntu version.<\/p>\n\n\n\n Be sure to choose AMI from us-east-1<\/strong> region because heroku instances are located exactly in this region, On moment of writing this article latest AMI is:<\/p>\n\n\n\n Run instance using haproxy-balancer<\/em> credentials into us-east-1<\/em> region on t1.micro<\/em> instance:<\/p>\n\n\n\n In order to find out status and IP of instance wait a while and run:<\/p>\n\n\n\n Connect to balancer instance:<\/p>\n\n\n\n Install HAProxy package:<\/p>\n\n\n\n (Optionally) Install git package:<\/p>\n\n\n\n Clone our HAProxy request balancer<\/a> configuration. For simplity we clone sample configuration and change it on the server:<\/p>\n\n\n\n Tune slow urls in:<\/p>\n\n\n\n Enable HAProxy by setting ENABLED=1<\/em> into \/etc\/default\/haproxy<\/em>:<\/p>\n\n\n\n Start HAProxy:<\/p>\n\n\n\n Check HAProxy:<\/p>\n\n\n\n Now we can try visit slow\/fast urls of our application using balancer IP address or perform some load\/stress testing. We recommend to allocate and assign elastic IP<\/a> for this purporse. It allows you easy switch to another balancer if something goes wrong.<\/p>\n\n\n\n Allocate new IP addresss:<\/p>\n\n\n\n Assign address to your Balancer IP address:<\/p>\n\n\n\n Check again by visiting your application via assigned elastic IP.<\/p>\n\n\n\n Finally change your DNS “A” record of to balancer IP according your DNS provider manual.<\/p>\n\n\n\n Also Heroku Custom Domains<\/a> article can be useful.<\/p>\n\n\n\n If something went wrong switch back DNS settings to use heroku application directly.<\/p>\n\n\n\n As we already mentioned this solution is “temporary”. It gives your application opportunity to survive under high traffic. We are hope you find this article helpful.<\/p>\n\n\n\n One of the most exciting things about Heroku is that you can scale in a single click – just add more dynos, and you can immediately handle higher load. This post is about our experience of hosting high load application on Heroku and situation when adding more dynos does not help. While throughput of our…<\/p>\n","protected":false},"author":38,"featured_media":9456,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["Maxim Bondaruk"],"class_list":["post-3887","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"categories_data":[{"name":"Engineering","link":"https:\/\/railsware.com\/blog?category=development"}],"post_thumbnails":"\/\/railsware.com\/blog\/wp-content\/uploads\/2012\/12\/queuing_time_average_and_max-1024x264.png","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/users\/38"}],"replies":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/comments?post=3887"}],"version-history":[{"count":65,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3887\/revisions"}],"predecessor-version":[{"id":13975,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3887\/revisions\/13975"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media\/9456"}],"wp:attachment":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media?parent=3887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/categories?post=3887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/tags?post=3887"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/coauthors?post=3887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}WebFrontend\/QueueTime<\/code> metric.<\/p>\n\n\n\n![\"custom_dashboard\"](\"https:\/\/railsware.com\/blog\/wp-content\/uploads\/2012\/12\/custom_dashboard-e1356443403835.png\")
<\/a><\/figure>\n\n\n\nThe cause in Heroku Routing Mesh<\/h2>\n\n\n\n
<\/a><\/figure>\n\n\n\nX-Heroku-queue-Wait-Time<\/code> HTTP header with the queuing time value for this particular request. That’s how New Relic knows what to show on its charts. The key point for us in the routing schema is the request distribution. The Heroku docs say:<\/p>\n\n\n\nPossible solutions<\/h2>\n\n\n\n
Heroku recommended one final workaround: drop all long-running requests within our application using Rack::Timeout<\/a> or, when we switched from Thin to Unicorn, by setting a timeout in Unicorn.<\/li>
Long-running requests, like uploading CSVs of contacts or downloading CSVs of contacts can be moved to background jobs<\/li>
Each application was an identical Heroku application that served different types of requests using a unique subdomain. So, for example, you can move database-heavy reports to second app.Problems:\n\n
Our solution: Slow-Fast Request Balancer<\/h2>\n\n\n\n
Do you have this problem?<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Preface<\/h2>\n\n\n\n
Prepare AWS tools<\/h2>\n\n\n\n
$ export AWS_ACCESS_KEY=your_AWS_ACCESS_KEY_ID\n$ export AWS_SECRET_KEY=your_AWS_SECRET_KEY\n<\/pre>\n\n\n\n
$ ec2-describe-instances\n<\/pre>\n\n\n\n
Create balancer ssh access key<\/h2>\n\n\n\n
$ ec2-add-keypair haproxy-balancer\nKEYPAIR haproxy-balancer 0a:ea:f9:f6:4c:29:80:33:0c:2e:af:7b:8c:dc:5c:5b:24:65:53:6f\n-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n<\/pre>\n\n\n\n
$ chmod 600 haproxy-balancer.pem\n<\/pre>\n\n\n\n
Configure Firewall<\/h2>\n\n\n\n
$ ec2-authorize default -P tcp -p 22\n$ ec2-authorize default -P tcp -p 80\n<\/pre>\n\n\n\n
Setup balancer on AWS<\/h2>\n\n\n\n
AMI<\/h3>\n\n\n\n
otherwise you will have network latency.<\/p>\n\n\n\nUbuntu 12.10 Quantal EBS boot ami-7539b41c\n<\/pre>\n\n\n\n
$ ec2-run-instances ami-7539b41c -k haproxy-balancer -t t1.micro --region us-east-1\n<\/pre>\n\n\n\n
$ ec2-describe-instances i-99de1ce8\n<\/pre>\n\n\n\n
HAProxy<\/h3>\n\n\n\n
$ ssh -i haproxy-balancer.pem ubuntu@ec2-174-129-156-71.compute-1.amazonaws.com\n<\/pre>\n\n\n\n
ubuntu@ip-10-112-71-4:~$ sudo apt-get install haproxy\n<\/pre>\n\n\n\n
ubuntu@ip-10-112-71-4:~$ sudo apt-get install git\n<\/pre>\n\n\n\n
You are free to use own HAProxy configuration just follow ideas in our configuration.
Anyway we strongly recommend you to keep your configuration under control version system (git. subversion, etc).
And then deliver changes to server from SCM repository instead of editing it directly on the server.<\/p>\n\n\n\nubuntu@ip-10-112-71-4:~$ sudo mv \/etc\/haproxy\/ haproxy.origin\nubuntu@ip-10-112-71-4:~$ sudo git clone git:\/\/github.com\/railsware\/haproxy-slow-fast-request-balancer.git \/etc\/haproxy\n<\/pre>\n\n\n\n
\/etc\/haproxy\/patterns\/*\n<\/pre>\n\n\n\n
ubuntu@ip-10-112-71-4:~$ sudo vim \/etc\/default\/haproxy\n<\/pre>\n\n\n\n
ubuntu@ip-10-112-71-4:~$ sudo vim \/etc\/init.d\/haproxy start\n<\/pre>\n\n\n\n
http:\/\/stats:qwerty@ec2-174-129-156-71.compute-1.amazonaws.com\/haproxystats\n<\/pre>\n\n\n\n
Switch to HAProxy balancer<\/h3>\n\n\n\n
When everything is ok we are ready to change application DNS host settings to balancer IP.<\/p>\n\n\n\n$ ec2-allocate-address\n$ ADDRESS 54.235.201.228 standard\n<\/pre>\n\n\n\n
$ ec2-associate-address 54.235.201.228 -i i-99de1ce8\nADDRESS 54.235.201.228 i-99de1ce8\n<\/pre>\n\n\n\n
Rollback<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
You may use “slow-fast” balancer until you fix your slow resources by moving them to background or changing your architecture.
Then you may switch back to direct application usage.<\/p>\n\n\n\nReferences<\/h2>\n\n\n\n