{"id":4222,"date":"2013-02-12T20:59:19","date_gmt":"2013-02-12T17:59:19","guid":{"rendered":"http:\/\/railsware.com\/blog\/?p=4222"},"modified":"2021-08-16T15:57:49","modified_gmt":"2021-08-16T12:57:49","slug":"chef-server-and-amazon-auto-scaling-groups","status":"publish","type":"post","link":"https:\/\/railsware.com\/blog\/chef-server-and-amazon-auto-scaling-groups\/","title":{"rendered":"Chef Server and Amazon Auto Scaling Groups"},"content":{"rendered":"\n<p>Amazon Auto Scaling Groups (ASG) present a new requirement for applications that will utilize this feature. When a new instance spins up, it has to know what to do.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Problem<\/h2>\n\n\n\n<p>Let\u2019s follow the steps leading to scaling up application. First, conditions set by one of your monitoring alarms should be met. Generally, this is the consumption of one of the core resources like CPU or system memory exceeding configured threshold. Then, scaling up policy takes part and launches a new instance using specified AMI. You have to craft that AMI in such a way that will bring new instance in a ready to serve condition.<\/p>\n\n\n\n<p>Presumably, you have nicely working preconfigured AMI, but you have to bring your application code to the up-to-date state. Otherwise&#8230; Well, there is no \u201cotherwise\u201d, things must be correct on a new instance, and you simply can\u2019t do this manually &#8211; this is ASG, and it should work automatically.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Chef Server to the rescue<\/h2>\n\n\n\n<p>Here comes Chef Server that will help you to deal with it. The solution described below will also work nicely for projects without ASG.<\/p>\n\n\n\n<p>When using Chef Server, you\u2019ll have chef-client agent on each instance. According to the documentation, you\u2019re presented with two options for running it: it could run either as a daemon or cron task&#8230; Nice! You\u2019re done. Your AMI is only required to have validation.pem key in order to register new instance on Server.<\/p>\n\n\n\n<p>Wait a second, does this mean that chef-client will perform useless runs most of the time? Yes, it is. Actually Chef architecture preserves you from useless work if the state of a node is up to date, but there\u2019s a Deploy resource which probably you\u2019ll be using actively. Provider for this resource <a href=\"https:\/\/railsware.com\/blog\/chef-dos-and-donts\/#chef-capistrano\" target=\"_blank\" rel=\"noreferrer noopener\">will run callbacks no matter what is the current state<\/a> of the deployed code on the node.<\/p>\n\n\n\n<p>Also keep in mind that you have to manage registering\/unregistering node on Server, otherwise you will end up with trash stored on a server.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Tweaking Chef-Client<\/h2>\n\n\n\n<p>So what would be the best to do? First, we decided to create our own init script for chef-client. Its responsibility is to register and converge node to the desired state during boot up and unregister node on Server during reboot\/shutdown. Second, we opted to run chef-client only when we need, so no daemons or cron tasks.<\/p>\n\n\n\n<p>If you\u2019ve installed Chef from package, before you proceed, you have to stop daemon and remove links from rc.d directories:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sudo \/etc\/init.d\/chef-client stop\nsudo mv \/etc\/init.d\/chef-client \/etc\/init.d\/chef-client.disabled # or remove it completely\nsudo update-rc.d chef-client remove\n<\/pre>\n\n\n\n<p>Here are the snippets from init script template used by our cookbook for its management:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">NODE_ENV=\"&lt;%= node.chef_environment %>\"\nINSTANCE_ID=`curl -s http:\/\/169.254.169.254\/latest\/meta-data\/instance-id`\nNODE_NAME=\"${NODE_ENV}___${INSTANCE_ID}\"\n\n...\nregister_node() {\n update_node_config\n\n # aggregate options etc\n options=\"-E $NODE_ENV -N $NODE_NAME -o $RUN_LIST -S $CHEF_SERVER_URL\"\n export SKIP_RESQUE_RESTART=1 # because we're at boot up stage\n\n # shoot!\n chef-client $options > $LOG_FILE\n...\n}\n\nunregister_node() {\n echo \"`date +[%FT%T%:z]` `knife node delete $NODE_NAME -y -u $CHEF_CLIENT -k $CLIENT_KEY -c $CLIENT_CFG`\" >> $LOG_FILE\n echo \"`date +[%FT%T%:z]` `knife client delete $NODE_NAME -y -u $CHEF_CLIENT -k $CLIENT_KEY -c $CLIENT_CFG`\" >> $LOG_FILE\n rm $CLIENT_KEY $CLIENT_CFG &amp;&amp; echo \"`date +[%FT%T%:z]` Removed client key at $CLIENT_KEY and config at $CLIENT_CFG\" >> $LOG_FILE\n}\n\nupdate_node_config() {\n > $CLIENT_CFG\n echo \"chef_server_url \\\"$CHEF_SERVER_URL\\\"\" >> $CLIENT_CFG\n echo \"node_name \\\"$NODE_NAME\\\"\" >> $CLIENT_CFG\n}\n\ncase \"$1\" in\n start)\n   log_daemon_msg \"Starting $DESC\"\n   errcode=0\n   register_node || errcode=$?\n   log_end_msg $errcode\n   ;;\n stop)\n   log_daemon_msg \"Stopping $DESC\"\n   errcode=0\n   unregister_node || errcode=$?\n   log_end_msg $errcode\n   ;;\n...\nesac\n<\/pre>\n\n\n\n<p>Overview of what it does:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>sets node name for instance (we have next naming convention <em>project-environment___instance_id<\/em>)<\/li><li>creates and configures client.rb used by chef-client (sets two vital attributes &#8211; node_name and chef_server_url)<\/li><li>performs initial run of chef-client during boot up<\/li><li>unregisters instance on Server during reboot\/shutdown<\/li><\/ul>\n\n\n\n<p>Then, this script must be placed to rc.d directories to be executed on different runlevels:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">chmod +x \/etc\/init.d\/chef-client\nupdate-rc.d chef-client defaults 80 20\n<\/pre>\n\n\n\n<p>After covering startup\/shutdown cases with this script, we can safely achieve the second goal &#8211; triggering chef-client via knife ssh command whenever we need.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusions<\/h2>\n\n\n\n<p>Summarizing above said, your AMI must have:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>validation.pem key in order to register an instance on Server<\/li><li>init script for chef-client<\/li><\/ul>\n\n\n\n<p>That\u2019s it! After completing these steps, you\u2019ll have a correctly working, predictable and easily maintainable ASG.<\/p>\n\n\n\n<p>For the general Chef tips please refer to <a href=\"https:\/\/railsware.com\/blog\/chef-dos-and-donts\/\" target=\"_blank\" rel=\"noreferrer noopener\">&#8220;Chef: DOs and DON\u2019Ts&#8221;<\/a> article.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Amazon Auto Scaling Groups (ASG) present a new requirement for applications that will utilize this feature. When a new instance spins up, it has to know what to do. The Problem Let\u2019s follow the steps leading to scaling up application. First, conditions set by one of your monitoring alarms should be met. Generally, this is&#8230;<\/p>\n","protected":false},"author":20,"featured_media":4232,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["Igor Antonyuk"],"class_list":["post-4222","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"categories_data":[{"name":"Engineering","link":"https:\/\/railsware.com\/blog?category=development"}],"post_thumbnails":"https:\/\/railsware.com\/blog\/wp-content\/themes\/railsware\/vendors\/images\/article-thumbnail-default.jpg","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/4222","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/comments?post=4222"}],"version-history":[{"count":40,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/4222\/revisions"}],"predecessor-version":[{"id":14161,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/4222\/revisions\/14161"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media\/4232"}],"wp:attachment":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media?parent=4222"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/categories?post=4222"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/tags?post=4222"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/coauthors?post=4222"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}