iOS acceptance testing with Calabash and Rspec

While investigating existing automated tools for mobile TDD, we have run into a well-documented and supported library by Xamarin. It’s called calabash, comes with iOS and Android support and has a wide scope of supported user actions, gestures and expectations. The only drawback we found significant is lack of native Rspec support (cucumber only for now). This post provides a simple tutorial for using calabash with Rspec to test your iOS applications.

Overview

Calabash library including сucumber step definitions is aggregated within single gem calabash-cucumber. Basic tutorial for integrating your project with Calabash testing tool is well described here. We use mixed approach for initial setup by adding Calabash pod and manually copying main project target in Xcode. As cucumber is a primary framework coming out of the box, some additional steps are required to use it with Rspec.

Setting up Rspec environment

First of all install rspec gem and initialize your project:

gem install rspec //or use Gemfile
rspec --init

This will generate the following file structure:

./spec folder for all your test examples.
./spec/spec_helper.rb file for all configuration and initial setup stuff.

Using predefined cucumber setup and a bit of source investigation, we came up with the following definitions within spec_helper.rb:

require 'sim_launcher'
require 'calabash-cucumber/operations'

include Calabash::Cucumber::Core
include Calabash::Cucumber::TestsHelpers
include Calabash::Cucumber::WaitHelpers
include Calabash::Cucumber::KeyboardHelpers

RSpec.configure do |config|
  config.treat_symbols_as_metadata_keys_with_true_values = true
  config.run_all_when_everything_filtered = true
  config.filter_run :focus
  config.order = 'random'

  config.before do
    @calabash_launcher = Calabash::Cucumber::Launcher.new
    unless @calabash_launcher.calabash_no_launch?
      @calabash_launcher.relaunch
      @calabash_launcher.calabash_notify(self)
    end
  end

  config.before(:each) do
    element_exists('view')
  end

  config.after do
    unless @calabash_launcher.calabash_no_stop?
      calabash_exit
      @calabash_launcher.stop if @calabash_launcher.active?
    end
  end
end

A before hook was set up to wait for any view to appear – such trick is required to avoid stupid failures because of initial simulator startup delay. There are a few additional method definitions required to make this setup work:

def escape_quotes(str)
  str.gsub("'", "\\\\'")
end
  
#### emulate cucumber reports
def embed(path, type, label)
  p "storing report: #{path} (type: #{type}, label:#{label})"
end

You can add them to your spec_helper.rb.

Sugarize it

Basic ruby modules included in calabash-cucumber gem have predefined steps for

querying elements
performing actions/gestures
matching expectations

We have added simple wrapper methods to make test examples readable and DRY. Here are some examples that are used in our test scenarios:Step definitions

module StepHelper
  WAIT_TIMEOUT = (ENV['WAIT_TIMEOUT'] || 30).to_f
  STEP_PAUSE = (ENV['STEP_PAUSE'] || 0.5).to_f

  def wait_to_see_view(expected_mark)
    wait_for_elements_exist([marked_view_query_string(expected_mark)], :timeout => WAIT_TIMEOUT)
  end

  #Touches
  def touch_the_button(name)
    touch("button marked:'#{name}'")
    sleep(STEP_PAUSE)
  end

  def touch_view(expected_mark)
    touch(marked_view_query_string(expected_mark))
    sleep(STEP_PAUSE)
  end

  def go_back
    touch("navigationItemButtonView first")
    sleep(STEP_PAUSE)
  end

  #Waiting
  def wait_for_seconds(seconds)
    sleep seconds.to_f
  end

  #Inputs
  def enter_text_into_textfield_by_placeholder(text_to_type, placeholder)
    touch("textField placeholder:'#{placeholder}'")
    keyboard_enter_text(text_to_type)
    sleep(STEP_PAUSE)
  end

  def enter_text_into_textfield_by_label(text_to_type, label)
    touch("textField marked:'#{label}'")
    keyboard_enter_text(text_to_type)
    sleep(STEP_PAUSE)
  end

  def marked_view_query_string(expected_mark)
    "view marked:'#{expected_mark}'"
  end

  ...

end

Custom matchers

RSpec::Matchers.define :show_view do |expected_mark|
  match do
    (element_exists( "view marked:'#{expected_mark}'" ) or
     element_exists( "view text:'#{expected_mark}'"))
  end

  failure_message_for_should do
    "No visible view marked '#{expected_mark}' found"
  end

  failure_message_for_should_not do
    "Found visible view marked '#{expected_mark}'"
  end
end

RSpec::Matchers.define :show_error_alert do |expected_error_message|
  match do
    error_message = query("view className:'UIAlertView' label className:'UILabel'", :text).first
    error_message == expected_error_message
  end

  failure_message_for_should do
    "No alert with text '#{expected_error_message}' found"
  end

  failure_message_for_should_not do
    "Found visible alert with text '#{expected_error_message}'"
  end
end

...

Conclusions

Adding Rspec support for your acceptance testing using calabash library is really simple. Approach described above helped us to build and support application test coverage for wide variety of flows and interactions. Here is a short example on login flow test scenario:

...

shared_examples_for "login error" do
  it do
    should show_error_alert(alert_error_message)
    should_not show_view('Asset list zero case')
  end
end

...

describe "login error" do
  before do
    navigate_to_login_screen
    stub_api_request_with_error("POST", "/users/sign_in", error_message);
    submit_login_form
  end

  context "wrong credentials" do
    let(:error_message) {'login_failed'}
    let(:alert_error_message) {wrong_email_or_password_error_message}

    it_should_behave_like "login error"
  end

  context "server exception/unavailable" do
    let(:error_message) {'some_error_message'}
    let(:alert_error_message) {server_unavailable_error_message}

    it_should_behave_like "login error"
  end
end

...

Using Rspec instead of cucumber gives you a better opportunity to setup context of test examples. It’s also very important to use any CI server you prefer as it takes some time for full test coverage to run. Do not hesitate to add your own step definitions and matchers to keep your test examples readable, DRY and clean.