willow-camp

End To End Tests with Minitest and Rails

Cassia Scheffer

— 3 min read

willow.camp has a small system test suite that runs on CI. It used to use the out-of-the-box settings that come with Rails: Capybara, Selenium, etc.

I noticed a flaky test on CI, which launched a journey to replace Selenium with Cuprite.

Fixing the Flaky Test

I worked through the flaky test with Claude Code, and as Claude was running the tests, I noticed this pop-up from Chrome.

A data breach on a site or app exposed your password…

I don’t use Chrome very often, so this jumped out at me. Of course, I use the oh-so-secret password “password” in my test suite, and Chrome flagged it as unsafe.

Disabling password leak detection fixed this for me.

browser_options: { "disable-features": "PasswordLeakDetection" }

But I still got timeouts on CI that looked like the tests were running before the browser was ready.

Waiting for CI is a bit painful. You can see my hilariously bad Git history with Claude’s gleeful overconfidence:

* 0c514d1 - nosandbox (15 minutes ago) <Cassia Scheffer>
...
* 57ac55b - bump timeout to 30 sec for ci (21 minutes ago)
* 9d6e86e - rewirite cuprite/capybara configs (26 minutes ago)
* 39f3238 - fix: simplify Cuprite configuration following best practices (10 hours ago)
* 68bfd53 - fix: simplify Cuprite configuration and fix CI timeout issues (24 hours ago)
* cebfae7 - fix: resolve Cuprite websocket timeout issues in CI (24 hours ago)
* dc288b7 - fix: make Cuprite configuration more robust for CI environments (24 hours ago)
...
* e5bb4df - fix: increase Cuprite timeout and add CI-specific settings for GitHub Acti>
...
* cb36eaa - feat: migrate system tests from Selenium to Cuprite (35 hours ago)
* 39f0ff1 - disable selenium chrome password detection (35 hours ago)

That Looks Bad, Cassia. Why Would You Let Claude Do That!?

Because Claude is faster at making mistakes than I am, and I am tired of waiting for CI. Making mistakes quickly means speedier feedback. So I sent Claude off on a fool’s errand to help me learn from the mistakes I would otherwise have made myself.

In the meantime, I read the Evil Martians article about better system tests.

Using Cuprite in Rails with Minitest

The Evil Martians article on system tests uses RSpec, but willow.camp uses Minitest. Here’s what I had to change to make it work with Minitest.

1. PrecompileAssets becomes a module and gets called when setup runs.

# Precompile assets before running tests to avoid timeouts.
# Do not precompile if webpack-dev-server is running (NOTE: MUST be launched with RAILS_ENV=test)

module PrecompileAssets
  def self.setup
    # Check if we're running system tests by looking at the test files being loaded
    running_system_tests = caller.any? { |line| line.include?("test/system") } ||
      ARGV.any? { |arg| arg.include?("test/system") } ||
      ENV["RAILS_TEST_TYPE"] == "system"

    unless running_system_tests
      puts "\n🚀️️  No system test selected. Skip assets compilation.\n"
      return
    end

    puts "\n🐢  Precompiling assets.\n"
    original_stdout = $stdout.clone

    start = Time.current
    begin
      $stdout.reopen(File.new(File::Constants::NULL, "w"))

      require "rake"
      Rails.application.load_tasks
      Rake::Task["assets:precompile"].invoke
    ensure
      $stdout.reopen(original_stdout)
      puts "Finished in #{(Time.current - start).round(2)} seconds"
    end
  end
end

# Run the setup when this file is loaded
PrecompileAssets.setup if defined?(Rails)

2. BetterRailsSystemTests Becomes a Moduel

module BetterRailsSystemTests
  # Make failure screenshots compatible with multi-session setup.
  def take_screenshot
    return super unless Capybara.last_used_session

    Capybara.using_session(Capybara.last_used_session) { super }
  end
end

3. Modules are included in ApplicationSystemTestCase

require "test_helper"
require "capybara/cuprite"
require_relative "system/system_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  include BetterRailsSystemTests
  include CupriteHelpers

  driven_by Capybara.javascript_driver

  def setup
    super
    # Use JS driver always

    # Store original host for cleanup
    @original_host = Rails.application.default_url_options[:host]
    # Make urls in mailers contain the correct server host.
    # This is required for testing links in emails (e.g., via capybara-email).
    Rails.application.default_url_options[:host] = Capybara.server_host
  end

  def teardown
    # Restore original host
    Rails.application.default_url_options[:host] = @original_host
    super
  end
end

Flakiness Still Included on CI

I still see a bit of system test flakiness on CI, but far less than before. I am running on the default GitHub Actions runner, which is pretty small. So there’s a good chance that the flakiness is due to low resources on the machine. So far, tests have consistently passed when I run these on my M1 MacBook Air.

I’ll dig into these flaky tests another day! But this looks like an improvement overall to me!

Read more