Lesson 7
Testing Rails Code

comments

In this episode, we are going to direct our attention towards finding and preventing bugs through testing in many of its various flavors. While Ruby and Rails lend themselves pretty nicely to writing short, concise code that is reasonably easy to follow with a little experience, sadly, it is still possible to create bugs. The trick is to recognize that this going to happen and to formulate a strategy to minimize the impact of bugs as soon as possible. The saying “an ounce of prevention is worth a pound of cure” definitely applies to finding and fixing bugs.

Like the earlier episodes in this first series of Learning Rails podcasts, we’ll stay at a mostly conceptual level to survey the testing philosophies and technical options at your command when developing a Ruby on Rails application.

Why Write Tests?

Writing code with Ruby is both productive, and for many developers in our community, fun. Ruby’s concise, expressive syntax makes it easy to code complex concepts with relatively few lines of code. The powerful collection of add-on libraries, including Ruby on Rails, makes it even easier to develop rich applications quickly by leveraging third party code.

At some point, though, things can and will go wrong. Assumptions about how those third party libraries work break down. Interactions between modules don’t always go as planned. Perhaps some update to the overall system breaks logic that you designed many months earlier and that you forgot about. And every now and then, we all simply make mistakes in our logic, use some tricky bit of code that doesn’t behave as we expect, or forget to consider some situations.

It has been demonstrated through a variety of studies that the cost to fix bugs rises quickly the further you go in time from the point the bug was introduced. Common sense alone reminds us that if you spot a problem right when you are creating a piece of code, you often have all of the information still at the forefront of your mind to address it. Most of us aren’t so lucky a few days, weeks or months later, when you have to hunt for a problem and recall what you were thinking. It gets even harder when you are tracking a problem down in someone else’s code.

Even if you aren’t tracking down a bug, the same problems arise when it is time to rewrite our code to introduce new functionality. All of those assumptions and interactions that you were juggling in your mind back when while you were in the “flow” of development the first time may not be there this time. A seemingly simple change might break a forgotten dependency and without an early warning system in place, it might be much later before it is discovered (at a potentially higher expense to you or your users).

The good news is that there lots of tools and practices that help you find and avoid bugs. In most open-source projects, including Rails, the community expects all contributors to provide broad and deep testing automation that demonstrates that the code functions as advertised. So code libraries that you use typically will already include extensive tests.

Best of all, bringing these practices to your own code is relatively easy, and something you can do incrementally or all at once, scaled to your needs and resources. Testing is development methodology agnostic, you will get many of the benefits we describe regardless of the approach you take. The Ruby on Rails community, however, seems to mostly lean towards Agile development influenced techniques, and we’ll call out a few of these during this podcast.

Testing Types

There are many types of tests, and as luck would have it, the naming conventions used in the testing world, and those used in Ruby on Rails use similar terms for slightly different things. We’ll try to clarify Rails use of terminology here.

The Ruby language has a handy library called Test::Unit that has been adopted by Rails as the primary means for testing code. Test::Unit is installed for you when you initially set up your Ruby environment, so you don’t need to configure additional software if you use the basic test strategies in a standard Rails application.

Ruby on Rails, like almost all open-source community projects, comes with a rich set of tests that are built in. Rails is opinionated software, and the core Rails team believes you should test your code too, so Rails makes it fairly easy to do so. When you originally create your project, Rails generates a number of utilities for you to build and run tests. You can take a peek inside of the automatically created test directory inside of your project, and you will see directories for unit tests, functional tests, integration tests, and something called fixtures (as well as one or more other supplemental files).

Unit tests in Rails are tests that exercise your models. Whenever you use a generator script that creates a model, you will also find a file is created under the test/unit directory that starts with your model name and ends with _test.rb. The purpose of a unit test for your model is to ascertain that it properly handles your data, that validations work the way you expect, and that any business logic that you might create inside of your model operates correctly. The initial unit test for your model simply subclasses the Test::Unit::TestCase class, but it is otherwise empty, with a simple sample function waiting for you to fill it in.

Functional tests live inside of the test/functional directory. These tests are designed to exercise your controllers. Like unit tests, when you generate controller code automatically, you will find new functional tests are created for you. Like unit tests, functional tests use a subclass of Test::Unit::TestCase. Unlike unit tests, however, functional tests are frequently populated automatically with some starter tests, especially if you use the Rails 2 scaffold generator to create RESTful controllers. These pre-created tests exercise all of the RESTful actions within your controller at a very basic level.

Integration tests are not automatically created for you, unlike unit and functional tests. If you wish to write an integration test, standard convention is to put them into a test/integration directory. Whereas unit tests exercise models and functional tests focus on single actions within a specific controller, integration tests are meant to test complete workflows, that likely step through multiple actions across one or more controllers.

Other types of tests exist too. One you may encounter, or wish to create some day, is an acceptance test. With a typical web application, an acceptance test usually means a test that exercises the program through a web browser as if a user were operating the application by clicking on buttons and filling in fields. The default tool set installed with Ruby and Rails doesn’t directly support this kind of testing, but there are a number of add-on tools, gems, and plugins that help you create this level of testing.

Peaking Inside Test::Unit and Friends

Let’s focus for now on Test::Unit and using it for unit and functional tests. As mentioned earlier, Test::Unit is a Ruby library that provides the infrastructure to construct tests for our code. You’ll use the class Test::Unit::TestCase as the basis for your own test classes. TestCase based classes use a naming convention to identify actual test code. Any method that starts with the name “test_” is considered to be a test method.

Test methods should be written so they focus on one specific test case. While you can put anything you want into a test method, it is just Ruby code after all, good testing practice is to write small, focused tests. You should name your tests something that is self describing too, so it is easy for anyone to understand the intent of a test. So, for instance, if I was unit testing a podcast model, and wanted to check if a validation that ascertains the podcast file exists is working, I might name my test method “test_check_podcast_filename_exists”. These names can get verbose, but it helps down the road when looking at failing tests to quickly know what was failing.

Test::Unit::TestCase is pretty smart about how it runs tests. Since you want to exercise code in an environment that is as absent of outside influences as possible, each test runs in a pristine environment, with new sample data each time. But what does that mean?

As it turns out, the test environment is clever about using dedicated resources that aren’t influenced by your development. Before a test is run, the test class runs a setup method that creates a clean set of data for you. The test can optionally run a teardown method that cleans up necessary resources after each test. Behind the scenes, the testing framework interacts with a special copy of your database when needed to make sure it is clean before each test method is run. We’ll look more at this a little later.

Inside of your test method, you use a series of methods called “assertions” that check the validity of various conditions. An assertion’s results are recorded for your review at the end of a test run, so you can see how many succeeded or failed. Basic assertions are things that test a boolean condition. Samples include assert_nil which checks whether an associated parameter is nil. Another is assert_equal, which checks whether two values are the same. There are many assert methods at your disposal, both in the Test::Unit library as well as add-ons that are Rails specific.

Using asserts is very simple. As an example, suppose you have a test that searches for a non-existent podcast episode; you’d want to assert that the result was nil. The test case code would read like:

def test_podcast_does_not_exist
  podcast = Podcast.find_by_title('Intro')
  assert_nil podcast
end

Another example checking to see if the test database contains three podcasts loaded from fixtures might read:

def test_three_podcasts_exist
  assert_equal, Podcast.find(:all), 3
end

You will also find that a default Rails application, as well as many add-on plugins and gems, provide testing helper methods to make writing tests easier. These helpers are not the same thing as view helpers introduced earlier in this podcast series. Testing helpers are simple methods that you can use to set up a specific condition. For instance, if you use Rick Olsen’s RESTful_authentication plugin, it supplies testing helpers that let you simulate logging in as a particular user…this is useful when testing controllers to see if an action should be available to a user who hasn’t logged in, for instance.

With all of these capabilities at your finger tips, it is easy to create very powerful tests. It is also possible to quickly write complicated tests. There are proponents for a multitude of testing styles, but usually the choice comes down to whether you write simple tests with one or only a few assertions in them, or write many related assertions within the body of a single test. Ultimately you need to decide what is right for you. In my experience, simpler tests are often more clear in their intent and force you to focus on specific test cases.

Another reason to stick to simple tests is to avoid one behavior quirk of Test::Unit. Namely, if something goes wrong in a test case method, the test case stops there. If you have lots of assertions and logic in a test method, you might find that the first failure masks future issues (until you fix the error), and that it takes multiple iterations of fixing and running the test to get through it. I like to minimize this masking issue so I know what is broken as soon as possible.

The Testing Environment: Settings, Databases, Fixtures, and Mocks

Let’s look at the surrounding testing ecosystem and what it takes to set up and support the tests that you write. Rails was designed from its earliest incarnations with testing in mind. As we mentioned earlier, testing is a core practice of the Rails community, and with little work on your part, you can get a lot of value out of your own tests.

Rails sets up a dedicated environment for testing. The idea here is that you will want to run your tests without disrupting production data and without the influence of your development environment either. You can tweak Rails core settings that will be used only for testing in the environments/tests.rb file. The default settings for your test environment maximize the number diagnostic log messages and ensure that features that may be resource expensive (such as sending email) or that would mask behaviors (such as caching) are turned off. Usually you don’t have to change anything initially in the standard test environment, but as you add functionality to your app (typically in the form of plugins), you might set up special settings there. For example, if your application authorizes credit card transactions through a web service gateway, you would probably want to set up test authorization credentials in the test environment file that would override the normal production settings.

An entire test database is at your disposal too. You will find default settings in your config/database.yml file that can be adjusted to point to a test database. The test database is completely independent of your other environments, and it is automatically wiped out and reloaded with test data before every test. Setting up an initial test database is as easy as leveraging the default rake tasks that Rails sets up for you (simply issue a one time rake db:test:prepare command and you are usually up and running).

But where does test data come from you might ask? The flip response might be “from where-ever you want it.” Usually, though, test data is generated either by extracting copies of production or development data from your databases, or by using data that you enter into fixtures.

The notion of fixtures comes from manufacturing where you have a tool or testbed that provides a baseline environment in which your product can be built and tested. In Ruby on Rails, fixtures are data files that capture objects in a known state, either with data that represents those objects in a good state or with bogus data that represents one of many failure states. Fixtures data can be static, specific values that you provide, or it can be dynamic, using embedded Ruby code to generate values as needed.

Fixture files are stored by default in your test/fixtures directory. When Rails generator scripts are used to create models, default fixture files are created for you. Fixture data can be stored in a variety of file formats, but they usually use either comma separated values (CSVs) or, more commonly, YAML, which is a very simple markup language for writing out the contents of an object.

Rails 2 introduces some nice shortcuts for creating and using fixture files. Some fields of your model are handled automatically. IDs, for instance, don’t need to be specified, they are instead derived from the name of the fixture entry. Certain date fields, like the automatic created_at and updated_at fields, are also populated for you. You can instead concentrate on providing your own important data.

Fixtures are also pretty intelligent about helping you test associations between models. We talked a bit about model associations such as has_many and belongs_to in episode 4. If you want your fixtures to help you set up data for such relationships, you can very easily specify them by name. For instance, if you have two models, one called Podcast and another called Listeners, and Podcast has_many Listeners, you can set this up in fixture files by listing out the listeners in the podcast fixture, and indicating the podcast that the listener belongs_to in the listener fixture.

An entry in the podcast.yml file might read:

episodeone:
  name: Introductory Podcast
  listeners: anne, bob, charlie

and entries in the listeners.yml file would read:

anne:
  name: Anne
  podcast: episodeone

bob:
  name: Bob 
  podcast: episodeone

charlie:
  name: Charlie
  podcast: episodeone

The Rails extensions to the fixture capabilities of the Test::Unit framework handle all of the hookups to the model for you, and give you an easy way to generate complicated test data.

Another testing tool concept you will eventually encounter is the idea of mock objects. Mocks, for short, are code modules that stand-in for expensive (in terms of resources), external, or state-changing components. You replace production logic with a mock so you can isolate the code under test. Traditional mock objects contain actual replacement logic for the item they are substituting for. A slightly different concept, stubbed code, can also be used to simply bypass real functionality in an object, often by just returning known values with no real logic behind it.

Rails provides a built-in mechanism for doing simple mocking, although its capabilities are often used for stubbing (see the show notes for links to essays on the differences). If you place code in the test/mocks directory, it can be used to override code elsewhere in your application. If you need more sophisticated capabilities, then you should turn to a dedicated mocking library such as Mocha or FlexMock.

Running Your Tests and Making Sense of the Results

You can write all of the tests in the world, but unless you run them regularly and maintain them, you fall prey to ignoring what Andy Hunt and Dave Thomas call “Broken Windows”—-small problems that can grow quickly into large ones.

The Test::Unit test cases that you write are simple Ruby programs. You can run them one-off at the command line, even one test method at a time. This is handy when first implementing testing code, but quickly becomes tedious. Fortunately, the default Rails application that you create comes with rake tasks that help you exercise all of your tests. Simply type “rake” at the command line, and by default, all your tests will be run. Almost all IDEs provide an equivalent menu command. You can also run only the tests that apply to a particular file.

Test::Unit will report back one of three kinds of responses. If your tests are successful, they’ll indicate that. In testing circles, we often call that “green”. If the tests fail, they will be reported as failing in one of two ways. An Error will occur if an exception or other logic problem occurred in your test case. An error will usually require that you check the programming of the test itself to make sure it is correct, then also check your code under test. On the other hand, you might get a Failure. A Failure is when one of your assertions is incorrect. Failures indicate, usually, that the code under test is faulty, and you’ve found a bug. Failures and Errors both are often also called “red”.

Hints as to what went wrong are captured in a couple of places for you. If you are running your tests from the command line, one off or with rake, you’ll see a summary display of results as well as some diagnostic information about the red tests. Rails also provides a special log file, called “test.log” in your logs directory, where the test run is also captured. This is similar to your development and production logs, and captures the runtime information of your application code while the tests are executing. IDEs typically capture all of this information and display it in a nice consolidated form.

The next step to keeping on top of tests is to get them to run automatically. If you don’t have to think about it, you can focus on just writing good tests, exercising new code, and quickly fixing things when alerted that a red condition exists. Several utilities exist that help in the automation side of things, including ZenSpider’s excellent “autotest,” which is part of the ZenTest gem. Autotest continuously monitors your code, and whenever you save a file, it checks for applicable tests and runs them automatically. Definitely a big boost to your quality safety net.

Beyond the Basics

As we discussed earlier, you should write and use tests no matter what development method or philosophy you follow. Whether you are a test-first/test driven adherent, or as I like to call it, a “test-after developer” (as in a TAD too late), writing testing code and applying it with automation tools gives you some assurance that your code is operating to a certain level of expectations, and is a great start to a larger quality plan.

The terms test first or test driven development (or any of the many other “fill-in-the-blank driven development terms” may be new to you if you are just learning about Agile programming practices. Test Driven Development (or TDD for short) is the practice of writing your testing code before you develop actual application logic. The suggested advantage is that by thinking through what your program does up front, and describing that functionality in the form of tests, you not only create an immediate automated test bed, but you also code the minimal effective set of features that are truly needed. You become a user of your own logic. TDD is sometimes also called Red-Green-Refactor programming, after the fact that running failing tests is called red (and they will fail, since there is no logic yet), implementing the API makes the tests go green, then you start the cycle again by changing your code in some fashion—often called refactoring.

Test::Unit based testing isn’t the only tool set available to you. Many developers are discovering that techniques such as Behavior Driven Development (or BDD), using tool libraries such as rSpec or test/spec, give them a higher level abstraction in which to define their code’s operating characteristics and to capture a more precise kind of documentation. Behavior Driven Development evolved out of test driven development. BDD simply provides a set of tools that implement a specialized, and concise, syntax that allows tests to be written that express both programming and business rules at the same time.

Acceptance testing, using tools such as Selenium or Watir, let you devise tests that exercise the actual display of your web application as the user would see it in the browser. This thorny area that includes GUI tests, is harder at times to automate, but just as valuable.

Using all of various techniques and tools mentioned so far is extremely helpful, but unless you have a very small code base, it is hard to tell if you covered all of the logic in your application. Code coverage tools, such as rcov, monitor test runs and analyze just how much of your program has been exercised. This can help you approach 100% test coverage, or alert you if you are decreasing in coverage if logic is added or changed without associated tests being amended.

Conclusion

No amount of automation and test tools replaces the need for you as a programmer to be alert to good development practices and pitfalls. It’s also important to have quality assurance testers (when available) and your users involved with the final assessment of your application. Hopefully, though, some of the tools we’ve discussed in this podcast will up-level your overall product quality and make your job easier, while enabling you to focus on developing cool new features, and not on finding and fixing bugs.

If you want to learn more about testing, and some of techniques and tools that we use at BuildingWebApps.com, visit our web site and read some of our articles and blog posts on the topic. Testing, like editors and IDEs, is a a topic that raises some passionate dialog in the various Ruby on Rails discussion group and forums, so you’ll always get an opinion from someone in the community.

This podcast concludes the first season of Learning Rails. We’re going to take a break for a few weeks, and then we’ll be back with a new season of multimedia podcasts, in which we’ll show you step-by-step how to build a Ruby on Rails application.

Please keep your podcast subscription so you’ll be notified as soon as the new series begins. If you also join our email list (you’ll find a place to enter your email address at the right side of most pages on the site), you’ll be sure to be notified as we announce new screencasts, articles, and more.

Thanks for listening.