Add comment about possible warnings in Template::Base and log levels

[catagits/Catalyst-Manual.git] / lib / Catalyst / Manual / Tutorial / Testing.pod

=head1 NAME

Catalyst::Manual::Tutorial::Testing - Catalyst Tutorial - Chapter 8: Testing


=head1 OVERVIEW

This is B<Chapter 8 of 10> for the Catalyst tutorial.

L<Tutorial Overview|Catalyst::Manual::Tutorial>

=over 4

=item 1

L<Introduction|Catalyst::Manual::Tutorial::Intro>

=item 2

L<Catalyst Basics|Catalyst::Manual::Tutorial::CatalystBasics>

=item 3

L<More Catalyst Basics|Catalyst::Manual::Tutorial::MoreCatalystBasics>

=item 4

L<Basic CRUD|Catalyst::Manual::Tutorial::BasicCRUD>

=item 5

L<Authentication|Catalyst::Manual::Tutorial::Authentication>

=item 6

L<Authorization|Catalyst::Manual::Tutorial::Authorization>

=item 7

L<Debugging|Catalyst::Manual::Tutorial::Debugging>

=item 8

B<Testing>

=item 9

L<Advanced CRUD|Catalyst::Manual::Tutorial::AdvancedCRUD>

=item 10

L<Appendices|Catalyst::Manual::Tutorial::Appendices>

=back


=head1 DESCRIPTION

You may have noticed that the Catalyst Helper scripts automatically 
create basic C<.t> test scripts under the C<t> directory.  This 
chapter of the tutorial briefly looks at how these tests can be used 
to not only ensure that your application is working correctly at the 
present time, but also provide automated regression testing as you 
upgrade various pieces of your application over time.

You can check out the source code for this example from the Catalyst
Subversion repository as per the instructions in
L<Catalyst::Manual::Tutorial::Intro|Catalyst::Manual::Tutorial::Intro>.


=head1 RUNNING THE "CANNED" CATALYST TESTS

There are a variety of ways to run Catalyst and Perl tests (for example,
C<perl Makefile.PL> and C<make test>), but one of the easiest is with the
C<prove> command.  For example, to run all of the tests in the C<t>
directory, enter:

    $ prove --lib lib t

There will be a lot of output because we have the C<-Debug> flag 
enabled in C<lib/MyApp.pm> (see the C<CATALYST_DEBUG=0> tip below for 
a quick and easy way to reduce the clutter).  Look for lines like this 
for errors:

    #   Failed test 'Request should succeed'
    #   in t/controller_Books.t at line 8.
    # Looks like you failed 1 test of 3.

The redirection used by the Authentication plugins will cause several 
failures in the default tests.  You can fix this by making the following
changes:

1) Change the line in C<t/01app.t> that reads:

    ok( request('/')->is_success, 'Request should succeed' );

to:

    ok( request('/login')->is_success, 'Request should succeed' );

2) Change the "C<request('/logout')-E<gt>is_success>" to 
"C<request('/logout')-E<gt>is_redirect>" in C<t/controller_Logout.t>.

3) Change the "C<request('/books')-E<gt>is_success>" to 
"C<request('/books')-E<gt>is_redirect>" in C<t/controller_Books.t>.

4) Add "C<use MyApp;>" to the top of C<t/view_TT.t>.

As you can see in the C<prove> command line above, the C<--lib> option
is used to set the location of the Catalyst C<lib> directory.  With this
command, you will get all of the usual development server debug output,
something most people prefer to disable while running tests cases.
Although you can edit the C<lib/MyApp.pm> to comment out the C<-Debug>
plugin, it's generally easier to simply set the C<CATALYST_DEBUG=0>
environment variable.  For example:

    $ CATALYST_DEBUG=0 prove --lib lib t

B<Note:> Depending on the versions of various modules you have 
installed, you might get some C<used only once> warnings -- you can 
ignore these.  If you want to elliminate the warnings, you can 
edit C<Template::Base> to disable and then re-enable warnings
are the C</usr/lib/perl5/Template/Base.pm> line in C<sub new>.
You can locate where C<Template::Base> is located with the 
following command (it's probably in a place similar to
C</usr/lib/perl5/Template/Base.pm>):

    perldoc -l Template::Base

Edit the file and modify C<sub new> to match:

    ...
    {   no strict qw( refs );
        # Disable warnings
        no warnings;
        $argnames = \@{"$class\::BASEARGS"} || [ ];
        # Turn warnings back on
        use warnings;
    }
    ...

During the C<t/02pod> and C<t/03podcoverage> tests, you might notice the
C<all skipped: set TEST_POD to enable this test> warning message.  To
execute the Pod-related tests, add C<TEST_POD=1> to the C<prove>
command:

    $ CATALYST_DEBUG=0 TEST_POD=1 prove --lib lib t

If you omitted the Pod comments from any of the methods that were
inserted, you might have to go back and fix them to get these tests to
pass. :-)

Another useful option is the C<verbose> (C<-v>) option to C<prove>.  It
prints the name of each test case as it is being run:

    $ CATALYST_DEBUG=0 TEST_POD=1 prove --lib lib -v t


=head1 RUNNING A SINGLE TEST

You can also run a single script by appending its name to the C<prove>
command. For example:

    $ CATALYST_DEBUG=0 prove --lib lib t/01app.t

Also note that you can also run tests directly from Perl without C<prove>.
For example:

    $ CATALYST_DEBUG=0 perl -Ilib t/01app.t


=head1 ADDING YOUR OWN TEST SCRIPT

Although the Catalyst helper scripts provide a basic level of checks
"for free," testing can become significantly more helpful when you write
your own script to exercise the various parts of your application.  The
L<Test::WWW::Mechanize::Catalyst|Test::WWW::Mechanize::Catalyst> module 
is very popular for writing these sorts of test cases.  This module 
extends L<Test::WWW::Mechanize|Test::WWW::Mechanize> (and therefore 
L<WWW::Mechanize|WWW::Mechanize>) to allow you to automate the action of
a user "clicking around" inside your application.  It gives you all the
benefits of testing on a live system without the messiness of having to
use an actual web server, and a real person to do the clicking.

To create a sample test case, open the C<t/live_app01.t> file in your
editor and enter the following:

    #!/usr/bin/perl
    
    use strict;
    use warnings;
    
    # Load testing framework and use 'no_plan' to dynamically pick up
    # all tests. Better to replace "'no_plan'" with "tests => 30" so it
    # knows exactly how many tests need to be run (and will tell you if
    # not), but 'no_plan' is nice for quick & dirty tests
    
    use Test::More 'no_plan';
    
    # Need to specify the name of your app as arg on next line
    # Can also do:
    #   use Test::WWW::Mechanize::Catalyst "MyApp";
    
    use ok "Test::WWW::Mechanize::Catalyst" => "MyApp";
        
    # Create two 'user agents' to simulate two different users ('test01' & 'test02')
    my $ua1 = Test::WWW::Mechanize::Catalyst->new;
    my $ua2 = Test::WWW::Mechanize::Catalyst->new;
    
    # Use a simplified for loop to do tests that are common to both users
    # Use get_ok() to make sure we can hit the base URL
    # Second arg = optional description of test (will be displayed for failed tests)
    # Note that in test scripts you send everything to 'http://localhost'
    $_->get_ok("http://localhost/", "Check redirect of base URL") for $ua1, $ua2;
    # Use title_is() to check the contents of the <title>...</title> tags
    $_->title_is("Login", "Check for login title") for $ua1, $ua2;
    # Use content_contains() to match on text in the html body
    $_->content_contains("You need to log in to use this application",
        "Check we are NOT logged in") for $ua1, $ua2;
    
    # Log in as each user
    # Specify username and password on the URL
    $ua1->get_ok("http://localhost/login?username=test01&password=mypass", "Login 'test01'");
    # Use the form for user 'test02'; note there is no description here
    $ua2->submit_form(
        fields => {
            username => 'test02',
            password => 'mypass',
        });
    
    # Go back to the login page and it should show that we are already logged in
    $_->get_ok("http://localhost/login", "Return to '/login'") for $ua1, $ua2;
    $_->title_is("Login", "Check for login page") for $ua1, $ua2;
    $_->content_contains("Please Note: You are already logged in as ",
        "Check we ARE logged in" ) for $ua1, $ua2;
    
    # 'Click' the 'Logout' link (see also 'text_regex' and 'url_regex' options)
    $_->follow_link_ok({n => 4}, "Logout via first link on page") for $ua1, $ua2;
    $_->title_is("Login", "Check for login title") for $ua1, $ua2;
    $_->content_contains("You need to log in to use this application",
        "Check we are NOT logged in") for $ua1, $ua2;
    
    # Log back in
    $ua1->get_ok("http://localhost/login?username=test01&password=mypass", "Login 'test01'");
    $ua2->get_ok("http://localhost/login?username=test02&password=mypass", "Login 'test02'");
    # Should be at the Book List page... do some checks to confirm
    $_->title_is("Book List", "Check for book list title") for $ua1, $ua2;
    
    $ua1->get_ok("http://localhost/books/list", "'test01' book list");
    $ua1->get_ok("http://localhost/login", "Login Page");
    $ua1->get_ok("http://localhost/books/list", "'test01' book list");
    
    $_->content_contains("Book List", "Check for book list title") for $ua1, $ua2;
    # Make sure the appropriate logout buttons are displayed
    $_->content_contains("/logout\">User Logout</a>",
        "Both users should have a 'User Logout'") for $ua1, $ua2;
    $ua1->content_contains("/books/form_create\">Create</a>",
        "Only 'test01' should have a create link");
    
    $ua1->get_ok("http://localhost/books/list", "View book list as 'test01'");
    
    # User 'test01' should be able to create a book with the "formless create" URL
    $ua1->get_ok("http://localhost/books/url_create/TestTitle/2/4",
        "'test01' formless create");
    $ua1->title_is("Book Created", "Book created title");
    $ua1->content_contains("Added book 'TestTitle'", "Check title added OK");
    $ua1->content_contains("by 'Stevens'", "Check author added OK");
    $ua1->content_contains("with a rating of 2.", "Check rating added");
    # Try a regular expression to combine the previous 3 checks & account for whitespace
    $ua1->content_like(qr/Added book 'TestTitle'\s+by 'Stevens'\s+with a rating of 2./, "Regex check");
    
    # Make sure the new book shows in the list
    $ua1->get_ok("http://localhost/books/list", "'test01' book list");
    $ua1->title_is("Book List", "Check logged in and at book list");
    $ua1->content_contains("Book List", "Book List page test");
    $ua1->content_contains("TestTitle", "Look for 'TestTitle'");
    
    # Make sure the new book can be deleted
    # Get all the Delete links on the list page
    my @delLinks = $ua1->find_all_links(text => 'Delete');
    # Use the final link to delete the last book
    $ua1->get_ok($delLinks[$#delLinks]->url, 'Delete last book');
    # Check that delete worked
    $ua1->content_contains("Book List", "Book List page test");
    $ua1->content_contains("Book deleted", "Book was deleted");
    
    # User 'test02' should not be able to add a book
    $ua2->get_ok("http://localhost/books/url_create/TestTitle2/2/5", "'test02' add");
    $ua2->content_contains("Unauthorized!", "Check 'test02' cannot add");

The C<live_app.t> test cases uses copious comments to explain each step
of the process.  In addition to the techniques shown here, there are a
variety of other methods available in 
L<Test::WWW::Mechanize::Catalyst|Test::WWW::Mechanize::Catalyst> (for 
example, regex-based matching). Consult the documentation for more
detail.

B<TIP>: For I<unit tests> vs. the "full application tests" approach used
by L<Test::WWW::Mechanize::Catalyst|Test::WWW::Mechanize::Catalyst>, see 
L<Catalyst::Test|Catalyst::Test>.

B<Note:> The test script does not test the C<form_create> and
C<form_create_do> actions.  That is left as an exercise for the reader
(you should be able to complete that logic using the existing code as a
template).

To run the new test script, use a command such as:

    $ CATALYST_DEBUG=0 prove --lib lib -v t/live_app01.t

or

    $ DBIC_TRACE=0 CATALYST_DEBUG=0 prove --lib lib -v t/live_app01.t

Experiment with the C<DBIC_TRACE>, C<CATALYST_DEBUG> and C<-v> 
settings.  If you find that there are errors, use the techniques 
discussed in the "Catalyst Debugging" section (Chapter 7) to isolate 
and fix any problems.

If you want to run the test case under the Perl interactive debugger,
try a command such as:

    $ DBIC_TRACE=0 CATALYST_DEBUG=0 perl -d -Ilib t/live_app01.t

Note that although this tutorial uses a single custom test case for
simplicity, you may wish to break your tests into different files for
better organization.

B<TIP:> If you have a test case that fails, you will receive an error
similar to the following:

    #   Failed test 'Check we are NOT logged in'
    #   in t/live_app01.t at line 31.
    #     searched: "\x{0a}<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Tran"...
    #   can't find: "You need to log in to use this application."

Unfortunately, this only shows us the first 50 characters of the HTML
returned by the request -- not enough to determine where the problem
lies.  A simple technique that can be used in such situations is to 
temporarily insert a line similar to the following right after the 
failed test:

    diag $ua1->content;

This will cause the full HTML returned by the request to be displayed.

Another approach to see the full HTML content at the failure point in 
a series of tests would be to insert a "C<$DB::single=1;> right above 
the location of the failure and run the test under the perl debugger 
(with C<-d>) as shown above.  Then you can use the debugger to explore 
the state of the application right before or after the failure.


=head1 SUPPORTING BOTH PRODUCTION AND TEST DATABASES

You may wish to leverage the techniques discussed in this tutorial to
maintain both a "production database" for your live application and a
"testing database" for your test cases.  One advantage to
L<Test::WWW::Mechanize::Catalyst|Test::WWW::Mechanize::Catalyst> is that
it runs your full application; however, this can complicate things when
you want to support multiple databases.  One solution is to allow the
database specification to be overridden with an environment variable.
For example, open C<lib/MyApp/Model/DB.pm> in your editor and
change the C<__PACKAGE__-E<gt>config(...> declaration to resemble:

    my $dsn = $ENV{MYAPP_DSN} ||= 'dbi:SQLite:myapp.db';
    __PACKAGE__->config(
        schema_class => 'MyApp::Schema',
        connect_info => [
            $dsn,
        ],
    );

Then, when you run your test case, you can use commands such as:

    $ cp myapp.db myappTEST.db
    $ CATALYST_DEBUG=0 MYAPP_DSN="dbi:SQLite:myappTEST.db" prove --lib lib -v t/live_app01.t

This will modify the DSN only while the test case is running.  If you
launch your normal application without the C<MYAPP_DSN> environment
variable defined, it will default to the same C<dbi:SQLite:myapp.db> as
before.


=head1 AUTHOR

Kennedy Clark, C<hkclark@gmail.com>

Please report any errors, issues or suggestions to the author.  The
most recent version of the Catalyst Tutorial can be found at
L<http://dev.catalyst.perl.org/repos/Catalyst/Catalyst-Manual/5.70/trunk/lib/Catalyst/Manual/Tutorial/>.

Copyright 2006-2008, Kennedy Clark, under Creative Commons License
(L<http://creativecommons.org/licenses/by-sa/3.0/us/>).