
The following is provided as a reference for those interested in understanding
how the test suite works, and how to add and maintain tests.

Overview of types of tests
==========================

Darcs has tests in three formats.  Unit tests that directly test Haskell
functions are written in Haskell and live in src/unit.lhs.  Functional
tests that test the darcs binary are written in Shell and Perl, and live
in tests/. Additionally, there are some functional tests which are
expected to fail, kept in bugs/ and used for documenting bugs and todos.

Haskell tests
--------------------------

These are QuickCheck tests primarily testing the Darcs core.

Shell tests
---------------------------

Shell tests are useful because they are easy to create from a copy/paste
of actual shell commands executed. They are considered successful if no
bad exit codes are returned from the commands in the script.

Perl tests
---------------------------

The Perl tests are useful because of the featureful testing framework that
Perl provides, and the potential they offer to create more portable tests. 

Right now both shell and perl tests are accepted by the project, but shell
tests are preferred by the main developers, as they are generally more
useful and easier to debug. See "The future of testing darcs" below.


How to run tests
=============================

"make unit" builds the unit tests, "./unit" runs them. They take a while.
Output like "good" and "OK, passed 100 tests." is good. Output like 
"*** Gave up! Passed only n tests." is a shortage of quickcheck test
cases, not a test failure.

"make test" causes all the functional tests in "tests/" to be run against
repositories in the old, hashed, and darcs2 formats.

"make bugs" runs the scripts in "bugs/", also on all three types of
repository, with the expectation that they currently fail.

Because "make test" can take a long time to run, it's useful to run fewer
tests at once.  To help with that, the following make targets are also
available:

  make test_shell           # darcs1 old format
  make test_shell_hashed    # darcs1 hashed format
  make test_shell_format2   # darcs2 hashed format
  make test_perl
  make test_perl_hashed
  make test_perl_format2
  make bugs_shell
  make bugs_shell_hashed
  make bugs_shell_format2
  make bugs_perl
  make bugs_perl_hashed
  make bugs_perl_format2


Running specific tests
------------------------------

Add test file names to tests_to_run to do only those tests:

  echo annotate.pl > tests/tests_to_run
  echo annotate.sh >> tests/tests_to_run
  make test_shell # or whichever you like!


Running a single test
-----------------------------

For consistent output, perl tests are usually run with the prove script:

  cd tests
  DARCS_KEEP_TEMPDIR=1 perl bin/prove annotate.pl

and shell tests are run with the shell_harness script:

  perl shell_harness annotate.sh

("chmod +x bin/prove shell_harness" to use them directly.)


Tips for writing (and reading) tests
====================================

- Avoid including a repo format type to "darcs init"
  This insures that all three repo formats will be tested.
  However, if you know that the test only passes under some
  repo formats, *do* explicity include a format option to "darcs init". 


Tips for writing Shell tests
-------------------------------

- Copy an existing test, which will already have the following properties:

- Simply call darcs using "darcs" as you would in the shell.  It is the
  responsibility of the test harness to ensure that the darcs we are
  testing is first in the path.

- Always use bash explicitly--this improves the portability of our tests.

- Always add this near the top of the script:

   set -ev 

  The "v" causes the contents of the script to be printed as part of the run,
  which is helpful for debugging.  The "e" causes the script to exit as soon as
  there is an error. 

- Try to avoid defining functions where possible (see below)

Tips for writing Perl tests
-------------------------------

- Be aware of the testing modules that are available. See the docs for each of
  these, in order of importance. They are in the tests/lib/perl directory.
  (Use perldoc file.pm to see the docs). 

   Test::More
   Test::Darcs
   Shell::Command
   File::Slurp

- Avoid direct system calls. It will be tempting to use backticks to drop in direct shell
  commands. By writing as much of the tests as possible in pure Perl, they will remain
  more portable. The modules above cover the common cases. 

- ./bin/prove has many options for running the Perl tests. 


The future of testing darcs
=================================

We would prefer to have all tests written in bash, as these tests are both
easier to write (since you ordinarily run darcs from a shell in any case),
and are more helpful when trying to track down bugs, for the same reason:
it is easy to run bash tests by hand in a shell, examining each
intermediate state.  For this same reason, try to avoid defining functions
in your test scripts.  This makes them harder to run and generally harder
to use.  There are certainly cases where it is appropriate to define a
function, but please do not do this just to avoid a little duplication.
