Step 3 of 6

Tests that catch bugs

A test suite is only as good as the bugs it would catch. Here's a classic: in the Gregorian calendar, a year is a leap year if it's divisible by 4, except years divisible by 100, except again years divisible by 400. So 2024 is a leap year, 1900 isn't, and 2000 is.

Suppose someone forgot the century rules:

fn is_leap_year(year: u32) -> bool {
    year % 4 == 0 // bug: ignores the 100 and 400 rules
}

fn main() {
    // "typical" checks, the kind people write first
    for (year, expected) in [(2024, true), (2023, false), (2020, true)] {
        println!("{year}: expected {expected}, got {}", is_leap_year(year));
    }
    println!("1900: got {} (should be false)", is_leap_year(1900));
}

2024: expected true, got true
2023: expected false, got false
2020: expected true, got true
1900: got true (should be false)

Every "typical" check passes, and the bug sails through. Tests that only use ordinary inputs prove very little.

Think in rules and boundaries

Good tests come from reading the specification and asking "what would a wrong implementation get wrong?":

• One test per rule. The leap year rules need at least: a normal leap year (2024), a normal non-leap year (2023), a century that isn't a leap year (1900) and a century that is (2000).
• Boundaries. Off-by-one errors live at the edges: if adults are 18 and over, test 17 and 18. For < versus <=, the boundary value itself is the one that tells them apart.
• Special inputs. Empty strings and slices, zero, negative numbers, the maximum value, non-ASCII text.

Mutation testing

How do you know your tests are good enough? Break the code on purpose and see if a test notices. Each broken copy is a mutant; a mutant your tests don't notice has survived, and it points at a missing test. That's what this site does with the planted bugs in each exercise, and at work the tool cargo-mutants does it automatically for a whole project (it flips < to <=, replaces function bodies with default values, and so on).

Common mistakes

• Testing the implementation instead of the rules: copying the formula into the test (assert_eq!(is_leap_year(y), y % 4 == 0)) repeats the bug instead of catching it. Write expected values you worked out by hand.
• One giant test. Small tests with clear names tell you which rule broke.
• Chasing 100% coverage. Running every line isn't the same as checking every rule: a test can execute code without asserting anything useful about it.

Your task

is_leap_year is correct. Write at least 3 tests that catch all the planted bugs: a missing 400-year rule, a missing 100-year rule, "every second year is a leap year", and "only multiples of 400 are leap years".

Challenge 2

grade turns a score into a letter: 90 and up is A, 80 to 89 B, 70 to 79 C, 60 to 69 D, below 60 F. Write at least 2 tests that catch the boundary bugs. Test the exact boundary values.

Challenge 3

Write at least 3 tests for fizzbuzz: multiples of 15 give FizzBuzz, other multiples of 3 give Fizz, other multiples of 5 give Buzz, and every other number is printed as itself. There's no tests module in the starter: write it yourself.

Previous: assert_eq!, messages, and comparing your own types Next: Testing failures: should_panic and Result