Blog

Mechanizing Git bisect: Bug hunting for the lazy

Written by Metal Toad Staff | Sep 7, 2010 12:00:00 AM
Filed under:

Like this article? Check out our top 5 Git tips and tricks

Git bisect is a powerful automated tool for searching deep into a project's history. Instead of searching for relevant commit messages (git log) or patches (git log -S), bisect actually allows you to run a functional test on each revision until the first bad commit is identified. (Okay, it doesn't test every revision, it performs a binary search, which results in at most log2(N) tests. This allows a relatively large history to be searched quickly.)

The test can be done interactively, with the human performing each check, or mechanically if you can supply a testing script. Randy Fay has done a nice screencast on the interactive method; this post will instead focus on mechanizing the process.

For an example, let's look at a core Drupal bug that impacts this very site: #812990: Search page title changes to Home. For the moment, we'll pretend the cause of this bug isn't already known, and hunt it with git-bisect.

Before running bisect, I'll need to find a known good revision. I picked an arbitrary revision on April 1, and manually confirmed the title "Search" is correct.

Note - Make sure you test that the <good> commit you specify is genuinely good, or git will simply report it's next child as the first bad commit.

Starting bisect

git bisect start HEAD 6e894d75b5160b703566ab674f7dd99d998ac4a6
git bisect run ./test.php

test.php

This is a basic test that uses PHP's DOM and SimpleXML extensions. There are many ways this test could be written; Selenium, SimpleTest, or even just wget.

#!/usr/bin/php
<?php
 
# Clean the repository to prevent collisions with untracked files.
# Make sure your .gitignore is up-to-date!
`git clean -f`;
 
# Re-install core.  This is necessary because HEAD
# revisions aren't necessarily compatible.
`drush -y site-install`;
 
# Grant "anonymous" permission to use the search page.
`drush sql-query "INSERT INTO role_permission (rid, permission, \   module) VALUES (1, 'search content', 'search')"`;
 
# Core likes a non-writeable sites directory, but this can interfere
# with git's ability to update default.settings.php.
# So the directory is made writeable again.
chmod('sites/default', '0755');
 
# The path to a search result page.
$url = 'http://d7.jasper/search/node/foo';
 
# Use SimpleXML to locate the page title.
$html = new DOMDocument();
if (@$html->loadHTMLFile($url)) {
  chmod('sites/default', '0755');
  if ($xml = simplexml_import_dom($html)) {
    $title = trim(array_shift($xml->xpath('//h1[1]')));
    if (!empty($title)) {
      print "title: $title\n";
      # Exit status of zero indicates the correct title was found.
      exit($title == 'Search' ? 0 : 1);
    }
  }
}
 
// Indicates this revision cannot be tested.
exit(125);
?>

Results

A few minutes after starting bisect, we have the answer:

Bisecting: 441 revisions left to test after this (roughly 9 steps)
[33f4bfc13d106f697a1f09fb6fedd9ec16696b74] - Patch #312144 by CorniI, dropcube, EvanDonovan, Damien Tournoud, David_Rothstein, tstoeckler: install fails when default.settings.php is not present.
running ./test.php
title: Home
Bisecting: 220 revisions left to test after this (roughly 8 steps)
[cddd4f8cf2dce0a0bad37186bd3c939c3200e724] #777738 by Garrett Albright: Fixed #states: Correct spelling of Dependent.
running ./test.php
title: Search
Bisecting: 110 revisions left to test after this (roughly 7 steps)
[83c375da758447e7336dc6c7693970f32b97fd41] #721400 follow-up by pwolanin: Remove unnecessary query strings from CSS/JS files.
running ./test.php
title: Home
Bisecting: 54 revisions left to test after this (roughly 6 steps)
[284cd3c472a5d561cff5e54f973fe6a72643e48c] #596614 by asimmonds: Rename node_update_7006() context parameter to sandbox, for consistency.
running ./test.php
title: Home
Bisecting: 27 revisions left to test after this (roughly 5 steps)
[f4dd428815e58b48ebd023ea155d5477036b7a1c] - Patch #555830 by effulgentsia: better code comments.
running ./test.php
title: Search
Bisecting: 13 revisions left to test after this (roughly 4 steps)
[3bd9de3c96751c853d370df76bdfc3f5b830570c] #679960 follow-up by lambic: Add test for 'Notice: undefined variable cids' when there are no comments available.
running ./test.php
title: Home
Bisecting: 6 revisions left to test after this (roughly 3 steps)
[1b426dc86a6cb5ee1fe1da689073489691484dfa] - Patch #719686 by duellj: tests for search weighting for HTML tags.
running ./test.php
title: Home
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[bf354f944559f21ee5cd0c3dd435cca0a417c8f5] #777100 by jhodgdon: Fixed  hook_field_storage_update_field() is not documented.
running ./test.php
title: Search
Bisecting: 1 revision left to test after this (roughly 1 step)
[f792269874c012edc0e2f28794ec8178c3403c07] #245103 by chx, jhodgdon, merlinofchaos, douggreen, jrbeeman: Fixed Search page tabs not highlighting.
running ./test.php
title: Home
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[6c2976e6b3f89a5c8427c8929da0c9198d046272] #783438 by sun: Fixed #states doesn't work for #type item.
running ./test.php
title: Search
f792269874c012edc0e2f28794ec8178c3403c07 is the first bad commit
commit f792269874c012edc0e2f28794ec8178c3403c07
Author: webchick <webchick>
Date:   Sat May 1 01:04:24 2010 +0000
 
    #245103 by chx, jhodgdon, merlinofchaos, douggreen, jrbeeman: Fixed Search page tabs not highlighting.
 
:040000 040000 2ef1c3ea6f6e198977257e1fc9825119e11eaf8f 9a04134995cfba314e2ebf72c4bb5315b20acd53 M	modules
bisect run success