chanson

I've updated my posts on creating and debugging unit tests for for Cocoa applications and frameworks for Xcode 3.0. If you have any additional questions about how unit testing works in Xcode, don't hesitate to ask!

Here are pointers to the updated articles:

• Xcode: Unit Testing Cocoa frameworks
• Xcode: Debugging Cocoa Framework Unit Tests
• Xcode: Unit Testing Cocoa Applications
• Xcode: Debugging Cocoa application unit tests

The most significant change is in debugging Cocoa application unit tests. There's an extra environment variable you'll need to add to your executable, DYLD_FALLBACK_FRAMEWORK_PATH, before you'll be able to debug it with Xcode 3.0. This is necessary because Xcode takes advantage of a new linker feature — runpath-relative install names — in order to support moving and renaming your Developer folder and having multiple versions of Xcode installed.

About a year ago, I wrote about unit testing target-action connections for Cocoa user interfaces. That covers the traditional mechanism by which user interfaces have typically been constructed in Cocoa since the NeXTstep days. However, with the release of Mac OS X 10.3 Panther we've had a newer interface technology available — Cocoa bindings — which has presented some interesting application design and testing challenges.

Among other hurdles, to properly use Cocoa bindings in your own applications, you need to ensure that the code you write properly supports key-value coding and key-value observing. However, since the release of Mac OS X 10.4 Tiger, the necessary APIs have been available to easily do test-driven development of your application's use of Cocoa bindings, following a trust, but verify approach. (It's also been quite easy from the start to test your support for key-value coding and key-value observing, to ensure that your code meets the necessary prerequisites for supporting bindings. I can write more on this topic in another post if anyone is interested.)

The key to writing unit tests for Cocoa bindings is the -infoForBinding: method in AppKit's NSKeyValueBindingCreation informal protocol. Using this simple method, you can interrogate any object that has a binding for all of the information about that binding! It simply returns a dictionary with three keys:

1. NSObservedObjectKey, which is the object that the binding is bound to;
2. NSObservedKeyPathKey, which is the key path that is bound — in Interface Builder terms, this is the controller key path combined with the model key path, with a dot in between them; and
3. NSOptionsKey, which is a dictionary of additional binding options unique to the binding. These are all of those additional checkboxes and pop-ups in the Interface Builder bindings inspector for setting things like a value transformer.

By specifying what this dictionary should contain for a particular binding, you can describe the binding itself and thus start doing test-driven development of your Cocoa bindings-based user interface. Note that all of the system-supported binding names — as well as the binding option names — are specified in <AppKit/NSKeyValueBinding.h> and are documented, too!

Let's take a simple example, like the one in last year's target-action example, of a window controller whose window has a static text field in it. The field should have its value bound to the name of a person through an object controller for that person. Assume that I've already created the test case and set up some internal methods on my window controller to refer to the contents of the window via outlets, and to load the window (without displaying it) in -setUp just like in the target-action example.

First, to see that my text field has a value binding, I might write something like this:

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];

    NSDictionary *valueBindingInfo = [personNameField infoForBinding:NSValueBinding];
    STAssertNotNil(valueBindingInfo,
        @"The person name field's value should be bound.");
}

Of course, this tells us nothing about how the binding should be configured, so it needs some fleshing out...

Let's check the object and key path for the binding.

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];

    NSDictionary *valueBindingInfo = [personNameField infoForBinding:NSValueBinding];
    STAssertNotNil(valueBindingInfo,
        @"The person name field's value should be bound.");

    NSObjectController *personController = [_windowController personController];
    STAssertEquals([valueBindingInfo objectForKey:NSObservedObjectKey], personController,
        @"The person name field should be bound to the person controller.");

    STAssertEqualObjects([valueBindingInfo objectForKey:NSObservedKeyPathKey], @"name",
        @"The person name field's value should be bound to the 'name' key.");
}

Not very exciting, and a little verbose, but it'll easily lead us through what needs to be set up in Interface Builder for this binding to work. If you want to cut down the verbosity, you can of course extract a method to do the basic checking...

- (BOOL)object:(id)object shouldHaveBinding:(NSString *)binding
            to:(id)boundObject throughKeyPath:(NSString *)keyPath
{
    NSDictionary *info = [object infoForBinding:binding];

    return ([info objectForKey:NSObservedObjectKey] == boundObject)
            && [[info objectForKey:NSObservedKeyPathKey] isEqualToString:keyPath];
}

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];
    NSObjectController *personController = [_windowController personController];
    
    STAssertTrue([self object:personNameField shouldHaveBinding:NSValue
                           to:personController throughKeyPath:@"name"],
    @"Bind person name field's value to the person controller's 'name' key path.");
}

If you're writing code that needs, say, a value transformer, it's a simple matter to extend this model to also check that the correct value transformer class name is specified for the NSValueTransformerNameBindingOption key in the binding options dictionary returned for NSOptionsKey.

You can even extract these kinds of checks into your own subclass of SenTestCase that you use as the basis for all of your application test cases. This will let you write very concise specifications for how your user interface should be wired to the rest of the code, that you can use to just walk through Interface Builder and connect things together — as well as use to ensure that you don't break it accidentally by making changes to other items in Interface Builder.

This is the real power of test-driven development when combined with Cocoa: Because you can trust that the framework will do the right thing as long as it's set up right, you simply need to write tests that specify how your application's interface should be set up. You don't need to figure out how to create events manually, push them through the run loop or through the window's -sendEvent: method, how to deal with showing or not showing the window during tests, or anything like that. Just ensure that your user interface is wired up correctly and Cocoa will take care of the rest.

In my previous post, I suggested turning off assertions for your Release build. However, if you want to be able to do so, you'll need to be careful to distinguish between assertions that can be safely turned off, and those that can't.

For example, say you've implemented your own ordered-set class, MyOrderedSet. Obviously it will have an interface similar to both the NSSet class and the NSArray class in Cocoa's Foundation framework. As part of making it work like the Foundation collections, you'll probably want to do things like make it throw an NSRangeException if the caller requests the object at an index that's at or beyond the collection's count.

You might be tempted to implement this using a form of assertion, since by default assertions in Cocoa will throw an exception. But remember, this is part of the API contract of your class, not just an internal detail that should never go wrong! Thus if you do implement this using an assertion, and you disable assertions in your Release build, your Release build will wind up implementing your API incorrectly.

The solution is to distinguish in your code between assertions that verify state that should never be incorrect, and checks that ensure preconditions that are specified as part of an API contract are valid. In the example above, you could easily use a check to cause a range exception to be thrown, and this would still be compiled into your Release build.

The Foundation framework in Cocoa doesn't have any pre-defined checks, but they're not hard to develop on your own, especially with C99 variadic macros. You'll probably want to use a syntax similar assertions in creating your check macro, for example, MyCheck(condition, exceptionName, format, ...) and you'll probably want it to just throw the named exception with the given reason. If you take the time to make your checks easy to use, you'll definitely use them more, and if you make them use exceptions, you'll make it easy to ensure your application's state is rolled back in a sane fashion.

The behavior of your checks can even be specified as part of your unit tests. (It's 2007, you are writing unit tests, right?) You can write tests that mimic the misuses of your API that your checks are designed to trap, and then note that these misuses cause the appropriate exception to be thrown via OCUnit's STAssertThrowsSpecificNamed assertion macro.

Typically, when you create a build of your software to release, you'll apply a whole bunch of additional optimizations to that version that you won't apply to the debugging builds. You'll strip debugging symbols — or, nowadays, build them to a separate DWARF with dSYM file — and crank up the compiler optimization and instruction scheduling settings that would normally make debugging hard.

One optimization that's often overlooked, though, is to turn off assertions. A lot of developers will use assertion macros as sort of a "continuous unit testing" technique within their code during development. They'll assert that methods' preconditions hold, that the postconditions of various operations continue to hold, that changes within methods produce expected results, and so on.

Standard C provides the assert(3) macro to perform a simple form of verification. By default, it just writes the failure information — including the failing file and line — and then calls abort(3); most environments also provide a way to extend its functionality by using assert as a wrapper for a sub-macro or other function.

Similarly, Cocoa's Foundation framework provides a number of NSAssert and NSParameterAssert macros that you can use in your code. (The primary difference is in the message of the exception they generate.) These normally throw an NSInternalInconsistencyException, but their behavior is extensible by subclassing NSAssertionHandler.

The thing is, you probably don't want the released versions of your applications terminating immediately or throwing exception in unexpected places — even though you probably do want them to do so during development. (After all, that's part of how you ensure they won't happen in production, right?) So you need to disable them in your release build.

Fortunately, Xcode makes this really easy. You just need to add two entries to the Preprocessor Macros build setting in your Xcode project, typically at the project level (since you'll want it to apply to all targets in your project). To turn off Standard C assertions, you just need to specify NDEBUG while for turning off Foundation assertions, just specify NS_BLOCK_ASSERTIONS. If you do this, you can put assertions literally everywhere in your code but be confident you'll only ever hit them during development.

Logging is in a similar boat. The release build of an application should, in general, never log in normal use. However, it can be extremely useful when things do go wrong to allow your users to enable logging by changing a hidden (or not-so-hidden) preference within your application. Ideally, you could even control the logging for different aspects of your application and different levels of severity, so enabling logging for one piece of functionality doesn't affect others, and so you can avoid spewing more log information than necessary.

Unfortunately, there's no built-in facility in Foundation for this sort of fine-grained logging. There's really just NSLog which, though useful during development, is both unconditional and doesn't have either aspects or levels. However, there's the Open Source (BSD licensed) Log4Cocoa project which aims to provide most of this functionality. It's currently based on the same design as Log4J, and it hasn't had a lot of work since its introduction, but it's a good starting point for working sophisticated logging into your own applications and it would be great to see developers pick it up again.

Bob Frank and Wolf Rentzsch presented on both of them for Uniforum a few years back, and made a PDF of their presentation available. Just be sure to tune the logging that actually gets compiled into your release builds by leveraging the Preprocessor Macros build setting...

Update (April 30, 2007): I've followed this up with another post on the importance of distinguishing between assertions and checks in your code. One of the main reasons I find that people don't want to turn off assertions before shipping is that they're using assertions as checks; if you distinguish between them, then this won't be a problem.

Jeremy Miller, The Shade Tree Developer, On Writing Maintainable Code:

Separation of Concerns is the Alpha and Omega of design principles. No other design principle that I'm going to discuss -- be it loose coupling, high cohesion, encapsulation, minimal duplication, or orthogonality -- is possible without Separation of Concerns. Simply put, strive to do one thing at a time in your code. Layering. Divide and conquer. A lot of the other principles are about enabling a system to change with minimal effort and risk. Before you can even think about that, you need to be able to build the system, and then understand that code.

Jeremy hits the nail on the head here: The single most important factor in ease of software maintenance is how well-factored the software is. If its code commonly does more than one thing at once or otherwise mixes concerns, apply the Extract Method, Extract Class, Extract Interface, and Replace Conditional with Polymorphism refactorings liberally until the software is broken up in such a way that each logical unit of code only does one thing.

Also, be sure to check out Working Effectively with Legacy Code by Michael Feathers. (Here's a good review by James Robertson.) It contains a lot of techniques for taking legacy code — which Feathers defines as any code without unit tests — and breaking it up in well-defined ways into code for which it's possible to write unit tests so further, safer refactoring can be done more quickly. It's based on this paper, Working Effectively with Legacy Code, available from the ObjectMentor web site.

It's really great to see that a lot of people are adopting unit testing for their projects and dramatically improving their quality. Test-driven development and agile development methodologies built around it are really taking off. However, a lot of people still feel that their user interface is difficult to test through code, and either requires a capture-playback tool or requires a different design approach based heavily on interfaces/protocols to get right.

In last year's post Trust, but verify. I tried to dispel some of the mystery of testing your application's user interface when using the Cocoa frameworks. However, I've still had a lot of (entirely well-justified!) requests for examples of how to put it into practice. So here's a simple example of what I'd do to write a unit test for a button in a window that's supposed to perform some action.

First, when implementing my window, I'd follow the standard Cocoa pattern of having a custom NSWindowController subclass to manage my window. This window controller will have an outlet connected to each of the views in the window, and will also wind up with a private accessor method — used only within the class and any subclasses, and in testing — for getting the value of each of its outlets. This design flows naturally from the test which I would write to specify that the window should contain a button. First, here's the skeleton into which I'd put tests:

// TestMyWindow.h

#import <SenTestingKit/SenTestingKit.h>

@class MyWindowController;

@interface TestMyWindow : SenTestCase {
    MyWindowController *_windowController;
    NSWindow *_window;
}
@end

// TestMyWindow.m

#import "TestMyWindow.h"
#import "MyWindowController_Private.h"

@implementation TestMyWindow

- (void)setUp {
    // MyWindowController knows its nib name and
    // invokes -initWithWindowNibName: in -init
    _windowController = [[MyWindowController alloc] init];

    // Load the window, but don't show it.
    _window = [_windowController window];
}

- (void)tearDown {
    [_windowController release];
    _window = nil; // owned by _windowController
}

@end

That's the infrastructure into which I'd put my other test methods for this window. For example, I'll want to specify the nib name for the window controller and ensure that it actually knows its window:

- (void)testNibName {
    STAssertEqualObjects([_windowController windowNibName], @"MyWindow",
      @"The nib for this window should be MyWindow.nib");
}

- (void)testWindowLoading {
    STAssertNotNil(_window,
      @"The window should be connected to the window controller.");
}

Now let's check that I have a "Do Something" button in the window, and that it sends an action directly to the window controller.

- (void)testDoSomethingButton {
    // _doSomethingButton is a private method that returns the button
    // conected to the doSomethingButton outlet
    NSButton *doSomethingButton = [_windowController _doSomethingButton];
    
    STAssertNotNil(doSomethingButton,
      @"The window should have a 'Do something' button.");
    
    STAssertEqualObjects([doSomethingButton title], @"Do Something",
      @"The button should be titled accordingly.");

    STAssertEquals([doSomethingButton action], @selector(doSomething:),
      @"The button should send -doSomething: to its target.");

    STAssertEquals([doSomethingButton target], _windowController,
      @"The button should send its action to the window controller.");
}

You'll notice something I'm not doing in the above: I'm not simulating interaction with the interface. This is the core of the trust, but verify approach to unit testing of your user interface.

I can trust that as long as I verify everything is hooked up properly that Cocoa will cause the button to send its action message to its target — whether it's a specific object or, if the target is nil, the responder chain — whenever the button is clicked while it's enabled and not hidden. I don't need to simulate a user event, and I don't even need to display the interface while running the unit tests. All I need to do is inspect, through code, that everything is wired up correctly.

Note that I can do way more than the above in testing my interface design, too. For example, I can ensure that the control layout is correct according to what my interface designer has specified, by checking bounding rectangles for example. But testing only the functionality of my interface has significant advantages, too. For example, it doesn't matter if I wind up using a custom kind of button to achieve exactly the kind of look and feel or behavior I need. It doesn't matter if I wind up changing the layout in response to feedback. No matter what I do, I'll know that functionality won't accidentally break while I'm messing around in Interface Builder — even if I completely rip out my interface and replace it with a new one!

This approach can also be used for testing Cocoa bindings using the -infoForBinding: method that was introduced in Mac OS X 10.4 Tiger. I hope to write up a post soon on how to approach Cocoa bindings using these same techniques, but it should be fairly straightforward given the above and the above documentation.

Update: I've struck through the check of the button's title above, because you may or may not want to do that. For example, if you're primarily running your unit tests against your development localization, you may want to put it in. But if you want to run your unit tests against a localized build of your application, you'll probably want to avoid checking a localized title against an English string. A "have your cake and eat it too" strategy might be to keep a variable somewhere in your application that can be used to selectively disable checks of only localized strings.

Update July 7, 2007: I've finally written a post, Unit testing Cocoa user interfaces: Cocoa bindings, on how to write tests for Cocoa bindings. Now there's no excuse for not doing test-driven development of your Cocoa user interfaces!

Noel Llopis (Games from Within) and Sean Houghton, Backwards Is Forward: Making Better Games with Test-Driven Development:

One of the questions we had when we jumped into TDD is whether it was going to hold for high-level code. We had seen in practice from previous projects that we can certainly do TDD to create low-level and intermediate-level libraries (math, collision, messaging, etc). But would it really work for high-level code that would build on low-level code?

The answer is an unconditional yes. We have developed a full codebase doing TDD from the start, and we had no difficulty writing high-level code with TDD. Things like character state machines, game flow, or specific game entities were done through TDD without any problems, and greatly benefited from the TDD approach.

Noel's blog is great, and this paper is being presented at the 2006 Game Developers Conference.

Noel and High Moon Studios have been a lot of pioneering work using Extreme Programming in the game development space. Game development can definitely make good use of Extreme Programming to manage the development process, dramatically improve the quality of code, and do wonders for scheduling accuracy and schedule predictability.

Extreme Programming in game development also presents special challenges due to the exploratory nature of a lot of the work, as well as the highly-interactive nature of the software itself. As Noel points out, you wind up writing much more finely-factored code when doing TDD, which will be alien to a lot of game developers but will help greatly with maintenance, debugging, and the ever-more-important portability. (Not just between Windows and Mac OS X, but also between Xbox and PlayStation 2 and Nintendo GameCube and Xbox 360 and PlayStation 3 and Nintendo Revolution...)

CocoaHeads Silicon Valley is meeting tonight on the Apple Campus in Cupertino, in the Town Hall auditorium of Infinite Loop 4.

I'll be speaking about Cocoa unit testing with Xcode.

Chris Liscio, Unit Testing and Code Coverage with Xcode:

At this point, your have instrumented your framework to support the auto-generation of code coverage data. When you run your unit test using the framework built in this manner, it will generate a dump of what lines of been hit (and how many times).

Chris talks about how to combine Xcode 2.1's unit testing support with gcov for code coverage analysis. Freakin' awesome!

Followup: Chris's article states that you need to ensure your target's "Library Search Paths" build setting includes $(SYSTEM_DEVELOPER_DIR)/SDKs/MacOSX10.4u.sdk/usr/lib/gcc/powerpc-apple-darwin8/4.0.0. This is not the case — you should not do this.

All you need to do is add -lgcov to the "Other Linker Flags" build setting in your Coverage configuration. The correct GCC library directory for your chosen compiler version and target architecture is added to your library search paths automatically. Since libgcov.a is part of the GCC library it will be found automatically.

Wil Shipley has just come out as a completely and utterly wrong when it comes to unit testing.

Talk about not getting it.

On the Extreme Programming mailing list, Rob Evans recently mentioned that he had been asked to define a software development methodology for his company's entire IT organization. To help him define this methodology, he asked list participants to name their top three agile practices to use when moving an organization to agile methods.

Here are my top three agile methods, all taken from Extreme Programming:

1. Test-Driven Development
2. Small Releases
3. The Planning Game

All of these practices work well together, they can be done in virtually any development environment on virtually any platform, and they don't require much adjustment in the actual developer workday (though they may require significant adjustments in the project planning and scheduling process).

Test-Driven Development gives you a huge amount of ROI when it comes to keeping code quality and maintainability high. This is particularly true if you refactor every time you make a test pass; if you get the tests passing but don't remove duplication and keep your design clean, you can wind up with very well tested spaghetti.

Small Releases — working in terms of short iterations and having a complete, production-quality deliverable at the end of every iteration — keep progress on a project extremely visible. You can tell exactly when and what progress is being made, allowing progressively more accurate decisions to be made about your project and making its return-on-investment extremely visible.

The Planning Game works with small releases to really put the business drivers of the project behind the wheel. They get to decide what is most important to work on in an iteration — and adjust this from iteration to iteration as their requirements change — and the developers performing the work get to decide exactly how much effort it will take to complete. How much work to actually schedule in an iteration is not dictated by either group, but rather by successively-more-accurate measurement of what can be accomplished in an iteration. This means that business can front-load a project's ROI, development isn't having all of schedule, features, and resources dictated to it, and the project's schedule actually gets more rather than less accurate over the life of the project.

With varying degrees of efficiency, all of the above can be done for a team of 1 or a team of 100. They can be done with a team all in one room or with every member on a different continent. And, in my experience, even if these are the only methods you adopt, they form a solid process with which to deliver quality software with accurate schedules and budgets.

Plug: Xcode Unit Testing can be used very effectively to do Test-Driven Development of applications and frameworks using a variety of Mac OS X technologies. You can even do Test-Driven Development of your software's Core Data data model and human interface!

I've also found OmniOutliner to be very effective for managing user stories on small projects.

A couple weeks ago as part of my Unit Testing Series I talked about how to use Xcode to write unit tests for Cocoa frameworks, debug unit tests in Cocoa frameworks, and write unit tests for Cocoa applications. However, I haven't yet described how to debug your unit tests in Objective-C Cocoa applications. I'll take care of that tonight.

After you've set up unit testing in your Cocoa application, debugging your unit tests is similar to debugging them in a Cocoa framework. All you have to do is adjust the arguments and environment variables your application's Executable is configured to use in Xcode. You don't even have to create a new executable.

To start, bring up the Info window for your application's executable (which is its entry in Xcode's Executable smart group). In the Arguments tab, add the argument -SenTest All. This tells the unit testing infrastructure that you want to run all of the unit tests, not just the ones that are built in to the executable. (After all, you don't have any unit tests in your executable itself.)

Now we'll need to engage in a little bit of environment variable magic. When you test an application, what you're really doing is injecting your unit test bundle into the application and telling it to run its tests at its first opportunity. This is accomplished through by telling dyld to insert a private framework, DevToolsBundleInjection.framework, into your application on launch, and telling that framework via an environment variable, XCInjectBundle, the path of the test bundle to inject.

You also have to tell the injection framework itself the full path to the application executable you want to inject the bundle into, via the XCInjectBundleInto environment variable. This is needed to avoid injecting your test bundle into other executables that are run by the application you're testing, or that are run as a side-effect of running the application you're testing. (For example, gdb generally wants to run applications from within a shell, so that environment variables are expanded in its environment and command-line parameters.)

In the Arguments tab of your application executable, first add an environment variable named DYLD_INSERT_LIBRARIES. Set its value to the path to the DevToolsBundleInjection.framework/DevToolsBundleInjection library in the framework of the same name that's included in the developer tools. Prior to Xcode 2.5, this was in $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks but as of Xcode 2.5 and Xcode 3.0, it has been moved to $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks

Then add a second environment variable, XCInjectBundle. Set its value to $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest.

Add a third environment variable, XCInjectBundleInto. Set its value to the full path to your application's executable — not just the application bundle — e.g. $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. This is the debugging equivalent of the Test Host build setting you used to tell Xcode what executable to inject your tests into when running them.

For Xcode 3.0 and later, add a final environment variable, DYLD_FALLBACK_FRAMEWORK_PATH to your executable. Set its value to $(DEVELOPER_LIBRARY_DIR)/Frameworks.

Why do you need to do this? In order to support moving and renaming the Developer folder, all of the frameworks within it — including OCUnit — use runpath search paths. This means that the internal name of the framework, including the one copied into your test bundle, will start with @rpath rather than an absolute path starting with /Developer/Library/Frameworks. Unfortunately this means that your unit tests won't find SenTestingKit.framework without some extra help. That's what DYLD_FALLBACK_FRAMEWORK_PATH does: It tells dyld to try an additional set of directories in place of @rpath when it can't be resolved. (More information on runpath-relative install names can be found in the ld(1) man page.)

Make sure the check marks next to all three of these environment variables — and your -SenTest All argument, of course — are set.

Troubleshooting note: Troubleshooting note: If this doesn't work — that is, if your test bundle isn't found and run — change the executable's working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Now if you choose Run Executable from the Debug menu, your application should launch, you should see the results of executing your unit tests in the Run Log, and as soon as your unit tests are complete your application should quit!

To debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. As with a Cocoa framework, do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test. Debug Executable will work as long as your executable itself is actually present.

Having done all this, you should be stopped at the breakpoint!

Just as any other time you use OCUnit, instead of -SenTest All you can specify -SenTest MyTestCaseClassName to run just the tests in the test case class MyTestCaseClassName, or -SenTest MyTestCaseClassName/testMethodName to run just a single test.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Update March 17, 2008: I've updated this a bit to handle some changes in the process introduced with Xcode 3.0.

Update September 5, 2008: I've updated this again to cover the changes that were made to bundle injection for Xcode 3.1; the change is the introduction of the XCInjectBundleInto environment variable.

Yesterday, I talked about how to add unit tests to Cocoa frameworks using Xcode. There's only a little more set-up required to add tests to Objective-C Cocoa applications.

First, turn off ZeroLink for the application target you want to test. Just as with a framework, your unit tests will be built as a test bundle containing only the tests, completely separate from the code being tested. The test bundle will access the code in your application by linking against it, and you can't link against something built with ZeroLink. Note: You only need to do this for Xcode 2.1 through Xcode 2.5. Xcode 3.0 removed support for ZeroLink, since the linker is now sufficiently fast as to obviate it.

Next, add a new Cocoa Unit Test Bundle target to your application project. This is the target that will actually contain your tests. When you add the test bundle target, Xcode shows its Info window. In the General tab of this window, press the + button and choose your application target from the sheet that appears. This will make your test bundle target depend on your application target, so you can just build your test bundle all the time and have your application automatically rebuild first.

Now you actually need to make your test bundle link against your application. You do this using the Bundle Loader property under Linker Options in your test bundle target's configurations. You need to set the value of this option to the path of your application executable and not just its wrapper: $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. The $(BUILT_PRODUCTS_DIR) variable is expanded at build time to be the path to the build products for the currently-selected configuration, which lets you avoid using an absolute path.

The final step is to tell the unit testing infrastructure how to run your tests. For a framework, the dynamic loader does the heavy lifting; when the test rig loads your test bundle, the loader will automatically load the frameworks it depends on including the one you're testing. However, for an application to be tested the application must be launched and the test bundle injected into it. You can specify that this should happen using the Test Host property under Unit Testing in your test bundle target's configurations. Just as with the Bundle Loader property, you need to set the value to the full path of your application executable. Since you've already done that once, you can just re-use the value of the Bundle Loader setting by specifying $(BUNDLE_LOADER) — this is the underlying variable that the property is associated with.

Now just like with frameworks you can add test cases to your test bundle just by adding new files based on the Cocoa Objective-C test case class template. Your application code can remain in your application target and your unit testing code can remain in your test bundle target. Whenever you build your test bundle target, once everything is built your application will launch, its tests will be run, and it will quit — and the test results will be reported via the Build Results window just like compiler and linker errors.

Scott Collins posted an entry a couple weeks ago about using Xcode 2.1 with Qt, including some discussion of using the new CPlusTest framework to write C++ unit tests.

So you've set up unit testing for your Objective-C Cocoa framework and it's been working great. But now you've written a test and it fails, and you can't figure out why. It's time to break out the debugger, but how, exactly, do you do that? Your unit tests are built as a bundle, and you can't run a bundle.

It's simple. All you have to do is set up an appropriate Executable in Xcode to run the test rig that runs your bundle, and then debug that. To get started, choose the Project > New Custom Executable menu item. For its name, specify otest — this is the name of the test rig used by OCUnit. Specify /Developer/Tools/otest as the path to your tool.

When you add the custom executable, Xcode will bring up its info window. Switch to the Arguments tab. Here you'll need to enter some command-line arguments to pass to the test rig and some environment variables to affect how it's run.

First, add the argument -SenTest Self to your executable. This tells otest to run all of the tests in your bundle. (It's actually a pair of arguments, but you can add it as one as a convenience.) Then for your second argument specify $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest where MyTestBundle is the name of your test bundle. This tells otest the path of the test bundle to load; $(BUILT_PRODUCTS_DIR) will be expanded to the appropriate build products directory for your project at run time. (If you get the order wrong, just drag the arguments around in the table.)

Troubleshooting note: If this doesn't work — that is, if otest complains it can't find your test bundle — change the executable's working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Next add a pair of environment variables named DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your executable. These will tell the loader to check your build products directory first for frameworks and libraries whenever the executable is run from within Xcode. Specify $(BUILT_PRODUCTS_DIR) for the value of each variable.

Finally, from the Project > Set Active Executable menu choose your new otest executable. This way any time you run or debug within Xcode, it will run otest with the arguments and environment you've specified.

To actually debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. Do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test; Debug Executable doesn't try to build first, it only cares whether an executable is present.

You should successfully stop at your breakpoint!

If your tests take a long time to run — they shouldn't, they're unit tests after all, but it can still happen — you may want to just run the tests for one test case, or just one individual test. This is easy too. Rather than specifying -SenTest Self in your arguments to otest, you can specify -SenTest MyTestCaseClassName to run the all the tests in the specified test case. To run just a single test, use -SenTest MyTestCaseClassName/testMethodName instead.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

It's straightforward to write unit tests for Objective-C Cocoa frameworks with Xcode 2.1 and later.

First, turn off ZeroLink when building your framework. ZeroLink is a great technology, but you can't link against something that's built with ZeroLink, and that's exactly what your unit tests are going to do. Note: You only need to do this for Xcode 2.1 through Xcode 2.5. Xcode 3.0 removed support for ZeroLink, since the linker is now sufficiently fast as to obviate it.

Next, add a new Cocoa Unit Test Bundle target to your framework project. This is the target that will actually contain your tests.

When you add the new test bundle target Xcode should bring up its Info window. Switch to its General tab and press the + button in the lower-left to add a new dependency. In the resulting sheet, choose your framework target. This tells the build system that your framework target needs to be built before your test bundle target.

Now that you've declared the dependency between the targets, you need to make sure your test bundle actually links against your framework. Make your test bundle target the active target by choosing it from the Project > Set Active Target submenu. Then highlight the Products group in the Groups & Files view at the left of the Xcode project window. In the detail view to the right, you'll see all of the products that are built by the various targets in your project; one of the rows will be for your framework. Click the checkbox at the very right of its row, in the Target Membership column, to make your test bundle link against your framework.

Now you can add test cases to your test bundle by just adding new files based on the Cocoa Objective-C test case class template. You don't need to add any of your framework code to your test bundle, and you don't need to add any testing code to your framework. And you're not restricted to using only header files with public visibility in tests, either; you should be able to use anything from your framework target, so long as it's exported at link time.

I've written a little on the support for unit testing in Xcode, but there are a bunch more things I'd like to write about. Specifically, I'd like to describe how to both set up and debug tests for Objective-C Cocoa and C++ Carbon applications and frameworks.

Also, be sure to check out the Unit Testing Guide.

Here are the posts in the series so far:

• Xcode: Unit Testing
• Xcode: Unit Testing Cocoa frameworks
• Xcode: Debugging Cocoa Framework Unit Tests
• Xcode: Unit Testing Cocoa Applications
• Xcode: Debugging Cocoa application unit tests

President Reagan, for all his faults, gave us a very useful aphorism in describing his approach to diplomacy with the Soviet Union: "Trust, but verify." This is also a very useful approach to take when writing unit tests when you're working with a framework, particularly when you're developing a human interface.

For example, Cocoa uses the target-action pattern for controls. When you create an NSButton you can specify which object to send a message to when it's clicked — the target — and what message to send — the action.

You can trust that when you click the button, Cocoa will cause the action message to be sent to the target object so long as they have been properly specified. Therefore you probably don't need to write a unit test that simulates clicking on the button. However, you should verify that your button has had its target and action properly specified, and you can write a test for this.

You can also apply this principal in your own code. Let's say you're implementing a new type of control that also has to follow the target-action pattern. In the tests for your control itself, you probably will want to simulate the appropriate user-interface events and see that a testing instance of your control behaves appropriately. However, you don't need to do this in the code that uses the control — you can trust that the control behaves correctly due to the tests you wrote for the control itself, and just verify that it's been properly configured.

Xcode 2.1 introduced integrated unit testing to the Xcode IDE. Xcode includes two unit testing frameworks, target templates for setting up test bundle targets, and infrastructure for running unit tests every time you build your project and reporting their results in the Build Results window just like compilers and linkers do.

With Xcode unit testing, you group your test cases into test bundles which are built by separate targets. This means that you don't need to build two versions of your software — one with tests and one without tests — and can run your tests against the actual software you want to deliver.

Xcode 2.1 and later include OCUnit for unit testing of Objective-C Cocoa software, and it includes a new framework called CPlusTest for unit testing of C++ software. Using either you should be able to test C code with relative ease. Corresponding target templates are included for creating unit test bundles appropriate for Cocoa and Carbon applications and frameworks, and file templates are included for creating OCUnit and CPlusTest test case classes.

The test bundle target templates have a shell script build phase at the very end that invokes /Developer/Tools/RunUnitTests. RunUnitTests looks at the build settings it's passed via its environment and determines from that information how to run the tests in your test bundle.

If you're testing a framework, RunUnitTests will run the appropriate test rig and tell it to load and run the tests in your bundle. Since your test bundle should link against your framework, your framework will be loaded when the test rig loads your bundle.

If you're testing an application, you need to specify the application as the Test Host and Bundle Loader for your test bundle in its configuration's build settings. The Bundle Loader setting tells the linker to link your bundle against the application that's loading it as if the application were a framework, allowing you to refer to classes and other symbols within the application from your bundle without actually including them in the bundle. The Test Host setting tells RunUnitTests to launch the specified application and inject your test bundle into it in order to run its tests.

There's even support in RunUnitTests for invoking a test rig of your own, rather than the test rig for one of the supplied frameworks. You just need to specify the Test Rig build setting for your test bundle; this should be the path to a tool to run. It will be passed the path to your test bundle as its first argument, and if it needs to generate failure information it can just generate it in a gcc/compiler-like format on stderr:

FailingTest.c: 10: error: (1 == 0) failed

This will cause it to show up in the Build Results window as an error, just like a compiler or linker error. You can see the RunUnitTests manpage for more information on the environment in which your test rig will be run.

There's a great Unit Testing Guide on the Apple Developer Connection web site that has lots of information on getting started with Xcode unit testing. Check it out!

WWDC 2005 is over, and damn was it a great week! Apple made some incredible announcements and shipped some incredible software, I got to see lots of old friends and make a lot of new ones, and I got to talk to lots of developers about things that I'm passionate about: Core Data, unit testing, setting up and streamlining your build process, and creating insanely great software to make users' lives better.

It was a wonderful, wonderful time. Thanks to everyone!