Previous Entry Share Next Entry
Xcode: Debugging Cocoa application unit tests
latest
chanson
A couple weeks ago as part of my Unit Testing Series I talked about how to use Xcode to write unit tests for Cocoa frameworks, debug unit tests in Cocoa frameworks, and write unit tests for Cocoa applications. However, I haven't yet described how to debug your unit tests in Objective-C Cocoa applications. I'll take care of that tonight.

After you've set up unit testing in your Cocoa application, debugging your unit tests is similar to debugging them in a Cocoa framework. All you have to do is adjust the arguments and environment variables your application's Executable is configured to use in Xcode. You don't even have to create a new executable.

To start, bring up the Info window for your application's executable (which is its entry in Xcode's Executable smart group). In the Arguments tab, add the argument -SenTest All. This tells the unit testing infrastructure that you want to run all of the unit tests, not just the ones that are built in to the executable. (After all, you don't have any unit tests in your executable itself.)

Now we'll need to engage in a little bit of environment variable magic. When you test an application, what you're really doing is injecting your unit test bundle into the application and telling it to run its tests at its first opportunity. This is accomplished through by telling dyld to insert a private framework, DevToolsBundleInjection.framework, into your application on launch, and telling that framework via an environment variable, XCInjectBundle, the path of the test bundle to inject.

You also have to tell the injection framework itself the full path to the application executable you want to inject the bundle into, via the XCInjectBundleInto environment variable. This is needed to avoid injecting your test bundle into other executables that are run by the application you're testing, or that are run as a side-effect of running the application you're testing. (For example, gdb generally wants to run applications from within a shell, so that environment variables are expanded in its environment and command-line parameters.)

In the Arguments tab of your application executable, first add an environment variable named DYLD_INSERT_LIBRARIES. Set its value to the path to the DevToolsBundleInjection.framework/DevToolsBundleInjection library in the framework of the same name that's included in the developer tools. Prior to Xcode 2.5, this was in $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks but as of Xcode 2.5 and Xcode 3.0, it has been moved to $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks

Then add a second environment variable, XCInjectBundle. Set its value to $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest.

Add a third environment variable, XCInjectBundleInto. Set its value to the full path to your application's executable — not just the application bundle — e.g. $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. This is the debugging equivalent of the Test Host build setting you used to tell Xcode what executable to inject your tests into when running them.

For Xcode 3.0 and later, add a final environment variable, DYLD_FALLBACK_FRAMEWORK_PATH to your executable. Set its value to $(DEVELOPER_LIBRARY_DIR)/Frameworks.

Why do you need to do this? In order to support moving and renaming the Developer folder, all of the frameworks within it — including OCUnit — use runpath search paths. This means that the internal name of the framework, including the one copied into your test bundle, will start with @rpath rather than an absolute path starting with /Developer/Library/Frameworks. Unfortunately this means that your unit tests won't find SenTestingKit.framework without some extra help. That's what DYLD_FALLBACK_FRAMEWORK_PATH does: It tells dyld to try an additional set of directories in place of @rpath when it can't be resolved. (More information on runpath-relative install names can be found in the ld(1) man page.)

Make sure the check marks next to all three of these environment variables — and your -SenTest All argument, of course — are set.

Troubleshooting note: Troubleshooting note: If this doesn't work — that is, if your test bundle isn't found and run — change the executable's working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Now if you choose Run Executable from the Debug menu, your application should launch, you should see the results of executing your unit tests in the Run Log, and as soon as your unit tests are complete your application should quit!

To debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. As with a Cocoa framework, do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test. Debug Executable will work as long as your executable itself is actually present.

Having done all this, you should be stopped at the breakpoint!

Just as any other time you use OCUnit, instead of -SenTest All you can specify -SenTest MyTestCaseClassName to run just the tests in the test case class MyTestCaseClassName, or -SenTest MyTestCaseClassName/testMethodName to run just a single test.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Update March 17, 2008: I've updated this a bit to handle some changes in the process introduced with Xcode 3.0.

Update September 5, 2008: I've updated this again to cover the changes that were made to bundle injection for Xcode 3.1; the change is the introduction of the XCInjectBundleInto environment variable.

Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-03 06:52 pm (UTC)

Hi Chris

Thanks for posting this guide. It has been very helpful to me in the past but that solution stopped working when I upgraded to Xcode 3.

I get an error stating that the DevToolsBundleInjection cannot be found or loaded. I have DYLD_INSERT_LIBRARIES set to $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection. I haven't changed it when I upgraded to Xcode 3.

My project use Xcode 2.4 compatibility mode and the target SDK is Mac OS X 10.4(Universal) as I dare not switch to Objective-C 2 for the moment for fear that OCUnit or OCMock will stop working entirely.

Would you have an idea about how to solve this by any chance?
thanks
eric

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-05 02:57 pm (UTC)

Hi Eric,

With XCode 3, you can do this in another way:

create a new executable and put the the path to 'otest'
(/Developer/Tools/otest)

Put as argument your unit test bundle name including the octest suffix.
(like MyTests.octest)

The rest can stay 'default'

option (right) click on the executable to run, or run with breakpoint.
This works for me.

Please note that if you have a garbage collection only app (objc 2), the otest binary can't handle that. If so, you need to fetch the OCUnit sources and build it with garbage collection supported (only the otest binary)

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

chanson

2007-11-05 06:55 pm (UTC)

This is not correct; Eric's question was about debugging application unit tests, whereas your instructions are for debugging framework unit tests.

Furthermore, you do not have to download OCUnit from Sen:te and build your own otest from their sources to run tests with garbage collection enabled. You do have to create your own test rig, but you can do that very straightforwardly by just creating a Foundation Tool that links SenTestingKit.framework, sets the SenTestTool user default to YES, and invokes SenSelfTestMain(); both of these are declared in <SenTestingKit/SenTestProbe.h>.

I'll document this in one of my posts on Xcode 3 unit testing.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

chanson

2007-11-05 06:50 pm (UTC)

All of the developer tools - including their frameworks - are self-contained in the Developer folder as of Xcode 2.5 and Xcode 3.0 so that it can be moved and renamed at will, and coexist with other versions of the developer tools. There are two things you need to add to your executable to debug your application unit tests under Xcode 3.0.

First, you need to change your DYLD_INSERT_LIBRARIES to refer to $(DEVELOPER_LIBRARY_DIR) rather than $(SYSTEM_LIBRARY_DIR). (You'd also need to do this step under Xcode 2.5.)

Second, you need to add a new environment variable, DYLD_FALLBACK_FRAMEWORK_PATH, to your executable and set its value to "$(DEVELOPER_LIBRARY_DIR)/Frameworks:$(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks" (without the quotes).

The second step is necessary because in order to support moving and renaming the Developer folder, the Xcode frameworks are linked using a new dyld feature called runpaths and have an install-name that starts with @rpath. Without the DYLD_FALLBACK_FRAMEWORK_PATH setting, Xcode will not necessarily be able to locate SenTestingKit.framework because your application doesn't necessarily define any runpaths that @rpath will resolve to (much less runpaths that contain SenTestingKit.framework).

I'm going to update and re-post the unit testing articles specifically for Xcode 3 at some point in the near future that point out these issues. Keep an eye out for that.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-18 03:21 pm (UTC)

One more thing... for me to get this working on Leopard, I needed to add 'set start-with-shell 0' to my ~./.gdbinit file. Chris is actually the source (via xcode mailing list) of that info, but I figured it would be good to add that here also.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-27 04:44 am (UTC)

Thank you very much for all the help.

I was finally able to get everything working again. In the end, I had to add the DYLD_FALLBACK_FRAMEWORK_PATH as Chris suggested and to add "set start-with-shell 0" to my ~./.gdbinit file as someone else kindly suggested.

I also had to remove $(BUILT_PRODUCTS_DIR) from my XCInjectBundle environment variable. I'm still not sure why though, but it works.

eric