Previous Entry Share Next Entry
Xcode: Debugging Cocoa application unit tests
latest
chanson
A couple weeks ago as part of my Unit Testing Series I talked about how to use Xcode to write unit tests for Cocoa frameworks, debug unit tests in Cocoa frameworks, and write unit tests for Cocoa applications. However, I haven't yet described how to debug your unit tests in Objective-C Cocoa applications. I'll take care of that tonight.

After you've set up unit testing in your Cocoa application, debugging your unit tests is similar to debugging them in a Cocoa framework. All you have to do is adjust the arguments and environment variables your application's Executable is configured to use in Xcode. You don't even have to create a new executable.

To start, bring up the Info window for your application's executable (which is its entry in Xcode's Executable smart group). In the Arguments tab, add the argument -SenTest All. This tells the unit testing infrastructure that you want to run all of the unit tests, not just the ones that are built in to the executable. (After all, you don't have any unit tests in your executable itself.)

Now we'll need to engage in a little bit of environment variable magic. When you test an application, what you're really doing is injecting your unit test bundle into the application and telling it to run its tests at its first opportunity. This is accomplished through by telling dyld to insert a private framework, DevToolsBundleInjection.framework, into your application on launch, and telling that framework via an environment variable, XCInjectBundle, the path of the test bundle to inject.

You also have to tell the injection framework itself the full path to the application executable you want to inject the bundle into, via the XCInjectBundleInto environment variable. This is needed to avoid injecting your test bundle into other executables that are run by the application you're testing, or that are run as a side-effect of running the application you're testing. (For example, gdb generally wants to run applications from within a shell, so that environment variables are expanded in its environment and command-line parameters.)

In the Arguments tab of your application executable, first add an environment variable named DYLD_INSERT_LIBRARIES. Set its value to the path to the DevToolsBundleInjection.framework/DevToolsBundleInjection library in the framework of the same name that's included in the developer tools. Prior to Xcode 2.5, this was in $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks but as of Xcode 2.5 and Xcode 3.0, it has been moved to $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks

Then add a second environment variable, XCInjectBundle. Set its value to $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest.

Add a third environment variable, XCInjectBundleInto. Set its value to the full path to your application's executable — not just the application bundle — e.g. $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. This is the debugging equivalent of the Test Host build setting you used to tell Xcode what executable to inject your tests into when running them.

For Xcode 3.0 and later, add a final environment variable, DYLD_FALLBACK_FRAMEWORK_PATH to your executable. Set its value to $(DEVELOPER_LIBRARY_DIR)/Frameworks.

Why do you need to do this? In order to support moving and renaming the Developer folder, all of the frameworks within it — including OCUnit — use runpath search paths. This means that the internal name of the framework, including the one copied into your test bundle, will start with @rpath rather than an absolute path starting with /Developer/Library/Frameworks. Unfortunately this means that your unit tests won't find SenTestingKit.framework without some extra help. That's what DYLD_FALLBACK_FRAMEWORK_PATH does: It tells dyld to try an additional set of directories in place of @rpath when it can't be resolved. (More information on runpath-relative install names can be found in the ld(1) man page.)

Make sure the check marks next to all three of these environment variables — and your -SenTest All argument, of course — are set.

Troubleshooting note: Troubleshooting note: If this doesn't work — that is, if your test bundle isn't found and run — change the executable's working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Now if you choose Run Executable from the Debug menu, your application should launch, you should see the results of executing your unit tests in the Run Log, and as soon as your unit tests are complete your application should quit!

To debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. As with a Cocoa framework, do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test. Debug Executable will work as long as your executable itself is actually present.

Having done all this, you should be stopped at the breakpoint!

Just as any other time you use OCUnit, instead of -SenTest All you can specify -SenTest MyTestCaseClassName to run just the tests in the test case class MyTestCaseClassName, or -SenTest MyTestCaseClassName/testMethodName to run just a single test.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Update March 17, 2008: I've updated this a bit to handle some changes in the process introduced with Xcode 3.0.

Update September 5, 2008: I've updated this again to cover the changes that were made to bundle injection for Xcode 3.1; the change is the introduction of the XCInjectBundleInto environment variable.

Testing for UnitTest?

(Anonymous)

2006-10-31 10:22 pm (UTC)

In my Obj-c program I retrieve some user credentials from the keychain at app startup. I would like to bypass this if the unit test is being run since it is probably being run -unattended- on our continuous build server and no-one is there to click the "Confirm Application Change" dialog from the system.

Is there a way to test if the unit-tests are being run and to then change behaviour like this?

Thanks

Re: Testing for UnitTest?

chanson

2008-06-13 04:26 pm (UTC)

Sorry I didn't see this for so long - for assistance with anything Xcode-related, please ask on the xcode-users mailing list.

You can check whether tests are likely to be run from your application by checking for the existence of a class named SenTest. That won't force tests to be run (unlike checking for SenTestProbe) but NSClassFromString will return Nil if no class with that name is present.

debugging tests...

(Anonymous)

2006-11-01 07:27 pm (UTC)

Hi Chris,

Thanks for the great info on how to get OCUnit integrated and then debug the tests. I'm able to debug but for some reason I had to use $(PROJECT_DIR)/$(BUILT_PRODUCTS_DIR)/MyTestFmk.octest instead of simply $(BUILT_PRODUCTS_DIR). Any idea why that env var would not be the full path but instead be relative?

Thanks again.

Re: debugging tests...

(Anonymous)

2007-03-21 06:43 am (UTC)

You probably left Xcode's build preferences at their default values, so the build products are in a sub directory of your project. Chris probably set his Xcode preferences to put his build directory in a specific place.

Re: debugging tests...

(Anonymous)

2008-07-29 02:45 pm (UTC)

And the secret to debug under Xcode 3.1 instead of 3.0 is?

Re: debugging tests...

(Anonymous)

2008-07-29 02:52 pm (UTC)

http://dwt.schwarz-online.org/cgi-bin/trac.cgi?year=2008&month=5

Debugging Unit Tests in Xcode 3.1 ¶
Since I haven't found this info anywhere in the Blogosphere and had to painfully gather it through a long debugging stare, I'm posting it here:

If you want do debug Unit Tests in Xcode 3.1 you have to do the usual, that is:

Make a custom Executable
Set it's argumet to -SenTest All
Set some Environment Variables
DYLD_INSERT_LIBRARIES to ${DEVELOPER_LIBRARY_DIR}/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection
DYLD_FALLBACK_FRAMEWORK_PATH to $(DEVELOPER_LIBRARY_DIR)/Frameworks
XCInjectBundle to $(BUILT_PRODUCTS_DIR)/InvocationBuilder Tests.octest This is of course where you have enter the name of your test bundle
So far nothing new.

What is new though is that you also set the variable XCInjectBundleInto to point to the executable that is going to load the tests (probably to avoid the 3.0 Bug that would load the tests into the shell that GDB used to start the Application). This should be the same as the $(TEST_HOST) variable that you have to set for the Unit Test bundle. In my case this is $(BUILT_PRODUCTS_DIR)/InvocationBuilder.app/Contents/MacOS/InvocationBuilder. (...)

Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-03 06:52 pm (UTC)

Hi Chris

Thanks for posting this guide. It has been very helpful to me in the past but that solution stopped working when I upgraded to Xcode 3.

I get an error stating that the DevToolsBundleInjection cannot be found or loaded. I have DYLD_INSERT_LIBRARIES set to $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection. I haven't changed it when I upgraded to Xcode 3.

My project use Xcode 2.4 compatibility mode and the target SDK is Mac OS X 10.4(Universal) as I dare not switch to Objective-C 2 for the moment for fear that OCUnit or OCMock will stop working entirely.

Would you have an idea about how to solve this by any chance?
thanks
eric

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-05 02:57 pm (UTC)

Hi Eric,

With XCode 3, you can do this in another way:

create a new executable and put the the path to 'otest'
(/Developer/Tools/otest)

Put as argument your unit test bundle name including the octest suffix.
(like MyTests.octest)

The rest can stay 'default'

option (right) click on the executable to run, or run with breakpoint.
This works for me.

Please note that if you have a garbage collection only app (objc 2), the otest binary can't handle that. If so, you need to fetch the OCUnit sources and build it with garbage collection supported (only the otest binary)

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

chanson

2007-11-05 06:55 pm (UTC)

This is not correct; Eric's question was about debugging application unit tests, whereas your instructions are for debugging framework unit tests.

Furthermore, you do not have to download OCUnit from Sen:te and build your own otest from their sources to run tests with garbage collection enabled. You do have to create your own test rig, but you can do that very straightforwardly by just creating a Foundation Tool that links SenTestingKit.framework, sets the SenTestTool user default to YES, and invokes SenSelfTestMain(); both of these are declared in <SenTestingKit/SenTestProbe.h>.

I'll document this in one of my posts on Xcode 3 unit testing.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

chanson

2007-11-05 06:50 pm (UTC)

All of the developer tools - including their frameworks - are self-contained in the Developer folder as of Xcode 2.5 and Xcode 3.0 so that it can be moved and renamed at will, and coexist with other versions of the developer tools. There are two things you need to add to your executable to debug your application unit tests under Xcode 3.0.

First, you need to change your DYLD_INSERT_LIBRARIES to refer to $(DEVELOPER_LIBRARY_DIR) rather than $(SYSTEM_LIBRARY_DIR). (You'd also need to do this step under Xcode 2.5.)

Second, you need to add a new environment variable, DYLD_FALLBACK_FRAMEWORK_PATH, to your executable and set its value to "$(DEVELOPER_LIBRARY_DIR)/Frameworks:$(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks" (without the quotes).

The second step is necessary because in order to support moving and renaming the Developer folder, the Xcode frameworks are linked using a new dyld feature called runpaths and have an install-name that starts with @rpath. Without the DYLD_FALLBACK_FRAMEWORK_PATH setting, Xcode will not necessarily be able to locate SenTestingKit.framework because your application doesn't necessarily define any runpaths that @rpath will resolve to (much less runpaths that contain SenTestingKit.framework).

I'm going to update and re-post the unit testing articles specifically for Xcode 3 at some point in the near future that point out these issues. Keep an eye out for that.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-18 03:21 pm (UTC)

One more thing... for me to get this working on Leopard, I needed to add 'set start-with-shell 0' to my ~./.gdbinit file. Chris is actually the source (via xcode mailing list) of that info, but I figured it would be good to add that here also.

Re: Worked fine with Xcode 2.4 but not working with Xcode 3

(Anonymous)

2007-11-27 04:44 am (UTC)

Thank you very much for all the help.

I was finally able to get everything working again. In the end, I had to add the DYLD_FALLBACK_FRAMEWORK_PATH as Chris suggested and to add "set start-with-shell 0" to my ~./.gdbinit file as someone else kindly suggested.

I also had to remove $(BUILT_PRODUCTS_DIR) from my XCInjectBundle environment variable. I'm still not sure why though, but it works.

eric

Debug Data.

(Anonymous)

2007-12-04 06:05 am (UTC)

I'm using xcode 3 and I followed the directions described in this Thread and running the tests results in taking breakpoints. However, I'm unable to view values for my variables only the memory addresses. Any idea how to get that working? I'm new to cocoa sorry if this is a stupid question.

Re: Debug Data.

(Anonymous)

2007-12-04 06:07 am (UTC)

Oh and my build configuration is set to debug.

Thanks

(Anonymous)

2008-01-04 11:59 pm (UTC)

This was a great help - thank you!

DO NOT talk about things under NDA!

chanson

2008-04-17 05:43 am (UTC)

I had to delete a couple of comments asking questions about things that are under non-disclosure.

Pay attention to what you agree to when you download pre-release or beta software. I cannot answer questions about — and, in fact, must delete comments mentioning — anything under NDA.

Also, don't just leave yourself completely anonymous. I don't care if you use a one-time pseudonym, but always sign your replies in some fashion. It makes it easier to have a real conversation. Thanks.

Applictions executable?

(Anonymous)

2008-06-13 03:16 pm (UTC)

These steps are followed for the applications executable, and not the otest executable? That's what the instructions say. If that is so, how do I debug my app without running the unit tests?

It's a bit unclear.

Re: Applictions executable?

chanson

2008-06-13 04:29 pm (UTC)

When running tests for an application, otest isn't used. The application itself is run, and the test bundle is injected into it. That's why you need to follow these steps for the application executable when debugging application unit tests.

If you want to debug your application without running tests, all you have to do is turn off the command-line arguments and environment variables you added for debugging your tests. Each argument and environment variable line in the Executable info panel has a checkbox next to it that tells Xcode whether it's used or not — just turn those off, and you'll be back to debugging your app without running the tests.

Again: Don't talk about things under NDA

chanson

2008-06-13 04:30 pm (UTC)

I had to delete another comment asking a question about something under NDA. Sorry, but I can't leave such questions up here or answer them here. I hope everyone understands.

Oh boy

(Anonymous)

2008-11-07 05:50 pm (UTC)

This makes you want to run away from unit testing. Thanks for the write-up, Chris. But this so should be easier.

Easier, you say?

(Anonymous)

2009-01-27 12:18 am (UTC)

Thanks so much Chris, your posts were life-savers. Here's an applescript to set up an xcode project for unit testing and debugging those tests, following your procedure. I had to remove the comments for length restrictions. The full script can be found at http://clipperadams.com/programming/xcode_unit_test_applescript.html.

(*
To debug the unit tests, in the "General" tab of the executable info window, the "Working Directory" must be set to "Build Products directory"
Modify the testFrameworkPath for your system
*)
set testFrameworkPath to "/Developer/Library/Frameworks/SenTestingKit.framework"

tell application "Xcode"
tell project of active project document
set projectName to name
set executablePath to "$(BUILT_PRODUCTS_DIR)/" & projectName & ".app/Contents/MacOS/" & projectName
tell executable name
make new launch argument with properties {name:"-SenTest All", active:true}
make new environment variable with properties {name:"DYLD_FALLBACK_FRAMEWORK_PATH", value:"$(DEVELOPER_LIBRARY_DIR)/Frameworks", active:true}
make new environment variable with properties {name:"XCInjectBundleInto", value:executablePath, active:true}
make new environment variable with properties {name:"XCInjectBundle", value:"Unit Tests.octest", active:true}
make new environment variable with properties {name:"DYLD_INSERT_LIBRARIES", value:"$(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection", active:true}
end tell
set unitTestTemplate to target template "Cocoa/Unit Test Bundle"
make new target at end of targets with data unitTestTemplate with properties {name:"Unit Tests"}
tell root group
make new group with properties {name:"Test Cases"} at beginning of groups
end tell
tell group "Linked Frameworks"
set testFrameworkFile to make new file reference with properties {full path:testFrameworkPath, name:"SenTestingKit.framework"}
end tell
add testFrameworkFile to (get link binary with libraries phase of target "Unit Tests")
set value of build setting "BUNDLE_LOADER" of build configuration "Debug" of target "Unit Tests" to executablePath
set value of build setting "TEST_HOST" of build configuration "Debug" of target "Unit Tests" to "$(BUNDLE_LOADER)"
set active target to target "Unit Tests"
end tell
end tell
activate application "Xcode"
tell application "System Events"
tell application process "Xcode"
tell outline 1 of scroll area 1 of splitter group 1 of group 1 of window projectName
tell (first row whose value of text field 1 of group 1 = "Targets")
set targetGroupDisclosureTriangle to UI element 1 of group 1
if value of targetGroupDisclosureTriangle = 0 then
perform action "AXPress" of targetGroupDisclosureTriangle
end if
end tell
select (first row whose value of text field 1 of group 1 = "Unit Tests")
end tell
perform action "AXPress" of menu item "Get Info" of menu 1 of menu bar item "File" of menu bar 1
tell window "Target “Unit Tests” Info"
click radio button "General" of tab group 1
perform action "AXPress" of button 1 of splitter group 1 of tab group 1
tell sheet 1
select (first row of outline 1 of scroll area 1 whose value of text field 1 = projectName)
click button "Add Target"
end tell
click button 1
end tell
end tell
end tell

Debugging doesn't work in Xcode 3.1

(Anonymous)

2009-03-18 10:32 am (UTC)

Hello,

You say 'choose' Debug Executable' from the 'Debug' menu.

I haven't go a Debug menu nor a Debug Executable item Xcode 3.1, what should I choose?

Thanks

Re: Debugging doesn't work in Xcode 3.1

(Anonymous)

2009-03-23 10:59 am (UTC)

I have the same problem. I'm trying to debug an iPhone unit test using XCode 3.1, but the Chris' procedure doesn't work.

Can you help me?

Access to application classes on the iPhone

(Anonymous)

2009-04-09 09:40 pm (UTC)

I followed all of this and my simple tests run fine. However using actual application classes in my tests results in link errors when building the test target. Any idea what I could be doing wrong?

XCode 3.1

(Anonymous)

2009-05-15 11:54 pm (UTC)

Great article. I had some problems getting it to work. I got to the point that SenTest would run, but wouldn't execute my test cases (only ran the 'SenInterfaceTestCase'). After some search, I stumbled over this on ADC http://developer.apple.com/mac/articles/tools/unittestingwithxcode3.html. Basically, the same, except it showed XCInjectBundle and XCInjectBundleInto without $(BUILT_PRODUCTS_DIR). The moment I removed the variable, everything worked like a charm.

If you can't find the Debug menu (for example using Xcode 3.1), select the executable *target*, and click on "Build and Debug".

Bundle injection with iPhone Simulator

(Anonymous)

2009-09-28 02:30 am (UTC)

Hi!

Has anyone figured how to achieve bundle injection in the context of the iPhone Simulator? After lot of experimentation and thinking, I realized that iPhone apps were not run and debugged from the $(BUILT_PRODUCTS_DIR), but instead from a temporary directory with a GUID that is unique every time! For example, using Activity Monitor, I was able to see the actual path of my running application:

/Users/username/Library/Application Support/iPhone Simulator/User/Applications/F9F1E9A6-EB8D-4B1C-94F5-6AD7B5BFE6AE/MyApp.app/MyApp

No wonder bundle injection does not occur, as it is filtered according to the built products directory copy of the app, which doesn't match the actual running app path.

Any clue anyone?

Regards,
Mathieu

Re: Bundle injection with iPhone Simulator

(Anonymous)

2010-05-02 09:22 pm (UTC)

I am not able to run application unit tests on xcode 3.2.2 following the above steps. Are these steps worked for anyone working on iphone simulator?

Re: Bundle injection with iPhone Simulator

(Anonymous)

2010-05-03 01:32 am (UTC)

To give more info -
I have tried with below set of options but none of the breakpoints set in unit tests are hit. Can some one please share their settings for iphone simulator.

-SenTest All
DYLD_FALLBACK_FRAMEWORK_PATH - $(DEVELOPER_LIBRARY_DIR)/Frameworks
DYLD_INSERT_LIBRARIES - $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection
XCInjectBundle - UnitTests.octest
XCInjectBundleInto - MyApp.app/MyApp


-SenTest All
DYLD_FALLBACK_FRAMEWORK_PATH - $(DEVELOPER_LIBRARY_DIR)/Frameworks
DYLD_INSERT_LIBRARIES - $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection
XCInjectBundle - $(BUILT_PRODUCTS_DIR)/UnitTests.octest
XCInjectBundleInto - $(BUILT_PRODUCTS_DIR)/MyApp.app/MyApp


Still works

(Anonymous)

2010-12-31 08:45 pm (UTC)

As of Xcode 3.2.5, I can confirm that these instructions still work.

Thanks!

I am using this to set up unit-testing of simple programs to teach myself some Objective-C. It works as expected. Thank you.

Thats really good...

Exceptional...!!