Previous Entry Add to Memories Share Next Entry
Xcode: Debugging Cocoa framework unit tests
latest
chanson
So you've set up unit testing for your Objective-C Cocoa framework and it's been working great. But now you've written a test and it fails, and you can't figure out why. It's time to break out the debugger, but how, exactly, do you do that? Your unit tests are built as a bundle, and you can't run a bundle.

It's simple. All you have to do is set up an appropriate Executable in Xcode to run the test rig that runs your bundle, and then debug that. To get started, choose the Project > New Custom Executable menu item. For its name, specify otest — this is the name of the test rig used by OCUnit. Specify /Developer/Tools/otest as the path to your tool.

When you add the custom executable, Xcode will bring up its info window. Switch to the Arguments tab. Here you'll need to enter some command-line arguments to pass to the test rig and some environment variables to affect how it's run.

First, add the argument -SenTest Self to your executable. This tells otest to run all of the tests in your bundle. (It's actually a pair of arguments, but you can add it as one as a convenience.) Then for your second argument specify $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest where MyTestBundle is the name of your test bundle. This tells otest the path of the test bundle to load; $(BUILT_PRODUCTS_DIR) will be expanded to the appropriate build products directory for your project at run time. (If you get the order wrong, just drag the arguments around in the table.)

Troubleshooting note: If this doesn't work — that is, if otest complains it can't find your test bundle — change the executable's working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Next add a pair of environment variables named DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your executable. These will tell the loader to check your build products directory first for frameworks and libraries whenever the executable is run from within Xcode. Specify $(BUILT_PRODUCTS_DIR) for the value of each variable.

Finally, from the Project > Set Active Executable menu choose your new otest executable. This way any time you run or debug within Xcode, it will run otest with the arguments and environment you've specified.

To actually debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. Do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test; Debug Executable doesn't try to build first, it only cares whether an executable is present.

You should successfully stop at your breakpoint!

If your tests take a long time to run — they shouldn't, they're unit tests after all, but it can still happen — you may want to just run the tests for one test case, or just one individual test. This is easy too. Rather than specifying -SenTest Self in your arguments to otest, you can specify -SenTest MyTestCaseClassName to run the all the tests in the specified test case. To run just a single test, use -SenTest MyTestCaseClassName/testMethodName instead.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Kind of complicated

ejalbert

2005-07-24 08:16 am (UTC)

Surely y'all can make this easier. :) Improving this would be a great feature for a future Xcode release.

Another approach is to create a Cocoa (Shell Tool) Target and inject something like the following ...

int main (int argc, const char * argv[])
{

NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

[SenTestObserver class];

SenTestSuite *suite= [SenTestSuite testSuiteForTestCaseClass:[LogTest class]];
[suite run];

[pool release];
return 0;
}

Note the passing of your test class instead of 'LogTest' should be the only changes you need to make. Works like a charm.

best regards,
Richard Long


How does the Unit Test bundle come into the shell tool?

Thanks. You saved me an hour and a fistful of hair.

I need to retweet this to my twitter.

Thanks Chris!

(Anonymous)

2006-04-18 11:01 pm (UTC)

I searched google for a solution and found it in a blog I read. :-)

Didn't work in XCode 2.2

(Anonymous)

2006-04-27 06:07 am (UTC)

To make this work in XCode 2.2 instead of having $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest you need ${BUILT_PRODUCTS_DIR}/${TARGET_NAME}.${WRAPPER_EXTENSION}

thanks for the help !
LeibowitzN

Re: Didn't work in XCode 2.2

(Anonymous)

2008-11-29 07:50 am (UTC)

Use $PRODUCT_NAME instead of $TARGET_NAME; your target can be named something different from the product it builds, which is useful if, for example, you want to build a framework, test bundle, and application all with the same basic name (but different extensions, obviously).

oops, disregard the above comment

(Anonymous)

2006-04-27 06:13 am (UTC)

That wasn't the real problem. Don't know exactly why it doesn't work for me.

otest has exited with status -1

edibiase

2006-11-06 12:55 am (UTC)

Thanks for the awesome guide! It's a bunch of concise information that I was only getting half of on my other Google searches.

The only issue I'm running in to is that I don't seem to be stopping at my breakpoint :-) Here's the output from the Run Log:
[Session started at 2006-11-05 19:21:13 -0500.]
2006-11-05 19:21:13.588 otest[4102] +[NSATSGlyphGenerator initialize] invocation.  The class is deprecated.
Test Suite 'Test.octest(Tests)' started at 2006-11-05 19:21:13 -0500
Test Suite 'EDAtomEntryTest' started at 2006-11-05 19:21:13 -0500

otest has exited with status -1.


The bottom of the debug window reads, "otest existed with status (1). The Debugger is still running. Use 'Restart' to debug again."

Any help on what I can do to resolve this would be greatly appreciated!

Re: otest has exited with status -1

edibiase

2006-11-06 01:54 am (UTC)

Hrm, nevermind. I was using a bastard form of the framework tests, and switching to an actual framework and following your instructions seems to have worked out, with the exception of changing the second argument on the otest executable to just be MyTestBundle.octest, which prevents otest from complaining, "launch path not accessible".

Re: otest has exited with status -1

chanson

2006-11-06 04:28 am (UTC)

Glad I could help! ;)

you might also need to set OBJC_DISABLE_GC to 'YES' in the executable's environment in more recent versions of Xcode

and great article, too, btw :-)

Also select the right arch

(Anonymous)

2008-12-10 04:03 pm (UTC)

Nowadays you might also have to select the arch you run otest for,
so that it matches your test bundle.

Replace /Developer/Tools/otest with /usr/bin/arch in your Xcode
custom executable, and insert the arguments -arch, i386 and
/Developer/Tools/otest and you should be good to go.

(use something other than i386 if needed of course).



Re: Also select the right arch

chanson

2008-12-11 10:13 am (UTC)

Don't do that. You will be debugging the invocation of arch that's invoking otest then, not debugging otest itself.

Instead, use the Active Architecture submenu of the Project menu to specify the architecture you want to debug your tests for. Note that you don't run your tests after you build, you run them by building; you debug them using an executable.

I mention this because Xcode 3.0-3.1.2 have a bug whereby if you set up an otest executable and attempt to run it, it will always use the architecture of otest that matches your machine rather than the Active Architecture you've set. However, if you attempt to debug your otest executable, it will work fine. A number of people try to run their executable first "to make sure it works" before debugging; this fails and then get confused. Don't bother trying to run, just debug. Your build runs your tests.

GC capability mismatch

(Anonymous)

2009-01-23 03:13 pm (UTC)

The full error is as follows:
2009-01-23 16:06:42.213 otest[76051:813] Error loading /Users/01oC/Desktop/EE/cxn/build/Debug/UnitTEST.octest/Contents/MacOS/UnitTEST:  dlopen(/Users/01oC/Desktop/EE/cxn/build/Debug/UnitTEST.octest/Contents/MacOS/UnitTEST, 265): no suitable image found.  Did find:
	/Users/01oC/Desktop/EE/cxn/build/Debug/UnitTEST.octest/Contents/MacOS/UnitTEST: GC capability mismatch
2009-01-23 16:06:42.238 otest[76053:203] *** NSTask: Task create for path '/Users/01oC/Desktop/EE/cxn/build/Debug/UnitTEST.octest/Contents/MacOS/UnitTEST' failed: 8, "Exec format error".  Terminating temporary process.
2009-01-23 16:06:42.241 otest[76053:203] Usage: otest [-SenTest Self | All | None | <TestCaseClassName/testMethodName>] 
2009-01-23 16:06:42.243 otest[76053:203] *** -[NSConcreteTask terminationStatus]: task not launched
NOTE: I also didn't include $(BUILT_PRODUCTS_DIR) in my -SenTest Self switch Any advice? I *DO* use garbage collection!

Re: GC capability mismatch

(Anonymous)

2009-01-23 04:13 pm (UTC)

I'll answer my own question...

"If you’re seeing “GC capability mismatch” and your code is build to require GC, you probably need to update your version of Xcode (3.1.1 comes with an otest that does work with GC)" -- alastairs-place.net

Can't debug test cases in XCode 3.1

(Anonymous)

2009-01-25 11:40 am (UTC)

Hi,
my project used to run ok with XCode 3.0. But I've updated to version 3.1 and I cannot debug the test cases any more. I get this error: "launch path not accessible". I have redone the configuration several times alredady, and still get the same :(

The environment variables are set to:
DYLD_INSERT_LIBRARIES = /Developer/Library/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection

XCInjectBundle = build/Debug/UnitTesting.octest

XCInjectBundleInto = $(BUILT_PRODUCTS_DIR)/TestApplication.app/Contents/MacOS/TestApplication

DYLD_FALLBACK_FRAMEWORK_PATH = $(DEVELOPER_LIBRARY_DIR)/Frameworks

OBJC_DISABLE_GC = YES

What is it that I'm doing wrong? Why is it so complicated to have the debugger connected to the test cases in XCode3.1? The same thing is so very simple in Java!!

Chris, I really appreciate your article! but I have to admit that for newbies like me, it's a bit hard to read when all the pieces are separate. Maybe you could put all the conclusions of the comments in ONE single piece of text?? And paste some screenshots?

Finally, WHY Apple is not documenting this properly themselves??

Thanks and best regards,
Gabriel


Re: Can't debug test cases in XCode 3.1

(Anonymous)

2009-01-26 12:00 am (UTC)

Hi again,
in my previous post I was quite frustrated (I couldn't debug test cases in XCode3.1). So I decided to start all over again and created a new project from scratch. The issues are fixed now. Unfortunately, I lack the knowledge to understand what was wrong...

So I'd like to share the solution; hopefully this helps others in the same situation.

To debug test cases using XCode 3.1.2 (my current setup), I followed the explanations available here:

http://developer.apple.com/mac/articles/tools/unittestingwithxcode3.html

And it works! But be careful when configuring the Unit test target: in the "Build" tab, you have to configure the value of "Test Host" and "Bundle Loader" TWICE:

1) In the "Configuration" drop down list, select "Release" and configure the value of "Test Host" and "Bundle Loader" as indicated in the article

2) In the "Configuration" drop down list, select "Debug" and configure again the value of "Test Host" and "Bundle Loader" as indicated in the article.

And that's it. Now you can use breakpoints and debug as usual and also run the test cases in release mode.

Best regards,
Gabriel



Re: Can't debug test cases in XCode 3.1

(Anonymous)

2009-02-07 12:22 pm (UTC)

Something tells me this pertains to your other blog entry, no?
http://chanson.livejournal.com/119097.html

problems debugging framework

mrcaron.myopenid.com

2009-03-03 02:03 am (UTC)

It seems that otest and my bundle are built for different architectures? Using XCode 3.1.2 and following the instructions for setting up a framework with a unit test bundle, then setting up otest as the debug executable, I get the following output from the debugger. It would seem that the framework was built with the wrong architecture? But my Architectures setting for the bundle and for the Framework are the same. Any ideas out there?

2009-03-02 19:56:03.414 otest[28059:10b] Error loading /Volumes/Biggie/Development/SonoPlot/DXFReader/build/Debug/Unit Tests.octest/Contents/MacOS/Unit Tests: dlopen(/Volumes/Biggie/Development/SonoPlot/DXFReader/build/Debug/Unit Tests.octest/Contents/MacOS/Unit Tests, 265): no suitable image found. Did find:
/Volumes/Biggie/Development/SonoPlot/DXFReader/build/Debug/Unit Tests.octest/Contents/MacOS/Unit Tests: mach-o, but wrong architecture
2009-03-02 19:56:03.561 otest[28059:10b] The test bundle at /Volumes/Biggie/Development/SonoPlot/DXFReader/build/Debug/Unit Tests.octest could not be loaded because it is built for a different architecture than the currently-running test rig (which is running as unknown).
2009-03-02 19:56:03.568 otest[28060:203] *** NSTask: Task create for path '/Volumes/Biggie/Development/SonoPlot/DXFReader/build/Debug/Unit Tests.octest/Contents/MacOS/Unit Tests' failed: 8, "Exec format error". Terminating temporary process.

Xcode 3.1.2

(Anonymous)

2009-04-09 07:35 pm (UTC)

I finally got this working in Xcode 3.1.2, but my settings ended up being a little different than what was detailed in the original article. My steps were:


  1. Project -> New Custom Executable

    1. Type otest for the executable name.

    2. Type /Developer/Tools/otest for the executable path.

    3. Press the "Next" button.

  2. Double click on the "otest" executable.

    1. Under "General" nothing should need to be changed.

    2. Add the following arguments (in order):
      -SenTest all
      "Unit Tests.octest"
      

      Note: If there is a space in the name of your bundle, put quotation marks around it.

    3. Add the following environment variable:
      OBJC_DISABLE_GC = YES | NO (depends on your project settings.)
      


That's it! Just do the separate build and debug steps as mentioned above, and breakpoints will be active.

Xcode 3.1.3

(Anonymous)

2009-04-26 08:59 am (UTC)

I've followed the example for Xcode 3.1.2 but for IPhone tests on IPhone OS 3. Whenever I run the otest with build and debug I receive the following error.



[Session started at 2009-04-26 20:44:25 +1200.]
2009-04-26 20:44:25.353 otest[11711:10b] Error loading /Users/todd/iPhone/Projects/AffiPhone/build/Debug-iphonesimulator/Tests.octest/Tests: dlopen(/Users/todd/iPhone/Projects/AffiPhone/build/Debug-iphonesimulator/Tests.octest/Tests, 265): no suitable image found. Did find:
/Users/todd/iPhone/Projects/AffiPhone/build/Debug-iphonesimulator/Tests.octest/Tests: mach-o, but wrong architecture
2009-04-26 20:44:25.361 otest[11711:10b] The test bundle at Tests.octest could not be loaded because it is built for a different architecture than the currently-running test rig (which is running as unknown).
2009-04-26 20:44:25.366 otest[11712:203] *** NSTask: Task create for path '/Users/todd/iPhone/Projects/AffiPhone/build/Debug-iphonesimulator/Tests.octest/Tests' failed: 8, "Exec format error". Terminating temporary process.

The Debugger has exited with status 5.The Debugger has exited with status 5.


Given that the build architecture is armv6, how do I set otest to run in that architecture (if possible)? Note that I've set garbage collection to off.


no luck debugging unit tests

(Anonymous)

2009-06-19 04:41 pm (UTC)

Anyone having any luck actually debugging unit tests in Xcode 3.1.3? I've successfully setup unit testing but I still can't figure out how to debug them. Really annoying actually but you can 'work around' this by copying and pasting your unit test into a UIApplicationDelegate but that's a pita.

Any additional help here would be appreciate.

My 2 cents

(Anonymous)

2009-07-24 04:22 am (UTC)

Hi,

As an Obj-C newcomer from the C#/.NET world, I spent quite some time (and sleep) getting OCUnit to trace in debug mode with my iPhone 3.0 project. Since web pages dealing with that topic are scarce, I thought I would contribute a few hints to spare other people's time.

Depending on what I tried, I was either getting the GC mismatch error, the architecture mismatch or the launch path not accessible. Here is what (I think) got me out of that crazy loop :

- Some posts repeatedly misspell the OBJC_DISABLE_GC environment variable (mistakenly spelled "OBJ_DISABLE_GC") which initially got me on the wrong path.

- Not sure, but I think that the only way I could get rid of the "launch path not accessible" error was with the following page, especially step 2 (I think) : http://www.sente.ch/s/?p=535&lang=en

- In step 2 of the above, I used the SenTestKit.framework from the iPhone Simulator SDK 3.0, instead of the specified 2.2.

- (If developing for iPhone:) Make sure to select the Active SDK to be "iPhone Simulator 3.0" (because I tried so many combinations everywhere, at some point it was not working just because of that being incorrecly set! So all planets must be aligned, otherwise no magic! ;-)

- For the architecture mismatch problem, look no further than Chris' explanation: only use "Debug" with otest, not "Run".

That's pretty much what I can think of.

Good luck everybody and let's hope that process will get simpler in upcoming versions of Xcode, as I consider debugging unit tests to be a must!

Regards,
Mathieu


EXC_BAD_ACCESS

(Anonymous)

2009-07-25 01:37 am (UTC)

When I build without the new otest executable, my unit tests run just fine, but when I follow the instructions here, the debugger outputs:

Test Suite 'build/Debug/UnitTests.octest(Tests)' started at 2009-07-24 21:31:45 -0400
Test Suite 'MyTests' started at 2009-07-24 21:31:45 -0400
Program received signal: “EXC_BAD_ACCESS”.

Stepping through the unit test, I find that it fails on:

MyObject *obj = [[MyObject alloc] init];

Where MyObject is a class in my framework project. What might I have forgotten to do? I'm new to XCode and scratchin' my head over it. In the meantime, I can go back to NSLog statements. Ugh.

Re: EXC_BAD_ACCESS

(Anonymous)

2009-07-25 01:48 am (UTC)

P.S., XCode 3.1.3

- Brian

Re: EXC_BAD_ACCESS

(Anonymous)

2009-08-12 04:56 am (UTC)

Never mind. Newbie mistake. I didn't realize there was a difference between a framework and an application, and didn't look closely at the title of your post. I was trying to debug an application. Your post entitled Debugging Cocoa Application unit tests worked perfectly.

working with XCode 3.2.1 ?

(Anonymous)

2009-11-24 09:37 am (UTC)

I am getting errors with XCode 3.2.1:

The test bundle at Unit Tests.octest could not be loaded because its Objective-C runtime information does not match the runtime information required by the test rig. This is likely because the test rig is being run with Objective-C garbage collection disabled, but the test bundle requires Objective-C garbage collection. To enable Objective-C garbage collection for the test rig, run it in an environment without the OBJC_DISABLE_GC environment variable.

NSTask: Task create for path 'My/Path' failed: 22, "Invalid argument". Terminating temporary process.

Anyone have an idea? I really want to debug my unit tests :)

Re: working with XCode 3.2.1 ?

(Anonymous)

2009-11-29 10:40 pm (UTC)

I got it working with instructions from here:
http://www.grokkingcocoa.com/how%5Fto%5Fdebug%5Fiphone%5Funit%5Fte.html

Re: working with XCode 3.2.1 ?

(Anonymous)

2009-12-23 02:10 am (UTC)

I'm following the same tutorial but it doesn't work for me.
I have the error:

The test bundle at LogicTests.octest could not be loaded because an unanticipated error occurred: Error Domain=NSCocoaErrorDomain Code=3587 UserInfo=0x10bd90 "The bundle “LogicTests.octest” couldn’t be loaded because it is damaged or missing necessary resources." (dlopen_preflight(/Users/me/test/build/Debug-iphonesimulator/LogicTests.octest/LogicTests): Library not loaded: /System/Library/Frameworks/UIKit.framework/UIKit
Referenced from: /Users/me/test/build/Debug-iphonesimulator/LogicTests.octest/LogicTests
Reason: image not found)


Any ideas?

not working on xcode 3.2

(Anonymous)

2009-11-25 05:31 pm (UTC)

I'm using xcode 3.2 but its not possible to debug test cases. if I click build and then debug -> nothing happens!

btw. ridiculous how complicated this is in xcode ...

works, but still one problem

(Anonymous)

2010-02-15 01:40 am (UTC)

Chris,

Great post. Everything worked as advertised. But I still have one annoying problem. The "otest" executable doesn't track changes to the TestCase classes and doesn't force a rebuild of the test bundle if needed before execution. So right now I have to first build the unit test bundle, and then in a separate step execute the otest to run/debug the tests.

Is there some way I can set up the unit test code as dependencies of the otest executable?

thanks.

Brian King

(Anonymous)

2010-05-11 08:21 pm (UTC)

I'm getting competing error messages! First I run it:

2010-05-11 15:57:22.312 otest[30594:a0f] The test bundle at RailsCacheTests.octest could not be loaded because its Objective-C runtime information does not match the runtime information required by the test rig. This is likely because the test rig is being run with Objective-C garbage collection enabled, but the test bundle does not support Objective-C garbage collection. To disable Objective-C garbage collection for the test rig, run it in an environment with the OBJC_DISABLE_GC environment variable set to YES.

So I do as described and I get:


objc[30626]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
2010-05-11 15:58:07.919 otest[30626:a0f] The test bundle at RailsCacheTests.octest could not be loaded because its Objective-C runtime information does not match the runtime information required by the test rig. This is likely because the test rig is being run with Objective-C garbage collection disabled, but the test bundle requires Objective-C garbage collection. To enable Objective-C garbage collection for the test rig, run it in an environment without the OBJC_DISABLE_GC environment variable.



Any thoughts? This is with XCode 3.2.2

Still struggling with XCode3.2.2 and debugging ocunit tests

(Anonymous)

2010-06-14 02:02 pm (UTC)

Anyone have any luck with this? I am also seeing the 'test bundle could not be loaded' because of a mismatch. I've set OBJC_DISABLE_GC to YES and NO and no avail!

Help! [Please drop me a note at cwoloszynski@gmail.com if you get this working]


I suggest using GH testing for iphone

(Anonymous)

2010-09-18 10:40 pm (UTC)

It looks to me like the best one can expect for debugging unit tests on iphone is tracing, which is just laughable. This dude has written a nice testing framework which runs on the device (or sim) and therefore provides debugging, without any of this crazy setup (which doesn't seem to even work correctly on iphone). URL is : http://github.com/gabriel/gh-unit

If I am wrong about this, then please correct me, but unless someone on the internet somewhere posts an xcode iphone project that I can download with breakpoints working in the unit tests my reaction is just "meh: Octest sucks for iphone use GH-unit instead". As they say "photos or it never happened" Not seen a single example of this after hours of trying and searching...

Peace, and great article by the way, thoroughly enjoy your blog.

Excellent Article

(Anonymous)

2011-01-09 02:42 am (UTC)

Chris,

I appreciate you taking the time to a) figure this out and, b) write it up.

I've successfully used your setup instructions on a framework in XCode 3.2.5

Thanks again,
Frank

Is there a way to debug iOS OCUnit tests??

Hiedi Utley

2011-01-13 10:12 pm (UTC)

I followed your instructions but can't seem to get this to work for iOS OCUnit tests in Xcode 3. There isn't a "debug executable" option on my IDE and neither run, run/debug, or debug seem to work. Any help would be appreciated!