According to forum reports the iPhone DFU mode, which is required to install custom firmware with PwnageTool, does not get detected properly after updating Mac OS X to 10.5.6.
That means you cannot Jailbreak or Unlock the iPhone currently. The iPhone Dev Team has investigated in to this issue and comes to the conclusion that this is not a countermeasure by Apple, but a bug in the new kernel.
The Phone Dev Team names possible fixes for 10.5.6. As always: you do this at your own risk!
1. Replace the following plugin kexts from within IOUSBFamily.kext with the ones from 10.5.5 and then rebuild kextcache (if you don’t understand this, then you shouldn’t attempt it!)
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBHub.kext
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBCompositeDriver.kext
2. Use a USB hub in-between the DFU device and the Mac and insert/reinsert the iPhone’s USB cable.
3. Use a PwnageTool created .ipsw on Windows! Oh the irony!
If you did not update yet to 10.5.6 better don’t do it and wait for a fix or for 10.5.7 to be released.