If the takedown lever locking block is all the way up (engaged), scratch a vertical line on the round (unbroken) end before trying to move it...You want to make sure it rotates a full 90�º before trying to drift it out...but you should be able to see it come down into the notch...another pic attached...
Another idea - If the broken end of the takedown lever is covering this notch, and you have a Dremel, carefully cut it off so you can see where/when it is positioned correctly...(cut about 95% through and then snap it off so you don't go through and hit the frame)...
If all else fails, I'd drill it almost all the way out and take it out in pieces...