What airline did you use? American is part of the OneWorld airlines as is Qantas so I thought I could use one for the other. But maybe I can't. See I don't even know about that.
You can use AA miles to book flights on other OneWorld airlines, but the availability isn't necessarily the same as if you were to use Qantas airlines to book the Qantas flight. It's just a little safer to get the miles on the airline you actually want to use, if possible.
And yes, BA usually has crazy expensive fees even when you're using miles.