I'm guessing that you don't have a job that you could disappear for 20 minutes without being noticed? Because I would probably just take a late lunch in my car for the phone interview.
If you want to take off for it, I would try to come up with an excuse that could reasonably require a follow up a week or so later, which you could use for the in person interview.