Internal auditors seem to get a lot of the same push-back from people they are auditing, that one does as an internal auditor. It might be different in a bank, but at the insurance company there seemed to be a fair amount of resentful feelings about people "watching" them doing their work, or "checking up" on them.
I fully confess this is not what I do, but what I do puts me in often daily communication with internal audit (public companies, I'm on the legal side) and this is how they describe it to me. That said, they have good work hours generally (except for quarter and year end filings) and most left external audit for the extensive travel reasons actualy. Good luck!