I would just talk to the recruiter. They may have some ideas or suggestions, or might be able to tell the hiring manager/interviewers to expect you to dress casually because you work for the other company. I am sure they'd be understanding that it's a weird spot for you to be in given the location and help you figure out how to impress their hiring managers while keeping your current job secure.
I wonder if they ever grant temporary visitor badges - maybe there is a way they could leave the badge with someone in the lobby (assuming there is security) and you could let yourself onto the floor. Or somehow otherwise send it to you.
Meeting outside of the building is also a great idea
You couldn't change in the ladies' room and sneak onto the elevator without someone seeing you?
OR, barring that -- claim you're going to traffic court to fight a ticket. Dress nice, leave early for the day, interview on your way out.
No way to change & sneak - especially since I'd need one of their employees to grant me access to their floor. (Typically people meet in the lobby with the interviewer and go up with them.)
Might be able to use that as an excuse for dressing nice - would still need to figure out how to get on their floor with no coworkers seeing me - ha.
SO COMPLICATED.
If you do it on the way out, would someone notice if you took the elevator to the other floor vs to the ground floor/exit?