We have 0 RTO expectations and never plan to at my company, but we DO have 3 physical locations. We are asked to let admins know when we'll be onsite, partially so they can ensure we have access and/or someone else will be there, and probably also for safety reasons.
We also have work chats, including one thread about who is going into each office. That's not a mandate, it's just so you can either let people know if you'll be in with a client (and therefore will be reserving the conference room, or need specific space, or want the office to look busy or whatever) or just so you can possibly go in on days other people will be there. Tues/Thurs tend to be popular, with Weds also bringing in people sometimes. Many people never go in, some people go in twice a week, voluntarily.
SO my suggestion is what others have said - email the team and remind them of the policy now that summer PTO is coming to an end, and that you had agreed Tuesday was good for the team, but if things have changed and they prefer another day, you can discuss what works best. If they blow it off, I'd just say one-on-one that they should let you know in advance if possible if they won't be coming in.
As for your boss, deny if anyone confronts you. "Hey PSU, have you seen Frank around here on Tuesdays?" "Frank? No. AFAIK he hasn't been coming in on Tuesdays. Why?" Never tell anyone you know about the badge swiping.
I asked my H about the badge swiping thing last night and he was mortified. He said he would absolutely report it, it would be a huge issue at his company and both boss and wife would be fired. He said it would be an ethics violation if he didn't report it.