FTR, my H is allergic to cats but when we found ours under our back deck and brought them in at 4 weeks old, he was able to get used to them as they gradually developed dander. He loves to cuddle with them and is fine. He takes a daily allergy medication but he was on that for non-pet allergies before we found the kitties.
If you can't keep him, I hope you find a good home for him. I highly suggest contacting the SPCA as they will be able to insure the home is a good one.