Hi Lula.
I know exactly how you feel, it feels confusing, scary and frustrating, well that's how I feel with my current situation.
I have two wonderful boisterous Doodles I've shared my life with since pups - they are 5 and 7 years now and the bond (connection) between us is massive. They are my entire life since my husband died 5 years ago. We are together 24 hours 7 days a week and we greatly rely on each other, even though at times they are far too noisy, far too smelly, far too messy, far too boisterous, and far too demanding. But I would never let them go, never.
On the other hand, 6 weeks ago I found a tiny young cat that had been abandoned in a local wood, next to it's discarded cat carrier - he was emaciated, cold, wet, terrified and in a dreadful way. I just couldn't leave him there, (hubby and I always kept cats) and I still have a huge passion for them. BUT, even though he is now plump, happy and funny, and playful, and my Doodles love him, him being a cat means he's very destructive, quite smelly at times, noisy and demands my attention all the time - sort of invades my space if you know what I mean, I am therefore quite anxious about him being in my house and I am still unsettled.
I think you need to do what's best for you, whatever that decision is, try not to feel bad, let your husband support you. I feel either decision will leave it's scars.
Me, I am keeping the little puss cat, I am quite fond of him, but (like you) I am finding it hard to bond/connect. I am always cuddling my Doodles and have all the affection I need from them - they are my best friends.
I do cuddle and play with the kitty (he's just out of kittenhood)- he has allotted times twice a day (he has complete freedom of my house, apart from during the night) - but it's sort of forced. I know intellectually that's what he needs and wants, but I am not feeling it emotionally - at the moment.
I know he would not have survived much longer outdoors, so I know I made the right decision for him, and he is thriving now.
Life has far too many hard decisions and sometimes, we are damned if we do and damned if we don't.
Good luck, with your decision, I hope all goes well.