At the moment you don't sound like you're on the wrong side of him and you're likely to just be a side thing that isn't too important to him, in fact he's probably got a few things on the go and he was most likely just busy when he didn't answer his phone. He might have left the shirts on purpose as an excuse to contact you because he is trying to take advantage of you, a lot of autistic people including myself can have "sucker" marked on our heads, I've have learnt the hard way a few times in my life. If you don't fall for any crap and ignore him as much as possible he will most probably just move onto someone else and other things, don't even try to call him and if he calls you don't be so keen to answer quickly, maybe even miss the odd call yourself. Also if you can post the shirts do it.
He's unlikely to come over especially to harm you, but if you feel safer keep a mobile phone near you so you can call 911 if you ever did feel threatened (assuming you live in the USA), they will trace the location of your phone if it became necessary. Just the thought of the cops coming would almost certainly frighten him off anyway.