I think there are others in your situation. One part of me would ask, "If you are physically capable, why is it morally OK for others to work and support you?" You are a part of society,...contribute if you can. Another part of me does understand that in some situations, the world around us "disables" us, due to sensory issues, communication difficulties, etc. It's not that we can't do things,...it's the environment that is disabling. Now, if it is an anxiety issue,...there are ways to address this,...self-aware meditation exercises, therapies, and medications. If your "fight-or-flight" isn't under control and isn't down-regulating appropriately, then anxiety is the least of your worries,...it's a physical health risk,...high blood pressure, high glucose, cancers, immune dysfunction, etc.
Sometimes these things are a "mind over matter" exercise. Most neurotypical people would never put themselves through the daily stresses that I put myself through (insane amounts of work at a children's hospital, plus teaching on my "days off"),...but I have a high tolerance for discomfort compared to most. Having said that, all things in moderation,...I don't think my work environment is mentally or physically healthy for anyone, and probably wouldn't push someone else the way I push myself.