Noo ooby, you are not an idiot! It's fine, we all have our bad days.. maybe you are just a bit tired, I was scared to go to pick up some text books the other day! I felt ridiculous, I still don't know what that barrier was that told me to stay put.
I thought back to what I could have done to get myself there, and came up with, (may sound a bit daft) but try putting yourself in someone elses shoes, maybe it could be someone you look up to and then tell yourself how would they act in this situation? There's a point when we need
our comfort zone, and a point when we so deperatley want
that comfort zone but we need to recognise when these irrational thoughts come up and put a plan into action, we need
our comfort zone when say for example, we have been completely embarrassed in front of loads of people, or we are in danger, and we want
it when these anxieties really come out of nowhere. We can't let these things defeat us, you say it's the first time, so try not to think about it too much or be harsh on yourself but I'd say after my experience, that you should give it a try, if things get too much then you can always go back home earlier, if you can't then that's perfectly fine, just try and figure out some strategies to cope better next time these feelings pop up.
You say it's at certain times of the day, do these times maybe come a few hours after eating? Maybe the feeling of hunger could be mistaken for that butterfly feeling in your stomach. The brain works in funny ways, I know when I've eaten enough I feel much more able to cope!
This is just another hurdle, and can get over it! Don't call yourself silly names too, even people without anxieties get these strange thoughts so please don't worry!