Perhaps you have experiences to be avoid by some Japanese people, when walking on the street, having a seat on the train and so on.
But it doesn’t mean that they don’t like you. They just worried about what to say.
Basically, many Japanese are not good at speaking foreign languages including English and most of us tend to think that we have to speak English to talk with foreigners.
Generally we respect the other position, but because of the problem of the language, many Japanese get so confused when seeing foreigners. I guess that the situation is changing because English is emphasized more and more in the educational institution.
Although more time is necessary to improve this situation.