Well, other than meaning some one is stupid, it also means penis. Thats not that offensive.
But you raise a key issue I have been saying for years.
Bugger.
Bugger is not considered offensive very much at all, but it means anal sex between two men. Yet fuck, which means to have sexual intercourse is considered to be alot worse.