No, it doesn't mean that the speed limit is different if you are going the other way. I don't think so, anyway.
It means that to the right of this sign (upstream) the speed limit is 4 knots (regardless of which way you are going).