Apply stricter speculative load restriction