The Maximum RPM in VESC Tool doesn’t do what you think it does. It does not prevent the motor from spinning faster. (only prevents power/brake while spinning faster)
Try setting Maximum Duty Cycle instead.