These you can connect in any order. If the motor spins the wrong direction, swap any two. You can also swap the motor direction in software.
As for setting up VESC software, there is enough material to fill up an entire college level course with. That’s difficult to answer with one forum post.
I’d say start here:
That’s an older version of a fork of the official software, but it works well.
However for maximum quality I would suggest not using sensors and using the newest firmware with FOC and HFI sensorless mode. But following that guide will get you up and running.