That’s probably a small part of it, but the main part is gearing disadvantage. Let’s just say 10T vs. 20T, each wheel rotation would need to spin the motor twice as many times on the 10T, which would create twice as much resistance.
There’s been some discussion about KV and gearing in the past, I think it came down to a tradeoff of copper losses and iron losses due to the thickness of the windings, and it probably doesn’t matter too much, although that was from a quick skim and you’d have to read deeper to figure out the answer.