The washer helps the bearings have clearance and make sure they aren’t pinched. You can also get or make 0.2mm thick ones instead of 1.0mm thick ones. I’d highly recommend at least something. You don’t have to use them. Got a photo of the hanger end?