The explanation is as I have proposed: it affects different beings different. Recall that I am Aesir, a being you would consider to be entirely 'of spirit'. Mortal, physical bodies were aged; is it so surprising that a 'body' which is neither of those would be differently affected?
no subject