The more I do calligraphy or any pen or brush work, the less it hurts, because I'm building up those very fine muscles in my hands/fingers.

It's like not lifting weights for a year, and then trying to lift weights again. If you keep up with it, you have the muscles to do it. If not, you'll get some pain.