well done awesome that you are putting new handles on
you are the real winner on melb cup day!

how hard have you hit it with the hammer? it looks like it is just stuck on with years of paint

It's most likely been installed when the paint was still wet so is stuck.
Just give the handle a quick, sharp hit down. Usually the palm of the hand is enough…