Joystick knob, how to unscrew?

(Back after a trip to the country side). I tried with a screwdriver, but to no use. I even tried twisting the wrong way. Just to be sure, there is no pushing or pulling required to unscrew the knob?