I'm surprised I didn't catch this sooner, it looks like the method responsible for polling ( _do_poll) gets no sleep, this will mean it'll run in an brutal, hog-all-the-resources loop to the detriment of anything else you want to do.
If you want to try hacking it, you should be able to just drop in a time.sleep(0.001) ( or whatever value is practical for you ) into the _do_poll method itself.
I've dropped this into the GitHub version of the Library: https://github.com/pimoroni/skywriter-hat/commit/184da1b99c9db105aaeab87ff8e459e467648061
I can't push it to pip until I've tested it though!