I had a similar experience. I think one of the issues is that if you go more than the "4" or "8" wide, it just continues onto the next row/column. There really isn't any definitive help for this module yet, it seems.
I recreated this as.
import unicornhat as uh
uh.set_layout(uh.PHAT)
uh.brightness(0.5)
uh.set_pixel(0,0,255,0,0) # red - top left
uh.set_pixel(7,0,0,255,0) # green - top right
uh.set_pixel(0,3,0,0,255) # blue - bottom left
uh.set_pixel(7,3,255,255,0) # yellow - bottom right
uh.show()
(the bottom right is yellower in real life).