The controller config in retropie only works with certain emulators (based on retroarch).
MAME has to be configured separately. To do this you need to have a keyboard plugged in. Launch any mame rom and then press the 'TAB' key on the keyboard. This will bring up the contol configuration for Mame. You need to configure the joysticks buttons (player1) and Coin1, buttons etc.
To change the binding just press enter on the keyboard and choose the desired input on the joystick (up, down, left, right etc) And the same for button1, 2 , 3
In short, The controller config in retropie only works in certain emulators (i.e SNES, Megadrive etc)
MAME and i beleive Neo Geo are done separately.
Hope this helps.