It's rather late (or early here), so will sleep on it. If you can, try a different monitor or cable. If you can send me the debug file of a game that's 'stable' I can check if it's something obvious, but if it reaches stage 4 (
https://irkenlabs.com/retro-scaler-a1/detailed-mode-of-operation), everything
should be ok.
To get a debug file, just create (an empty) file called "rsa1debug.txt", put it on a sdcard and power up the scaler with it.
Also, check power - I assume you're powering it from the Jammafier, then it's just checking that the 5V is not way off.