Final update on this. I tried about 6 different SD cards I had (Sandisk, Samsung, Kingston), ranging from 8GB to 32GB, still same behavior (in game resets, boot looping, behavior worse closer to 5.0v I went, etc).
I decided to open the cart and check the SD card pins, but they were fine...