@arcadeWC, based on my further investigation, the problem is mostly voltage related. Last night, and this morning, again after 5-6 hours of frustrating try-error and measurement. I seem to pinpoint the problem.
By voltage, I mean the voltage supplied to the CF card itself and the voltage that CF card internally operates (and factors related with voltage). These details make the card work or not. And these details make "which" card works or not and also which "game" loads better.
During one of my tests, I took out the CF card and hold it in my hand for a while and immediately plugged it back and powered on. Guess what, the game loaded. I first thought it might be static electricity but it was not because I tested it further. I put the card on an aluminum plate to make it cold. Then took it and plugged it back, it didn't worked again. Then while it was in the CF reader, I heated it slightly with my hot-air solder (almost close to body temperature) and powered on the setup, it started working again... Weird isn't it? Actually not...
Further, I took the time to discuss with one of my friends who works for an electronics company mainly on embedded systems (i.e. industrial equipments, meters, data loggers etc). He said that Compact Flash media, although its extremely simple compatibility with legacy IDE technology, is extremely unreliable. It is affected by temperature and manufacturing quality.
Although, in theory, CF media should operate in normal room temperature more or less effectively, my example of heating it juuuust a little only make sense IF the voltage supplied into the card is "barely" around the "safely operable" edge
Please take a look at this photo;
This is the voltage I measured ON the CF card adapter. As you can see how voltage is fluctuating (almost 2 volts!) although the average (see lover left - 3.28v) seems adequate...
I was taking this voltage from the CN2 on Naomi. CN2 is actually a INPUT plug (2x3 plug which only 4 pins populated). It is connected in parallel to the main plug next to it (the 2 x 4 plug). The problem of taking voltage out of this connector is that it is not "filtered". I mean the "filter" board on Naomi which everything else is connected to it is indeed a necessary piece. I stopped using CN2 and took voltage right from the pins sticking out just above the CN2 connector. These pins are the solder legs of the two main connectors which Naomi motherboard is actually connected to the filter board. The 3.3V on these pins are filtered and more stable.
Also, I'm using a (fairly quality) ATX power supply. Although it might be adequate, my major problem is that I cannot adjust voltages, specifically the 3.3V line. I totally rely on the voltage at that time of the day I'm getting.
Anyway, when I left CN2 alone and took voltage out of the pins I mentioned, I got rid of the ripples above (Didn't have the time to take a photo of it but I will). Also, since I do not have control on the voltage of my ATX power supply I do many tests different times. During morning and after dinner time I get better voltage levels (weird but true).
So, when the voltage I measure, on the CF adapter is like 3.35V, my CF cards starts to operate ALL THE TIME.
If it is just around 3.3V and lower (3.2-3.28V) it "sometime" works... Please keep in mind that these numbers are highly related with my equipment, i.e. the osciloscope I'm using. But the bottom line, is, if I see 3.35V I'm confident that it'll work.
During boot, like after 10-15 seconds, my CF adapter activity LED flashes for a split second. This seems to be an indication that DIMM board actually checks the CF card. If it does not do that flash, I know I'll end up with a black screen (this is 100% confirmed). After that flash, like 30 seconds or so, I see Naomi logo on screen...
On the otherhand, net dimms seems to perform better. I mean specifically the option board is different and if the setup does not work on normal dimm at that specific time, when I change the option board with a netdimm option board, it works... I think its because they are newer tech and communication voltage levels might be a bit more tolerant.
Regarding the "mods" I made on my CF adapter was merely to help clean and condition the voltage supplied to the CF card, thats all. Again it helps if my voltages are on the edge.
Here they are, input capacitors for voltage conditioning (100uF 16V electrolitic and 100nF ceramic);
The 10K resistor (right in the middle)
Although at first I thought the original CF adapter that Sega designed connected the way I connected, I'm not sure about it because the picture on the pcb otaku do have shadow right there and I cannot figure out if THAT 10K resistor is connected parallel to the filter capacitor OR it is the pull-up resistor for pin 37 (READY).
I did observed a benefit after connecting it BUT I now think that I might connect it for pin 37 for better results. I'll test that tonight... Btw, most of the CF adapters do not use it. Some have empty solder points for that. It is not a "must" requirement I guess but most of the microcontroller projects recommend that piece.
Anyway, lastly this is my setup... (Note that MVC2 is my "problematic game" that I used mostly during my tests. If it loads, everything loads
Everybody on the net shows their setup with Gigawing2. Sure Gigawing loads since its a small game to load like 30 MB something. Try MVC2 or Melty Blood Actress which these games are above 128 MB and takes longer to load, meaning that there is more chance you get bad voltage etc.)
And in operation...
My tests will continue. My next tests will be these;
1) Use a small high-precision LDO voltage regulator chip (LT1086 or LM3940). I do not thrust the classic 1117 since my input voltage is not so adequate. I want to make this CF adpater operate with 5V (internally reduced and regulated to 3.3V - just like the Sega one and mitsurugi-w's adapter).
2) Grab an official Naomi PSU (Sun) which has 3.3V adjustment.
I'm beginning to think that if I created my setup with an official SUN psu, I wouldn't be troubled that much... But then again, I wouldn't have that much experience either.
I'll continue to update this thread with my failures and hopefully success
(My success criteria is; the setup should work day and night for at least a week and at least on 100 re-boots without even a miss!)