Netboot Naomi With EEPROM Presets!

[DragonMinded]

Another update, this one's a big one:

- Fully fleshed out and finished the game settings menu (hold start on a game for 1+ second to enter the game settings menu).
- EEPROM settings are loaded and saved on the host, EEPROM settings are remembered and applied every time a game is booted, settings are editable on the game settings menu.
- Found and fixed a bug causing intermittent message transfer failures between Naomi and host. Communications should be more reliable now. I haven't had a message failure since fixing this.
- Enabled compression between Naomi and host, as well as changed the transfer protocol to be approximately 3x faster to send data to the Naomi. This was necessary as EEPROM settings are so much bigger than anything else that was being transferred up to this point. It is still very slow but it is usable.

There's a few remaining issues that I need to deal with:
- I've seen the occasional failure to send a game to the net dimm after selecting a game, resulting in a communications error page. I haven't been able to duplicate this, it only happens when I don't look for it. But I'll need to track this down and fix it.
- There's a bug in the EEPROM settings apply trojan and associated editor with regards to games that don't have a settings definition file for their settings. I'm working on this but I haven't got a fix for it yet.

 

[DragonMinded]

Okay, so the bug I found last night leads me to this realization: there is no reliable way to initialize EEPROM settings for a game while still allowing you to make further changes in the test menu. It's just not possible. There is no way to tell the difference between a valid EEPROM that was written by my initialization program versus written by the game itself versus written by some other game. The BIOS will initialize the system section before getting to the game and before getting to my initialization trojan, so the EEPROM will always have the right system CRC and serial. So you can't use validity of system settings as an indicator. You can try to check the length of the game settings as this tends to change between games. However, games tend to use the game section in chunks of 4 and there's only 42 available bytes, meaning in practice there's only 11 unique combinations of length across all games. Detection collision between two games is highly likely. You can't even ask if the game section is valid, because when the BIOS initializes the system section it leaves the game section alone, so it passes CRC checks but it is for the wrong game. Only the game itself knows what it should expect and when to reset its own EEPROM.

In practice, many games stick their own serial or something else identifiable in the game section so they know that the EEPROM is for them and not another game. They don't do this in a standard way, however, and not all of them even do this. So it is not possible to examine an existing EEPROM and determine that yes it is for a particular game. So you can't key off of the EEPROM being for the wrong game. You can't key off the EEPROM system section being invalid (it will always be valid, no matter what, the BIOS ensures this). You can't key off the EEPROM game section being invalid (it will often be valid but for the wrong game if you just switched games). You also can't write down something in the EEPROM that the BIOS and game won't touch, because....they won't touch it! It'll still be there if you switch games and then switch back and you still know nothing!

Therefore, the feature originally announced on the first page where the settings trojan would initialize your EEPROM but then leave it alone if you messed with test settings is not possible to do reliably. I'm going to have to change it to always initialize the EEPROM every boot. This means that the test menu and game test menu will no longer have any effect if you are using my settings trojan. That sucks, but it is life. There's no way to fix this that I can think of without patching the BIOS or patching other games. In theory, I could put a bit of code out that always runs before every game, even if you are not initializing the EEPROM, and which stamps the game serial number somewhere in the end of the EEPROM that says what game is actually initialized. Then I could check and see if that was different than what I wanted to initialize and if so write a new EEPROM and if it is the same leave it alone to preserve modified settings. But, asking people who want to use settings trojan to change every single ROM they use is a bit out of the question.

The best I can do is to always write the system section, and if the game section exists, to always write it as well. If it does not exist (no settings definition file for it, or the game does not use the game EEPROM section) then do not write the game section at all which will preserve the game test menu functionality. However, system settings will be read-only if you use the setting trojan. you can modify them all you want in the test menu but they will revert to what you set up before you net booted the game. In practice, this is not a big deal for me because I never go into the test menu once I've set up a game. I like that the settings are guaranteed. I'm not sure how much of a dealbreaker it is for anyone else however. It would mean if you netboot something and then a friend wants to change an option, you would have to net boot it again with that option updated. It seems to make the settings trojan much less useful.

Thoughts?

 

[Mrhide]

There's no way to fix this that I can think of without patching the BIOS or patching other games.

We are all silently holding our breath!!

The best I can do is to always write the system section, and if the game section exists, to always write it as well. If it does not exist (no settings definition file for it, or the game does not use the game EEPROM section) then do not write the game section at all which will preserve the game test menu functionality. However, system settings will be read-only if you use the setting trojan. you can modify them all you want in the test menu but they will revert to what you set up before you net booted the game. In practice, this is not a big deal for me because I never go into the test menu once I've set up a game. I like that the settings are guaranteed. I'm not sure how much of a dealbreaker it is for anyone else however. It would mean if you netboot something and then a friend wants to change an option, you would have to net boot it again with that option updated. It seems to make the settings trojan much less useful.

Thoughts?

Hey the purpose is to NOT have to go in the test menu right? How many friends do you have that know and want to change options anyway?

 

[DragonMinded]

Yeah that's what I was thinking. Personally, I couldn't care less. Once I set up my options I'm good to go forever. I'm just thinking of something like, okay you have friends over and are playing a fighter and somebody wants to change the difficulty and you gotta tell them sorry test menu doesn't work anymore. Not sure if this is how anyone wants to use the settings trojan.

 

[Aurich]

I'm in camp "set how you want and never change them again" personally. I'm turning on free play, turning off demo sounds, and moving on with my life.

I've never actually randomly changed difficulty settings or anything on the fly like that for any game.

 

[DragonMinded]

Okay, pushed a smaller update today. There's now a toggle similar to available patches for choosing whether to force system settings or not, and the settings get grayed out if the checkbox is not checked. The bug where games with no settings definition file couldn't apply system settings has been fixed as well, and the settings trojan has been updated to fix reliability issues. I also updated my web interface but I don't think anyone cares about that but me.

EDIT: If you were waiting for a good point to try the netdimm menu, now is the time! I am at a point where I consider it "good enough" and I am hoping for confirmation that it works for others, or bug reports about when it doesn't.

 

[Mrhide]

Nice menu !

 

[DragonMinded]

Another update, this one with a LOT of fixes behind the scenes but not much obvious changed up front. After a solid week of coding, I now have interrupts and threads working (including a scheduler and thread library written from scratch by yours truly) on the Naomi which means I can properly respond to DIMM communications requests without polling. For most of you, this means that sending/updating settings is about 10x faster now. It also means that entering and exiting the test menu after running the netdimm menu no longer gives an ERROR 22 screen. So, I've also added an option to the menu config to enter the system settings menu. This is mostly useless unless you've done a bone-headed thing like run a vertical game on a horizontal cabinet and now your menu is sideways.

Aside from those fixes, everything should be the same in the menu. So, basically a quality of life upgrade with one bug fixed (ERROR 22) and one massive speed-up (loading and saving settings).

For the @bobbydilley folks among us that are actually setting up the toolchain to write homebrew, there's a lot of exciting changes under the hood. Proper timer support (theoretically infinite timers), threads, semaphores, mutexes, global counters, no more dimm command polling, exception screen, ability to resolve exceptions to line of code using addr2line, and updates to lots of libnaomi to make it thread-safe by default. There's still lots to do and I have some cool ideas. Namely, I'm going to move the packet and message libraries into the netdimm python package and incorporate packet/message handlers into libnaomi. This will let me write a stdin/stdout/stderr redirector stub that can be run on the naomi so that you can get debug printing on a PC connected to the net dimm. It should also allow me to implement a GDB stub for live on-target debugging of homebrew while your naomi is executing code.

 

[Mrhide]

after this update, trying to open the settings menu for games I have saved settings, I get the message below.
It does load the settings menu for non saved games .... should I delete the old settings somewhere?

equested settings for UNDER DEFEAT...
Traceback (most recent call last):
File "./netdimm_menu", line 11, in <module>
runpy.run_module(f"scripts.{name}", run_name="__main__")
File "/usr/lib/python3.8/runpy.py", line 210, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/mrhide/Desktop/netboot-trunk/scripts/netdimm_menu.py", line 756, in <module>
sys.exit(main())
File "/home/mrhide/Desktop/netboot-trunk/scripts/netdimm_menu.py", line 495, in main
parsedsettings = manager.from_eeprom(eepromdata)
File "/home/mrhide/Desktop/netboot-trunk/naomi/settings/settings.py", line 1099, in from_eeprom
eeprom = NaomiEEPRom(data)
File "/home/mrhide/Desktop/netboot-trunk/naomi/eeprom.py", line 203, in __init__
raise NaomiEEPRomException("Invalid EEPROM CRC!")
naomi.eeprom.NaomiEEPRomException: Invalid EEPROM CRC!

 

[mv1f]

This is all super awesome! Thanks so much for all your hard work @DragonMinded . This code is all very high-quality.

I did have to make a couple of changes to the homebrew env setup Makefile to get it to work on MacOS and made a couple of small quality of life changes (adding a prefix var I can override on the commandline and ignore the patch if it's already been applied).

 

[MetalliC]

@DragonMinded impressive work!
but, don't you think building all the framework from scratch might be not the best idea ? perhaps it might be better to fork KOS and add there all the NAOMI-specific stuff ? in this case you'll be able to use tons of already ported libraries like SDL, GL, pthreads etc, and also give the way to DC homebrew gamedevs to (re)build shit ton of new games for NAOMI platform.

 

[DragonMinded]

after this update, trying to open the settings menu for games I have saved settings, I get the message below.
It does load the settings menu for non saved games .... should I delete the old settings somewhere?

equested settings for UNDER DEFEAT...
Traceback (most recent call last):
File "./netdimm_menu", line 11, in <module>
runpy.run_module(f"scripts.{name}", run_name="__main__")
File "/usr/lib/python3.8/runpy.py", line 210, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/mrhide/Desktop/netboot-trunk/scripts/netdimm_menu.py", line 756, in <module>
sys.exit(main())
File "/home/mrhide/Desktop/netboot-trunk/scripts/netdimm_menu.py", line 495, in main
parsedsettings = manager.from_eeprom(eepromdata)
File "/home/mrhide/Desktop/netboot-trunk/naomi/settings/settings.py", line 1099, in from_eeprom
eeprom = NaomiEEPRom(data)
File "/home/mrhide/Desktop/netboot-trunk/naomi/eeprom.py", line 203, in __init__
raise NaomiEEPRomException("Invalid EEPROM CRC!")
naomi.eeprom.NaomiEEPRomException: Invalid EEPROM CRC!

Yeah, unfortunately. There was a bug in the older EEPROM code which generated incompatible, broken EEPROM stuff. If you saved settings and then updated, your stuff will be bugged. So unfortunately you need to delete your settings and start fresh Sorry.

If you run netdimm_menu with --help it will tell you where the default settings is stored and you can delete that and run again. It should fix that issue.

@DragonMinded impressive work!
but, don't you think building all the framework from scratch might be not the best idea ? perhaps it might be better to fork KOS and add there all the NAOMI-specific stuff ? in this case you'll be able to use tons of already ported libraries like SDL, GL, pthreads etc, and also give the way to DC homebrew gamedevs to (re)build shit ton of new games for NAOMI platform.

I originally wanted to use KOS for a lot of stuff, but there's enough difference between DC and Naomi that it wasn't practical or feasible when I started. Also, a lot of the stuff in KOS is hand-rolled and then hacked into GCC/newlib such as pthreads (its a partial re-implementation in KOS) and much of it centers around having a BIOS to go back to (DC menu) and syscalls (default font, rebooting, etc) as well as tools surrounding that. Also, back when I started all of this years ago, KOS didn't support Naomi at all and nobody had made a working Naomi homebrew as far as I could find on the whole public internet. KOS claimed they supported Naomi but they had no header tools, no way of choosing the subarch, no input or eeprom support and their instructions were "just patch the raw binary on top of some commercial game it might work" and that was super hacky and relied on having the same entrypoint. I tried to play with KOS and it wouldn't even compile, so I made my own toolchain based on theirs and got started that way.

Ultimately its something I'm doing for fun, and I find trying to figure out how to get undocumented, sometimes poorly written open source to work on esoteric platforms to be such a buzzkill that I often give up and walk away for good. Lots of KOS has literally no comments as to why they do something or set some register and it doesn't make sense unless you probably understand much of the other libraries they wrote and know the preconditions they need setup. Even after reading several of the official SH-4 documents some stuff they do does not make sense and I do not know how it works reliably. I am not trying to shit-talk KOS, it is an impressive platform with lots of cool stuff and I even use some of their knowledge to help me figure stuff out. But hacking on somebody else's project to try to get stuff running on an unsupported platform does not make for a fun time IMO.

 

[Aurich]

Ultimately its something I'm doing for fun

hacking on somebody else's project to try to get stuff running on an unsupported platform does not make for a fun time IMO.

Your priorities sound perfectly sorted to me.

I don't even have a Naomi setup, but I'm really enjoying watching all your updates. And living vicariously through your test partner's excitement. I don't really have anything helpful to add to your thread, so I just don't post as a rule, but I am reading and appreciating, and I'm sure I'm not alone.

 

[DragonMinded]

Hmm, so I found out that the settings trojan no longer works with some games. I'm going to remove most of the threading stuff and make a small standalone trojan and hopefully that will fix things.

 

[DragonMinded]

Okay, so I have an incredibly whittled down settings trojan now and it modifies the literal bare minimum of hardware to do its thing and move on. However, even when I introduce only a simple jump from the trojan directly to the main executable without changing anything else, both Crazy Taxi and Jambo Safari crash during the attract loop. Given that they do this when the simplest of trojans is applied, but they work fine with no trojan, I am beginning to suspect that they both have buggy code relating to how many executable sections exist in the ROM header. I have no other idea as to why even having a second section that does literally nothing causes them to crash.

I really don't want to fuck around with MAME to get it to take my images for me to debug this, so I'm going to give up on that for now. However, I realized that both of these games have the default settings string in their main and test executable, probably so they can initialize with defaults when needed. I suspect that many games have this particular setup as I know that at least MVC2 does as well (that's what the unlock edits mess with). The interesting thing is with the existing EEPROM creation code, I can craft both the defaults (which I can use to find the section in the binary) and the updated settings (which I can use to patch the section in the binary). I'm going to give this method a try. If I can auto-patch the default creating section of a ROM with the settings that were applied, then that means that ROMs can be made so they have your settings by default, but can be modified later, and it should not break games which seem to have some bug like Crazy Taxi/Jambo Safari.

I'm going to give this a try.

 

[DragonMinded]

Okay, so the idea is sound. Both Crazy Taxi and Jambo Safari take the settings and initialize them. However, while Jambo Safari can be asked to go into freeplay/silent attract just fine, asking Crazy Taxi to do so using the header flags causes it to freeze on the SEGA logo before the attract screen. I'm getting the feeling that this is a particularly buggy game where there is no real option to use supported BIOS features. Pity, I'm going to have to keep digging.

 

[DragonMinded]

Okay, for Jambo Safari and Crazy Taxi I checked in some updated patches for freeplay/no attract that don't use the header mod since that seems to crash both games. Instead, I found two new patterns that can be patched and patch the binary directly. I can also patch the default settings. So tomorrow I'll work on a compatibility mode setting for the netdimm menu as well as my web netboot setup.

 

[rtw]

It might not be bugs but protection. SEGA like to checksum things.

 

[DragonMinded]

It might not be bugs but protection. SEGA like to checksum things.

I thought about that. However, if that was the case then they are only checksumming the header? Because my modifications to the actual code stick just fine. So its quite possible they are validating the header since both games DO transfer it and it is available to the game code when running (some games do this, some games don't). I guess it might be worth it to track down any code that might be doing that because it's going to be generic enough to apply broadly (the patches for free play/no attract were generic between the two games).

 

[DragonMinded]

Okay I dug in, and found where they were looking at the first 0x500 (header length) bytes and running some PRNG over it and comparing to a copy. If it doesn't match for a header byte, it ends up going into an infinite loop where it just reads from the EEPROM constantly. I can get it to crash in MAME, and if I change the comparison to enter the false case to return true, it immediately stops freezing in MAME and goes back to working. I then looked at Jambo and was able to find the exact same code (well, slight difference, but still, identical functionality) and the same patch should apply. Going to try that downstairs, but if so then that means some games just need a protection patch. I still don't quite understand what the code is doing, but hopefully this fixes things for both games.