DragonMinded
Professional
Okay, this update is more for the people who implement net boot setups and less for the rest of everyone. I've done a bunch of work on the underlying python code that powers all of this with the purpose of making it easier to integrate into existing projects. At the request of @LazerFlux I have worked to make all of the packages more RPI-friendly. My old methods involved loading the whole ROM into memory, manipulating it with patches and settings enhancements and then sending it to the net dimm. This works great on high-powered computers but the RPI only has 512 MB or 1 GB of RAM depending on the version. When some ROMs are pushing 512MB on their own, that means that this doesn't really work all that well and certainly fails when you use a RPI to manage more than one net dimm at once. So I did a bunch of work creating a "FileBytes" class that you can give to any of the net boot libraries where you previously could only give bytes.
The "FileBytes" class is basically a magic class that pretends a file is fully loaded. It only needs to load parts of the file and keeps track of what changes were made. That means that my utilities can still load a ROM, patch it on the fly, add settings on the fly and send it to a net dimm just as it could before. But this time it only takes a fraction of the memory. Comparing before and after with Monkey Ball, originally it took about 200 MB of RAM to patch, add settings and send to the net dimm. Now, it takes about 25 MB. That's more than plenty of room to run on even the smallest RPI setup. As an added bonus, not loading the whole file to make some minor edits also speeds up many of the utilities. That means that not only are things faster to load but they are easier on the RPI.
If you were using my packages to patch data and netboot, you might have done something like this:
Now, you would do something like this instead:
Basically, the code is identical, you just use "FileBytes" to handle the magic of loading things when needed instead of reading the whole file. You get speed and memory benefits automatically.
I've also separated out and created a "netdimmutils" package that includes a class for net booting. It has all the enhancements and fixes that I've put in from reverse engineering fw 3.17, transfergame.exe, transE.exe and checkoff.exe as well as my experiments. Basically if you saw me talk about a net boot option in this thread its available in this package. Together with "naomiutils" which includes all of the ROM and EEPROM stuff that I made in this thread as well as "arcadeutils" which includes all of the patching utilities and "FileBytes" magic class, these can be used to form the basis of a web-based, console-based or LCD-based net boot solution running on a RPI or even on a computer. Of course, they are all used in my "netboot" project to provide just such a web-based netboot solution that I use on my own setups but that is meant for servers so I don't expect many people to be interested in it.
All of these packages are available in PyPI so you can pip install them or depend on them in your own projects. The arcadeutils package is https://pypi.org/project/arcadeutils/, the naomiutils package is https://pypi.org/project/naomiutils/ and the newly created netdimmutils package is https://pypi.org/project/netdimmutils/. You can see how I use them to make various utilities like the Naomi Menu and the web based loader here: https://github.com/DragonMinded/netboot. And of course, the instructions in the first page of this thread for how to set up the utilities and run all the stuff I've done is on page 1 and can still be used by everyone else.
CC both @LazerFlux and @chunksin because I think that the new layout for a lot of this code should be much easier to consume and integrate. @LazerFlux I know you wrote your code in python so you can just take dependencies on these packages and integrate them. For @chunksin I think you base off of the old python scripts with PHP for the frontend? You should be able to use the https://github.com/DragonMinded/netboot project's "netdimm_send" or "netdimm_ensure" commands in place of the existing scripts as they take the same arguments (save for extra arguments for supplying settings and patches).
The "FileBytes" class is basically a magic class that pretends a file is fully loaded. It only needs to load parts of the file and keeps track of what changes were made. That means that my utilities can still load a ROM, patch it on the fly, add settings on the fly and send it to a net dimm just as it could before. But this time it only takes a fraction of the memory. Comparing before and after with Monkey Ball, originally it took about 200 MB of RAM to patch, add settings and send to the net dimm. Now, it takes about 25 MB. That's more than plenty of room to run on even the smallest RPI setup. As an added bonus, not loading the whole file to make some minor edits also speeds up many of the utilities. That means that not only are things faster to load but they are easier on the RPI.
If you were using my packages to patch data and netboot, you might have done something like this:
Code:
with open("someromfile.bin" "rb") as fp:
# Read the contents
romdata = fp.read()
# Apply some random patch
romdata = BinaryDiff.patch(romdata, somepatch)
# Apply some random EEPROM settings
patcher = NaomiSettingsPatcher(romdata, get_default_naomi_trojan())
patcher.put_settings(somesettings)
romdata = patcher.data
# Send the patched ROM to a net dimm
netdimm.send(romdata)
netdimm.reboot()Now, you would do something like this instead:
Code:
with open("someromfile.bin" "rb") as fp:
# Magic wrapper to the file contents, but don't read it
romdata = FileBytes(fp)
# Apply some random patch
romdata = BinaryDiff.patch(romdata, somepatch)
# Apply some random EEPROM settings
patcher = NaomiSettingsPatcher(romdata, get_default_naomi_trojan())
patcher.put_settings(somesettings)
romdata = patcher.data
# Send the patched ROM to a net dimm
netdimm.send(romdata)
netdimm.reboot()Basically, the code is identical, you just use "FileBytes" to handle the magic of loading things when needed instead of reading the whole file. You get speed and memory benefits automatically.
I've also separated out and created a "netdimmutils" package that includes a class for net booting. It has all the enhancements and fixes that I've put in from reverse engineering fw 3.17, transfergame.exe, transE.exe and checkoff.exe as well as my experiments. Basically if you saw me talk about a net boot option in this thread its available in this package. Together with "naomiutils" which includes all of the ROM and EEPROM stuff that I made in this thread as well as "arcadeutils" which includes all of the patching utilities and "FileBytes" magic class, these can be used to form the basis of a web-based, console-based or LCD-based net boot solution running on a RPI or even on a computer. Of course, they are all used in my "netboot" project to provide just such a web-based netboot solution that I use on my own setups but that is meant for servers so I don't expect many people to be interested in it.
All of these packages are available in PyPI so you can pip install them or depend on them in your own projects. The arcadeutils package is https://pypi.org/project/arcadeutils/, the naomiutils package is https://pypi.org/project/naomiutils/ and the newly created netdimmutils package is https://pypi.org/project/netdimmutils/. You can see how I use them to make various utilities like the Naomi Menu and the web based loader here: https://github.com/DragonMinded/netboot. And of course, the instructions in the first page of this thread for how to set up the utilities and run all the stuff I've done is on page 1 and can still be used by everyone else.
CC both @LazerFlux and @chunksin because I think that the new layout for a lot of this code should be much easier to consume and integrate. @LazerFlux I know you wrote your code in python so you can just take dependencies on these packages and integrate them. For @chunksin I think you base off of the old python scripts with PHP for the frontend? You should be able to use the https://github.com/DragonMinded/netboot project's "netdimm_send" or "netdimm_ensure" commands in place of the existing scripts as they take the same arguments (save for extra arguments for supplying settings and patches).
