Cooked editor engine changes

As previously mentioned, due to the existence of UEFN, Epic Games have invested a lot into making the editor able to handle cooked content fairly well - and the later the engine version, the better it will be.

Without any engine changes, this is how Epic describes working with cooked content in the editor - it is quite limited (there are some hacky ways to make is not so bad, but it's annoying).

To maximise the potential of the cooked content in the editor, some engine changes will be necessary - but they are really not that complicated changes. Most of the changes are simply necessary to guard against editor code paths that aren't expecting cooked content.

Editor only data

Before I go into the engine changes, I want to mention about editor only data.

In your research, you may notice some code mentioning "editor data", "editor only data" or WITH_EDITORONLY_DATA. This is an option for packages to be cooked with all the extra metadata that the game itself doesn't need - most notably kismet graph data (rather than just the compiled bytecode) for blueprints, material shader code for materials, and niagara effect node definitions in niagara assets. I believe this exists again due to UEFN as it is enabled for that.

Editor only data does inflate the size of assets considerably, but it does give the obvious benefit that you can still use cooked editor while still having blueprint/material source code show up in the modkit!

Editor only data does not replace the need to make most of the engine changes below, so please keep reading on for that information.

Engine changes

I will explain each engine change I had to make in UE5.6.1 for the Subnautica 2 modkit - what they are and how they work. Your results will vary depending on the engine version, but that is for you to figure out. I will visit each topic in detail:

  • Serialisation
  • Mounting game containers
  • Prioritise loading loose files over mounted containers
  • Enable premade asset registry
  • Enabling all cooked blueprint references
  • Allowing all cooked assets to be openable
  • Miscellaneous small changes
  • Loading cooked levels
  • Loading the compiled shaders
  • Extra utilities

At the time of writing (check the sn2-v.0.10.3-2 tag), the cooked editor is very stable as I was able to make mods referencing all kinds of asset types and having a bunch of assets open, for over 3 hours, without the editor crashing once.

Note that all modkit-related engine changes should be wrapped with WITH_EDITOR compiler guards so that the modkit changes don't exist in the game. While most of my engine changes are already doing this, one thing most changes are not taking into account is operability between a modkit editor and a source/non-modkit editor - as modders don't have access to the source editor so obviously there is no need to for me to support anything but the modkit editor.

Serialisation

When a cooked package is created, its binary structure is dependant off the sizes and offsets of the reflected properties of native class schemas. In UE5+, cooked content is additionally cooked as "unversioned", meaning that it does not contain any information in its header about how to parse the package. This saves a lot of space across all assets and reduces access time in the game as it does not need to spend time looking up the data in the header to get info about parsing the package - now the game can just directly load in data using the offsets known to it from the types in the engine.

If there is a mismatch of the format of the cooked asset binary, the game nor editor would be able to load it properly, as data would eventually shift out of alignment, thus allowing garbage data to be read into properties, leading to crashes.

All of this is to say, that for the editor to load the cooked packages correctly, your custom engine must also include all of the reflected class changes that you have made for your game.

In practicality, you should just include all engine patches for the modkit, not only for serialisation, but also so that shaders load in (more on this later) correctly, editor binaries work, etc. I'm also not really in a place to comment on your build system, but I would see it as much easier to just use the same engine build that the game itself uses for the modkit with the aforementioned compiler guards in place.

Mounting game content containers (works with IoStore and non-IoStore games similarly)

So in the engine, there is an existing project startup command line flag -UsePaks. This flag allows you to directly mount container files to the editor.

Engine edit -UsePaks flag to always be enabled (optionally, flip it so that you can supply -NoPaks flag to disable mounting).

While the engine does already support mounting containers from within the project, there are a couple drawbacks:

  • It expects the container files to be at a relative location to the engine install. You should change this behaviour anyway though, as if you are distributing an installed engine build seperately, then the user may choose to install the engine to a different location, thus causing it to break
  • You would need to either distribute your modkit project already containing the game content (inflating download size massively) or have some utility to copy the game content from the game into the project location (extra hoops, slow to copy, duplicate data)

So what I did is to make an engine change in IPlatformFilePak.cpp that reads in a txt file in the project root containing the path to the game install directory. It's a very simple to ask the user to supply the path manually during modkit setup, or if you had a method of reliably finding the game install location you could have such an algorithm in the engine change and then fallback to a txt file in case it fails.

Engine change

When reading through this change you may notice some code relating to load priority...

Prioritise loading loose files over mounted containers

In your project you may have some loose game files that work better as loose/source files than used as cooked files. Example of these in Subnautica 2 modkit:

  • Source .ufont files - (at least with IoStore) these do not resolve correctly as cooked files only as the ufont files are stored in a seperate container path, so I extracted the ufont files from the game's pak file and placed them directly into the project content folder under the correct directory path and names. If you just load them from containers, any widgets or text using the cooked font files will look like [A][A][A][A][A] (missing font source).
  • FMOD banks - FMOD bank files are looked up as non UFS and as such are located as loose files in the game install folder, not in packaged containers. FMODStudio plugin then unpacks these banks at the first editor startup into the correct folders, as source assets. Therefore, the loose assets need priority over the cooked, non-working ones in the mounted containers (note that I did need to fix a bug in this copy process related to mounted containers here)

You may have other loose assets that you define in your project as "Directories to package as non UFS" such as movies, textures, animations, models (if using Interchange plugin pipeline). You may choose to either distribute these loose assets as part of the modkit download, or have them copied in from the game install files with some startup script.

Luckily, the engine already provides a way to do this: bLookLooseFirst. I hardcoded this to always be true in the editor as there is no reason for it not to be as far as I can tell.

Once you've made these changes, you may notice that there are no assets showing up in the content browser (aside from any loose ones)...

Enable premade asset registry

The content browser does not directly mirror the contents of packages on disk or mounted - instead, it builds a virtual view of the packages known to it at editor startup or when refreshed due to actions from content browser (such as creating, deleting or renaming an asset). Since loose assets are there on disk at startup, it can find these files immediately. However, since the mounting happens later in the engine init than the content registry read, it is missing all those in the mounted container.

Thankfully, again the engine already provides a neat way to to do this - an editor startup commandline flag -EnablePremadeAssetRegistry. This looks for an AssetRegistry.bin file in the project root and then loads up the content registry with all packages from it. Simply supply your game's AssetRegistry.bin file in the project with this flag and you should be able to see all cooked content in the editor. Make sure that the asset registry file that is in the project root is always the same version as from the installed game files - as otherwise it may show assets in the content browser that do not exist in the game or not show ones that do.

In this engine change, I do same as I did with -UsePaks -> -NoPaks flag - flip it to -DisablePremadeAssetRegistry so that bUsePremadeInEditor is true by default with the option to disable it if needed.

If there are still no cooked assets showing up in the editor, make sure you have these configs set in DefaultEngine.ini.

[/Script/UnrealEd.CookerSettings]
cook.AllowCookedDataInEditorBuilds=True
s.AllowUnversionedContentInEditor=1

I also found a bug that when deleting an asset in the content browser, the registry would refresh and "loose" all of the packages from the mounted container - because the refresh logic was simply only looking for packages on the disk - thus the cooked content would disappear. So to fix that I created a helper to check if an asset is from mounted container and then used it in the code paths related to regenerating the registry (also present in the same commit).

Enabling all cooked blueprint references

This is arguably the most important part of modkit - when a modder is creating their blueprint logic, 99% of the time they will need to get references to a game asset, for example in:

  • Casting
  • Get all actors/widgets of class
  • Getting/setting a property of a blueprint
  • Calling a blueprint's function
  • Binding to a blueprint's delegate

The main reason to provide a modkit is so that all the references for the mod are just there, readily available for the modder - no need for the modder to manually create dummy assets just to get their references.

In the vanilla engine, cooked blueprints are only referencable from blueprint code in asset list dropdowns such as on the get all actors of class dropdown. Any of the other referencing examples above aren't doable without an annoying workaround - creating a child blueprint of a cooked blueprint, which does a deep copy of the blueprint's component tree and saves it to an uncooked package on the disk. Since it's a child BP, defaults can still be accessed/modified as well as the copied component tree. However, it does not copy any of the properties, functions or events. So those still need to be manually dummied. Also, creating a child introduces some additional complexity in code as it's not actually the game blueprint they're referencing directly.

The fix for this turned out to be insanely simple - a single if statement change. In a nutshell, the code that builds the actions database (which is the stuff that appears in the context menu when you right click in a blueprint graph) was that the package's Class->ClassGeneratedBy property was never null (it is for cooked assets), thus was going down a code path that would silently fail. Once the change is made, the actions database is built using a seperate code path that doesn't rely on Class->ClassGeneratedBy and then simply works.

Engine change

Allowing all cooked assets to be openable

By default, trying to open a cooked asset will lead to a notification message saying something like "Cannot modify cooked assets". Obviously this is not useful, so you need to change this to allow opening them.

First, disable the logic for the above check in the content browser (note that this change is commented out code, obviously you should be implementing it with proper checks etc).

Next, set bCanBeModified to true (additional change) for cooked packages. The reason for this change is so that in certain asset types, you can temporarily make changes to the asset in that editor session, for example:

  • Experimenting with assigning different materials on a mesh (so modder does not need to spend the time testing it at runtime)
  • Experimenting with assigning different skeletons or physics assets to skeletal meshes
  • Assigning a skeletal mesh to an animation or vice versa - as sometimes this link is not set by default, depending on how the original project had it setup

It is important to note that any changes to the cooked assets are still temporary to that editor session - no data is written back to the cooked package - so all changes to them are lost on editor shutdown.

Once the changes have been made, the majority of asset tyes should be openable (with a caveat) without crashing, sound waves should be playable and overall the usefulness of the modkit has skyrocketed. The caveat is that most asset types that might have a graph view or viewport will open into a fallback asset editor that only lets you view and edit properties (it looks like a data asset view).

Miscellaneous small changes

There are a bunch of additional small changes that need to be done to fix code paths that are not expecting cooked data - but please review all changes to check if they will apply to you, as well as any changes that may be different on older/newer engine versions than UE5.6.1.

  • This commit, this commit, this commit contains a few small changes (please ignore all the header changes, I was unprofessional here and committed unrelated changes together). I also apologise that I'm showing some changes that were later reverted, moved about and stuff, as this was active in development. It might be best to just check the diffs from here

    • Allow cooked packages to be duplicated
    • Allow user defined structs to load
    • Fixes various issues loading animation based assets
    • Downgrades some checks and fatal errors to ensures and non-fatal/warnings so that editor does not crash on serialization changes. Note: these changes should not be necessary as long as you are supplying your custom engine with all your engine patches for the game (due to the nature of reverse engineering engine changes, some changes are inevitably missed/done incorrectly so it was beneficial for me to brute force down some code paths to get more data).
    • Fixes to issues related to the limitations of Suzie, the tool I was using to generate the UHT class schemas in the project. If you are including source binaries, you should not have these problems either.

  • This commit reflects the GameInstance.ReferencedObjects property to blueprint to allow modifications to default objects to persist across level changes. See more here.

  • This commit fixes opening sparse volume texture assets crashing the editor

Loading cooked levels

Once all blueprints and other instancable actors are all opening without crashing the editor, you should now be able to open cooked levels that do not contain any landscape data.

But if you have any levels that do, you should fix the ability to open levels that contain landscape data. This is something some thought wasn't easy with cooked editor, as even in UEFN you cannot open cooked levels.

I spent time looking into it as it is extremely beneficial to allow the cooked levels to be openable:

  • Modders can use them as references of where to spawn their mod actors at runtime (such as adding a new area in a level)
  • They can be used to see existing actor instances - where they are, how they're configured etc.
  • They can show actors that weren't previously noticed due to being invisible in-game, such as splines
  • They can be copied to create entirely new levels based on existing game ones, as of course mods can load levels from blueprint

As it turns out, at least in 5.6, there are only two small changes necessary! Disabling world partition streaming (more on this in a sec), and avoiding hash creation for weight maps as that data is stripped from the cooked asset.

Once these changes are done, all levels should be openable. However, as above, there is a major caveat: if a level is using world partition streaming, none of the partition regions will be loaded when you open it - you will only be able to view the persistent objects.

That being said, I think it should be possible to allow region streaming to exist, as if non-world partition levels with landscape can load, why not generated partitioned levels? It would take some more engine changes for sure. I'll change this guide if I figure it out (I intend on trying).

Something for you to experiment with (which I can't do as a modder) is to try copying in your uncooked levels as loose files and seeing if they all work fine? In theory, I think this should work perfectly without any of the above fixes required, as ultimately levels are either self-contained (landscape data) or contain references to assets in the project.

Loading the compiled shaders

During project cook, shader code from materials are compiled into shader archive files, which are stored in the cooked containers at the root directory.

This means that the cooked materials themselves do not contain any of their shader information, which explains why all the cooked materials look just black or white or grainy in the editor.

This is not great because it doesn't actually show the modder what the material looks like if they are inheriting from a game material when writing their own shader code/playing with game material parameters - they would have to package their mod and load into the game to test - making iteration painfully slow.

So, it is possible to load in the compiled shader files from the mounted container, because the game already does that.

Engine changes

Due to the ordering of engine init, the shader library init is done before mounting the game containers. Therfore, the shader library is populated only with the engine information by the time mounting is done. To workaround this, after the container mounting is done, the code opens the shader library, ready to receive the compiled shaders from the game - both the global shader map and the game ones. Note that since Subnautica 2 uses shader sharing (bShareMaterialShaderCode=true), all shaders not in global are in one file - so I haven't tested to see if this code would just work if a game does not use shader sharing (which produces many shader files).

Then, when a material is opened or referenced in an opened/loaded asset, the shader library will read in the shader maps from the compiled shaders as it comes. This also means that loads of time doesn't need to be spent at editor startup loading in the entire shader file if some of it isn't even going to be used in that session.

And as usual, I made it enabled by default but with the ability to be disabled with a startup flag -DisableCookedShaderLibrary.

Note (TODO remove when fixed): You also need to close the shader library on engine shutdown otherwise the editor will crash when being closed.

Extra utilities

Since you are making engine changes anyway, it might be worth to add useful little utilities to help facilitate working with cooked content even better:

  • Cooked niagara asset viewer
  • Duplicate cooked widget to uncooked widget
  • Duplicate cooked blueprint to uncooked blueprint

Technically all of these can be implemented in editor plugins using the engine API, (aside from a couple of tiny engine patches to make them work properly) but I think that its much easier to implement them in the engine directly to avoid being limited by engine API without big changes required.

Cooked niagara asset viewer

While the engine already provides relatively solid code paths for viewing cooked content for most asset types, one type that (as of 5.6) has no read-only viewer is niagara effect. Like blueprints and materials, the editor-only metadata (such as kismet node graph) is stripped from cooked assets. So when you try to open this asset, it will just crash, as the editor only has code paths for trying to directly load its metadata as if its uncooked.

Therefore, I decided to take a page out of the read-only blueprint code by implementing my own read-only viewer for niagara assets. This viewer shows all of the properties of the asset as well of each effect created inside of it. This is useful for providing more easily obtainable info about the asset in the editor rather than having to rely on third party tools like FModel (which is also much harder to read/understand than in the editor) - for stuff like copying the effect's behaviours or to modify at runtime.

Engine change

Create uncooked blueprint/widget from cooked blueprint/widget

When working with cooked blueprints & widgets, you need to create a child to open it up to see what's inside (which is also uncooked), as otherwise opening it directly just shows the fallback asset editor view. In the case of widgets, since it's a child, you cannot see the original widget tree nor modify it. But at least for blueprints, you can see the component tree as that is copied across into the child.

So what would be nice, is to have an option to create a copy of it as an uncooked widget. This allows for much easier widget modding because:

  • Mods can make their own widget using the basis or the styling of an existing game widget, without having to create and modify a copy of the game widget at runtime
  • It's way, way easier to know how to modify a game widget at runtime if you can actually see the widget tree in the editor, as the alternative is to use UE4SS live viewer or SDK dumps which are not easy to read at all!

This engine change adds a button to the right click menu on a cooked widget in the content browser. At the top of the context menu, there is a "Make Uncooked Widget Copy" button which asks for destination folder and deep copies the full cooked widget tree and animations into an uncooked widget. Notice that I also needed to make a small engine patch to fix a bug relating to BindWidget properties - but you can ignore this as this is another limitation of Suzie which you will not be using.

I would like to do the same thing for blueprints which copies the component tree, the functions, properties and events. It could even be possible to reconstruct the kismet graph code from the script bytecode, though many have tried in the past to do it from script bytecode JSON produced by FModel, but as its a lot of work it has not been achieved before.

Monolithic editor

A monolithic editor is when the entire engine and project are built into a single executable. This is what UEFN uses as it is then much easier to distribute - with the downside that every time the game updates, the entire executable needs rebuilding and updating, whereas with a standalone editor and engine, the engine may not need to be redistributed if it has not changed since the last version.

That being said I have not tried it and know nothing about how it works or is created, so I am just mentioning it as a potential research point. I have also heard that a modder once created a monolithic build for a UE4 cooked editor project but they deleted the code so I can't verify if that was really the case.