Going Commando

The stunt I’m about to describe was born of poor engineering judgment and bad coding decisions. Come, and bear witness the game development equivalent of Underwater Fart Fire.

The goal

• Run a bare Unreal Engine module (completely outside the engine)
• In a console app (on Windows, but no Windows)

The purpose

• Unit Tests
  …and, to avoid sorting through all the broken links and blog posts that once upon a time enumerated correct ways to Unit Test on UE projects (via plugins, engine support, etc etc). Also, because the UE Editor takes a really really long time to load, and then, really, how much are you isolating your target code when it’s hosted inside 7gb of DLLs running 57 threads of stuff thousands of developers have contributed to?

Plus, it’s a lot more fun and edutainment fodder tinkering with the inner workings of UE hands on, as opposed to reading a bunch of boring stuff about the “right” way to do this.

TL;DR Version

If you’d like to skip the details and methodologies, feel free to skip to the TL;DR version at the end.

LoadLibrary, GetProcAddress

When you write C++ code for an Unreal Engine project, whether or not you know it you’re developing an Unreal Engine module. On Windows, Unreal Engine implements modules as Windows DLLs.

DLL’s can be loaded at runtime via the Windows API LoadLibrary. Once a DLL is loaded you can get its exported functions via GetProcAddress. If we can load our module’s DLL and call a method we’ll validate that the whole idea here - running a UE module outside of UE - just might be crazy enough to work.

LoadLibrary from Console (command-line) app

So, we create the most basic console app we can in Visual Studio and flesh out the main function:

#include "CoreMinimal.h"
#include "Modules\ModuleInterface.h"

#include <libloaderapi.h>

typedef IModuleInterface* (*FInitializeModuleFunctionPtr)(void);

int main()
{
    HMODULE hDLL = LoadLibrary(TEXT("UnrealEditor-Spice.dll"));

    FInitializeModuleFunctionPtr InitializeModuleProc = (FInitializeModuleFunctionPtr)reinterpret_cast<void*>(GetProcAddress(hDLL, "InitializeModule"));
    
    IModuleInterface* ModuleInterface = InitializeModuleProc();

    FreeLibrary(hDLL);
}

The module I’d like to run outside of Unreal Engine is the Spice module, from MaxQ. This module contains precision algorithms and code originally authored by NASA/JPL. Unit Testing the UE5 wrapper around Spice will be helpful to ensure the UE wrappers do not break SPICE.

Your fake Unreal Engine “host application” will need to keep the Unreal Engine header files happy via a few preprocessor definitions. An example:

#define UE_BUILD_DEBUG 1
#define WITH_EDITOR 0
#define WITH_ENGINE 0
#define WITH_UNREAL_DEVELOPER_TOOLS 0
#define WITH_PLUGIN_SUPPORT 0
#define IS_MONOLITHIC 1
#define IS_PROGRAM 1
#define WITH_SERVER_CODE 0

#define UBT_COMPILED_PLATFORM Windows
#define PLATFORM_WINDOWS 1

#define ENGINE_API __declspec( dllimport )
#define CORE_API __declspec( dllimport )
#define COREUOBJECT_API __declspec( dllimport )

#define _AMD64_

// UE5 EA declares a local variable named "X64" in two header files, which collides on the X64 def.
// https://github.com/EpicGames/UnrealEngine/pull/8989
#if defined(X64)
#undef X64
#define X64_UNDEFINED
#endif

#include "CoreMinimal.h"

#if defined(X64_UNDEFINED)
#undef X64_UNDEFINED
#define X64
#endif

#include "Modules/ModuleInterface.h"

#include "UObject/ObjectMacros.h"
#include "UObject/ScriptMacros.h"
#include "UObject/Class.h"
#include "Kismet/BlueprintFunctionLibrary.h"

Ensuring our console app can find the UE Module DLL file

For now, the easiest way to ensure the DLL is findable by the host app is to just drop the DLL into the same folder. Also, we don’t want to #include the full allotment of Windows header files - they’ll conflict with the Unreal Engine headers that do such a fine job of insulating us from the actual platform. Fortunately, we can get away with just including libloaderapi.h here rather than windows.h – and LoadLibrary will work fine.

What time is it? It’s moment of truth time! So, hit F5 and see if this works!

Whomp whomp…. due to a lack of forethought (on my part), this first try is a total whiff. 0x00000000. No dice. No DLL. Nothing. We watch the VS debug output for any sign of a DLL load - nothing.

So, what’s gone wrong here? Welp, VS defaulted the console app’s platform to X86. And of course, our dll is X64. They’re incompatible.

Even though you see the module’s DLL is clearly sitting a directory named Binaries\Win64, it is useful to know how to verify, yes, a DLL is truly a 64-bit DLL, It’s time to introduce a favorite Windows development tool: dumpbin.exe.

Determining DLL platform (X86 vs X64)

dumpbin.exe /HEADERS

If you’re still learning C++, Unreal Engine, etc and noticed your Visual Studio installed a couple of VS “Command Prompt” links - and didn’t know what they might be useful for - now you’ll know. The links launch a command prompt window with environment variables pointing to some of the VS command line tools (the command line compiler, etc etc). One of the tools easily accessible from these command prompts is dumpbin.exe. If you launch of the “Visual Studio Developer Command Prompt” links, you’ll get a command prompt window:

**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.5
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>

C:\>dumpbin /?

From the VS Command prompt, type dumpbin /? you’ll see it’s help info:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>dumpbin /?
Microsoft (R) COFF/PE Dumper Version 14.29.30136.0
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: DUMPBIN [options] [files]

   options:

      /ALL
      /ARCHIVEMEMBERS
      /CLRHEADER
      /DEPENDENTS
      /DIRECTIVES
      /DISASM[:{BYTES|NOBYTES}]
      ...
      /UNWINDINFO

TL;DR - we can verify what’s in our module’s DLL using the dumpbin. For instance, dumpbin /HEADERS <file.dll>.

C:\>dumpbin /HEADERS C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

Dump of file C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
            8664 machine (x64)
               8 number of sections
        623854D5 time date stamp Mon Mar 21 18:35:01 2022
               0 file pointer to symbol table
               0 number of symbols
              F0 size of optional header
            2022 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   DLL

And, there it is - machine (x64) - conclusive proof our DLL is a 64-bit DLL. The directory name was telling the truth.

Debugging DLL Load Failures - GetLastError()

Alrighty, now that we’ve changed the project’s platform X64, we try our little console-app-that-can’t again, and this time the DLL loads. And, then the DLL immediately unloads. Like, immediately.

The VS output window will look something like:

Loaded 'UnrealEditor-Spice.dll'.
Unloaded 'UnrealEditor-Spice.dll'.

Normally, if loading the DLL encountered an error you’d get the error code by calling Windows API ‘GetLastError()'. However, this might be a PITA to get to given our iffy windows header situation. Recall, we can’t load the full Windows header set because we’re sneaking a little platform-specific functionality into our (mostly) platform-agnostic Engine. And, we can’t use Unreal Engine’s normal platform-agnostic stuff because our whole goal is to avoid running Unreal Engine… This situation is very frankenstenian! Since we’re in all out hack mode, apparently, … One solution would be to just inline our own declaration for GetLastError() so we can invoke it. Fair. However, a word of warning first.

Annoying bug: an ill declaration

One particularly annoying bug enountered while working at Microsoft was due to the above. The bug was caused by creating a duplicate declaration of a function - presumably to avoid including the original header file and a bunch of stuff that it in turn depended on. Let’s just sneak an alternate declaration into our source file!. Unfortunately, this created a situation in which the function call, when compiled against the correct header file is defined as using __cdecl calling conventions. But, the inlined function declaration resulted in a function call invocation that used the __stdcall calling convention. The resulting bug took some time to find, and was completely hidden at the source code level of the debugger. Anyways. Alternative declarations bad.

What Assembly Language Continues to be useful for

Back then (about 20 years ago, wow) it wasn’t TOO rare to sneak a small assembly language block into some code. This was particularly true for a few unusual assembly language instructions that weren’t supported by compilers yet… (Hello again, cpuid!)

Unless you work on a platform (like the smartest guy I ever worked with at Microsoft) you’ll probably never need to write a single line of .asm, and if you ever do, it will probably be a poor decision (not unlike this whole blog post). But, just like with the tangential story above - knowing some asm continues to be extremely helpful in debugging and determining wtf is happening inside the a bunch of black boxes our code always sits on top of. In this case, a working knowledge of assembly language lets us completely avoid calling GetLastError() to get an error code.

DLL Dependency Chains

Knowing a little assembly language, it’s easy enough to step into the LoadLibrary call then capture the error code out of a register. It’s error 3221225781. Aka 0xC0000135. This error generally happens when you’re attempting to load a DLL, but that DLL depends on another DLL - which cannot be found. Of course! We’ve built this UE module. But it, of course depends on some UE Engine modules. From Spice.Build.cs in MaxQ:

  PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine" });

That line allows our module to use the Unreal Engine Core, CoreUObject, and Engine modules.

So, to load the Spice module we also need the DLL’s for THESE modules.

Manually walking DLL dependency chains with dumpbin.exe

How can you find the EXACT list of DLL’s & their filenames the Spice module depends on? Back to our old friend, dumpbin - this time, with the /DEPENDENTS option. We can use dumpbin to list all the other dll’s the Spice module depends on.

C:\>dumpbin /DEPENDENTS C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

Dump of file C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

File Type: DLL

  Image has the following dependencies:

    UnrealEditor-Core.dll
    UnrealEditor-CoreUObject.dll
    UnrealEditor-Engine.dll
    MSVCP140.dll
    VCRUNTIME140.dll
    VCRUNTIME140_1.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    KERNEL32.dll

Ah, there’s the DLL module’s from the build settings: Core, CoreUObject, and Engine. And, KERNEL32.dll is an operating system DLL full of important stuff. The rest is VC runtime stuff.

Examining a DLL’s imports

So, how MUCH stuff in each of these modules is our Spice module using? Can we reduce it’s footprint a bit, or are we going to need all the DLLs listed above? Well, there’s a way to find out EXACTLY what, from each DLL, our module is using. Can you guess how? Can you can you? RIGHT! dumpbin /IMPORTS. We can use the option dumpbin /IMPORTS:UnrealEditor-Core.dll will list everything it needs from the Core module, or we can omit a filename and just get it all.

C:\>dumpbin /IMPORTS C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

Dump of file C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

File Type: DLL

  Section contains the following imports:

    UnrealEditor-Core.dll
                         ...
                         ...
                         428 ??1FStringFormatArg@@QEAA@XZ
                         274 ??0FStringFormatArg@@QEAA@AEBU0@@Z
                         27A ??0FStringFormatArg@@QEAA@VFString@@@Z
                         DDC ?Format@FString@@SA?AV1@PEB_WAEBV?$TArray@UFStringFormatArg@@V?$TSizedDefaultAllocator@$0CA@@@@@@Z
                         871 ?AppendChars@FString@@QEAAXPEB_WH@Z
                         86F ?AppendChars@FString@@QEAAXPEBDH@Z
                         F49 ?Get@IFileManager@@SAAEAV1@XZ
                        2014 ?UtcNow@FDateTime@@SA?AU1@XZ
                         6CC ??8@YA_NUFNameEntryId@@W4EName@@@Z
                         601 ??4FString@@QEAAAEAV0@AEBV0@@Z
                         271 ??0FString@@QEAA@AEBV0@@Z
                         26F ??0FString@@QEAA@$$QEAV0@@Z
                        ...

    UnrealEditor-CoreUObject.dll
                        ...
                        15E2 ?StaticClass@FFloatProperty@@SAPEAVFFieldClass@@XZ
                        15F0 ?StaticClass@FObjectPropertyBase@@SAPEAVFFieldClass@@XZ
                        1843 ?Z_Construct_UScriptStruct_FTransform@@YAPEAVUScriptStruct@@XZ
                        1810 ?Z_Construct_UScriptStruct_FColor@@YAPEAVUScriptStruct@@XZ
                        ...
                        1848 ?Z_Construct_UScriptStruct_FVector@@YAPEAVUScriptStruct@@XZ
                        ...
                         F55 ?GetTransientPackage@@YAPEAVUPackage@@XZ
                         B37 ?Get@FObjectInitializer@@SAAEAV1@XZ
                         40E ??1UObject@@UEAA@XZ
                 ...
                        183B ?Z_Construct_UScriptStruct_FQuat@@YAPEAVUScriptStruct@@XZ

    UnrealEditor-Engine.dll
                        1182 ??0UBlueprintFunctionLibrary@@QEAA@AEBVFObjectInitializer@@@Z
                        4AC0 ?GetFunctionCallspace@UBlueprintFunctionLibrary@@UEAAHPEAVUFunction@@PEAUFFrame@@@Z
                        3C61 ?DrawDebugLine@@YAXPEBVUWorld@@AEBUFVector@@1AEBUFColor@@_NMEM@Z
                        53E2 ?GetPrivateStaticClass@UBlueprintFunctionLibrary@@CAPEAVUClass@@XZ
                        980E ?Z_Construct_UClass_UBlueprintFunctionLibrary@@YAPEAVUClass@@XZ
                        9672 ?Z_Construct_UClass_AActor_NoRegister@@YAPEAVUClass@@XZ
                        1181 ??0UBlueprintFunctionLibrary@@QEAA@AEAVFVTableHelper@@@Z

    MSVCP140.dll
                         ...
                          2F ??0?$basic_streambuf@DU?$char_traits@D@std@@@std@@IEAA@XZ
                          89 ??1?$basic_streambuf@DU?$char_traits@D@std@@@std@@UEAA@XZ
                         ..

    VCRUNTIME140.dll
                          3C memcpy
                          ...

    VCRUNTIME140_1.dll
                           0 __CxxFrameHandler4

    api-ms-win-crt-math-l1-1-0.dll
                         112 sin
                          90 cos
                         116 sqrt
                         ...

    api-ms-win-crt-string-l1-1-0.dll
                          91 strnlen
                          89 strcpy_s
                          ...

    api-ms-win-crt-runtime-l1-1-0.dll
                          21 _errno
                          55 exit
                          ...

    api-ms-win-crt-heap-l1-1-0.dll
                          1A realloc
                          17 calloc
                          19 malloc
                          18 free

    api-ms-win-crt-stdio-l1-1-0.dll
                          ...

    api-ms-win-crt-filesystem-l1-1-0.dll
                          3F remove
                           0 _access

    api-ms-win-crt-convert-l1-1-0.dll
                          4F atof
                          50 atoi

    api-ms-win-crt-time-l1-1-0.dll
                          23 _localtime64
                          30 _time64

    KERNEL32.dll
                         27A GetModuleHandleW
                          86 CloseHandle
                         133 EnterCriticalSection
                         3BB LeaveCriticalSection
                         ...

Hmm… I think we’re going to need it all.

OS and CRT dependencies

The Runtime and OS dependencies - the DLL Loader knows where those files are. (Unless something’s gone horribly wrong.) Windows knows where these files are, and how to load them. The issue here that the loader can’t fine the Unreal Engine modules - Core, CoreUObject, and Engine.

DLLs usually depend on yet more DLLs

So, we copy UnrealEditor-Core.dll and friends into the same directory as our console-app-that-can’t, and try again. The result? 0xC0000135. Again.

But why? OH. Well, because THESE modules in turn depend on OTHER Unreal Engine modules. Okay, let’s set dumpbin.exe, the console-app-that-can, loose on those modules and see how much they depend on:

Core:

Dump of file C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64\UnrealEditor-Core.dll

File Type: DLL

  Image has the following dependencies:

    UnrealEditor-BuildSettings.dll
    UnrealEditor-TraceLog.dll
    <OS, CRT dlls>

  Image has the following delay load dependencies:

    WinPixEventRuntime.dll
    dbghelp.dll

The BuildSettings and TraceLog modules, and a bunch of OS and CRT stuff. Not too bad. And what do BuildSettings and TraceLog depend on?

Dump of file C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64\UnrealEditor-BuildSettings.dll

File Type: DLL

  Image has the following dependencies:

    <CRT, OS dlls>

Dump of file C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64\UnrealEditor-TraceLog.dll

File Type: DLL

  Image has the following dependencies:

    <CRT, OS dlls>

Both are UE module leaf nodes! Neither depends on any other UE modules. Yay! So, that’s 5 modules so far: Core, CoreUObject, Engine, BuildSettings, and TraceLog. What does CoreUObject depend on?

Dump of file C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64\UnrealEditor-CoreUObject.dll

File Type: DLL

  Image has the following dependencies:

    UnrealEditor-Projects.dll
    UnrealEditor-Json.dll
    UnrealEditor-Core.dll
    UnrealEditor-TraceLog.dll
    <CRT, OS dlls>

Core, CoreUObject, Engine, BuildSettings, TraceLog, Json now. Json is another leaf node IIRC. For modules that don’t depend on UnrealEditor-Engine.dll, we can just copy this subset of UE modules to our directory and run a bit of code.

But, the Spice module implements a Blueprints function library. And, if you recall from the Spice module’s imports above:

    UnrealEditor-Engine.dll
                        1182 ??0UBlueprintFunctionLibrary@@QEAA@AEBVFObjectInitializer@@@Z
                        4AC0 ?GetFunctionCallspace@UBlueprintFunctionLibrary@@UEAAHPEAVUFunction@@PEAUFFrame@@@Z

Reading through the name mangling, we can tell the base class for Blueprint Function Libraries is implemented in the Engine module. Yes, we definitely need UnrealEditor-Engine.dll. So, we take a deep breath and check it’s dependencies:

Dump of file C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64\UnrealEditor-Engine.dll

File Type: DLL

  Image has the following dependencies:

    UnrealEditor-EditorAnalyticsSession.dll
    UnrealEditor-AppFramework.dll
    UnrealEditor-Landscape.dll
    UnrealEditor-UMG.dll
    UnrealEditor-Projects.dll
    UnrealEditor-TypedElementFramework.dll
    UnrealEditor-TypedElementRuntime.dll
    UnrealEditor-MaterialShaderQualitySettings.dll
    UnrealEditor-Analytics.dll
    UnrealEditor-AudioMixer.dll
    UnrealEditor-SignalProcessing.dll
    UnrealEditor-CrunchCompression.dll
    UnrealEditor-TraceLog.dll
    UnrealEditor-RawMesh.dll
    UnrealEditor-EditorStyle.dll
    UnrealEditor-PerfCounters.dll
    UnrealEditor-ImageCore.dll
    UnrealEditor-DeveloperToolSettings.dll
    UnrealEditor-ClothingSystemEditorInterface.dll
    UnrealEditor-Core.dll
    UnrealEditor-CoreUObject.dll
    UnrealEditor-NetCore.dll
    UnrealEditor-ApplicationCore.dll
    UnrealEditor-Json.dll
    UnrealEditor-SlateCore.dll
    UnrealEditor-Slate.dll
    UnrealEditor-InputCore.dll
    UnrealEditor-RenderCore.dll
    UnrealEditor-AnalyticsET.dll
    UnrealEditor-RHI.dll
    UnrealEditor-AssetRegistry.dll
    UnrealEditor-EngineMessages.dll
    UnrealEditor-EngineSettings.dll
    UnrealEditor-GameplayTags.dll
    UnrealEditor-PacketHandler.dll
    UnrealEditor-AudioPlatformConfiguration.dll
    UnrealEditor-MeshDescription.dll
    UnrealEditor-StaticMeshDescription.dll
    UnrealEditor-PakFile.dll
    UnrealEditor-PhysicsCore.dll
    UnrealEditor-AudioExtensions.dll
    UnrealEditor-DeveloperSettings.dll
    UnrealEditor-PropertyAccess.dll
    UnrealEditor-UnrealEd.dll
    UnrealEditor-Kismet.dll
    UnrealEditor-Chaos.dll
    UnrealEditor-ClothingSystemRuntimeInterface.dll
    <CRT, OS dlls>

  Image has the following delay load dependencies:

    libvorbisfile_64.dll

FUUUUUUUUUU….

CK.

To Unit Test our wrapper around NASA’s SPICE library, … we have to load up Landscaping? Yes. And ClothingSystemEditorInterface. ClothingSystemRuntimeInterface. AudioMixer. For the love of GEZ we have to load up Slate, UMG and half of UE’s Physics? Where the hell have we gone wrong?????

And these DLL’s depend on more DLL’s which depend on yet more DLL’s which depend on more. And more. Etc. It is like the world’s WORST pyramid scheme!! A pyramid scheme that buries you in DLL dependencies!! I’d rather have money!!

Another way to get the full list of dependencies a dll needs.

An alternative approach is just to make all the needed DLL’s available, and then copy the list of DLL’s that loaded from the VS Debug Output window. Three possibilities:

• Update your file system path to add the UE Binaries directory
  C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64 for 5.0 Early Access/Windows
• Copy all the binaries from Engine/Binaries/Win64 to the console app’s binary directory
• Temporarily copy your console app to Engine/Binaries/Win64 and launch it under a debugger.

The full set of dependencies is ~135 files. It’s easy enough to massage the list extracted in the debug window into a little batch file that copies them wherever needed etc.

FINALLY, after copying the full set of 100+ dll’s into our console app’s directory – we can load the Spice module DLL. Onerous, sure, but still progress!! We’ve lost some of the advantage we were hoping for in that it takes a second or two or three to load all these DLL’s when we launch our console app - but it’s nothing like a full launch of the Unreal Engine editor!!!

Done! check(); please

Great! Now that we can actually load our DLL, let’s try calling something in Unreal Engine itself. As a test, We can call check() to DebugBreak if the DLL failed to load. Note, check() is intended for development-time validation, NOT checking for runtime errors!!… But that’s okay, it’s a simple engine call we can try out just for kicks.

#include "CoreMinimal.h"
#include "Modules\ModuleInterface.h"

#include <libloaderapi.h>

typedef IModuleInterface* (*FInitializeModuleFunctionPtr)(void);

int main()
{
    HMODULE hDLL = LoadLibrary(TEXT("UnrealEditor-CelestialMath.dll"));
    check(hDLL);

    FInitializeModuleFunctionPtr InitializeModuleProc = (FInitializeModuleFunctionPtr)reinterpret_cast<void*>(GetProcAddress(hDLL, "InitializeModule"));
    
    IModuleInterface* ModuleInterface = InitializeModuleProc();

    FreeLibrary(hDLL);
}

Now, if we try to build we’ll get a linker error:

error LNK2001: unresolved external symbol "__declspec(dllimport) private: static void __cdecl FDebug::CheckVerifyFailedImpl(struct FDebug::FFailureInfo const &,wchar_t const *,...)" (__imp_?CheckVerifyFailedImpl@FDebug@@CAXAEBUFFailureInfo@1@PEB_WZZ)

This makes sense. The Spice module’s already laid out which DLL’s it needs in its build settings, but we’ve done nothing to add libs to the console App. It doesn’t know where to find this function (yet).

Explicit vs implicit dll linking

So, let’s think about this. Should we, then, also load the UE modules via LoadLibrary and GetProcAddress? Well, we could. But especially with name decoration etc that would be… a bit much. This is called “explicit linking”. It’s a PITA, and avoided in cases where it’s avoidable.

Instead, what we want to do is “implicit linking”. This means we link our console app to a corresponding static library, which contains stubs for the DLL… For instance, UnrealEditor-Core.lib. This links our console app as if the core module were statically linked. The Windows OS then takes responsibility for loading the DLL at runtime and binding its entry points to the static library stubs… All the while being seamless to us as if the whole lot was actually inside a static lib. And, if the DLL load fails Windows will display a nice dialog box informing you of the missing DLL (as opposed to failing quietly inside a Windows API).

Taking this idea a bit farther, then, …. why the hell are we explicitly linking the Spice module via LoadLibrary and GetProcAddress? Good question! We were just doing some experimental programming to prove/disprove the concept here. Again, the concept is - we’re trying to run a module built for Unreal Engine - completely outside of Unreal Engine. Anyways, let’s implicitly link the Unreal Engine modules, and also our own modules - so that we don’t have to manage LoadLibrary and GetProcAddress type stuff anymore. We have the .lib files, soooo let’s use them!

Goodbye, LoadLibrary! Hello, Success!!

So, does this mean we need to link to all ~135 Unreal Engine static .lib files corresponding to all the DLL files?

No!!

It does NOT mean we need to link to 135 UE .lib files. We only need to link to .lib files containing any entry points we actually use.

So, we want to call Unreal’s check(). This macro is implemented as CheckVerifyFailedImpl. And where is this function defined? Per dumpbin, the module we need is Core:

    UnrealEditor-Core.dll
...
          9FA ?CheckVerifyFailedImpl@FDebug@@CAXAEBUFFailureInfo@1@PEB_WZZ

Easy. We add UnrealEditor-Core.lib to our linker inputs, and viola, the linker error disappears. We can run our console app, step through code… If we add a check(false); we can verify it behaves correctly. And, it does!!

Great, so, … It’s time to get aggressive here!

We try a few things out, … and quickly find we won’t be able to create UObjects.

NewObject<UObject>(outer, FName("My Object"))' fails. It hits an assertion that allocating a UObject would exceed the maximum allowable number of UObjects, which is set to 0. This makes sense, as we’ve done nothing to initialize garbage collection, etc etc etc. If you look through the engine code you’ll find this is really a bridge too far. To get UObjects working, you’ll pretty much need to recreate the entire engine inside the console app host, which defeats the purpose of going commando. If you really really felt the need to make UObjects work - you could do it using an officially supported path… You’d build your UE project as a library. See:

https://docs.unrealengine.com/4.27/en-US/SharingAndReleasing/BuildAsALibrary/

In this case, we’re just going to have to live with the limitation that we can’t actually allocate any UObjects.

And in this case, that’s okay. The only things we’re interested in testing are:

• USpice Methods (USpice is a Blueprint function library, and all its methods are static)
• Spice Struct types

Nearly all the functionality in MaxQ is static with respect to UObject instances. This is an unusual case, but thanks to circumstances this whole scheme is viable (yet still ill advised. Remember, this is just an edutainment venture. Nothing more!!)

GoogleTest

We can use Google’s unit test framework to do some unit testing. It’s integrated with MSVC (assuming you’ve installed GoogleTest via the VS Installer).

And, as we start writing a few tests, we find a few more wrinkles. There are a few minor things that will throw an exception at us due to the engine not running. For instance, the Engine API to return the project directory throws an exception. No big deal though.

Unreal Engine Macro Magic (Broken)

The other thing that happens is - anytime we spot something in the main Spice header files (Spice.h, SpiceTypes.h, etc) and fix it… The build console app build seems to break. We get errors such as.

SpiceTypes.h(1732,5): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int

If we look at the line of code, we just see GENERATED_BODY(). So, apparently the macro expands to something, and the something probably references a few things that were supposed to be defined by SpiceTypes.generated.h, but it’s going wrong now.

But what’s going wrong? Ugh a bunch of macro magic. We can try to find it by code-inspecting the UE macros, but there’s an easier way. We can tell VS to emit a pre-processed version of our source file, then see exactly how the macro mis-expanded.

Viewing pre-processed source files

There’s a great trick in deciphering where the UE macro magic is going wrong. The VS `/P/ option will expand all the preprocessor definitions and write the result to a file.

Using this option, and then taking a view at the point of failure in the preprocessed file, we see:

MaxQ_Source_MaxQ_Spice_Public_SpiceTypes_h_1732_GENERATED_BODY;

The GENERATED_BODY() macro is looking for this definition. Apparently, SpiceTypes.generated.h was supposed to define it. But, it doesn’t. Let’s look inside and see what it DOES define, though.

SpiceTypes.generated.h:

#define MaxQ_Source_MaxQ_Spice_Public_SpiceTypes_h_1729_GENERATED_BODY \
	friend struct Z_Construct_UScriptStruct_FSMassConstant_Statics; \
	SPICE_API static class UScriptStruct* StaticStruct();

GENERATED_BODY() uses its source file line # in definitions

In the above, we can see the .generated.h file defined something, while making reference to the source file and line number of something that the GENERATED_BODY() macro in our header file is going to look for later. That’s how all this mixing and matching of our actual header file, and the .generated.h header file work. The .generated file defines something by line number. The GENERATED_BODY() macro looks for it by line number. Editing your header file then changes the line numbers and breaks the links - until it’s fixed up again.

This means….

If you change the header file, you have to go back and compile the original project to re-generate the matching *.generated.h. Then, external-from-UE projects will correctly interpret GENERATED_BODY() and compile correctly.

Anyways, all we need to do is regenerate SpiceTypes.generated.h. We can do this by rebuilding the original project, MaxQ.

Manually invoking Unreal’s Header Tool

But, there’s a shortcut if we’re interested. We don’t need to rebuild the whole MaxQ project. Rather, to rebuild the autogenerated headers we can invoke the UnrealEngine Header Tool manually. If we look at the compilation output window when we build MaxQ, we can spot the command line that the Unreal Build System used to invoke the Header Tool:

UnrealHeaderTool "C:\git\spaceflight\MaxQ\Spice.uproject" "C:\git\spaceflight\MaxQ\Intermediate\Build\Win64\SpiceEditor\Development\SpiceEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -abslog="C:\Users\imake\AppData\Local\UnrealBuildTool\Log_UHT.txt" -installed

No problem. Any time we change the MaxQ headers while working on the console app, we just re-run the header tool for MaxQ - and viola, the console app is buildable again.

C:\Users\cn>cd "C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64"

C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64>
C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64>UnrealHeaderTool "C:\git\spaceflight\MaxQ\Spice.uproject" "C:\git\spaceflight\MaxQ\Intermediate\Build\Win64\SpiceEditor\Development\SpiceEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -WarningsAsErrors -abslog="C:\Users\cn\AppData\Local\UnrealBuildTool\Log_UHT.txt" -installed

C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64>type C:\Users\imake\AppData\Local\UnrealBuildTool\Log_UHT.txt
LogInit: Build: ++UE5+Release-5.0-EarlyAccess-CL-16682836
LogInit: Engine Version: 5.0.0-16682836+++UE5+Release-5.0-EarlyAccess
LogInit: Compatible Engine Version: 5.0.0-16433597+++UE5+Release-5.0-EarlyAccess
...

OH. EMM. GEE! It’s really working!

The SPICE module, as well as most of what we want from UE are now callable just fine - completely outside of Unreal Engine. Enough of it works, that we could GoogleTest the Spice Module and validate that the wrapper around SPICE isn’t introducing errors. Hooray! It’s working!!

And with all that… MaxQ has some (ill-advised) unit tests. You can see a working example of the whole scheme at the MaxQ github. Look for the secondary .sln, MaxQTests, in the MaxQ\ExternalTests directory.

Crazy. Just crazy. Am I right? :-D.

[==========] Running 77 tests from 66 test cases.
[----------] Global test environment set-up.
[----------] 1 test from clear_all_test
[ RUN      ] clear_all_test.DefaultsTestCase
[       OK ] clear_all_test.DefaultsTestCase (3 ms)
[----------] 1 test from clear_all_test (4 ms total)

...

[----------] 3 tests from init_all_test
[ RUN      ] init_all_test.DefaultsTestCase
[       OK ] init_all_test.DefaultsTestCase (1 ms)
[ RUN      ] init_all_test.Execution_ClearsKernel
[       OK ] init_all_test.Execution_ClearsKernel (2 ms)
[ RUN      ] init_all_test.Execution_ClearsError
C:\git\spaceflight\MaxQ\ExternalTests\MaxQ\Spice\init_all.cpp(51): error: Expected equality of these values:
  errorOutput.Len()
    Which is: 40
  0
[  FAILED  ] init_all_test.Execution_ClearsError (28009 ms)
[----------] 3 tests from init_all_test (28015 ms total)

...

[----------] Global test environment tear-down
[==========] 77 tests from 66 test cases ran. (28092 ms total)
[  PASSED  ] 76 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] init_all_test.Execution_ClearsError

 1 FAILED TEST

Working USpice Unit Tests!!! Runnable outside of UE! (Almost) zero load time!

Looks like I have an issue to fix :-D.

TL;DR Recap

Yes, you can run a subset of your UE modules outside of Unreal Engine. You won’t be able to allocate any new UObjects, however. Any UE API calls that require the engine to be running will obviously fail.

The steps host your UE module, commando style:

1 Create your host

Create the application to host your UE module. In the case above, we developed a windows console app as a host application.

2 Copy UE module DLLs (Core, CoreUObject, Engine, etc)

You’ll need to ensure the host application can load whatever subset of Unreal Engine modules it requires. In the example above, the Visual Studio solution was configured to copy the dll’s into the build directory - see the example on GitHub to see how this worked. Copying the Unreal Engine module DLL’s to the same directory as the host app is one way of ensuring the host app can find them at load time. For the UE 5.0 Early Access build, the files are here: C:\Program Files\Epic Games\UE_5.0EA\Engine\Binaries\Win64 The full set was ~135 dll files - UnrealEditor-Core.dll, UnrealEditor-CoreUObject.dll, UnrealEditor-Engine.dll, etc. The full list was obtained by temporarily copying all of them, then copying the list of files from the Debug Output window, which logs every time a DLL is loaded.

3 Link to UE module .lib files

You don’t need to link your project to the .lib file for every DLL it uses. You only need to link to the modules you call from your host app. In the example above, these libraries where linked (via the VS Linker options):
"C:\Program Files\Epic Games\UE_5.0EA\Engine\Intermediate\Build\Win64\UnrealEditor\Development\Core\UnrealEditor-Core.lib"
"C:\Program Files\Epic Games\UE_5.0EA\Engine\Intermediate\Build\Win64\UnrealEditor\Development\CoreUObject\UnrealEditor-CoreUObject.lib"

4 Repeat the above for your own UE modules

In the example above, we added UnrealEditor-Spice.dll, and linked to UnrealEditor-Spice.lib.

5 Include some header stuff in your host application.

You’ll need some definitions the UE header files look for. These will look like:

#define UE_BUILD_DEBUG 1
#define WITH_EDITOR 0
#define WITH_ENGINE 0
#define WITH_UNREAL_DEVELOPER_TOOLS 0
#define WITH_PLUGIN_SUPPORT 0
#define IS_MONOLITHIC 1
#define IS_PROGRAM 1
#define WITH_SERVER_CODE 0

#define UBT_COMPILED_PLATFORM Windows
#define PLATFORM_WINDOWS 1

#define ENGINE_API __declspec( dllimport )
#define CORE_API __declspec( dllimport )
#define COREUOBJECT_API __declspec( dllimport )

#define _AMD64_

// UE5 EA declares a local variable named "X64" in two header files, which collides on the X64 def.
// https://github.com/EpicGames/UnrealEngine/pull/8989
#if defined(X64)
#undef X64
#define X64_UNDEFINED
#endif

#include "CoreMinimal.h"

#if defined(X64_UNDEFINED)
#undef X64_UNDEFINED
#define X64
#endif

#include "Modules/ModuleInterface.h"

// C:\Program Files\Epic Games\UE_5.0EA\Engine\Source\Runtime\CoreUObject\Public
#include "UObject/ObjectMacros.h"
#include "UObject/ScriptMacros.h"
#include "UObject/Class.h"
#include "Kismet/BlueprintFunctionLibrary.h"

You’ll also need to define the _API macros for any of your own modules you intend to call, as well as any other UE modules. In the example above using the Spice module, a line was added for:

#define SPICE_API __declspec( dllimport )

6 Fix up *.generated.h files via UE Header Tool

Any time the headers in the original project change, you’ll either need to rebuild the original project - or minimally, invoke the UE header tool. In the example above the header tool was invoked as:

UnrealHeaderTool "C:\git\spaceflight\MaxQ\Spice.uproject" "C:\git\spaceflight\MaxQ\Intermediate\Build\Win64\SpiceEditor\Development\SpiceEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -abslog="C:\Users\imake\AppData\Local\UnrealBuildTool\Log_UHT.txt" -installed

Ta da!!

That’s the bulk and general gist of making a UE module work outside of UE. (Well… Tbh only a small subset “works”). You won’t be able to allocate any UObjects and a bunch of other stuff will be broken. And, the whole scheme could break at any moment due to Epic’s changes, either intentionally or not. It’s very ill advised, and only applicable to a narrow range of circuimstances. Such as… Learning how the engine works, hands-on.

But it’s still kinda cool calling into your own module, as well as into UE, from fully outside the engine’s host environment, ya? Ya.

Happy Unit Testing, edutainment, or whatever else you may seek :).

PS (Random stuff)

Viewing Exports, via dumpbin.exe /EXPORTS

We saw how to get a dll’s imports earlier. Sometimes you need to know the reverse - what a dll exports.

C:\>dumpbin /EXPORTS C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

Dump of file C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

File Type: DLL

  Section contains the following exports for UnrealEditor-Spice.dll

        1442 number of functions
        1442 number of names

    ordinal hint RVA      name

          1    0 0003BEA0 ??$StaticClass@VUSpice@@@@YAPEAVUClass@@XZ = ??$StaticClass@VUSpice@@@@YAPEAVUClass@@XZ (class UClass * __cdecl StaticClass<class USpice>(void))
          2    1 00066A10 ??$StaticClass@VUSpiceK2@@@@YAPEAVUClass@@XZ = ??$StaticClass@VUSpiceK2@@@@YAPEAVUClass@@XZ (class UClass * __cdecl StaticClass<class USpiceK2>(void))
          3    2 00068750 ??$StaticClass@VUSpiceOrbits@@@@YAPEAVUClass@@XZ = ??$StaticClass@VUSpiceOrbits@@@@YAPEAVUClass@@XZ (class UClass * __cdecl StaticClass<class USpiceOrbits>(void))
          4    3 0006EEE0 ??$StaticClass@VUSpiceTypes@@@@YAPEAVUClass@@XZ = ??$StaticClass@VUSpiceTypes@@@@YAPEAVUClass@@XZ (class UClass * __cdecl StaticClass<class USpiceTypes>(void))
          5    4 0006EEF0 ??$StaticEnum@W4ES_AberrationCorrection@@@@YAPEAVUEnum@@XZ = ??$StaticEnum@W4ES_AberrationCorrection@@@@YAPEAVUEnum@@XZ (class UEnum * __cdecl StaticEnum<enum ES_AberrationCorrection>(void))
          6    5 0006EF30 ??$StaticEnum@W4ES_AberrationCorrectionForOccultation@@@@YAPEAVUEnum@@XZ = ??$StaticEnum@W4ES_AberrationCorrectionForOccultation@@@@YAPEAVUEnum@@XZ (class UEnum * __cdecl StaticEnum<enum ES_AberrationCorrectionForOccultation>(void))
          7    6 0006EF70 ??$StaticEnum@W4ES_AberrationCorrectionFov@@@@YAPEAVUEnum@@XZ = ??$StaticEnum@W4ES_AberrationCorrectionFov@@@@YAPEAVUEnum@@XZ (class UEnum * __cdecl StaticEnum<enum ES_AberrationCorrectionFov>(void))
          8    7 0006EFB0 ??$StaticEnum@W4ES_AberrationCorrectionLocus@@@@YAPEAVUEnum@@XZ = ??$StaticEnum@W4ES_AberrationCorrectionLocus@@@@YAPEAVUEnum@@XZ (class UEnum * __cdecl StaticEnum<enum ES_AberrationCorrectionLocus>(void))
          ...
          ... ...
          ... ... ...

PPS (Random stuff)

Assembly language dump, via dumpbin.exe /DISASM

Dumpbin is also useful if you want a quick glance at a disassembly… Just redirect the output into a file, and you’ll have an .asm dump – of the entire file.

C:\>dumpbin /DISASM:NOBYTES C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

Dump of file C:\git\spaceflight\MaxQ\Binaries\Win64\UnrealEditor-Spice.dll

File Type: DLL

  0000000180001000: CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC  ÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌ
VectorSinConstantsSSE::`dynamic initializer for 'A'':
  0000000180001010: movss       xmm0,dword ptr [1802F5008h]
  0000000180001018: shufps      xmm0,xmm0,0
  000000018000101C: movaps      xmmword ptr [18031A3C0h],xmm0
  0000000180001023: ret
  0000000180001024: CC CC CC CC CC CC CC CC CC CC CC CC              ÌÌÌÌÌÌÌÌÌÌÌÌ
GlobalVectorConstants::`dynamic initializer for 'AllMask'':
  0000000180001030: movdqa      xmm0,xmmword ptr [__xmm@ffffffffffffffffffffffffffffffff]
  0000000180001038: movdqa      xmmword ptr [18031A1D0h],xmm0
  0000000180001040: ret
  0000000180001041: CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC     ÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌ
GlobalVectorConstants::`dynamic initializer for 'AnimWeightThreshold'':
  0000000180001050: movaps      xmm0,xmmword ptr [__xmm@3727c5ac3727c5ac3727c5ac3727c5ac]
  0000000180001057: movaps      xmmword ptr [18031A300h],xmm0
  000000018000105E: ret
  000000018000105F: CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC  ÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌ
  000000018000106F: CC                                               Ì
VectorSinConstantsSSE::`dynamic initializer for 'B'':
  0000000180001070: movss       xmm0,dword ptr [1802F500Ch]
  0000000180001078: shufps      xmm0,xmm0,0
  000000018000107C: movaps      xmmword ptr [18031A330h],xmm0
  0000000180001083: ret
  0000000180001084: CC CC CC CC CC CC CC CC CC CC CC CC              ÌÌÌÌÌÌÌÌÌÌÌÌ
GlobalVectorConstants::`dynamic initializer for 'BigNumber'':
  0000000180001090: movaps      xmm0,xmmword ptr [__xmm@7f7fc99e7f7fc99e7f7fc99e7f7fc99e]
  0000000180001097: movaps      xmmword ptr [18031A250h],xmm0
  000000018000109E: ret
  000000018000109F: CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC  ÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌ
  00000001800010AF: CC                                               Ì
Chaos::`dynamic initializer for 'ChaosVersionString'':
  00000001800010B0: sub         rsp,28h
  00000001800010B4: mov         edx,25h
  00000001800010B9: lea         rcx,[18031A490h]
  00000001800010C0: call        ?ResizeTo@?$TArray@_WV?$TSizedDefaultAllocator@$0CA@@@@@AEAAXH@Z
  00000001800010C5: mov         edx,dword ptr [18031A498h]
  00000001800010CB: lea         eax,[rdx+25h]
  00000001800010CE: mov         dword ptr [18031A498h],eax
  00000001800010D4: cmp         eax,dword ptr [18031A49Ch]
  00000001800010DA: jle         00000001800010E8
  00000001800010DC: lea         rcx,[18031A490h]
  00000001800010E3: call        ?ResizeGrow@?$TArray@_WV?$TSizedDefaultAllocator@$0CA@@@@@AEAAXH@Z
  00000001800010E8: mov         r8d,4Ah
  00000001800010EE: lea         rdx,[??_C@_1EK@LGIINMEI@?$AA6?$AA6?$AA7?$AA1?$AA1?$AAB?$AA0?$AA3?$AA?9?$AAE?$AA7?$AAB?$AA8?$AA?9?$AA4@]
  00000001800010F5: mov         rcx,qword ptr [18031A490h]
  00000001800010FC: call        qword ptr [__imp_?Memcpy@FGenericPlatformString@@CAPEAXPEAXPEBX_K@Z]
  0000000180001102: nop
  0000000180001103: lea         rcx,[18020C230h]
  000000018000110A: add         rsp,28h
  000000018000110E: jmp         atexit
  0000000180001113: CC CC CC CC CC CC CC CC CC CC CC CC CC           ÌÌÌÌÌÌÌÌÌÌÌÌÌ
GlobalVectorConstants::`dynamic initializer for 'DEG_TO_RAD'':
  0000000180001120: movaps      xmm0,xmmword ptr [__xmm@]
  0000000180001127: movaps      xmmword ptr [18031A230h],xmm0
  000000018000112E: ret
  000000018000112F: CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC  ÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌ
  000000018000113F: CC                                               Ì
GlobalVectorConstants::`dynamic initializer for 'DEG_TO_RAD_HALF'':
  0000000180001140: movaps      xmm0,xmmword ptr [__xmm@3c0efa353c0efa353c0efa353c0efa35]
  0000000180001147: movaps      xmmword ptr [18031A3A0h],xmm0
  000000018000114E: ret

You know why memory is padded with the value ‘CC’, right? If that’s the only assembly language thing you know, at least know that one :-D.