Adding Third-Party Libraries to Unreal Engine : NASA's SPICE Toolkit (Part 3)

post-thumb

Note: This is the 3rd post of a 5 part series.

Previous Post : Adding Third-Party Libraries - Part 2

Adding a makefile-based build dependency into an Unreal Engine 5 project

In the last post, we looked at a few options for integrating NASA’s SPICE Toolkit library into Unreal Engine 5. Whew. That was a lot to cover!

In this post, we’ll discuss how to build the CSPICE library as part of building our project.

Of Spice and Modules

We’re going to be dealing with no less than 3 UE modules to implement this cleanly.

  • CSpice_Library: A module that builds/links to cspice.lib (the CSPICE toolkit library)
  • Spice: A module that exposes CSPICE functionality to other UE modules and translates between the two data models
  • Normal UE modules

So, Spice acts like a shim between the SPICE world and the UE world. It understands UE data types and SPICE datatypes and it’s responsible for translating between the two.

We don’t really want regular UE modules to make calls directly into cspice.lib. That would turn messy, quickly. We don’t want 1000 callers trying to transform between UE and CSPICE data types, right?

CSPICE itself will be contained fully in the CSpice_Library module. It will only be accessed through the Spice module shim layer.

And note that, since cspice.lib is a static library every .dll module that uses it would have its own instance of the static lib. That means state, such as the spice kernel pool, is not shared between modules. If you launch the editor from the debugger and ‘break’, you’ll see a UnerealEditor-Spice-Win64-DebugGame.dll or similar, same for your other modules, but no dll for the CSpice_Library module. And if you use the Visual Studio DumpBin.exe tool to examine the exports of each .dll your project builds - you’ll see none of them export the base CSPICE functions. And how could they? We didn’t change the original code to add anything that causes a dll export to be generated. We’d either need to modify the code to include __declspec(dllexport), or include a DEF file that would tell a linker to export some functions… But we didn’t. So unlike our other modules, this module truly is a static library. All the other modules have their own ‘instance’ of CSpice_Library, kernel files loaded from one module will not be seen by others.

And the rest of our UE modules that want to do anything SPICEy can do so through the Spice Module.

So, how do we do build this?

Linking to cspice.lib

Forget about how cspice.lib is built for now, we’ll get to that in a bit. But, once we have it, what do we do with it?

In our CSpice_Library module we’re going to add a few things from the CSPICE toolkit. We need the src\cspice and include directories. We don’t need any of the other source under src, that code is for other stuff. And, we don’t need the lib directory, because we’re going to build the library ourselves.

CSpice_Library
    cspice
        include
        src
            cspice

To set up the CSpice_Library module we will flesh out CSpice_Library.Build.cs

public class CSpice_Library : ModuleRules
{
	public CSpice_Library(ReadOnlyTargetRules Target) : base(Target)
	{
        // ???
    }
}

CSpice_Library.Build.cs

We’ll start off with a little bit of path wrangling per the above:

        bAddDefaultIncludePaths = false;

        string cspiceDir = Path.Combine(ModuleDirectory, "cspice/");
        string includeDir = Path.Combine(cspiceDir, "include/");
        string libFile = Path.Combine(cspiceDir, "lib/win64/cspice.lib");

We don’t need the /src directory in this phase because by the time it’s invoked we’ll already have our cspice.lib library.

After rummaging through the UnrealBuildTool source code in the previous post, we have a good idea where to find a few things we need.

This will tell UE we don’t want it to look for and compile any source for this module:

        Type = ModuleRules.ModuleType.External;

Our Spice module will need access to the library’s header files. We can add this to expose the header files to the other modules:

        PublicIncludePaths.Add(includeDir);

And, we don’t use PublicAdditionalLibraries as suggested in the previous post (part 2). We use PublicPreBuildLibraries, because we’re going to have UE build it as a Pre-Build step:

        PublicPreBuildLibraries.Add(libFile);

If the module Type is set to ModuleType.External the UE build system will not look for any source code to compile.

PublicIncludePaths allows UE modules to access this module’s public header files.

PublicPreBuildLibraries tells the UE build system to expect a library to be built either via TargetRules.PreBuildSteps or TargetRules.PreBuildTargets.

That’s all the important stuff. We can do this while we’re here:

        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            PublicDefinitions.Add("_COMPLEX_DEFINED");
            PublicDefinitions.Add("MSDOS");
            PublicDefinitions.Add("NON_ANSI_STDIO");
        }

PublicDefinitions are used to add preprocessor definitions to the compilation of other modules that use this module. This can be necessary to ensure the other modules interpret our public header files correctly. While other modules will include these definitions, this module will be compiled with PublicDefinitions in addition to PrivateDefinitions. There is no need to define these in both places.

We only do this for platform Win64. How did we know we need to define those?

Well, because we peeked into the build files for the NAIF executables that link to the library, and that’s what they defined.

From src/brief_c/mkprodct.bat:

set cl= /c /O2 -D_COMPLEX_DEFINED -DMSDOS -DNON_ANSI_STDIO

I’m using the Win64 distribution of CSPICE. Presumably the definitions would be set up correctly for the other platforms CSPICE is available for. But… No clue what we’d need for Playstation etc. MSDOS? For playstation? Erm. How about NON_ANSI_STDIO? This would require some further investigation. The NAIF website says to consult them if building for other platforms, but they’re not likely to care about us gamedevs… They support real life spacecraft missions, so… we’re likely on our own.

Slam all that together, and you have a working module. Now we just need to get that library built, too.

And, to build the library we’re going to flesh out our TargetRules implementation, in Spice.Target.cs

The main thing we’re going to do is check if the library exists. If it doesn’t we build it. Otherwise we don’t, because we already have it.

		if (!File.Exists(pathToCSpiceLib))
		{
			System.Console.WriteLine("Rebuilding cspice toolkit lib");

			// If the prebuild step fails...
			// ...and you need to rebuild this module in order to fix the prebuild...
			// ...you'll need to comment out the next line to rebuild the module.
			targetRules.PreBuildSteps.Add("$(ProjectDir)\\" + RelativePathToCSpiceToolkit + "makeall_ue.bat \"$(ProjectDir)\\" + RelativePathToCSpiceToolkit + "\"");
		}
		else
		{
			System.Console.WriteLine("cspice toolkit lib is up to date");
		}

Spice.Target.cs

The UE build system invokes PreBuildSteps before building the project. This includes invoking the pre-build steps before recompiling any *.Target.cs files and *.Build.cs files.

There’s some jankiness in this scheme, in that this is a pre-build step. If this step fails, the project isn’t built. So, this module is not rebuilt. Which means the Spice.Target.cs file containing the scheme isn’t rebuilt. So, if you code an error which causes the Pre-build step to fail into Spice.Target.cs, you’re in a catch-22. To fix the error, you’ll need to comment out items in the Pre-Build step to remove the failure, thus the rest of the project will build and your code here will be recompiled. Sometimes you’ll need to trigger builds twice to get everything built. Please let me know if you find a better solution!

So, what exactly are we triggering in the PreBuildSteps.Add above? To build cspice.lib we just call a batch file, makeall_ue.bat. So, what’s in it? The original batch file provided by NAIF builds the entire source tree. We just want to build src/cspice and then stash the resulting library somewhere. So, our batch file is a trimmed down version of code from the original NAIF makeall.bat file.

cd src
rem
rem Creating cspice
rem
cd cspice
call mkprodct.bat

makeall_ue.bat

And, let’s copy the library build artifact somewhere handy:

copy .\lib\cspice.lib ..\lib\win64

Not much else to it.

But, wait, there’s one little problem…

The problem

When building from the command line, Microsoft’s compiler and Linker both print out a banner:

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30136 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.
Microsoft (R) Incremental Linker Version 14.29.30136.0
Copyright (C) Microsoft Corporation.  All rights reserved.

UNBELIEVABLY, these are sent to the error stream. Darn you, Microsoft!!!! We do want to catch actual errors, right? So we don’t just want to clear the errors after they occur and continue.

Microsoft’s cl.exe and link.exe direct a copyright banner to the error stream.

Microsoft’s cl.exe and link.exe have a /nologo option that suppresses the copyright banners.

Luckily we can use the /nologo option that suppresses the banner. That way only legitimate errors trigger a build failure.

Great! How do we set this option for the makefile build? Let’s take a closer look at the NAIF’s makefiles.

The NAIF-provided mkprodct.bat script invokes the compiler like so:

set cl= /c /O2 -D_COMPLEX_DEFINED -DMSDOS -DOMIT_BLANK_CC -DNON_ANSI_STDIO
...
for %%f in (*.c) do cl %%f 

mkprodct.bat

So, it sets the environment variable (cl) that the compiler looks for to find its options, then calls the compiler to actually compile them. How can we set the /nologo compiler option without modifying NAIF’s mkprodct.bat batch file? We should avoid modifications if at all possible.

Luckily, there’s a second environment variable the compiler looks for as well, _CL_. The compiler uses the union of CL and _CL_ as the command line options.

REM Suppress the compiler's banner
SET _CL_=/nologo

And, same deal for the linker.

REM Suppress the linkers's banner
SET _LINK_=/NOLOGO /MD

The compiler and linker options environment variables _CL_ and _LINK_ are applied in addition to options set in CL and LINK. You can use these for funneling additional options into a makefile build without modifying the makefile build scripts.

What’s with the /MD option? It avoids linker errors, what else? :-D

Debug Symbols

If you need to debug through the CSPICE source code, you’ll want debug symbols, so change the lines above as so:

SET _CL_=/nologo /Zi /Od
SET _LINK_=/NOLOGO /MD /DEBUG
...
copy .\lib\cspice.lib ..\lib\win64
copy .\src\cspice\*.pdb ..\lib\win64

Normally debug symbols are stored in the .obj files. The makefile build, however, deletes these files after linking. The compiler’s /Zi option tells the compiler to store debug symbols in a .pdb file instead.

The /Od compiler option tells the compiler to skip all optimizations so you can actually step through code linearly and watch variables. (You’ll need to ignore some warnings since the NAIF makefile already specified /O2, but you’ll figure it out). If you’re comfortable debugging through optimized code + feel at home in the disassembly window OR if you don’t really need to source-level debug you can omit /Od… Which would be preferable, as long as you don’t need it.

The /DEBUG linker option tells link.exe to embed the .pdb symbol filename into the library so when the .exe is linked later the symbols for it can be found and included.

Unreal Engine won’t keep the symbols without a few more changes, however. You’ll also need to add these to the TargetRules (*.Target.cs):

	public SpiceEditorTarget( TargetInfo Target) : base(Target)
	{
    ...
		bUseFastPDBLinking = false;      // <- Not necessary but avoids potential issues
		bPublicSymbolsByDefault = true;  // <- Forced to true on Windows anyways
		WindowsPlatform.bStripUnreferencedSymbols = false;  // <- Necessary.
    ...
	}

If we wanted to be all slick, we’d name the library differently for unoptimized-debug/debug/release so we could switch between them without rebuilding the library. IMO it’s not worth the effort/complexity tho.

And with all that, finally, … If cspice.lib doesn’t exist, unreal will call the NAIF build scripting to build the .lib. Then, the UE build system will link it and let all the other modules know where the NAIF header files live. HOORAY! Major accomplishment!


Completed Source

For reference, the final file listings for this post.

CSpice_Library.Build.cs:

// Copyright 2021 Gamergenic. All Rights Reserved.
// Author: chucknoble@gamergenic.com

using System.IO;
using UnrealBuildTool;

public class CSpice_Library : ModuleRules
{
	public CSpice_Library(ReadOnlyTargetRules Target) : base(Target)
	{
        bAddDefaultIncludePaths = false;

        string cspiceDir = Path.Combine(ModuleDirectory, "cspice/");
        string includeDir = Path.Combine(cspiceDir, "include/");
        string libFile = SpiceEditorTarget.CSpiceLibPath(Target);

        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            PublicDefinitions.Add("MSDOS");
            PublicDefinitions.Add("OMIT_BLANK_CC");
            PublicDefinitions.Add("NON_ANSI_STDIO");
        }
        /*
        Add conditionals for any other platforms you want to support via recompilation here:
        else if (Target.Platform == UnrealTargetPlatform.XXXX)
        {
            // either include a naif pre-build lib, or allow it to recompile
            // (for C spice platforms, see: https://naif.jpl.nasa.gov/naif/toolkit_C.html)
        }
        */
        else
        {
            string Err = string.Format("cspice SDK not found for platform {0}", Target.Platform.ToString());
            System.Console.WriteLine(Err);
            throw new BuildException(Err);
        }

        Type = ModuleRules.ModuleType.External;
        PrecompileForTargets = PrecompileTargetsType.Any;
        PublicPreBuildLibraries.Add(libFile);
        
        if (!Directory.Exists(includeDir))
        {
            string Err = string.Format("cspice headers not found at {0}.  A copy of cspice/include must be present", includeDir);
            System.Console.WriteLine(Err);
            throw new BuildException(Err);
        }

        PublicIncludePaths.Add(includeDir);
    }
}

Spice.Target.cs:

// Copyright 2021 Gamergenic. All Rights Reserved.
// Author: chucknoble@gamergenic.com

using UnrealBuildTool;
using EpicGames.Core;
using System.IO;
using System.Collections.Generic;

public class SpiceTarget : TargetRules
{
	public const string RelativePathToCSpiceToolkit = "Source\\CSpice_Library\\cspice\\";
	public const string RelativePathToCSpiceLib = "Source\\CSpice_Library\\lib\\Win64\\cspice.lib";

	public SpiceTarget(TargetInfo Target) : base(Target)
	{
		Type = TargetType.Editor;
		DefaultBuildSettings = BuildSettingsVersion.V2;
		ExtraModuleNames.AddRange( new string[] { "MainModule", "Spice" } );

		BuildCSpiceLib(this);
	}

	static public string CSpiceLibPath(ReadOnlyTargetRules targetRules)
    {
		string relativePathToCSpiceLib = "Source\\CSpice_Library\\lib\\" + targetRules.Platform.ToString() + "\\cspice.lib";

		return Path.Combine(targetRules.ProjectFile.Directory.FullName, relativePathToCSpiceLib);
	}

	static public void BuildCSpiceLib(TargetRules targetRules)
    {
		string pathToCSpiceLib = CSpiceLibPath(new ReadOnlyTargetRules(targetRules));

		if (!File.Exists(pathToCSpiceLib))
		{
			// Note :  If the step fails, since it's a prebuild step, these rules will not be rebuilt.
			// So, don't cause a failure here, if you're iterating on these rules.
			// Also, changes to the invocation won't be seen until the following build!
			System.Console.WriteLine("Rebuilding cspice toolkit lib");
			targetRules.PreBuildSteps.Add("$(ProjectDir)\\" + RelativePathToCSpiceToolkit + "makeall_ue.bat \"$(ProjectDir)\\" + RelativePathToCSpiceToolkit + "\"");
		}
		else
		{
			System.Console.WriteLine("cspice toolkit lib is up to date");
		}
	}
}

makeall_ue.bat:

echo Compiling CSpice Toolkit - this takes a while, please wait!

rem fix a canceled build that left zzsecprt.c renamed as zzsecprt.x
if not exist "%1\src\cspice\zzsecprt.c" (
    rename "%1\src\cspice\zzsecprt.x" zzsecprt.c
)

SET _CL_=/nologo
SET _LINK_=/NOLOGO /MD
cd "%1"

echo push
pushd .

echo cd
rem from makeall.bat
cd src
rem
rem Creating cspice
rem
cd cspice
call mkprodct.bat

popd

if not exist "..\lib\win64" (
    mkdir "..\lib\win64"
)

copy .\lib\cspice.lib ..\lib\win64

In the next post (Part 4) of the series, we get to the exciting stuff. We’ll start making use of CSPICE in Unreal Engine!!

Next Post : Adding Third-Party Libraries - Part 4

Comments are disabled. To share feedback, please send email, or join the discussion on Discord.