Cecal Asset Archives & Virtual File System

The Popcorning Guinea Pig engine ships a single-file asset packaging format called .cecal and a runtime virtual file system that resolves assets from packed archives or loose files transparently. Together they enable IP-protected single-file builds while keeping loose-file development workflow unchanged.

This article covers the format, the runtime, the build-time packer, and how to opt a game project into packed mode.

Why a packed asset archive?

Shipping assets as loose files exposes three problems for production builds:

  1. IP exposure. Anyone who unzips the build can copy the textures, fonts, and audio out of the assets/ folder.
  2. Slow startup. Hundreds of small file opens trash the OS file cache compared to a single mmap.
  3. Scattered installs. Players and modders can accidentally delete or modify loose files.

A .cecal archive collapses every packable asset into one memory-mapped binary blob with an O(1) table-of-contents lookup. Loose-file mode still works for development — the runtime falls through to disk for anything not in the archive.

Architecture overview

┌────────────────────────────────────────────────────────────────┐
│ Engine loaders (Scene, AssetRegistry, TMX, Quake, Localization)│
│    • Texture loads → vfs.TryReadAllBytes → LoadImageFromMemory │
│    • Font loads    → vfs.TryReadAllBytes → LoadFontFromMemory  │
│    • Model loads   → vfs.ResolveToTempFile → Raylib.LoadModel  │
│    • Sound loads   → vfs.ResolveToTempFile → FileSource(temp)  │
│    • Scene/CSV/JSON → vfs.TryReadAllText                       │
└──────────────────────────┬─────────────────────────────────────┘
                           │  IVirtualFileSystem
                           ▼
              ┌─────────────────────────────┐
              │ CompositeVirtualFileSystem  │   (when game.cecal exists)
              │  ├─ CecalVirtualFileSystem  │   ← mmap-backed
              │  └─ LooseFileVirtualFileSystem (root: null)│
              └─────────────────────────────┘

              ┌─────────────────────────────┐
              │       (null VFS)            │   (when no archive present)
              │  loaders use legacy File.*  │   ← byte-for-byte legacy behavior
              └─────────────────────────────┘

The composite tries the archive first. Loose files override on miss — useful for hot-replacing a single asset during dev without rebuilding the archive. When no archive is present, Game.VFS is null and every loader takes its pre-existing File.* codepath, so games that haven't opted in see zero behavioral change.

The .cecal binary format

The on-disk layout is defined in CecalFormat.cs (internal to the engine). All multi-byte integers are little-endian.

Header (26 bytes)

Offset Size Field Description
0 6 magic ASCII "CECAL\0"
6 4 version Format version (currently 1)
10 8 toc_offset Absolute byte offset of the TOC
18 4 toc_count Number of entries in the TOC
22 4 reserved Zero — reserved for future flags

Per-entry TOC record (38 fixed bytes + variable path)

Offset Size Field Description
0 4 flags Per-entry flags (currently always 0)
4 8 offset Absolute byte offset of the asset blob
12 8 length Byte length of the asset blob
20 16 hash XXH3-128 hash of the asset bytes
36 2 path_len UTF-8 byte length of the logical path
38 var path_bytes Logical path, UTF-8, forward slashes, no leading /

Logical paths use forward slashes; lookups are case-insensitive. Writers normalize backslashes to forward slashes at write time, and the runtime applies the same normalization on every lookup so callers can pass either.

Layout on disk

[ Header (26 bytes) ]
[ Asset blob 1     ]
[ Asset blob 2     ]
[ Asset blob N     ]
[ TOC entry 1      ]
[ TOC entry 2      ]
[ TOC entry N      ]

Asset blobs are written first; the TOC is written after every blob so the writer can record final offsets in a single pass without seeking. The header toc_offset is patched in place once the TOC position is known. This makes the writer streaming-friendly — CecalWriter never buffers a full asset in memory regardless of size.

What gets packed

The engine recognizes asset extensions in AssetExtensions.cs. Categories that return true from IsPackedAsset are written into the archive; everything else is skipped:

Category Extensions Packed?
Texture .png .jpg .jpeg .bmp .gif .tga ✅
Model .obj .gltf .glb .iqm .vox ✅
Sound .wav .ogg .mp3 .flac ✅
Font .ttf ✅
Shader .vs .fs .glsl ✅
Scene .pgpscn ✅
Data .csv .json ✅
Code .cs ❌

.cs files are recognized for editor browsing but never packed — game code ships as compiled DLLs. .json is packed because scene files (.pgpscn) and data files use JSON; user-writable settings.json consumed by SettingsService is read and written via File.* directly (bypassing VFS), so the loose copy stays authoritative even when an archive is present.

Walked directories exclude bin, obj, .git, .vs, .idea, and node_modules.

Building an archive: the cecalpack tool

tools/CecalPack is a thin command-line wrapper around the engine's CecalPacker. Usage:

cecalpack <project-root> <output.cecal>

The packer recursively walks <project-root>, hashes every packable asset (using a fast-path mtime+size check against the previous build's manifest), and writes the archive plus a sidecar <output.cecal>.manifest.json for incremental rebuilds. If the previous archive exists, all hashes match, and mtimes haven't drifted, the pack is skipped entirely — useful for incremental builds where only code changed.

Wiring into a game project's MSBuild

Add an AfterBuild target to your .csproj to pack assets into the build output every time the project compiles. The reference example is in MainMenu.Pgpscn.csproj:

<Target Name="PackCecalArchive" AfterTargets="Build">
  <PropertyGroup>
    <CecalPackProject>$(MSBuildThisFileDirectory)..\..\..\tools\CecalPack\CecalPack.csproj</CecalPackProject>
    <CecalProjectRoot>$(MSBuildThisFileDirectory.TrimEnd('\'))</CecalProjectRoot>
    <CecalOutputFile>$(TargetDir)game.cecal</CecalOutputFile>
  </PropertyGroup>
  <Exec Command='dotnet run --project "$(CecalPackProject)" -c $(Configuration) --no-launch-profile -- "$(CecalProjectRoot)" "$(CecalOutputFile)"'
        WorkingDirectory="$(MSBuildThisFileDirectory)" />
</Target>

Point CecalProjectRoot at the project directory, not the assets/ subfolder — the packer walks the whole tree and preserves the leading folder name in each logical path. So a file at assets/kenney/panel.png ends up keyed as assets/kenney/panel.png in the archive, which is exactly how scene JSON references it.

Consuming cecalpack as a prebuilt redistributable (external repos)

cecalpack is the engine's canonical asset packer and is meant to be consumed by external game repos that reference the engine via DLLs rather than a project reference. Build it once (dotnet build tools/CecalPack/CecalPack.csproj -c Release) and copy its Release output — cecalpack.dll, cecalpack.runtimeconfig.json, cecalpack.deps.json, the engine assembly, and the engine's managed dependency DLLs — into a folder beside the consumer's other engine libs. The consumer then invokes the framework-dependent assembly directly with dotnet "<dll>" "<projectRoot>" "<outDir>\game.cecal" from an AfterTargets="Build"/AfterTargets="Publish" MSBuild target, instead of using dotnet run --project. The minimum runtime closure is cecalpack.dll + PopcorningGuineaPig.Engine.dll + Microsoft.Extensions.Logging.Abstractions.dll + System.IO.Hashing.dll, but shipping the full Release dependency set is recommended so packing never breaks when the engine's type graph changes.

Runtime: how Game.VFS finds and uses an archive

On Game.Init(), the engine calls VfsBootstrap.CreateDefault(), which probes for an archive next to AppContext.BaseDirectory in this order:

  1. {exeName}.cecal
  2. game.cecal
  3. assets.cecal

The first existing file wins. If found, Game.VFS is set to a CompositeVirtualFileSystem(cecal, loose). If no archive is found — or if every probed archive fails to open — Game.VFS is null and loaders use their legacy paths.

Note

The archive must sit alongside the executable (AppContext.BaseDirectory), not the working directory. For typical desktop launches these are the same; for unusual launchers they may differ.

You can hear what happened by watching the console at startup:

[VFS] Loaded .cecal archive: C:\game\bin\Release\net10.0\game.cecal (14 entries)

or:

[VFS] No .cecal archive found — using legacy loose-file mode

The IVirtualFileSystem surface

IVirtualFileSystem is intentionally minimal — five methods, all Try-style for graceful misses:

bool Exists(string logicalPath);
bool TryOpen(string logicalPath, out Stream stream);
bool TryReadAllBytes(string logicalPath, out byte[] bytes);
bool TryReadAllText(string logicalPath, out string text);
string? ResolveToTempFile(string logicalPath);

Every miss is non-throwing — loaders can decide whether to fall back to a placeholder, log a warning, or hard-fail. This is the engine's graceful-degradation contract: a missing asset should never crash the game.

ResolveToTempFile exists because raylib-cs 7.0.1 has no LoadModelFromMemory and OwnaudioNET.Sources.FileSource accepts only a file path. For these two asset types, CecalVirtualFileSystem extracts the bytes to a cached temp file under %TEMP%/pgp-vfs/{archive-hash}-{pid}/ on first request and reuses the same file for subsequent loads. The loose-file VFS just returns the on-disk path unchanged. The temp directory is cleaned up on Dispose and on AppDomain.ProcessExit.

Important

Temp-file extraction is the one place where IP protection leaks: GLB models and audio files are briefly written to %TEMP% unencrypted. A motivated attacker with runtime access can read them. Textures and fonts stay in memory and are never extracted.

Implementations

Class Purpose
CecalVirtualFileSystem Memory-mapped reader for .cecal archives. O(1) lookup. Used at runtime.
LooseFileVirtualFileSystem Reads directly from disk. Optional root dir; null root = CWD-relative File.* semantics.
CompositeVirtualFileSystem Tries each layered VFS in order; first hit wins.
CecalReader Plain-FileStream reader used by tooling and tests. Same TOC parser, no mmap.
CecalWriter Streaming archive writer. Used by CecalPacker.
CecalPacker Walks a project root and produces .cecal + sidecar manifest. Used by the cecalpack build tool.

End-to-end example: opting main-menu-pgpscn into packed mode

The examples/main-menu-pgpscn example demonstrates the full pipeline:

  1. Csproj target invokes cecalpack on the project root after build, writing game.cecal next to MainMenu.Pgpscn.dll.
  2. At launch, Game.Init() calls VfsBootstrap.CreateDefault() which finds game.cecal and constructs a CompositeVirtualFileSystem(cecal, loose).
  3. Asset loads route through VFS:
    • assets/HeroicForgeGames.png → LoadImageFromMemory from the mmap'd archive bytes
    • assets/Roboto-Regular.ttf → LoadFontFromMemory
    • assets/localization.csv → vfs.TryReadAllText → splits to lines
    • scenes/MainMenuScene.pgpscn → vfs.TryReadAllText → JSON deserialize
  4. settings.json stays loose. SettingsService reads/writes via File.* directly so user preferences persist normally.

After the build:

$ ls bin/Debug/net10.0/
MainMenu.Pgpscn.dll
MainMenu.Pgpscn.exe
game.cecal             ← 14-entry archive
game.cecal.manifest.json
assets/                ← loose copies still present (composite fallback)
scenes/
settings.json

To force archive-only mode for a clean validation run, temporarily rename the loose assets/ and scenes/ folders. Watch for the [VFS] Loaded .cecal archive: ... line in the console to confirm the runtime opened the archive.

Cross-platform notes

  • Linux/macOS via Proton. MemoryMappedFile.CreateFromFile is implemented natively across all .NET-supported platforms. Wine/Proton handles MMF via the same NT API path, mapping ultimately to mmap() on Linux. The VFS has been verified on Windows; Proton/Wine validation is part of the standard ship checklist.
  • Path separators. All loaders normalize logical paths to forward slashes before lookup, so callers on Windows can pass assets\foo.png and the VFS will still resolve it.
  • Temp directories. Path.GetTempPath() is honored on every platform — Windows %TEMP%, Linux /tmp (or Z:\tmp under Wine), macOS $TMPDIR.

Limitations and known gaps

  1. Raylib model loading still requires a file path. Models (.glb/.gltf/.obj) and audio (via OwnaudioNET.Sources.FileSource) are extracted to temp files at first use. This is the only IP-protection compromise.
  2. settings.json is intentionally not VFS-routed. User settings are writable; routing them through VFS would mask the live on-disk copy with a stale archive snapshot.
  3. Only game.cecal-scope archives. There is currently no support for layered DLC packs or multiple .cecal files; the composite VFS supports it conceptually but VfsBootstrap only probes a single archive.
  4. No archive encryption. The .cecal format trades obfuscation for speed and simplicity. A custom build of the format with encrypted blobs is possible but out of scope for v1.