05 — File Formats & Original Source Insights

For the broader engine-completeness inventory across Dune II, Tiberian Dawn, Red Alert 1, Red Alert 2 / Yuri’s Revenge, the Remastered Collection, and Generals / Zero Hour, see Complete Resource Format Support Map. This page remains the narrower canonical scope for the current cnc-formats / ic-cnc-content split.

Canonical Completeness Bar

At the engine level, Iron Curtain’s resource-support bar is broader than the current cnc-formats crate surface: the engine must be able to directly load the original resource families of Dune II, Tiberian Dawn, Red Alert 1, Red Alert 2 / Yuri’s Revenge, the Remastered Collection, and Generals / Zero Hour.

This is a support requirement, not a requirement that a single parser crate owns every family. The split is:

1. cnc-formats is canonical for the clean-room classic Westwood / Petroglyph 2D family and closely related standalone tooling formats.
2. ic-cnc-content and game-module-specific loaders own game-specific families that depend on EA-derived details or exceed the classic Westwood 2D scope.
3. Additional sibling parser crates are allowed when a game family is structurally distinct enough that forcing it into cnc-formats would reduce clarity. The current obvious candidates are Dune II-specific resource families and SAGE-family formats for Generals / Zero Hour.

So the completeness question is always: “can IC load the original resource directly?” not “did cnc-formats personally absorb every format?”

Formats to Support (cnc-formats + ic-cnc-content)

Binary Formats (from original game / OpenRA)

Tiberian Sun / Red Alert 2 Formats

Binary format parsers for the TS/RA2 game family. All clean-room implementations in cnc-formats from community documentation (XCC Utilities, Ares, Phobos projects). These are additive to the FormatRegistry (D018) — loaded by TS/RA2 game modules, not the core engine.

Generals / Zero Hour Formats (SAGE Engine)

Binary format parsers for the Generals/ZH SAGE-era game family. Clean-room implementations in cnc-formats from EA’s GPL-released source code and community documentation (OpenSAGE project). These are additive to the FormatRegistry — loaded by Generals game modules.

Remastered Collection Formats (Petroglyph)

HD asset formats from the C&C Remastered Collection (EA, 2020). Format definitions derived from the GPL v3 C++ DLL source and community documentation. See D075 for full import pipeline and legal model.

Text Formats

Canonical Asset Format Recommendations (D049)

New Workshop content should use Bevy-native modern formats by default. C&C legacy formats are fully supported for backward compatibility but are not the recommended distribution format. The engine loads both families at runtime — no manual conversion is ever required.

Why modern formats: (1) Bevy loads them natively — zero custom code, full hot-reload and async loading. (2) Security — OGG/PNG parsers are fuzz-tested and browser-audited; our custom .aud/.shp parsers are not. (3) Multi-game — non-C&C game modules (D039) won’t use .shp or .aud. (4) Tooling — every editor exports PNG/OGG/WAV/WebM; nobody’s toolchain outputs .aud. (5) WASM — modern formats work in browser builds out of the box.

The Asset Studio (D040) converts in both directions. See decisions/09e/D049-workshop-assets.md for full rationale, storage comparisons, and distribution strategy.

Crate Goals (cnc-formats + ic-cnc-content)

D076 splits format handling into two crates with distinct roles:

cnc-formats (MIT OR Apache-2.0, separate repo — Tier 1, Phase 0):

1. Clean-room parsers for C&C binary formats across three game families:

   • Classic Westwood 2D (TD/RA1): .mix, .shp, .tmp, .pal, .aud, .vqa, .vqp, .wsa, .fnt, .cps (screen pictures), .eng/.ger/.fre (string tables), .lut (color lookup tables), .dip (installer data)
   • Tiberian Sun / Red Alert 2: .shp TS variant (scanline RLE, separate ShpTsFile parser), .vxl (voxel models), .hva (voxel animation transforms), .csf (compiled string format)
   • Generals / Zero Hour (SAGE): .big (BIG archives), .w3d (Westwood 3D mesh — recursive IFF chunk parser)
   • Petroglyph (Remastered): .meg/.pgm archives (Phase 2, behind meg feature flag — clean-room from OS Big Editor/OpenSAGE community docs)
   • Audio/music: Standard MIDI .mid parsing, writing, SoundFont rendering, and real-time playback behind midi feature flag (via midly/nodi/rustysynth — pure Rust; midly is Unlicense, nodi and rustysynth are MIT). Westwood-lineage audio: AdLib .adl (Dune II) parsing behind adl feature flag; XMIDI .xmi (Kyrandia / Miles Sound System) parsing and XMI→MID conversion behind xmi feature flag (implies midi). WAV→MID transcription behind transcribe + convert feature flags (implies midi; WAV decoding via hound requires convert); ML-enhanced via transcribe-ml feature
   • Format detection: Content-based magic-byte sniffing (identify CLI) auto-detects all supported formats without relying on file extension — covers all families above plus TS/RA2/Generals variants
   • Streaming APIs: Zero-copy Read-based streaming for large files — MIX, BIG, MEG, AUD, and VQA all support range-based member access without loading entire archives into memory

2. Clean-room parsers for C&C text configuration formats: .ini (classic C&C rules, always enabled) and MiniYAML (OpenRA rules, behind miniyaml feature flag)

3. Clean-room encoders for standard algorithms — LCW compression, IMA ADPCM encoding, VQ codebook generation (median-cut quantization), SHP frame assembly, PAL writing. These use publicly documented algorithms with no EA source code references. Sufficient for community tools, round-trip conversion, and Asset Studio basic functionality.

4. Bidirectional format conversion (behind convert feature flag, gates png/hound/gif dependencies):

   Conversion	Direction	Dependency
   SHP ↔ PNG	Bidirectional	png crate
   SHP ↔ GIF	Bidirectional	gif crate
   PAL → PNG (swatch)	Export	png crate
   TMP → PNG	Export	png crate
   WSA ↔ PNG (frame sequence)	Bidirectional	png crate
   WSA ↔ GIF (animated)	Bidirectional	gif crate
   AUD ↔ WAV	Bidirectional	hound crate
   MID → WAV	Export	midly + rustysynth (behind midi feature)
   MID → AUD	Export	midly + rustysynth + aud::encode_adpcm() (behind midi feature)
   XMI → MID	Export	midly (behind xmi feature, implies midi)
   XMI → WAV	Export	Via XMI→MID then MID→WAV SoundFont render (behind xmi feature)
   XMI → AUD	Export	Via XMI→MID→WAV→AUD pipeline (behind xmi + convert features)
   WAV → MID	Export	DSP transcription pipeline (behind transcribe + convert features — WAV decoding requires hound under convert); ML-enhanced via transcribe-ml feature
   VQA ↔ AVI	Bidirectional	custom AVI codec
   VQA → MKV	Export	custom Matroska encoder (V_UNCOMPRESSED per RFC 9559 or V_MS/VFW/FOURCC legacy codec); optional PCM audio track. Higher fidelity archival than AVI
   FNT → PNG	Export	png crate

   cnc-formats convert is the standalone CLI for community utility; Asset Studio (D040) provides the visual GUI via ic-cnc-content (which wraps cnc-formats).

5. CLI tool with phased subcommand rollout:

   • Phase 0: identify (content-based format sniffing via magic bytes — auto-detects format type without relying on file extension; reports detected format and confidence), validate (structural correctness), inspect (dump contents/metadata, --json for machine-readable output), convert (extensible --format/--to format conversion — --to required, --format overrides auto-detected source format from extension; text: --format miniyaml --to yaml behind miniyaml feature; binary: SHP↔PNG, SHP↔GIF, AUD↔WAV, VQA↔AVI, VQA→MKV, WSA↔PNG/GIF, TMP→PNG, PAL→PNG, FNT→PNG behind convert feature; MIDI: MID→WAV, MID→AUD behind midi feature; XMIDI: XMI→MID, XMI→WAV, XMI→AUD behind xmi feature; Transcribe: WAV→MID behind transcribe + convert features — WAV decoding via hound requires convert, ML-enhanced via transcribe-ml)
   • Phase 1: extract (decompose .mix archives to individual files; .meg/.pgm support added Phase 2 via meg feature), list (quick archive inventory — filenames, sizes, types; .meg/.pgm support added Phase 2)
   • Phase 2: check (deep integrity verification — CRC validation, truncation detection, validate --strict equivalent), diff (format-aware structural comparison of two files of the same type), fingerprint (SHA-256 canonical content hash for integrity/deduplication)
   • Phase 6a: pack (create .mix archives from directory — inverse of extract)

   CLI usage examples:

   # Auto-detect format from file content (no extension needed)
   cnc-formats identify mystery_file.dat
   
   # Convert MiniYAML rules to standard YAML (explicit --format because .yaml is ambiguous)
   cnc-formats convert --format miniyaml --to yaml rules.yaml
   
   # Auto-detection works when extension is unambiguous
   cnc-formats convert --to yaml openra-rules.miniyaml
   
   # Explicit --format required for pipe/stdin usage
   cat rules.yaml | cnc-formats convert --format miniyaml --to yaml -o rules-converted.yaml
   
   # Validate any supported format
   cnc-formats validate main.mix
   cnc-formats validate rules.yaml
   
   # Inspect archive contents (human-readable)
   cnc-formats inspect main.mix
   
   # Machine-readable JSON output for tooling
   cnc-formats inspect --json conquer.mix
   
   # Extract archive to directory (Phase 1)
   cnc-formats extract main.mix -o assets/
   
   # Quick archive inventory (Phase 1)
   cnc-formats list main.mix
   
   # Deep integrity check (Phase 2)
   cnc-formats check main.mix
   
   # Format-aware diff between two MIX archives (Phase 2)
   cnc-formats diff original.mix modded.mix
   
   # Content-hash for deduplication (Phase 2)
   cnc-formats fingerprint infantry.shp
   
   # MEG archive support (Phase 2, requires `meg` feature)
   cnc-formats list data.meg
   cnc-formats extract data.meg -o remastered-assets/
   
   # Create MIX archive from directory (Phase 6a)
   cnc-formats pack assets/ -o custom.mix
   
   # Binary format conversions (requires `convert` feature)
   # Convert SHP sprites to PNG (requires palette)
   cnc-formats convert --format shp --to png infantry.shp --palette temperat.pal -o infantry/
   
   # Convert PNG back to SHP
   cnc-formats convert --format png --to shp infantry/ --palette temperat.pal -o infantry.shp
   
   # Convert AUD audio to WAV
   cnc-formats convert --format aud --to wav speech.aud -o speech.wav
   
   # Convert WAV back to AUD
   cnc-formats convert --format wav --to aud recording.wav -o recording.aud
   
   # Render MIDI to WAV using SoundFont (requires `midi` feature)
   cnc-formats convert --format mid --to wav soundtrack.mid --soundfont default.sf2 -o soundtrack.wav
   
   # Render MIDI to AUD for original game engine modding (requires `midi` feature)
   cnc-formats convert --format mid --to aud soundtrack.mid --soundfont default.sf2 -o soundtrack.aud
   
   # Convert VQA cutscene to AVI (interchange format)
   cnc-formats convert --format vqa --to avi intro.vqa -o intro.avi
   
   # Convert VQA cutscene to MKV (archival quality, RFC 9559 uncompressed video)
   cnc-formats convert --format vqa --to mkv intro.vqa -o intro.mkv
   
   # Convert animated WSA to GIF
   cnc-formats convert --format wsa --to gif menu.wsa --palette temperat.pal -o menu.gif
   
   # Ambiguous format hint (TD vs RA .tmp files — use --format tmp to override auto-detection)
   cnc-formats convert --format tmp --to png tiles.tmp --palette temperat.pal
   

6. Extensive tests against known-good OpenRA data

7. No EA-derived code — permissive licensing enables adoption by any C&C tool or modding project

8. Released open source as a standalone crate on day one (Phase 0 deliverable)

9. Uses std — enables std::io::Read streaming for large files (.mix archives, .vqa video), std::error::Error ergonomics, and HashMap without extra dependencies. The &[u8] parsing API remains the primary interface; streaming is an additional option.

10. Serves as a clean-room feasibility proof that the engine is not technically dependent on EA’s GPL code — all formats parse and encode correctly from community docs alone (see D051 § “GPL Is a Policy Choice” and D076 rationale #6)

ic-cnc-content (GPL v3, IC monorepo):

1. Thin wrapper over cnc-formats (with miniyaml feature enabled) — adds EA-specific details (compression tables, game-specific constants) that reference EA’s GPL-licensed C&C source (D051)
2. Bevy AssetSource integration for IC’s asset pipeline
3. Remastered-specific format support (.tga+.meta megasheet splitting, .dds terrain import, .bk2 video conversion, .mtd MegaTexture) — GPL-derived or proprietary formats that stay in ic-cnc-content (see D076 § Remastered format split). Note: .meg/.pgm archive parsing is in cnc-formats (Phase 2, meg feature flag) — ic-cnc-content depends on it
4. EA-derived encoder enhancements (Phase 6a): Extends cnc-formats’ clean-room encoders for pixel-perfect original-game-format matching where EA GPL source provides authoritative edge-case details. Encrypted .mix packing (Blowfish key derivation + SHA-1 body digest — extends cnc-formats pack‘s unencrypted archives). Only needed when exact-match reproduction of original game file bytes is required — cnc-formats’ clean-room encoders are sufficient for all standard community workflows. Encoders reference the EA GPL source code implementations for the EA-specific enhancements only (see § Binary Format Codec Reference)
5. ADL→WAV rendering (OPL2 synthesis): Renders .adl register data to PCM audio via OPL2 chip emulation using opl-emu (GPL-3.0, pure Rust). cnc-formats provides the .adl parser (behind adl feature, permissive license); ic-cnc-content adds the rendering step because the only viable pure Rust OPL emulator is GPL-licensed. Pipeline: .adl → register writes → OPL2 emulation → PCM → WAV. Accessible via Asset Studio (D040) and IC SDK CLI. If a permissively-licensed pure Rust OPL2 emulator becomes available, rendering can migrate to cnc-formats

Non-C&C Format Landscape

The cnc-formats crate provides clean-room parsers for the entire C&C format family (binary codecs, .ini, and feature-gated MiniYAML); ic-cnc-content wraps it with EA-derived details and IC asset pipeline integration. But the engine (D039) supports non-C&C games via the FormatRegistry and WASM format loaders (see 04-MODDING.md § WASM Format Loader API Surface). Analysis of six major OpenRA community mods (see research/openra-mod-architecture-analysis.md) reveals the scope of formats that non-C&C total conversions require:

Key insight: Non-C&C games on the engine need 0–15+ custom format decoders, and there is zero format overlap with C&C. This validates the FormatRegistry design — the engine cannot hardcode any format assumption. ic-cnc-content (wrapping cnc-formats) is one format loader plugin among potentially many.

Cross-engine validation: Godot’s ResourceFormatLoader follows the same pattern — a pluggable interface where any module registers format handlers (recognized extensions, type specializations, caching modes) and the engine dispatches to the correct loader at runtime. Godot’s implementation includes threaded loading, load caching (reuse/ignore/replace), and recursive dependency resolution for complex assets. IC’s FormatRegistry via Bevy’s asset system should support the same capabilities: threaded background loading, per-format caching policy, and declared dependencies between assets (e.g., a sprite sheet depends on a palette). See research/godot-o3de-engine-analysis.md § Asset Pipeline.

Content Source Detection

Games use different distribution platforms, and each stores assets in different locations. Analysis of TiberianDawnHD (see research/openra-mod-architecture-analysis.md) shows a robust pattern for detecting installed game content:

#![allow(unused)]
fn main() {
/// Content sources — where game assets are installed.
/// Each game module defines which sources it supports.
pub enum ContentSource {
    Steam { app_id: u32 },           // e.g., Steam AppId 2229870 (TD Remastered)
    EaApp { registry_key: String },  // EA App (formerly Origin) — Windows registry path to install dir
    Gog { game_id: String },         // GOG Galaxy game identifier
    OpenRA { install_path: PathBuf }, // OpenRA installation — mods, maps, community assets
    Directory { path: PathBuf },     // Manual install / disc copy
}
}

TiberianDawnHD detects Steam via AppId, EA App (formerly Origin) via Windows registry key, and GOG via standard install paths. IC should implement a ContentDetector that probes all known sources for each supported game and presents the user with detected installations at first run. The OpenRA source kind detects existing OpenRA installations to access community mods, maps, and assets. This handles the critical UX question “where are your game assets?” without requiring manual path entry — the same approach used by OpenRA, CorsixTH, and other reimplementation projects.

Phase: Content detection ships in Phase 0 — format parsing (binary + .ini + MiniYAML) in cnc-formats, IC-specific asset pipeline integration (including content source probing) in ic-cnc-content. Game module content detection in Phase 1.

Browser Asset Storage

The ContentDetector pattern above assumes filesystem access — probing Steam, Origin, GOG, and directory paths. None of this works in a browser build (WASM target). Browsers have no access to the user’s real filesystem. IC needs a dedicated browser asset storage strategy.

Browser storage APIs (in order of preference):

• OPFS (Origin Private File System): The newest browser storage API (~2023). Provides a real private filesystem with file/directory operations and synchronous access from Web Workers. Best performance for large binary assets like .mix archives. Primary storage backend for IC’s browser build.
• IndexedDB: Async NoSQL database. Stores structured data and binary blobs. Typically 50MB–several GB (browser-dependent, user-prompted above quota). Wider browser support than OPFS. Fallback storage backend.
• localStorage: Simple key-value string store, ~5-10MB limit, synchronous. Too small for game assets — suitable only for user preferences and settings.

Storage abstraction:

#![allow(unused)]
fn main() {
/// Platform-agnostic asset storage.
/// Native builds use the filesystem directly. Browser builds use OPFS/IndexedDB.
pub trait AssetStore: Send + Sync {
    fn read(&self, path: &VirtualPath) -> Result<Vec<u8>>;
    fn write(&self, path: &VirtualPath, data: &[u8]) -> Result<()>;
    fn exists(&self, path: &VirtualPath) -> bool;
    fn list_dir(&self, path: &VirtualPath) -> Result<Vec<VirtualPath>>;
    fn delete(&self, path: &VirtualPath) -> Result<()>;
    fn available_space(&self) -> Result<u64>; // quota management
}

pub struct NativeStore { root: PathBuf }
pub struct BrowserStore { /* OPFS primary, IndexedDB fallback */ }
}

Browser first-run asset acquisition:

1. User opens IC in a browser tab. No game assets exist in browser storage yet.
2. First-run wizard presents options: (a) drag-and-drop .mix files from a local RA installation, (b) paste a directory path to bulk-import, or (c) download a free content pack if legally available (e.g., freeware TD/RA releases).
3. Imported files are stored in the OPFS virtual filesystem under a structured directory (similar to Chrono Divide’s 📁 / layout: game archives at root, mods in mods/<modId>/, maps in maps/, replays in replays/).
4. Subsequent launches skip import — assets persist in OPFS across sessions.

Browser mod installation:

Mods are downloaded as archives (via Workshop HTTP API or direct URL), extracted in-browser (using a JS/WASM decompression library), and written to mods/<modId>/ in the virtual filesystem. The in-game mod browser triggers download and extraction. Lobby auto-download (D030) works identically — the AssetStore trait abstracts the actual storage backend.

Storage quota management:

Browsers impose per-origin storage limits (typically 1-20GB depending on browser and available disk). IC’s browser build should: (a) check available_space() before large downloads, (b) surface clear warnings when approaching quota, (c) provide a storage management UI (like Chrono Divide’s “Options → Storage”) showing per-mod and per-asset space usage, (d) allow selective deletion of cached assets.

Bevy integration: Bevy’s asset system already supports custom asset sources. The BrowserStore registers as a Bevy AssetSource so that asset_server.load("ra2.mix") transparently reads from OPFS on browser builds and from the filesystem on native builds. No game code changes required — the abstraction lives below Bevy’s asset layer.

Phase: AssetStore trait and BrowserStore implementation ship in Phase 7 (browser build). The trait definition should exist from Phase 0 so that NativeStore is used consistently — this prevents filesystem assumptions from leaking into game code. Chrono Divide’s browser storage architecture (OPFS + IndexedDB, virtual directory structure, mod folder isolation) validates this approach.

Sub-Pages