The Database Black Box: Reverse Engineering Undocumented Schemas
Think you need a manual to understand a database? Discover how to peel back the layers of undocumented storage formats.
Have you ever inherited a legacy database with zero documentation, wondering if the data is just a pile of random bits?
The myth: Databases are readable puzzles
Many assume that if you have access to the files on disk, you can simply ‘open’ them like a text file or a spreadsheet. The belief is that data is stored in a clean, logical structure that tools can automatically interpret, making reverse engineering a simple matter of finding the right ‘viewer’ software.
The reality: It is a game of binary archeology
In reality, databases are highly optimized for performance, not human readability. Data is often stored in custom binary formats, using techniques like page-level compression, pointer arithmetic, and slot arrays. When you look at the raw bytes, you aren’t seeing tables; you are seeing memory offsets, header flags, and serialized objects. To understand the schema, you have to reconstruct the logic of the engine that wrote those bytes, which often involves mapping how the database handles variable-length records and index trees.
The nuance: Metadata is your anchor
While the raw data is cryptic, the database engine almost always leaves ‘breadcrumbs’ in the form of system catalogs or metadata tables. Even in undocumented systems, these internal tables define the structure of the user-facing tables. If you can find the entry point to the system catalog—often a specific fixed-offset file—you can automate the extraction of the entire schema definition without needing the original source code.
A better mental model: Think in layers
Stop looking for a ‘file format’ and start looking for the ‘engine’s intent.’ Treat the database as a state machine. The data is the current state, and the binary format is the serialization protocol. By analyzing the headers (the ‘what’) and the pointers (the ‘where’), you can map the relationship between storage blocks and logical entities.
Closing takeaway
If you find yourself staring at an undocumented database, don’t try to decode the data first. Map the metadata tables, identify the record headers, and build a schema map before you ever try to query a single row.
