Format Provenance
This page records where PBIXRay’s knowledge of the PBIX file format comes from — structure by structure, with the published Microsoft specification cited where one exists, and an honest marker where none does.
It covers the open source pbixray Python library: the parser itself, which anyone can read, run, and check against their own files.
It exists because “how do you know this?” is a fair question, and because it is usually asked at the wrong moment: during a security review, by a legal team, or by someone deciding whether an unofficial parser is safe to put in a pipeline. The answer should be a document, not a conversation.
The short version
Most of the structures PBIXRay reads are documented in Microsoft’s published specifications, chiefly [MS-XLDM], and both Xpress decompressors are built on source code Microsoft released under the MIT licence. The main gaps are the binary .idfmeta format and its numeric compression identifiers, the Connections manifest, the schema inside metadata.sqlitedb (including its storage and aggregation tables), and the XPress9 container around the DataModel stream.
PBIXRay is read-only. It does not modify files, does not bundle any proprietary Microsoft binary, and needs no Microsoft account, licence or network access to run.
How to read the tables
Every structure carries one of four provenance markers.
| Marker | Meaning |
|---|---|
| S — Specified | Structure and semantics are defined in a published Microsoft Open Specification. The implementation follows the document. |
| C — Microsoft source | Implemented against source code Microsoft has publicly released under an OSI-approved licence. |
| P — Partially specified | A specification names and describes the structure, but the on-disk layout is incomplete or diverges from what shipping writers actually produce. The gap was closed by observation and is documented below. |
| O — Observed | No published description found. PBIXRay’s description of the structure is tested by parsing real files. |
Specifications referenced throughout:
- [MS-XLDM] — Spreadsheet Data Model File Format
- [MS-SSAS-T] — SQL Server Analysis Services Tabular Protocol
- [MS-XCA] — Xpress Compression Algorithm
- [MS-QDEFF] — Query Definition File Format
- [MS-WUSP] — Windows Update Services: Client-Server Protocol, cited by [MS-XLDM] §2.7.5 for Xpress compression
- Dynamic Management Views in Analysis Services — Microsoft’s overview of the DMVs, which also applies to Fabric and Power BI Premium
- ECMA-376 Part 2 — Open Packaging Conventions
[MS-XLDM] §1.5 and [MS-QDEFF] §1.5 say they apply to spreadsheet files, and neither mentions Power BI. For a PBIX, an S means the specification describes the structure and PBIX files follow that description, not that Microsoft has stated the specification covers PBIX.
The O rows were searched for in the full text of [MS-XLDM], [MS-SSAS-T], [MS-SSAS], [MS-XCA], [MS-WUSP] and [MS-QDEFF]. None of those documents describes them.
Container layer
| Structure | Provenance | Source |
|---|---|---|
| PBIX package (ZIP envelope) | S | ZIP APPNOTE 6.3.x; OPC conventions per ECMA-376 Part 2 |
XLSX package, xl/model/item.data | S | ECMA-376; the embedded model is the [MS-XLDM] payload |
Connections manifest | O | No published schema. A plain JSON document; the keys and their meanings were read from real files. |
DataMashup part | S | [MS-QDEFF] §2.2 and §2.3: top-level binary stream and package parts |
Section1.m document structure | S / P | [MS-QDEFF] §2.3.2 defines the part and refers to the Power Query M language specification for its content. The lexical splitter that separates individual queries is ours. |
DataModel stream | P | Once decompressed, it follows the storage format in MS-XLDM §2.1. The DataModel part name and the XPress9 container around it are not described there. |
Backup header (byte order mark, stream storage signature, BackupLogHeaderType) | S | MS-XLDM §2.1.1, including the signature, §2.1.1.2 |
| Backup log | S | MS-XLDM §2.1.2.3 |
| Virtual directory | S | MS-XLDM §2.1.3 |
Model metadata layer
| Structure | Provenance | Source |
|---|---|---|
metadata.sqlitedb | P | The SQLite file format is public. The schema Power BI uses inside it is not published, but most of its tables correspond to objects defined in [MS-SSAS-T]. |
| XML model metadata (XLSX / older models) | S | [MS-XLDM] §2.6.2 to §2.6.9: DataSource, DataSourceView, Database, Cube, Dimension, MeasureGroup, Partition, MdxScript |
| Tables, columns, measures, hierarchies, levels | S | [MS-SSAS-T] §2.2.5.3, §2.2.5.4, §2.2.5.8, §2.2.5.9, §2.2.5.10, and the matching TMSCHEMA rowsets in §3.1.5.1.1 |
| Relationships | S | [MS-SSAS-T] §2.2.5.7 and TMSCHEMA_RELATIONSHIPS (§3.1.5.1.1.7) |
| Row-level and object-level security | S | [MS-SSAS-T] §2.2.5.21 to §2.2.5.23 and §2.2.5.27: Role, RoleMembership, TablePermission (FilterExpression, MetadataPermission), ColumnPermission |
| Perspectives and refresh policies | S | [MS-SSAS-T] §2.2.5.16 to §2.2.5.20 and §2.2.5.34 |
| Other model objects | S | [MS-SSAS-T] §2.2.5: including Partition, DataSource, Annotation, KPI, Culture, Expression, CalculationGroup, CalculationItem, FormatStringDefinition, Function |
Aggregations (AlternateOf) | O | Not in [MS-SSAS-T]. The parser reads the table from metadata.sqlitedb. |
Storage tables (StorageFile, DictionaryStorage, ColumnPartitionStorage, RelationshipIndexStorage) | O | Not in [MS-SSAS-T], which only exposes references such as ColumnStorageID. These tables link model objects to their column files. [MS-SSAS] §3.1.4.2.2.1.3.58 to §3.1.4.2.2.1.3.60 documents the DISCOVER_STORAGE_TABLE* rowsets for storage tables, columns and segments, but not these tables. |
Newer model tables (CalendarColumnGroup, CalendarColumnReference, GroupByColumn, RelatedColumnDetails, AnalyticsAIMetadata) | O | Not in [MS-SSAS-T]. The parser reads them from metadata.sqlitedb when present. |
Set | O | Not in [MS-SSAS-T], and not listed in Microsoft’s DMV overview, which notes that its list may be incomplete. The engine does expose it through the TMSCHEMA_SETS DMV, so parsed values can be compared with what the engine reports. |
[MS-SSAS-T] is a protocol specification: it defines these objects as a client sees them, not how they are stored on disk. So an S in this table covers the object and its meaning, while the row for metadata.sqlitedb covers its storage. Parsed metadata can be compared directly with what the engine reports for the same objects.
VertiPaq column storage layer
[MS-XLDM] §2.3 and §2.7 specify the column store in detail. The main gap is the binary .idfmeta format: the specification describes column metadata as XML (§2.5), which is what XLSX models use, and does not mention .idfmeta.
| Structure | Provenance | Source |
|---|---|---|
.idf segment file, overall layout | S | [MS-XLDM] §2.3.1 (Column Data Storage) |
RLE primary segment, SegmentSize | S | [MS-XLDM] §2.3.1.1 |
| Hybrid RLE layout (primary + bit-packed sub-segment) | S | [MS-XLDM] §2.3.1.1.2 |
| Trailing-slot padding in real files | P | [MS-XLDM] §2.3.1.1 states unused trailing bytes are zero-padded. Some shipping Power Pivot writers instead leave stale memory there. A correct decoder must stop at the row count from the companion metadata, not at SegmentSize. |
| Bit-packing entries | S | [MS-XLDM] §2.7.3: the first value is a negated 1-based offset, starting at -1, and each later entry subtracts the previous entry’s count. 0xFFFFFFFF - offset is the same value read as unsigned. |
Column metadata XML (XLSX XMRawColumn) | S | [MS-XLDM] §2.5 |
.idfmeta — tag serialization | O | Binary tag framing (<1:CP\0 … CP:1>\0) and the CP/CS/SDOs nesting |
Compression encodings (XMRENoSplit, XM123, XMHybridRLE) | S | Class definitions in [MS-XLDM] §2.5.2; algorithms in §2.7.1 to §2.7.3 |
Numeric compression class identifiers in .idfmeta | O | Bit widths 0x000aba37–0x000aba56, hybrid RLE 0x000aba5a, and the general and XM123 encodings. These values are not in the specification. |
.dictionary — overall structure | S | MS-XLDM §2.3.2 |
| Dictionary page markers | S | [MS-XLDM] §2.3.2.1.2.4 and §2.3.2.1.2.4.3: 0xAABBCCDD opens a string store, 0xABCDABCD closes it |
| Uncompressed string pages | S | [MS-XLDM] §2.3.2.1.2.4.1: the remaining and used counts are in characters, the allocation size in bytes. Only used characters are strings. |
| Huffman-compressed string pages | S | [MS-XLDM] §2.3.2.1.2.4.2 and §2.7.4. Character set mode 0x000aba91 is XM_HUFFMAN_SINGLECHARSET (703121) in §2.3.2.1.3.4. |
.hidx hash index | S | [MS-XLDM] §2.3.3 |
Compression layer
Both Xpress layers decode with Microsoft’s own code. What is not published is the XPress9 container around them.
| Structure | Provenance | Source |
|---|---|---|
| Xpress8 (individual files inside the backup) | C | Microsoft’s own Xpress8 implementation, MIT licensed, from microsoft/Extensible-Storage-Engine |
| Xpress8 chunk wrapper | P | [MS-XLDM] §2.1.1.3 says individual files can be compressed but gives no format. PBIXRay reads 16-bit size fields in front of each raw [MS-XCA] buffer, which differs from the 32-bit header in [MS-WUSP] §2.1.1. See Windows Already Ships the Decompressor Power Pivot Needs. |
| Xpress LZ77 algorithm | S | [MS-XCA] §2.3 and §2.4 (Plain LZ77) |
Xpress9 (the whole DataModel stream, and compressed ABF files) | C | Microsoft’s own Xpress9 implementation, MIT licensed, from the same repository |
| XPress9 container, single-threaded | O | UTF-16LE signature followed by a flat chunk stream. [MS-XLDM] §2.7.5 names Xpress as specified in [MS-WUSP] §2.1.1 for the whole file, and does not mention XPress9 or this container. |
| XPress9 container, multithreaded | O | 40-byte thread-distribution header, then prefix and main chunk groups |
The Xpress8 result deserves restating: the compression Power Pivot uses is raw [MS-XCA] LZ77 behind a four-byte wrapper, and Windows has shipped a decoder for it — RtlDecompressBuffer with COMPRESSION_FORMAT_XPRESS — the entire time. Nothing was broken. Something documented was recognised.
Method
The tables above rest on the four sources below. This page records the published sources and how the results are tested. It is not a full history of the project.
Neither the specifications nor the files would have been enough alone. The specifications describe structures that shipping writers sometimes diverge from, and the files on disk mean little without the specifications to name what is in them.
1. Published specifications. [MS-XLDM] for the model file format, [MS-SSAS-T] for the tabular object model, [MS-XCA] for compression, [MS-QDEFF] for Power Query. [MS-WUSP] is the Xpress format that [MS-XLDM] cites.
2. Microsoft’s own open-source code. Both Xpress codecs come from the Extensible Storage Engine repository, which Microsoft publishes under the MIT licence. Xpress8 lives in dev/ese/src/_xpress, Xpress9 in dev/ese/src/_xpress9. Both are adapted to build outside ESE and wrapped as Cython extension modules.
Compression is the part of a format people most often assume must have been broken into. Here, both Xpress layers decode with Microsoft’s code, under a licence that permits it. Huffman-compressed dictionary strings use PBIXRay’s own xmhuffman decoder, and that scheme is specified in [MS-XLDM] §2.7.4.
3. Cross-checking against the engine’s own answers. Analysis Services reports the semantic model through Dynamic Management Views, a SQL interface to the schema rowsets. Microsoft’s overview points to [MS-SSAS-T] and [MS-SSAS] for their definitions, so a structure read from disk can be compared with what the engine reports for the same object.
4. Observing what the product writes. Until recently, Power BI Desktop expanded the model into a working folder on the user’s own disk while a file was open — apparently as memory-mapped files — where the individual column artifacts (.idf, .idfmeta, .dictionary, .hidx) existed as discrete, decompressed files rather than as offsets inside an opaque stream.
Much of the file-type taxonomy in the tables above dates from that period, and it is why the taxonomy is granular: the engine had already separated the format into named parts, on the user’s own machine, in the ordinary course of opening a file. Those artifacts were then described declaratively as Kaitai Struct definitions and validated by re-parsing real files, so that every claim about layout is executable and testable rather than prose.
Where no specification covers a structure, its definition is tested against real files. A wrong field layout shows up as a file that fails to parse, or as a column whose values do not match what the engine reports.
Each structure was revisited many times over several years, because new files kept breaking earlier assumptions. Power BI is a moving target: a format description that holds for every file on your own machine in one year will not necessarily hold the next. Most of the elapsed time in this project went into that iteration rather than into any single discovery.
Basis
These specifications are published through Microsoft’s Open Specifications programme. The notice on [MS-XLDM] allows copies to be made in order to develop implementations, and says Microsoft claims no trade secret rights in the documentation. It also says Microsoft patents might cover implementations and that the notice itself grants no patent licence. Whether a document falls under the Open Specifications Promise is a separate question.
Where Microsoft code is used, it is code Microsoft released under an OSI-approved licence, used in accordance with that licence. No Microsoft source is reproduced in PBIXRay beyond that, and no proprietary Microsoft binary is bundled, redistributed or required at runtime.
PBIXRay reads files the user already has. It does not bypass licence checks or access controls.
This section describes the basis on which this work was carried out. It is not legal advice.
Track record
Public work on this format began in late 2022 and has been documented continuously since — first notes on the internals, early parsing work, and twenty-odd long-form articles on this site covering the container, the metadata layer, the column store and the compression. The parser has been on PyPI since May 2024, is MIT licensed, and the format definitions are published as Kaitai Struct files in the repository so that anyone can check the claims above against their own files.
Nothing here depends on taking our word for it. That is the point.
What PBIXRay does not do
- It does not write, modify or re-save PBIX files.
- It does not circumvent licensing or access control.
- It does not bundle or require any proprietary Microsoft binary. Where it uses Microsoft code, that code is open source and is used under its licence.
- It does not require Power BI Desktop, Excel, a Microsoft account, or a network connection.
- It does not transmit file contents anywhere. The library is offline by construction and makes no network calls.
Last reviewed: November 2026. This page is versioned with the site — corrections and additions are tracked in its history.