# Validating a 5 GB XML file in the browser

Source: https://xmlvalid.com/blog/validating-5-gb-xml-in-the-browser

> libxml2 compiled to WebAssembly, fed from a File one megabyte at a time: a 5 GB document checked against its XSD in 66 seconds, nothing uploaded, memory flat

Most online XML validators stop at a few megabytes. They upload the file, parse it into a tree on the server and walk the tree against the schema. That is fine for a config file and hopeless for a 2 GB data export, a year of bank statements or a product feed.

xmlvalid validates in the browser instead: the document never leaves the machine. To see how far that goes, I built a 5.2 GB file (5,242,879,782 bytes) out of a book catalog, broke one `<year>` on line 38,269,200 so it read `20x5`, and opened it in the validator with its XSD. Sixty-six seconds later the page said:

```
Document is invalid: 1 issue
line 38,269,200  Element 'year': '20x5' is not a valid value of the atomic type 'xs:gYear'.

```

The tab's memory stayed flat the whole time. Here is how that works, and the few places where size still bites.

## Why a tree does not work

The obvious way to validate XML in JavaScript is to read the file into a string and hand it to a parser. Two walls stand in the way long before 5 GB:

- **Strings.** V8 caps a string at 229 characters, a little over 500 million, so `file.text()` fails on anything near a gigabyte.
- **Memory.** A WebAssembly module compiled for 32 bits addresses at most 4 GB, and Emscripten's default ceiling is 2 GB. A DOM of the document is several times the document's size anyway.

So the document has to stream: read a piece, validate it, forget it, read the next one. Memory then depends on how deep the document nests, not on how long it is.

## A streaming validator

libxml2 has done this for twenty years. Its `xmlTextReader` is a pull parser that can validate against XSD, RelaxNG or a DTD while it reads, and it frees each node once the reader has moved past it. It gets its bytes from whatever you give it: a file, a socket, or a callback.

The engine behind xmlvalid is a small Rust program around that reader. The same code runs natively on the server (for the API and the command-line client) and, compiled with Emscripten, in the browser. In the browser build the byte source is a function the page provides:

```rust
struct JsReader;

impl Read for JsReader {
    fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
        match unsafe { xv_js_read(buf.as_mut_ptr(), buf.len()) } {
            n if n >= 0 => Ok(n as usize),
            _ => Err(io::Error::other("host read failed")),
        }
    }
}

```

`xv_js_read` is a two-line Emscripten library function that calls back into JavaScript. libxml2 asks for a few kilobytes at a time and never sees more than that.

## Feeding it from a File

The engine runs in a Web Worker, which matters for two reasons: the page stays responsive, and workers have `FileReaderSync`. libxml2's read callback is synchronous, and in a worker a `Blob` can be read synchronously, so there is no need for Asyncify or SharedArrayBuffer tricks:

```js
const reader = new FileReaderSync();
const CHUNK = 1024 * 1024;

function byteReader(blob) {
    let offset = 0;
    let buffer = new Uint8Array(0);

    return (len) => {
        if (!buffer.length) {
            if (offset >= blob.size) return null;
            const end = Math.min(offset + Math.max(len, CHUNK), blob.size);
            buffer = new Uint8Array(reader.readAsArrayBuffer(blob.slice(offset, end)));
            offset = end;
        }
        const out = buffer.subarray(0, len);
        buffer = buffer.subarray(out.length);
        return out;
    };
}

```

`blob.slice()` does not read anything; it only describes a range of the file. Each `readAsArrayBuffer` call has a fixed cost, so the reader takes a megabyte at a time and serves libxml2's small requests from it. The callback copies the bytes into the module's heap with `HEAPU8.set` and returns the count; zero means end of file. The engine's heap stays around 20 MB whether the file is 5 MB or 5 GB.

Progress and errors go the other way as JSON lines, posted to the page as they happen, so the first error shows up while the rest of the file is still being read.

## Things that break past 4 GB

Streaming removes the obvious limit and leaves a few quieter ones:

- **Counters.** In wasm32 `usize` is 32 bits. Anything that counts bytes in it wraps at 4 GB, so the engine's byte and error counters are `u64`, and the file size crosses from JavaScript as a double, exact up to 253.
- **Line numbers.** libxml2 keeps a node's line number in 16 bits. The engine passes `XML_PARSE_BIG_LINES`, which tells it to keep the real number past line 65,535, so an error on line 38 million says so.
- **Huge text.** By default libxml2 refuses a single text node over 10 MB, a guard against hostile input. `XML_PARSE_HUGE` raises that to 1 GB, which is the one hard limit left: one text node, not the document.
- **Encodings.** The browser build of libxml2 has no iconv, so a document in windows-1251 or Shift\_JIS is transcoded to UTF-8 with a streaming `TextDecoder` on the way in, and its XML declaration rewritten to match. Line numbers stay the same.

## The numbers

Same 5.2 GB file and XSD, same Mac:

| Engine | Time | Throughput | Memory |
|---|---|---|---|
| Native (Rust + libxml2) | 42 s | 118 MB/s | 4 MB resident |
| Browser (WebAssembly, headless Chromium) | 66 s | 79 MB/s | flat, about 20 MB of heap |

The browser time is from the click to the verdict on the page, so it includes the worker, the copies into the heap and the page updating its progress bar. WebAssembly at two thirds of native speed is about what you would expect for a parser.

## Where size still hurts

Two things do not stream as nicely.

**Business rules.** E-invoices are checked against Schematron rules (EN 16931, Peppol, XRechnung) after the schema. Those are XSLT, run with SaxonJS, and XSLT wants the whole document as a tree. That costs about 2.7 seconds per megabyte of invoice, and a 50 MB invoice takes a gigabyte of memory, so the rules stop at 50 MB. Real invoices are a few hundred kilobytes; the large ones are almost always a PDF attachment in base64, so past 50 MB those runs are cut to a stub before the rules see the document.

**Big text nodes.** Those base64 attachments were also where libxml2 itself slowed down: validating a single large text node against XSD took time quadratic in its length, 12 seconds for a 30 MB attachment. We patched it (now 0.09 seconds) and sent the fix upstream; that is a story for another post.

## Try it

The validator is at [xmlvalid.com/validate](https://xmlvalid.com/validate). It takes files up to 50 MB without an account and up to 1 GB with a free one. The 1 GB is where we drew the line for the page, not where the engine stops: if you need to check something larger, [write to us](mailto:support@xmlvalid.com).