Metadata Extraction

The metadata stage adds metadata to each pipeline Document. It supports PDF, HTML, plain text, and Markdown files. The stage can also write a separate JSON Lines metadata report; the normal export stage can then save the complete documents, including their metadata.

The PDF path has an important external dependency: it reads existing MonkeyOCR output from a fixed directory. Running the EVE metadata stage does not run OCR itself.

Quick start

Place your input documents under a directory and configure the metadata stage in the repository's config.yaml:

pipeline:
  batch_size: 10
  inputs:
    path: "input_data"
  stages:
    - name: metadata
      config:
        enabled_formats: ["pdf", "html", "txt", "md"]
        fallback_to_filename: true
        export_metadata: true
        metadata_destination: "./output"
        metadata_filename: "metadata.jsonl"

    - name: export
      config:
        format: "jsonl"
        output_dir: "./output/documents"

Run from the repository root:

eve run

The CLI loads config.yaml from the current working directory. The metadata stage is run in the order listed in stages; if there is no export stage, the pipeline adds one with its defaults. Prefer specifying export explicitly so the output format and directory are clear.

What the stage does

The stage filters the incoming documents by enabled_formats, processes supported documents concurrently within each pipeline batch, and passes unsupported documents through unchanged. If the extractor returns no metadata (or raises an error), fallback_to_filename can add a title derived from the filename (for example, mission-report_2024.txt becomes mission report 2024).

Format behavior

PDF

The current PDF extractor reads already-created MonkeyOCR artifacts below:

server/MonkeyOCR/output/<subdirectory>/<subdirectory>.md
server/MonkeyOCR/output/<subdirectory>/<subdirectory>_content_list.json

The root is hard-coded as server/MonkeyOCR/output relative to the process working directory. For each subdirectory, it looks for a DOI, arXiv identifier, or ISBN in the Markdown file, and looks for title candidates in the content-list JSON. A title candidate is a row with text_level == 1; the longest candidate is used. It then requests metadata from Crossref by DOI, or from the arXiv API for an arXiv identifier. If that lookup returns no metadata, it tries Crossref's bibliographic title search. The network requests have a 50-second timeout and the extractor allows up to 40 concurrent lookups.

Prequisitre: Running MonkeyOCR

Run MonkeyOCR separately to produce the expected files. The setup helper is server/setup_monkeyocr.sh; it clones MonkeyOCR into server/MonkeyOCR when run from the server directory.
To OCR only the first page before prediction, scripts/extract_first_pages.py can create a first-page PDF tree. For example: python scripts/extract_first_pages.py --input-dir ./pdfs --output-dir ./pdf_first_pages. Its default input and output paths are /data/pdfs and /data/pdf_first_pages.

HTML

For HTML, the extractor reads the source file as UTF-8 and replaces document.content with its raw HTML. It extracts the contents of the first <title> element using regular expressions, strips tags inside the title, replaces named and numeric HTML entities with spaces, collapses whitespace, and rejects titles shorter than three characters or made only of digits.

Configuration

All options below go under the metadata stage's config key.

Option	Type	Default	Meaning
`enabled_formats`	list of strings	`["pdf", "html", "txt", "md"]`	File formats the stage handles. Format is inferred from the file extension, lowercased.
`fallback_to_filename`	boolean	`true`	Add a filename-derived `title` when extraction returns no metadata or raises an error.
`debug`	boolean	`false`	Enable debug logging for the stage and extractors.
`export_metadata`	boolean	`true`	Append a metadata summary record for each document to a separate JSONL file.
`metadata_destination`	path	`"./output"`	Directory for the separate metadata report. Created if missing.
`metadata_filename`	string	`"metadata.jsonl"`	Name of the separate report file.

The separate report is append-only: a rerun with the same destination and filename adds more records rather than replacing the old report. Set export_metadata: false to disable that report. This does not disable the main pipeline export stage.

Output and schema

The separate report has one JSON object per line. The stage writes base fields and then includes fields from document.metadata:

{
  "filename": "report.html",
  "file_path": "input_data/report.html",
  "file_format": "html",
  "content_length": 1234,
  "has_extracted_metadata": true,
  "extracted_metadata": {
    "title": "Example report",
    "content_length": 1234
  }
}