Document Extraction Comparator – Findings Report
24 PDFs tested . Judge: gpt-5.4 Generated 2026-03-11
Overall Extractor Leaderboard
| Extractor | Avg Score | Wins | PDFs Tested | Avg Time (s) |
|---|---|---|---|---|
| llm_direct | 4.31 | 20 | 24 | 28.0 |
| marker | 3.67 | 0 | 1 | 4.0 |
| mineru | 3.5 | 0 | 1 | 42.2 |
| docling | 3.48 | 0 | 24 | 5.1 |
| pymupdf4llm | 3.36 | 2 | 24 | 0.9 |
| markitdown | 3.33 | 1 | 24 | 0.5 |
| unstructured | 3.02 | 1 | 23 | 10.3 |
llm_direct ranked highest overall with an average score of 4.31 across 24 documents, winning 20 of 24 head-to-head comparisons. However, it averaged 28.0s per document. The fastest extractor was markitdown at 0.5s average, scoring 3.33 overall.
2022-Lobo-Softball-Schedule.pdf
Winner: llm_direct (avg 5.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 5 | 5 | 5.00 | 7.0 |
| docling | 4 | 2 | 3 | 2 | 4 | 5 | 3.33 | 3.5 |
| unstructured | 2 | 2 | 2 | 2 | 2 | 4 | 2.33 | 2.3 |
| markitdown | 1 | 1 | 1 | 1 | 1 | 5 | 1.67 | 0.0 |
| pymupdf4llm | 1 | 1 | 1 | 1 | 1 | 5 | 1.67 | 0.7 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` .
2024-25_District_Calendar.pdf
Winner: llm_direct (avg 4.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 3 | 3 | 5 | 4.00 | 16.2 |
| markitdown | 3 | 2 | 3 | 2 | 3 | 4 | 2.83 | 0.1 |
| docling | 2 | 2 | 3 | 3 | 2 | 2 | 2.33 | 9.8 |
| unstructured | 2 | 2 | 3 | 2 | 3 | 2 | 2.33 | 2.1 |
| pymupdf4llm | 2 | 2 | 2 | 1 | 2 | 4 | 2.17 | 0.5 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
2025-2026_SHUTTLE_BUSES.pdf
Winner: llm_direct (avg 5.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 5 | 5 | 5.00 | 7.6 |
| docling | 4 | 3 | 5 | 5 | 4 | 5 | 4.33 | 1.7 |
| unstructured | 3 | 2 | 2 | 5 | 4 | 5 | 3.50 | 1.7 |
| pymupdf4llm | 2 | 2 | 2 | 5 | 3 | 5 | 3.17 | 0.2 |
| markitdown | 2 | 1 | 2 | 1 | 3 | 4 | 2.17 | 0.1 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key, :
4-STRATEGIC-PLAN.pdf
Winner: llm_direct (avg 4.7)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 4 | 5 | 5 | 4 | 5 | 4.67 | 9.3 |
| markitdown | 3 | 3 | 3 | 4 | 4 | 2 | 3.17 | 0.1 |
| docling | 3 | 3 | 3 | 3 | 3 | 2 | 2.83 | 10.0 |
| unstructured | 2 | 2 | 2 | 3 | 3 | 1 | 2.17 | 12.1 |
| pymupdf4llm | 1 | 1 | 1 | 1 | 1 | 3 | 1.33 | 0.4 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
CPSD_2024-2025_Transportation_Snow_Routes.pdf
Winner: llm_direct (avg 5.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 5 | 5 | 5.00 | 19.4 |
| docling | 4 | 4 | 5 | 4 | 4 | 5 | 4.33 | 6.2 |
| pymupdf4llm | 4 | 4 | 5 | 4 | 5 | 3 | 4.17 | 0.4 |
| markitdown | 4 | 3 | 3 | 2 | 4 | 2 | 3.00 | 0.2 |
| unstructured | 3 | 2 | 3 | 1 | 3 | 4 | 2.67 | 6.6 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`
CSL_Webpage_Design.pdf
Winner: llm_direct (avg 4.7)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 4 | 5 | 5 | 4 | 5 | 4.67 | 5.1 |
| markitdown | 4 | 4 | 4 | 5 | 3 | 5 | 4.17 | 0.2 |
| pymupdf4llm | 3 | 3 | 3 | 5 | 4 | 4 | 3.67 | 0.2 |
| unstructured | 3 | 2 | 3 | 5 | 3 | 2 | 3.00 | 2.2 |
| docling | 2 | 2 | 2 | 5 | 2 | 3 | 2.67 | 2.4 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
DPS-Calendar-2026-27_ENGLISH.pdf
Winner: pymupdf4llm (avg 3.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| pymupdf4llm | 3 | 3 | 3 | 2 | 3 | 4 | 3.00 | 0.5 |
| llm_direct | 3 | 3 | 4 | 1 | 2 | 5 | 3.00 | 9.2 |
| markitdown | 2 | 2 | 2 | 1 | 3 | 4 | 2.33 | 0.2 |
| docling | 2 | 2 | 1 | 1 | 2 | 4 | 2.00 | 2.4 |
| unstructured | 1 | 2 | 2 | 1 | 2 | 3 | 1.83 | 3.3 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`
DPS-Calendar-2026-27_FRENCH.pdf
Winner: llm_direct (avg 3.7)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 3 | 4 | 5 | 2 | 3 | 5 | 3.67 | 33.5 |
| pymupdf4llm | 3 | 2 | 3 | 1 | 3 | 3 | 2.50 | 0.5 |
| markitdown | 2 | 2 | 2 | 1 | 3 | 4 | 2.33 | 0.3 |
| docling | 2 | 2 | 2 | 1 | 2 | 3 | 2.00 | 2.6 |
| unstructured | 2 | 2 | 3 | 1 | 1 | 2 | 1.83 | 3.4 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`
Denver_Schools_Thrive_Initiative_Regional_Meet.pdf
Winner: llm_direct (avg 4.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 4 | 5 | 4.83 | 9.7 |
| docling | 4 | 4 | 5 | 5 | 4 | 5 | 4.50 | 2.6 |
| markitdown | 4 | 3 | 5 | 5 | 4 | 3 | 4.00 | 0.4 |
| pymupdf4llm | 4 | 4 | 5 | 5 | 4 | 2 | 4.00 | 0.2 |
| unstructured | 3 | 3 | 5 | 5 | 4 | 2 | 3.67 | 3.9 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` .
Director_Sia_Community_Engagement.pdf
Winner: llm_direct (avg 3.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 3 | 3 | 4 | 3.83 | 31.2 |
| markitdown | 4 | 4 | 5 | 3 | 3 | 2 | 3.50 | 0.3 |
| docling | 3 | 3 | 3 | 4 | 3 | 2 | 3.00 | 9.8 |
| pymupdf4llm | 3 | 3 | 4 | 2 | 3 | 2 | 2.83 | 1.2 |
| unstructured | 3 | 3 | 4 | 2 | 3 | 2 | 2.83 | 25.6 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` .
English_One_Pager_for_New_Families.pdf
Winner: llm_direct (avg 4.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 4 | 5 | 4 | 4 | 5 | 4.50 | 7.8 |
| docling | 4 | 4 | 5 | 3 | 4 | 4 | 4.00 | 2.4 |
| unstructured | 4 | 3 | 4 | 2 | 4 | 5 | 3.67 | 2.2 |
| pymupdf4llm | 4 | 3 | 4 | 2 | 4 | 3 | 3.33 | 0.3 |
| markitdown | 2 | 2 | 3 | 2 | 3 | 4 | 2.67 | 0.1 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
FAQ_School_Choice_OSCPE.pdf
Winner: llm_direct (avg 4.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 5 | 2 | 4.50 | 8.0 |
| pymupdf4llm | 4 | 4 | 5 | 5 | 5 | 3 | 4.33 | 0.2 |
| docling | 4 | 4 | 5 | 5 | 3 | 4 | 4.17 | 2.4 |
| markitdown | 4 | 3 | 4 | 5 | 4 | 2 | 3.67 | 0.1 |
| unstructured | 3 | 2 | 4 | 5 | 4 | 2 | 3.33 | 3.9 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
Graduation_Requirements_584.10.pdf
Winner: llm_direct (avg 4.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 4 | 5 | 3 | 4.50 | 7.5 |
| docling | 4 | 4 | 5 | 4 | 5 | 4 | 4.33 | 2.0 |
| markitdown | 5 | 4 | 5 | 4 | 5 | 3 | 4.33 | 0.1 |
| pymupdf4llm | 5 | 4 | 5 | 4 | 5 | 2 | 4.17 | 0.2 |
| unstructured | 4 | 3 | 4 | 3 | 4 | 2 | 3.33 | 3.6 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`
LRSD_Ready_for_Learning_Plan_FY23.pdf
Winner: pymupdf4llm (avg 3.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| pymupdf4llm | 4 | 4 | 5 | 3 | 3 | 2 | 3.50 | 2.1 |
| markitdown | 4 | 4 | 4 | 3 | 3 | 2 | 3.33 | 2.2 |
| docling | 3 | 3 | 3 | 2 | 3 | 2 | 2.67 | 11.7 |
| unstructured | 3 | 3 | 4 | 2 | 2 | 2 | 2.67 | 47.7 |
| llm_direct | 3 | 4 | 4 | 1 | 1 | 3 | 2.67 | 28.0 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` .
Lady_Lobo_Volleyball_Booster_Club_Membership.pdf
Winner: llm_direct (avg 4.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 4 | 5 | 5 | 5 | 5 | 4.83 | 6.3 |
| pymupdf4llm | 3 | 3 | 4 | 5 | 3 | 4 | 3.67 | 0.2 |
| docling | 3 | 3 | 4 | 5 | 4 | 2 | 3.50 | 3.0 |
| markitdown | 3 | 2 | 2 | 5 | 4 | 4 | 3.33 | 0.2 |
| unstructured | 3 | 3 | 3 | 5 | 3 | 3 | 3.33 | 2.1 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` .
Lobo_Student-Athlete_Discipline_Policy.pdf
Winner: llm_direct (avg 4.7)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 4 | 5 | 5 | 4 | 5 | 4.67 | 6.4 |
| docling | 4 | 4 | 5 | 5 | 4 | 5 | 4.50 | 3.1 |
| markitdown | 4 | 4 | 5 | 5 | 4 | 5 | 4.50 | 0.1 |
| pymupdf4llm | 4 | 4 | 5 | 5 | 4 | 5 | 4.50 | 0.2 |
| unstructured | 4 | 3 | 5 | 5 | 5 | 4 | 4.33 | 2.1 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`
Private_Lesson_Scholarship_Application.pdf
Winner: llm_direct (avg 5.0)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 5 | 5 | 5 | 5 | 5 | 5 | 5.00 | 5.7 |
| markitdown | 5 | 4 | 5 | 5 | 5 | 5 | 4.83 | 0.1 |
| pymupdf4llm | 5 | 4 | 5 | 5 | 5 | 5 | 4.83 | 0.3 |
| unstructured | 5 | 3 | 5 | 5 | 4 | 5 | 4.50 | 1.8 |
| docling | 4 | 4 | 5 | 4 | 4 | 5 | 4.33 | 3.2 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
RI.1_Equity.pdf
Winner: llm_direct (avg 3.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 3 | 4 | 3 | 3.83 | 30.7 |
| docling | 4 | 4 | 4 | 3 | 4 | 3 | 3.67 | 4.6 |
| markitdown | 4 | 4 | 4 | 3 | 4 | 3 | 3.67 | 0.9 |
| pymupdf4llm | 4 | 3 | 4 | 2 | 4 | 3 | 3.33 | 0.8 |
| unstructured | 3 | 3 | 4 | 2 | 3 | 3 | 3.00 | 11.7 |
marker: In order to use GoogleGeminiService, you must set the configuration values gemini_api_key, :
Retirement_of_Dr_James_Wilcox.pdf
Winner: llm_direct (avg 4.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 5 | 4 | 5 | 4.50 | 7.1 |
| pymupdf4llm | 3 | 3 | 4 | 5 | 4 | 3 | 3.67 | 0.2 |
| docling | 3 | 3 | 2 | 5 | 4 | 3 | 3.33 | 4.1 |
| markitdown | 2 | 2 | 2 | 5 | 3 | 4 | 3.00 | 0.4 |
| unstructured | 2 | 2 | 1 | 5 | 3 | 2 | 2.50 | 2.4 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,` :
Short_History_of_the_LRSD.pdf
Winner: unstructured (avg 4.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| unstructured | 5 | 4 | 5 | 5 | 5 | 5 | 4.83 | 3.3 |
| llm_direct | 5 | 4 | 5 | 5 | 5 | 5 | 4.83 | 8.9 |
| docling | 4 | 4 | 5 | 5 | 5 | 5 | 4.67 | 1.9 |
| pymupdf4llm | 4 | 4 | 5 | 5 | 5 | 5 | 4.67 | 0.2 |
| markitdown | 4 | 4 | 5 | 5 | 5 | 3 | 4.33 | 0.1 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
Student_Handbook_GCB.pdf
Winner: llm_direct (avg 4.5)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 4 | 5 | 5 | 4.50 | 74.8 |
| docling | 4 | 4 | 5 | 3 | 5 | 4 | 4.17 | 5.9 |
| pymupdf4llm | 4 | 4 | 5 | 3 | 5 | 2 | 3.83 | 1.1 |
| markitdown | 4 | 3 | 4 | 2 | 4 | 2 | 3.17 | 0.7 |
| unstructured | 3 | 3 | 4 | 2 | 3 | 3 | 3.00 | 24.2 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`.
Wellness-Committee-Member-2025-2026.pdf
Winner: llm_direct (avg 4.3)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 4 | 4 | 5 | 4 | 4 | 5 | 4.33 | 7.1 |
| docling | 3 | 3 | 4 | 3 | 4 | 5 | 3.67 | 5.7 |
| markitdown | 3 | 3 | 4 | 2 | 4 | 5 | 3.50 | 0.1 |
| pymupdf4llm | 4 | 3 | 4 | 2 | 4 | 3 | 3.33 | 0.3 |
| unstructured | 4 | 2 | 3 | 1 | 3 | 5 | 3.00 | 2.8 |
marker: In order to use GoogleGeminiService, you must set the configuration values gemini_api_key, .
txschools-one-pager-25-web.pdf
Winner: markitdown (avg 4.3)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| markitdown | 4 | 4 | 5 | 5 | 3 | 5 | 4.33 | 0.1 |
| llm_direct | 4 | 4 | 5 | 5 | 3 | 5 | 4.33 | 5.0 |
| marker | 3 | 3 | 4 | 5 | 2 | 5 | 3.67 | 8.6 |
| mineru | 3 | 3 | 4 | 5 | 2 | 4 | 3.50 | 42.2 |
| pymupdf4llm | 3 | 3 | 2 | 5 | 3 | 4 | 3.33 | 0.2 |
| docling | 2 | 3 | 3 | 5 | 3 | 2 | 3.00 | 2.8 |
unstructured: Unknown model type: layout_v1.0.0
voice-winter2025-digital.pdf
Winner: llm_direct (avg 2.8)
| Extractor | text_accuracy | structure_preservation | reading_order | table_fidelity | completeness | noise_artifacts | Avg | Time (s) |
|---|---|---|---|---|---|---|---|---|
| llm_direct | 2 | 3 | 3 | 3 | 2 | 4 | 2.83 | 319.6 |
| docling | 2 | 2 | 2 | 2 | 3 | 2 | 2.17 | 19.1 |
| markitdown | 2 | 2 | 2 | 2 | 3 | 1 | 2.00 | 5.3 |
| unstructured | 1 | 2 | 2 | 2 | 3 | 1 | 1.83 | 75.7 |
| pymupdf4llm | 2 | 2 | 1 | 1 | 2 | 2 | 1.67 | 10.6 |
marker: In order to use GoogleGeminiService, you must set the configuration values `gemini_api_key,`