Data Sources

Local inventory, cloud sync readiness, scraping status, and provenance.

Every record in this prototype starts with a repository-local file path before moving to Neon and scheduled public-source refresh jobs.

48
Total files

Complete inventory across the sourcedata tree.

48
Parsed files

CSV and JSON files currently feed typed API responses.

Ready
Neon sync

Schema is defined without requiring a local cloud connection.

Live
Runtime refresh

The app checks file path, size, and modified time on each request.

Source inventory slice

Loading
Inventory view

Source inventory slice: Folder

Data Source And Lineage Research

Local-source-first architecture with Treasury transparency, USSGL, and DoD source governance.

The Data Sources page should teach how raw files become governed analytic records and how those records later migrate to Neon, Data Broker-style validations, and public-source refreshes.

Research-backed

Source governance model

Every parsed fact needs a lineage record before it becomes a dashboard, AI context item, or database row.

  • Inventory: path, source family, file type, fiscal year, agency, domain, size, modified time, parser status, and checksum/signature.
  • Extract: parser version, extraction timestamp, row count, columns detected, validation result, rejected rows, and warning count.
  • Normalize: canonical field names, data types, units, dollars/thousands conversion, fiscal-year handling, and source primary key.
  • Validate: schema completeness, required fields, duplicate detection, referential checks, reasonableness tests, and reconciliation totals.
  • Publish: API payload, dashboard chart, AI context, database table, refresh status, and audit trail.

Migration and automation playbook

Move from local files to production data services in controlled stages.

  • Stage 1: local snapshot with source signature and generated gzip artifact for Vercel deployment.
  • Stage 2: Neon schema for source_documents, budget_lines, award_transactions, audit_documents, findings, anomalies, and model_runs.
  • Stage 3: ingestion jobs for DoD budget materials, USAspending downloads/API, Treasury Fiscal Data, GAO/OIG reports, and agency AFRs.
  • Stage 4: validation jobs similar to Data Broker thinking: load, validate, publish, certify, reconcile.
  • Stage 5: lineage API for every dashboard number and AI answer.
Treasury Data TransparencyGovernmentwide spending data model, Data Broker, and USAspending publication path.Treasury USSGLUniform chart of accounts and accounting scenarios.DoD FMR 7000.14-RDoD financial management regulation by volume and chapter.USAspending Data DictionaryAward and federal spending data element definitions.

Source files by source family

SourceTypeDomainFYModifiedStatus
sourcedata/audit/FY2025_DoD_Agency_Financial_Report.pdfPDFdocument20256/9/2026, 1:02:32 AMparsed
sourcedata/audit/FY2025_DoD_Audit_Finding_DODIG-2026-032.pdfPDFdocument20256/9/2026, 1:03:03 AMparsed
sourcedata/dod_1/FY2026/FY2026_c1.xlsxXLSXexhibit20266/9/2026, 12:35:54 AMparsed
sourcedata/dod_1/FY2026/FY2026_m1_display.xlsxXLSXexhibit20266/9/2026, 12:35:42 AMparsed
sourcedata/dod_1/FY2026/FY2026_o1_display.xlsxXLSXexhibit20266/9/2026, 12:35:43 AMparsed
sourcedata/dod_1/FY2026/FY2026_p1r_display.xlsxXLSXexhibit20266/9/2026, 12:35:50 AMparsed
sourcedata/dod_1/FY2026/FY2026_p1_display.xlsxXLSXexhibit20266/9/2026, 12:35:47 AMparsed
sourcedata/dod_1/FY2026/FY2026_r1_display.xlsxXLSXexhibit20266/9/2026, 12:35:52 AMparsed
sourcedata/dod_1/FY2026/FY2026_rf1_display.xlsxXLSXexhibit20266/9/2026, 12:35:45 AMparsed
sourcedata/dod_1/FY2027/FY2027_c1_display.xlsxXLSXexhibit20276/9/2026, 12:42:21 AMparsed
sourcedata/dod_1/FY2027/FY2027_m1_display.xlsxXLSXexhibit20276/9/2026, 12:42:09 AMparsed
sourcedata/dod_1/FY2027/FY2027_o1_display.xlsxXLSXexhibit20276/9/2026, 12:42:10 AMparsed
sourcedata/dod_1/FY2027/FY2027_p1r_display.xlsxXLSXexhibit20276/9/2026, 12:42:17 AMparsed
sourcedata/dod_1/FY2027/FY2027_p1_display.xlsxXLSXexhibit20276/9/2026, 12:42:14 AMparsed
sourcedata/dod_1/FY2027/FY2027_r1_display.xlsxXLSXexhibit20276/9/2026, 12:42:19 AMparsed
sourcedata/dod_1/FY2027/FY2027_rf1_display.xlsxXLSXexhibit20276/9/2026, 12:42:12 AMparsed
sourcedata/dod_json/FY2027/FY2027_Drug_Interdiction_and_Counter-Drug_Activities.jsonJSONbudget20276/9/2026, 12:43:11 AMparsed
sourcedata/dod_json/FY2027/FY2027_Pacific_Deterrence_Initiative.jsonJSONbudget20276/9/2026, 12:43:02 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_Budget_Request_Overview_Book.pdfPDFdocument20266/9/2026, 12:37:27 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_DoD_FY2026_Mandatory_Funding_Overview.pdfPDFdocument20266/9/2026, 12:38:09 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_DoD_Revolving_Funds_J-Book_FY2026.pdfPDFdocument20266/9/2026, 12:39:36 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_ENV-30.pdfPDFdocument20266/9/2026, 12:39:06 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_Force_Structure_Changes_Exhibit.pdfPDFdocument20266/9/2026, 12:37:52 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_Military_Construction_Defense-Wide_Consolidated.pdfPDFdocument20266/9/2026, 12:39:40 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_OM_Overview.pdfPDFdocument20266/9/2026, 12:37:45 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_OP-31.pdfPDFdocument20266/9/2026, 12:39:16 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_OP-34.pdfPDFdocument20266/9/2026, 12:39:21 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_OP-8.pdfPDFdocument20266/9/2026, 12:39:11 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_OSD_OP-5.pdfPDFdocument20266/9/2026, 12:39:01 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_PPBE_Reform_Activities.pdfPDFdocument20266/9/2026, 12:37:39 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_PROC_OSD_PB_2026.pdfPDFdocument20266/9/2026, 12:39:26 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_RDTE_OSD_PB_2026.pdfPDFdocument20266/9/2026, 12:39:31 AMparsed
sourcedata/dod_pdf/FY2026/FY2026_Weapons.pdfPDFdocument20266/9/2026, 12:37:33 AMparsed
sourcedata/dod_pdf/FY2026/FY26-30 DoW Strategic Plan 6Apr2026 r18.pdfPDFdocument20266/9/2026, 12:38:54 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_Budget_Request_Overview_Book.pdfPDFdocument20276/9/2026, 12:45:14 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_DoW_Revolving_Funds_J-Book_FY2027.pdfPDFdocument20276/9/2026, 12:45:04 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_Force_Structure_Changes_Exhibit.pdfPDFdocument20276/9/2026, 12:45:31 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_Mandatory_Funding_Overview.pdfPDFdocument20276/9/2026, 12:45:39 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_Military_Construction_Defense-Wide_Consolidated.pdfPDFdocument20276/9/2026, 12:45:09 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_OM_Overview.pdfPDFdocument20276/9/2026, 12:45:24 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_OSW_OP-5.pdfPDFdocument20276/9/2026, 12:44:50 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_PROC_OSD_PB_2027.pdfPDFdocument20276/9/2026, 12:44:55 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_RDTE_OSW_PB_2027.pdfPDFdocument20276/9/2026, 12:45:00 AMparsed
sourcedata/dod_pdf/FY2027/FY2027_Weapons.pdfPDFdocument20276/9/2026, 12:45:19 AMparsed
sourcedata/USASPENDING/All_Assistance_PrimeTransactions_2025-03-30_H05M14S35_1.csvCSVawards20253/30/2025, 5:18:48 AMparsed
sourcedata/USASPENDING/All_Assistance_Subawards_2025-03-30_H05M14S21_1.csvCSVawards20253/30/2025, 5:18:48 AMparsed
sourcedata/USASPENDING/All_Contracts_PrimeTransactions_2025-03-30_H05M14S25_1.csvCSVawards20253/30/2025, 5:18:48 AMparsed
sourcedata/USASPENDING/All_Contracts_Subawards_2025-03-30_H05M14S04_1.csvCSVawards20253/30/2025, 5:18:48 AMparsed