Built with SupabaseData quality monitored by Sodabase
Stoop

Data health

Stoop mirrors NYC HPD's housing complaint data. Here is how fresh it is, what the pipeline has done recently, and the checks that keep bad data out.

Pipeline
Healthy
Last caught up 5 min ago. Checks every 15 min.
City data as of
Nov 10, 10:04 AM
HPD republishes once a day. Between publishes there is nothing new to fetch.
Records
111,942
complaints across 19,702 buildings this heat season

Data sources

  • HPD housing complaints
    data.cityofnewyork.us/resource/ygpa-z7cr
    Healthy

    Up to date as of 5 min ago.

    Synced through the city's Nov 10, 10:04 AM update · last successful contact 5 min ago

  • HPD building registry
    data.cityofnewyork.us/resource/kj4p-ruqc
    Healthy

    Fills BIN and BBL the complaint feed leaves out. Last lookup worked 5 min ago.

Recent syncs

Last 24 hours: 96 runs (95 caught up, 1 failed). 34,546 rows read, 34,546 written, 0 rejected. Re-reading unchanged rows writes nothing, and a run that runs out of time saves its place for the next one.

StartedResultReadWrittenRejectedTook
Nov 10, 1:30 PMcaught up0005.4s
Nov 10, 1:15 PMcaught up0003.7s
Nov 10, 1:00 PMcaught up0003.8s
Nov 10, 12:45 PMcaught up0008.5s
Nov 10, 12:30 PMcaught up0006.7s
Nov 10, 12:15 PMcaught up0003.5s
Nov 10, 12:00 PMcaught up00016.0s
Nov 10, 11:45 AMcaught up0003.0s
Nov 10, 11:30 AMcaught up0007.2s
Nov 10, 11:15 AMcaught up0004.6s
Nov 10, 11:00 AMcaught up16,54616,54601.2m
Nov 10, 10:45 AMfailed18,00018,00001.2m

How bad data is kept out

  • Validated before it lands

    Every row is checked in one database transaction: IDs numeric, borough real, required fields present. Rows that fail go to a private quarantine table with the reason, never into the app.

  • Constraints the database enforces

    BBL and BIN formats, coordinates inside NYC, open ≤ total, a closed report must have a resolved date. Bad values are impossible to store, not just unlikely.

  • No duplicates, no going backwards

    Each complaint is keyed by HPD's problem ID. An older city update can never overwrite a newer one, so re-running a sync is always safe.

  • Freshness you can see

    We record the city's publish time and our own sync time separately, and an outage of the city's API is shown as one, not as a broken pipeline. Every page of a sync saves its place in the same transaction as its rows, so an interrupted sync resumes exactly where it stopped.

  • Watched by Sodabase

    10 recurring rules (valid statuses, unique IDs, coordinates in NYC, counts that add up) plus freshness and volume monitors on every table.

  • Fixed at the source

    Sodabase flagged buildings missing a BBL and a BIN. The city feed omits them on some rows, so the pipeline derives BBL from block and lot, treats the city's placeholder BINs (1000000, 2000000, …) as missing, and fills real BINs from HPD's building registry. Missing BBL: 0. Missing BIN: 119 (not published by the city for those buildings).