Method
How we know
Every grade comes from two public sources: what the agency scheduled and where its trains actually were. Nothing is estimated by hand and every number can be recomputed from the downloads.
What we collect
The schedule. Every system here publishes its timetable as open data. We download it on the agency's own refresh cycle and keep each copy, so any day can be re-scored against the timetable that was in force.
The trains. Once a minute, each agency's live feed reports every train in service and its next stop. We save every response.
Alerts. Every 10 minutes, each agency's service alerts. Only significant unplanned disruptions count. Planned work and elevator outages do not.
When a train counts as seen
When a train’s next stop changes from A to B, it has just passed A. Stations skipped between two polls are filled in from the scheduled stop order. A train’s direction comes from the order of the stops it passes, because some feeds change a train’s direction code partway through a trip. Terminal stations are left out because trains appear and disappear there. Feeds often first show a train a stop or two after it leaves the terminal, so those stops are counted when it first appears. When a train drops out of the feed with only the terminal left ahead, the last stop before the terminal is counted too.
The measures
Trains ran. At every station, each scheduled train is matched to a train seen between 5 minutes early and 15 minutes late. Each seen train matches once. Extra unscheduled trains do not make up for missing ones.
Extra wait. For someone arriving at a random moment, the average wait is the sum of each gap squared, divided by twice the total time. We work that out for the trains seen and for the schedule, hour by hour, and report the difference in minutes. Long gaps make it grow fast, which is how riders feel them.
Bunching. The share of gaps between trains shorter than a quarter of the scheduled gap. Any gap under a minute also counts. Lines that never schedule 3 or more trains at once in a direction, such as short shuttles, are not graded on bunching.
Long waits. A gap counts as a long wait when it is at least twice the longest scheduled gap it overlaps, and at least 10 minutes.
Delays. Each agency reports lateness its own way, so delays are shown but never graded:
- Chicago: flagged delayed, the share of runs flagged delayed by Train Tracker.
- New York: late trips, the share of trips more than 5 minutes behind schedule by their last station seen.
The grade
Each measure earns 0 to 100 points on a fixed scale. Trains ran: 80% earns 0 and 99% earns 100. Extra wait: 5 minutes earns 0 and half a minute earns 100. Bunching: 25% of gaps earns 0 and 2% earns 100. The score weighs them 50% / 30% / 20%. If a line isn't graded on bunching, the other two are reweighted.
The scale is the same for every city and does not curve. A city's grade is its lines' scores weighted by scheduled station stops, so a short line counts for less than a long one. A line whose every stop is a terminal, like a two-station shuttle, has nothing in between to time and is not graded. Rankings use the last 7 days. A system gets its first grade after 7 full days of data. "Most improved" compares them with the 7 days before.
This week's best and worst
To appear in a city's "This week" cards, a line needs at least 200 scheduled trips in the week, so one bad day on a tiny line can't decide it.
When the collector misses time
If the collector goes more than 150 seconds without an answer, that stretch is left out of both sides: no scheduled trains are counted against the line and no gaps that span it are measured.
| Day | Chicago | New York |
|---|---|---|
| Sat Oct 10 | 35.97% | 31.81% |
The report card
The report card puts every city side by side, so the conditions behind a grade are easy to compare. It explains. It is not a grade.
- From our data: the time of day and the station with the lowest share of trains that ran, the causes named in the agency's significant alerts (sorted into a few groups by keywords), and the share of missing trains with no alert posted on that line within half an hour. Hours come from the timetable itself.
- Background: when the first line opened and its major rebuilds, whether other railways' trains share the track, the signal system, and the share of track under slow zones. Each value links to its source.
- Funding: the share of operating costs paid by fares, operating cost and subsidy per ride, and capital spending, in US dollars. US figures are from the National Transit Database; others from the operators' own accounts, adjusted to the same rules (depreciation left out) where we can. The notes under the card list every adjustment.
- Not published vs none: "Not published" means we found no regular public figure. "None" or "No" means the source says the answer is zero. A city never gets credit for not publishing something.
What this can't see
Positions are checked once a minute, so times are accurate to about a minute. The live feeds don't say how full each train is, so crowding isn't measured.
City by city
Chicago
The schedule. CTA publishes its full timetable as a GTFS feed, refreshed every week or two. We download it weekly and use the timetable for each day.
The trains. Once a minute, CTA Train Tracker reports every 'L' train in service: its run number, direction, next station, and whether it is flagged delayed.
Limits. Train Tracker gives run numbers, not timetable trip ids, so trains are matched to the schedule by time at each station rather than one-to-one by trip. Train Tracker's direction codes change partway through Loop trips, so direction is worked out from the order of stops instead.
New York
The schedule. The MTA publishes a supplemented GTFS schedule every hour that already includes the planned work for the coming week, so a weekend reroute is graded against the reroute timetable, not the normal one. We download it daily and use the timetable for each day.
The trains. Once a minute, the MTA's eight subway feeds report every train that has left its terminal: its scheduled trip, direction and the stops it still has ahead. Trips that are scheduled but have no train assigned are ignored, so a train only counts once it is really moving.
Limits. Lateness is measured trip by trip, using the trip id in the feed. Whether a train ran is still matched by time at each station, the same way as every other city, because trips are sometimes renumbered mid-route.