Market Data Pipeline

A client wanted the closing price of every company on the Taiwan exchange, back to June 2019, sitting in their own database rather than in somebody else's dashboard.

The collector walks the full list of roughly 1,700 tickers, and before fetching anything it compares the dates it ought to have against the dates already logged, so a run only pulls what is missing — a job that dies halfway can simply be run again. Prices come from a market data API rather than screen scraping, are shaped with pandas, and land in MySQL. Python throughout.

What is preserved here is the prototype. The version that shipped ran on a schedule on their server, appending each day's close as it happened — the step that turns a one-off backfill into a dataset worth building on.