Python for financial data pipelines (pandas, NumPy, SQLAlchemy, Apache Airflow)

1. Use `requests` to fetch raw JSON/CSV data from the API for each ticker. 2. Use pandas to parse the data, standardize column names, handle missing values, and set a DateTimeIndex. 3. Use SQLAlchemy to define a table schema (ticker, date, open, high, low, close, volume) and write the DataFrame to SQLite. 4. Create a simple Python script that runs this process and schedule it with a system cron job or `schedule` library.

1. Create two separate Airflow DAGs: one for raw price ingestion (`prices_raw_dag`) and one for corporate action ingestion (`corp_actions_dag`). 2. Define a third DAG (`price_adjustment_dag`) that depends on the completion of both upstream DAGs using Airflow's `TriggerDagRunOperator` or `ExternalTaskSensor`. 3. In the adjustment task, use pandas to merge price and corporate action data, then apply adjustment factors using cumulative product logic to generate an adjusted price series. 4. Load both raw and adjusted prices into distinct database tables, with metadata tagging the adjustment version.

1. Architect an Airflow pipeline with dynamic task generation to parallelize ingestion across instruments, pulling data from a streaming API (e.g., Polygon.io WebSocket) or bulk files. 2. Implement a two-layer storage strategy: land raw data in a data lake (e.g., S3) as Parquet files partitioned by date and symbol, then use Delta Lake or Apache Iceberg to create a merged, versioned table that handles upserts (for corrections) and provides ACID transactions. 3. Integrate data quality checks (e.g., `Great Expectations`) as Airflow tasks that validate schemas and statistical properties before promoting data to the 'gold' layer. 4. Build a caching layer using `redis` or `fastapi` to serve frequent queries for backtesting frameworks, reducing load on the primary data lake.