Dwarves Foundation’s Post

RECONSTRUCTING HISTORICAL TRADING PnL: A DATA PIPELINE APPROACH Bien Vo (our engineer) outlines a robust solution for missing historical trading profit and loss (PnL) data. Gaps in records lead to: - Traders: Unable to validate past strategies or measure performance. - Platforms: Struggling with compliance and reporting. 1/ The Challenge: Missing historical records - High Volume: 43,200 data points per token per month. - API Limits: Binance caps requests to 1,500 points each. - Complexity: 300+ positions per account amplify risks. 2/ Steps to Reconstruct PnL: We approached this as a data pipeline challenge, avoiding risky codebase changes. - Data Collection: Fetch trades, fees, and Binance Kline data. - Mapping: Group data by tokens for efficient processing. - PnL Calculations: 1. Realized PnL: Closed trades minus fees. 2. Unrealized PnL: Position size × (Mark Price - Entry Price). - Aggregation: Sum PnL across user accounts. 3/ Architecture highlights: - Time-series reconstruction: Restore accurate trade timelines. - Parallel processing: Distribute workloads across trading pairs. - Cumulative validation: Track PnL over trade sets for accuracy. 4/ Implementation: - Simultaneously process data for each trading pair, reducing overhead. - Use historical snapshots to rebuild missing PnL states without inconsistencies. - Validate trade sets by tracking cumulative PnL for accuracy and completeness. 5/ Real results: - Accurate reconstruction of historical PnL records. - Processing time reduced from 5 hours to minutes per account. - Designed to handle multiple trading pairs and large datasets seamlessly. Check out the full read: https://lnkd.in/gFMRM8uS #dataengineering #CryptoFinance #dwarves #Software — Dwarves Notes (https://memo.d.foundation/) combine our team’s collective know-hows, R&Ds, and operation approaches. Connect and learn alongside other tech fellows: - Discord: discord.gg/dwarvesv - Github: https://lnkd.in/gZZ2eZMu - Website: https://d.foundation

  • diagram

To view or add a comment, sign in

Explore topics