Overview
Built an automated data pipeline that fetches healthcare volume pricing data from an external API and uploads it to SharePoint Online for clinical and financial teams. The pipeline handles API unreliability with intelligent retry logic and runs unattended on a daily schedule.
Key Features
- Configurable API integration with date offset support for historical data pulls
- Intelligent retry logic (3 attempts, 10-minute delays) for rate-limited APIs
- Detection of rate limit responses embedded in HTTP 200 bodies
- JSON data preservation with 100-level depth for complex nested structures
- UTF-8 no BOM file output with dated filenames
- Microsoft Graph API upload to SharePoint with dynamic site/drive discovery
- Automatic folder creation during upload operations
- Comprehensive error logging with full Graph API response details
Technical Highlights
- OAuth 2.0 client credentials flow for Microsoft Graph authentication
- Azure AD app registration with Sites.ReadWrite.All permissions
- PowerShell data file (.psd1) for environment-specific configuration
- Dynamic SharePoint path parsing for /sites/SiteName/Library/Folder structures
- Hardcoded Site/Drive ID fallback for faster execution on known targets
- Exponential backoff retry mechanism for API resilience
- Scheduled execution via Windows Task Scheduler