The feeds were used by thousands of analytical jobs to calculate risk measures. Performance, stability and scalability were the main focus areas. The objective is to move to high end operating system that improves ease of operations as well as data quality implementation. 

• Migration from Solaris to Linux operating system 
• Removing hardcoded security credentials from scripts
• Creating a robust environment for ETL
• Building different environments for dev & beta in addition to the production instance
• Restricted access to ETL developers during business hours
• Create a centralized metadata for: Source Vendor like connection string, username & password, whether be for FTP, HTTP or Email
• Autosys job information including the business user details, SLA etc.
• Create a separate code repository for ETL and move all code

• ~ 945 feeds were converted to new metadata driven framework
• Dedicated environments for Dev, Beta, Prod, DR (Linux)
• Parallel job execution supported on Dev, Beta, Prod, DR (on demand)
• Dedicated file storage for each environment (Dev, Beta, Prod, DR)
• Dedicated data schema in Oracle

Step 1-Start enhancing the script on TEST environment and perform unit testing. Setup job in TEST autosys and execute.

Step 2 Compare data loaded in TEST tables vs PROD tables for a week using automated reporting tools.

Step 3- Stakeholders signoff on test reports.

Step 4- Production Release.

• Core Technologies: C++, Python, csh, Bash, Perl, Autosys
• Databases: Oracle SQL Developer, SQL Server Management Studio
• ETL Tool- Informatica, Talend
• QA: Query Surge, ICEDQ
• Reporting Tools: Micro-strategy, Power BI