Building Data Pipelines for near Real-Time Integration with Reltio MDM
What is Master Data Management
Master data management (MDM) involves creating a single master record for each person, place, or thing in a business, from across internal and external data sources and applications. This information has been de-duplicated, reconciled and enriched, becoming a consistent, reliable source. Once created, this master data serves as a trusted view of business-critical data that can be managed and shared across the business to promote – accurate reporting, – reduce data errors, – remove redundancy, – and help workers make better-informed business decisions.
High Level Goals MDM
1. Enhance Data Quality & Content
- Ensure consistent content across systems
- Standardize/normalize all attributes
- Improve match rates and reduce duplicates
- Data stewardship capability to correct invalid information
- Allow for new categories such as data centers
- Ability to increase field sizes and expand content
2. Streamline Processes
- Simplify and consolidate data processing
- Remove data redundancy
- Eliminate legacy systems such as CKMS
- Reduce operational costs
3. Data Accessibility & Analytics
- Easy access to master data with filtering and reporting
- Audit trail to show source and history of changes to content
- Dashboards to track metrics on quality, volume, error rates over time
4. Enable Future Innovation
- Ability to expand content with new types of data
- Incorporate user-provided content such as contractor profiles, project data
- Provide more accurate analytics as well as leads to customers
- Speed up time to market
Proposed Phases by Domain
Phase 1: Companies & Contacts
- Organization
- Person
- Contact
Phase 2: Reference Data
- Geographies
- Project Types
- Company Types
- Other Attributes
Phase 3: Products
- Products
- Manufacturers
- CAD, BIM, Specs
Phase 4: Projects
- Efeeds
- Dodge Reports
- Primary companies
- Plan Holders, Bidders, Awards
- Documents
Client Challenges
1. Siloed Data
- The company had data scattered across different systems or departments, making it hard to access and share.
2. Maintenance and Monitoring
- ETL pipelines and data integrations required ongoing maintenance and monitoring to ensure they keep working smoothly.
3. Resource Constraints
- The company had limited resources, including skilled staff and technical infrastructure, which made it challenging to manage the onboarding and meeting timelines effectively.
Data Integration
- Developed ETL Routes/ Jobs in Talend to extract data from legacy systems.
- Cleansed and transformed data to ensure consistency, accuracy, and quality.
- Performed end-to-end testing to ensure that the data integration process works seamlessly.
- Ensured that the client’s data was synchronized across all of their systems, which improved efficiency and reduced errors.
- Conduct thorough data validation to ensure that the data on Reltio matches the source system. Developed test cases and validated the data against them.
ETL Jobs Logs & Stats Mangement
- Implemented logging mechanisms within ETL jobs to track their execution status and any errors.
- Stored these logs in SumoLogs for easy access and analysis.
- Set up monitoring and alerting mechanisms to notify team about issues with ETL jobs in real-time.
Project Management
Activities
- Ensured that the project was executed efficiently, meeting deadlines, and staying within budget.
- This included coordinating activities, managing resources, and addressing any challenges that emerged during the project.
- These solutions collectively facilitated a successful integration with Reltio MDM for the client’s company, ensuring data accuracy, efficiency, and cost-effectiveness in the migration process, while also enhancing the overall integration and management of data across systems.
Conclusion
The implementation of data pipelines for real-time integration with Reltio MDM has led to significant improvements in data management. Addressing challenges like siloed data and resource constraints, the project enhanced data quality and streamlined processes. Through meticulous planning and execution, various domains were successfully onboarded. ETL routes, data cleansing, and logging mechanisms ensured consistency and reliability. Effective project management ensured timely delivery. Overall, these solutions have resulted in a successful integration with Reltio MDM, empowering the client with efficient data management and scalability.
