Migrating Lotus Notes to Dataverse and SharePoint Online

Joel Plaut

December 20, 2024

All Post

Project Overview

This case study explores the successful migration of approximately 160,000 records from a legacy Lotus Notes database to a modern, cloud-based solution leveraging Dataverse and Power Apps. The primary goal of this migration was to enhance data management, accessibility, and overall efficiency while transitioning to a more scalable and flexible platform, like the benefits of migrating from manual processes to automated systems.

Project Details

Tools and Technology:

Migration Strategies

• Smaller Batch Migration: Due to repeated failures during large batch exports, the team opted for a smaller batch migration strategy. This approach allowed for the migration of data in manageable segments, reducing the risk of failures and facilitating easier troubleshooting.
• Exporting as Data Items: Instead of exporting records as separate documents, the new strategy involved migrating them as “Data Items” into SharePoint Lists. This method simplified the process of handling attachments and comments, ensuring that they were directly associated with the corresponding records.
• Power Automate Implementation: To address issues with metadata mapping and the export of comments, the team is exploring the use of Power Automate. This tool will read the exported XML metadata files from Tzunami and migrate them to SharePoint lists, ensuring accurate mapping of approximately 167 fields.

Challenges and Solutions

Data Migration Challenges

• XML Data Parsing: The migration process involved extracting data from the Legacy Database as XML files. Parsing these files presented challenges, particularly when dealing with complex data structures like people fields, choice fields, and handling null values.
• Large Dataset Migration: The sheer volume of data (160,000 records) posed significant challenges in terms of processing time and potential failures during the migration process.

Solutions Employed

• Power Automate for Data Extraction: Power Automate flows were utilized to read the XML files, extract relevant data, and insert it into Dataverse tables.
• Custom Logic for Data Handling: Specific logic was developed to handle people fields by referencing user data from Dataverse, choice fields by using global choice IDs, and null values appropriately.
• Batch Processing: To mitigate the impact of large datasets, the migration was divided into smaller batches, reducing the risk of failures and improving overall efficiency.

Form Development and Dataverse Integration

• Model-Driven App Development: A Model-Driven App was created in Power Apps to replicate the structure and functionality of the Legacy Database.
• Dataverse Schema Design: Tables and columns were designed in Dataverse to accurately represent the migrated data, including fields for cascading, static headers, and timeline tracking.
• Business Rules and Logic: Business rules were implemented to control field visibility and behaviour, ensuring the application adheres to specific business requirements.

Key Achievements and Benefits

• Successful Data Migration: The migration of 160,000 records from the Legacy Database to Dataverse was successfully completed.
• Modernized Application: The Power Apps application provides a more modern and user-friendly interface, enhancing productivity and accessibility.
• Improved Data Management: Dataverse offers robust data management capabilities, including features for data governance, security, and integration with other Microsoft 365 services.
• Enhanced Scalability: The cloud-based nature of Dataverse and Power Apps ensures scalability and flexibility to accommodate future growth.

Lessons Learned and Future Considerations

• Batch Processing: The importance of dividing large datasets into smaller batches for migration was emphasized to reduce the risk of failures and improve efficiency.
• Custom Logic Development: The need for custom logic to handle specific data types and scenarios was highlighted, particularly when dealing with legacy systems.
• Data Validation and Quality: Implementing robust data validation mechanisms is crucial to ensure data integrity and accuracy during the migration process.
• Continuous Improvement: The migration project should be viewed as an ongoing process, with opportunities for future enhancements and optimizations as technology evolves.

Conclusion

The migration of Legacy Database data to Dataverse and Power Apps showcases how modern cloud platforms can revolutionize legacy applications. By addressing challenges in data migration, form development, and integration, this case study illustrates the advantages of transitioning to flexible and scalable solutions.