Optimizely

Ensuring GDPR Compliance with Scalable Data Solutions

Folio3 Data Engineering Enables Secure, Automated Data Retention and Deletion

1994 - New York

Information Technology & Services

1K-5K Employees

Overview

Optimizely, a global leader in digital experience optimization, required a robust data management solution to comply with GDPR. The company needed a secure way to enforce data retention policies while providing flexibility to retain specific datasets based on legal requirements. To meet this challenge, Optimizely partnered with Folio3 Data to develop a customized, scalable data automation solution using Apache Airflow. This solution efficiently managed data retention, deletion, and user data requests, ensuring compliance without compromising operational efficiency.

The Challenge

Implementing GDPR-Compliant Data Retention and Deletion

Optimizely faced several hurdles in implementing a GDPR-compliant data policy within its cloud-based infrastructure:

Automated Data Retention & Deletion – A mechanism was required to automatically remove expired data while preserving datasets that needed to be retained due to government laws or court orders.

Massive Data Volumes – Some datasets in BigQuery exceeded petabytes, making full-scale scans inefficient.

Granular Data Management: Users occasionally request data deletions, access, or downloads, which requires an efficient way to handle such requests without disrupting business operations.

Optimized Processing – A scalable solution was needed to perform retention-based operations without excessive compute overhead or system slowdowns.

The Solution: Automated GDPR Compliance with Apache Airflow

Folio3 Data designed and implemented a robust, automated data retention and deletion system leveraging Apache Airflow to address these challenges. Key components of the solution included:

Custom GDPR Retention & Deletion Workflow

To help the client maintain data compliance, Folio3 developed a customized data pipeline that identifies data eligible for deletion based on retention policies, retains legally required datasets, and automates deletion jobs to meet compliance standards. The pipeline also logs all data deletion and retention activities, providing a clear and reliable audit trail.

Scalable Data Processing with Airflow

Folio3 used Airflow to schedule and orchestrate all retention and deletion jobs, ensuring efficient and reliable data processing. By handling data in 500GB increments, the system minimized the risk of overload while maintaining accurate deletion tracking.

Optimized BigQuery Operations

Folio3 optimized BigQuery processes by segmenting petabyte-scale datasets, reducing query costs and improving execution time. Custom job scheduling staggered resource-intensive operations, preventing performance bottlenecks and ensuring efficient data processing.

Automated User Data Requests Handling

Folio3 streamlined user data requests using Airflow workflows, enabling customers to request data deletion, access, or downloads seamlessly. Secure logging and tracking ensured compliance with GDPR's Right to Access and Erasure requirements.

Technologies Involved In This Case

Results & Achievements

Through its collaboration with Folio3, Optimizely achieved significant operational improvements and full GDPR compliance. The implemented solution delivered measurable results:

100% GDPR Compliance

The automated system ensured compliance with strict data protection laws without manual intervention.

85% Reduction in Data Processing Time

Optimized Airflow jobs dramatically improved retention and deletion operations.

Efficient Handling of Petabyte-Scale Data

Intelligent job segmentation prevented excessive computational costs and improved system performance.

Seamless User Data Management

Automated workflows allowed for swift response to user data requests while maintaining compliance.

Enhanced Data Governance & Security

Full audit trails for every retention and deletion action ensured complete transparency.

Redefining Innovation in Digital Experience

Optimizely’s partnership with Folio3 Data showcases how advanced workflow automation and cloud-based data engineering can drive regulatory compliance at scale. The custom-built Apache Airflow solution ensures GDPR adherence and enhances overall data governance. Optimizely can handle evolving data regulations and business expansion needs with a future-proof and scalable architecture.