I have done a data analysis project earlier which the data was around 10GB while my laptop had only 8GB of RAM, as it is obvious I could not handle such amount of data with regular methods, I used disk.frame package in R which is very suitable for such scenario. It splits the data into short files with fst format, and the structure of coding is tidyverse which makes it suitable to use and easy to learn. All of processing in this package runs in parallel. In my experiment, it was better than Apache arrow, although, disk.frame package uses arrow package as the core of it implementation and running.

While analyzing food insecurity trends in NYC, I encountered challenges handling ~10GB of data (9.2M rows) efficiently. Our goal was to study food prices across neighborhoods and estimate how far people traveled to grocery stores. To overcome performance bottlenecks. Optimized Data Storage: Instead of CSVs, we leveraged GCP Cloud Storage with Apache Parquet, reducing read/write times and storage overhead. Processing large datasets sequentially wasn’t feasible. We used PySpark (RDDs & DataFrames) on Google Cloud Dataproc, distributing the workload across multiple nodes. By implementing these strategies, we processed ~10GB of data in just 1 minute 30 seconds, enabling scalable insights into food pricing disparities and accessibility.

Data Preprocessing Optimization: 1. Data Sampling: Instead of using the entire dataset for mining, you can sample smaller subsets of the data to speed up processing. Depending on the task, this can give you approximate results in a much shorter time. 2. Feature Engineering: Reduce the dimensionality of the data by selecting only the most relevant features. This minimizes the complexity of the data, helping the model to perform faster.

• Optimize Data Preprocessing: Clean and preprocess data to remove noise, outliers, and irrelevant features. Use dimensionality reduction techniques like PCA or autoencoders to reduce complexity. • Leverage Distributed Computing: Use frameworks like Apache Spark or Hadoop for parallel processing to handle large-scale data efficiently. • Efficient Query Optimization: Analyze query execution plans, avoid unnecessary joins, and use partitioning or indexing to improve database performance. • Resource Allocation: Ensure hardware and software resources (e.g., memory, storage) are sufficient and aligned with project needs. Automate repetitive tasks to free up resources. • Sampling and Caching