the portable Python dataframe library

Implementing best practices for PySpark ETL jobs and applications.

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Jupyter magics and kernels for working with remote Spark clusters

Lightweight and extensible compatibility layer between dataframe libraries!

pyspark🍒🥭 is delicious，just eat it!😋😋

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

PySpark test helper methods with beautiful error messages

pyspark methods to enhance developer productivity 📣 👯 🎉

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Process Common Crawl data with Python and Spark

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

A boilerplate for writing PySpark Jobs

A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models

Code for "Efficient Data Processing in Spark" Course

Delta Lake helper methods in PySpark