|
| 1 | +Setting up the Apache Spark on Visual Code IDE |
| 2 | +---------------------------------------------- |
| 3 | + |
| 4 | +1. Create virtual environment (`.venv`) for the project. |
| 5 | +2. Install pyspark to the virtual environment using `pip install pyspark` |
| 6 | + |
| 7 | + |
| 8 | +Reading csv data using spark |
| 9 | +---------------------------- |
| 10 | +``` |
| 11 | +spark_df = spark.read |
| 12 | + .format("csv") |
| 13 | + .option("header", "true") |
| 14 | + .option("inferSchema, "true") |
| 15 | + .load("/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv") |
| 16 | +``` |
| 17 | + |
| 18 | +Reading csv data using spark csv method |
| 19 | +--------------------------------------- |
| 20 | +``` |
| 21 | +spark_df = spark.read |
| 22 | + .csv("/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv", |
| 23 | + header="true", |
| 24 | + inferSchema="true") |
| 25 | +``` |
| 26 | + |
| 27 | +Creating globalTemporaryView of the dataframe |
| 28 | +--------------------------------------------- |
| 29 | +``` |
| 30 | +spark_df.createGlobalTempView("fire_service_calls_view") |
| 31 | +``` |
| 32 | + |
| 33 | +With this view, we can run Sql queries such as |
| 34 | +``` |
| 35 | +select * from global_temp.fire_service_calls_view |
| 36 | +``` |
| 37 | + |
| 38 | +Creating database |
| 39 | +----------------- |
| 40 | +``` |
| 41 | +create database if not exists demo_db |
| 42 | +``` |
| 43 | + |
| 44 | +Create Table |
| 45 | +------------ |
| 46 | +``` |
| 47 | +create table if not exists demo_db.fire_service_calls_tbl( |
| 48 | + CallNumber integer, |
| 49 | + UnitID string, |
| 50 | + IncidentNumber integer, |
| 51 | + CallType: string |
| 52 | +) using parquet |
| 53 | +``` |
| 54 | + |
| 55 | +Insering data into table |
| 56 | +------------------------ |
| 57 | +``` |
| 58 | +insert into demo_db.fire_service_calls_tbl |
| 59 | +values(1234, null, null, null) |
| 60 | +``` |
| 61 | + |
| 62 | +Viewing data from table |
| 63 | +----------------------- |
| 64 | +``` |
| 65 | +select * from demo_db.fire_service_calls_tbl |
| 66 | +``` |
| 67 | + |
| 68 | + |
0 commit comments