Skip to content

Commit 18a83d5

Browse files
committed
Update repo
1 parent 33d16ad commit 18a83d5

File tree

2 files changed

+74
-0
lines changed

2 files changed

+74
-0
lines changed

.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,9 @@
99
*.crc
1010
*.parquet
1111
*/*/_SUCCESS
12+
13+
# Venv
14+
.venv/
15+
16+
# System
17+
.DS_Store

01-HelloSpark/README.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
Setting up the Apache Spark on Visual Code IDE
2+
----------------------------------------------
3+
4+
1. Create virtual environment (`.venv`) for the project.
5+
2. Install pyspark to the virtual environment using `pip install pyspark`
6+
7+
8+
Reading csv data using spark
9+
----------------------------
10+
```
11+
spark_df = spark.read
12+
.format("csv")
13+
.option("header", "true")
14+
.option("inferSchema, "true")
15+
.load("/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv")
16+
```
17+
18+
Reading csv data using spark csv method
19+
---------------------------------------
20+
```
21+
spark_df = spark.read
22+
.csv("/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv",
23+
header="true",
24+
inferSchema="true")
25+
```
26+
27+
Creating globalTemporaryView of the dataframe
28+
---------------------------------------------
29+
```
30+
spark_df.createGlobalTempView("fire_service_calls_view")
31+
```
32+
33+
With this view, we can run Sql queries such as
34+
```
35+
select * from global_temp.fire_service_calls_view
36+
```
37+
38+
Creating database
39+
-----------------
40+
```
41+
create database if not exists demo_db
42+
```
43+
44+
Create Table
45+
------------
46+
```
47+
create table if not exists demo_db.fire_service_calls_tbl(
48+
CallNumber integer,
49+
UnitID string,
50+
IncidentNumber integer,
51+
CallType: string
52+
) using parquet
53+
```
54+
55+
Insering data into table
56+
------------------------
57+
```
58+
insert into demo_db.fire_service_calls_tbl
59+
values(1234, null, null, null)
60+
```
61+
62+
Viewing data from table
63+
-----------------------
64+
```
65+
select * from demo_db.fire_service_calls_tbl
66+
```
67+
68+

0 commit comments

Comments
 (0)