Apache Spark: Reading Data - Parquet Files

Preparing for Databricks Certified Associate Developer for Apache Spark 2.4 with Python 3

https://parquet.apache.orgColumn-OrientedSee also

• https://parquet.apache.org
• https://en.wikipedia.org/wiki/Apache_Parquet

Technical Accomplishments:

• Read data from:

  • Parquet files without a schema.
  • Parquet files with a schema.

parquetFile = "/mnt/training/wikipedia/pageviews/pageviews_by_second.parquet/"
(spark.read              # The DataFrameReader
  .parquet(parquetFile)  # Creates a DataFrame from Parquet after reading in the file
  .printSchema()         # Print the DataFrame's schema
)

• We do not need to specify the schema - the column names and data types are stored in the parquet files.
• Only one job is required to read that schema from the parquet file’s metadata.
• Unlike the CSV or JSON readers that have to load the entire file and then infer the schema, the parquet reader can “read” the schema very quickly because it’s reading that schema from the metadata.

Read in the Parquet Files w/Schema

If you want to avoid the extra job entirely, we can, again, specify the schema even for parquet files:

WARNING Providing a schema may avoid this one-time hit to determine the DataFrame's schema.
However, if you specify the wrong schema it will conflict with the true schema and will result in an analysis exception at runtime.

# Required for StructField, StringType, IntegerType, etc.
from pyspark.sql.types import *
parquetSchema = StructType(
  [
    StructField("timestamp", StringType(), False),
    StructField("site", StringType(), False),
    StructField("requests", IntegerType(), False)
  ]
)
(spark.read               # The DataFrameReader
  .schema(parquetSchema)  # Use the specified schema
  .parquet(parquetFile)   # Creates a DataFrame from Parquet after reading in the file
  .printSchema()          # Print the DataFrame's schema
)