AWS Glue and Lake Formation

By the end of this lesson you will be able to

• Explain what AWS Glue is (serverless ETL on Apache Spark), how a job is started (on demand, on a schedule, by a trigger or an event) and how it is billed in DPU-hours.
• Build the classic “convert CSV to Parquet for Athena” flow, and avoid the job that triggers itself.
• Describe the Glue Data Catalog and crawlers, and why Athena, Redshift Spectrum, EMR and Lake Formation all depend on the same table definitions.
• Use job bookmarks to stop reprocessing old data, and know when bookmarks do not save you.
• Place Glue Studio, Glue DataBrew, Glue Streaming ETL and Glue Data Quality, and recognise retired features (Elastic Views, governed tables).
• Explain what AWS Lake Formation adds on top of Glue: blueprints, one place for permissions, column-, row- and cell-level security, LF-Tags and cross-account sharing.
• Spot the two classic holes in a Lake Formation setup (direct S3 access and the IAM-only default) and answer the exam’s “centralised, fine-grained data lake security” questions.

#Why a data lake needs glue

Chapter 10 showed that S3 is the cheapest, most durable place to keep data, and lesson 17.1 showed that Athena can query files in S3 directly with SQL. In practice, the data that lands in S3 is rarely ready for that. Orders arrive from the shop as CSV, app events as JSON, a nightly export from the accounting database as a dump, IoT readings as thousands of tiny files. Nobody wrote down the column names, the formats differ, dates are written three ways, and every query has to scan everything.

Three jobs have to be done before analytics is cheap and safe:

1. Transform: clean the data and convert it into a query-friendly format (columnar Parquet, compressed, partitioned). That is ETL, and AWS’s serverless ETL service is AWS Glue.
2. Describe: record what each dataset is: where it lives, its columns and types, its partitions. That is the Glue Data Catalog, filled automatically by crawlers.
3. Govern: decide who may see which tables, columns and rows, in one place, for every query engine. That is AWS Lake Formation, built on top of Glue.

#AWS Glue: serverless ETL

AWS Glue is a fully managed, serverless ETL (extract, transform, load) service, useful to prepare and transform data for analytics. You never create a cluster: you give Glue a script and a capacity, Glue starts the workers, runs the job and stops them.

• Extract: read from S3, from databases over JDBC (RDS, Aurora, Redshift, on-premises databases), from DynamoDB, from streams (Kinesis Data Streams, Kafka, MSK) and many SaaS sources through connectors.
• Transform: a script in Python (PySpark) or Scala running on Apache Spark. It can rename and cast columns, drop bad rows, join datasets, mask PII, deduplicate and change the format, for example CSV to Parquet.
• Load: write the result to S3 (the data lake), to Redshift (the warehouse) or to another database, and optionally update the Data Catalog.

Besides Spark jobs (batch and streaming) Glue also runs small Python shell jobs for light tasks that do not need a cluster, and Ray jobs.

The slides show the basic picture: S3 and Amazon RDS on the left, Glue ETL in the middle doing extract and transform, Redshift as the data warehouse on the right.

#How a Glue job starts: not only by events

The notes call Glue a “serverless event-driven ETL service”. That needs a correction: a Glue job does not react to events by itself. A job runs when something starts it:

So Glue can be part of an event-driven design, but the event plumbing (S3 event notifications from lesson 10.7, Lambda, EventBridge) is yours to add.

#The classic flow: convert CSV to Parquet for Athena

This is the slide’s Glue pattern and one of the exam’s favourite pictures. Files land in an input bucket as CSV; an S3 event notification calls a Lambda function (EventBridge works as an alternative) that triggers the Glue ETL job; the job imports the CSV and writes Parquet to an output bucket; Athena analyses the Parquet.

Why bother? Athena charges by the data it scans. CSV is row-based and uncompressed: to sum one column, Athena reads every byte of every row. Parquet is columnar and compressed: Athena reads only the columns the query needs, and skips whole row groups using the min/max statistics in the file. Add partitioning by date and a query for one day reads one folder. Lesson 17.1 has the numbers; a 70% to 90% cut in scanned data is common.

#Workers, DPUs and the bill

Glue measures capacity in DPUs (data processing units). One DPU is 4 vCPUs and 16 GB of memory. You choose a worker type (G.1X is 1 DPU per worker, G.2X is 2 DPUs, and there are larger ones for memory-hungry jobs) and the number of workers; a Spark job needs at least 2 DPUs. Glue can also auto scale the number of workers during the run.

You pay per DPU-hour, billed per second with a 1-minute minimum for Spark jobs on Glue 2.0 and later. The list price in us-east-1 is US$0.44 per DPU-hour, and Flex execution (for jobs that are not urgent, run on spare capacity) is US$0.29. Glue 6.0 (August 2026) lowers the Spark price by about 30%, but only for jobs set to version 6.0.

A worked example: a nightly job on 10 G.1X workers (10 DPUs) runs for 15 minutes. That is 10 × 0.25 = 2.5 DPU-hours, so 2.5 × 0.44 = US$1.10 a night, about US$33 a month. There is no cluster sitting idle for the other 23¾ hours, which is the point of serverless ETL.

Crawlers are billed the same way (per DPU-hour, per second). The Data Catalog is free for the first million objects and the first million requests each month, then US$1 per 100,000 objects per month.

#The Glue Data Catalog: one catalog of datasets

The Glue Data Catalog is a central metadata repository: a catalogue of every dataset you have, not the data itself. For each table it stores:

• where the data is (an S3 path, or a database table reached over JDBC, or a DynamoDB table);
• the schema: column names and data types;
• the format and how to read it (CSV with a comma separator, JSON, Parquet, ORC, Avro…);
• the partitions (for example one per dt= folder) and table properties.

Tables are grouped into databases (just folders of metadata, nothing to do with RDS databases). There is one Data Catalog per account per Region, and it is compatible with the Apache Hive metastore, which is why so many engines can use it.

The slide’s picture: S3, RDS, DynamoDB and JDBC sources are scanned by a Glue crawler, which writes metadata into the Data Catalog; Glue jobs use those tables for ETL; and Athena, Redshift Spectrum and EMR use the same catalog for data discovery: they look up the table, then read the data where it lives. QuickSight queries through Athena, and Lake Formation stores its permissions against these same tables.

#Crawlers: let Glue work out the schema

A crawler is a process that connects to a data store, samples the files or tables, uses classifiers (built-in ones for CSV, JSON, Parquet, ORC, Avro, XML and more, or your own) to recognise the format, infers the columns and types, detects partitions from folder names like dt=2026-10-30/, and creates or updates tables in the Data Catalog.

• Run it on demand, on a schedule (for example hourly), or in S3 event mode, where it crawls only the folders that changed, using S3 event notifications delivered through SQS.
• A schema change policy says what to do when the data changes: update the table, add new columns only, or just log the change.
• You do not have to use a crawler: a Glue job can create and update catalog tables itself, and you can define tables by hand with DDL in Athena.

#Glue features to know at a high level

#Job bookmarks: do not process the same data twice

A daily job that reads s3://shop-raw-example/orders/ reads every file in the folder, yesterday’s included, unless it remembers where it stopped. A job bookmark is that memory: Glue saves state after each successful run (for S3, which files were already processed; for JDBC sources, the highest value of a key column) and the next run reads only new data.

• Enable it per job: --job-bookmark-option job-bookmark-enable (the other values are job-bookmark-disable and job-bookmark-pause).
• In the script, each source needs a transformation_ctx name, and the script must call job.commit() at the end; without those, nothing is saved.
• A failed run does not move the bookmark, so the next run retries the same files.
• You can reset a bookmark to reprocess everything after a bug fix, or rewind it to an earlier run.
• For JDBC sources the bookmark key must only ever increase (an auto-increment ID or a created-at time); updated rows are not seen again.

#Glue Studio, DataBrew, Streaming ETL and Data Quality

DataBrew is priced separately: US$1 per 30-minute interactive session and US$0.48 per node-hour for jobs.

#AWS Lake Formation: a data lake in days

A data lake is a central place to keep all your data for analytics, structured (tables) and unstructured (logs, images, documents), in its raw and its curated form, usually in S3. Building one by hand means many steps: create buckets, write ingestion jobs, crawl, convert, catalogue, deduplicate, set permissions in IAM, in S3 bucket policies and in the catalog. AWS Lake Formation is a fully managed service that automates these steps so a secure data lake is ready in days instead of months.

What it gives you:

• Register S3 locations as part of the lake; Lake Formation then manages access to them.
• Blueprints: ready-made ingestion workflows. A blueprint generates Glue crawlers, jobs and triggers that load data into the lake on demand or on a schedule:
  • Database snapshot: load or reload all tables from a JDBC source (RDS, Aurora, an on-premises SQL Server or Oracle…).
  • Incremental database: load only new rows, tracked by bookmark columns.
  • Log file: bulk load CloudTrail, Elastic Load Balancing and ALB logs.
  • The slides add S3 and NoSQL sources: data already in S3 is simply registered and crawled, and other sources are loaded with Glue jobs and connectors.
• Discover, cleanse, transform, catalogue and deduplicate, using Glue underneath (crawlers, ETL jobs, the Data Catalog, FindMatches).
• One place for permissions, down to column, row and cell, enforced for Athena, Redshift Spectrum, EMR, Glue and QuickSight (through Athena). This is the part the exam asks about most.

Lake Formation is built on top of AWS Glue: its tables are Glue Data Catalog tables, its workflows are Glue workflows. The permissions themselves are free; you pay for Glue, S3 and the query engines.

Most companies use both: the lake keeps everything, and the warehouse holds the curated slice that dashboards hit all day. Redshift Spectrum (lesson 17.2) joins the two.

#Centralised permissions

Without Lake Formation, giving an analyst access to one table in the lake means touching several places, each with its own language:

• an IAM policy for Athena, Glue catalog actions and s3:GetObject;
• the S3 bucket policy (and KMS key policy if the data is encrypted with a customer key);
• a Glue Data Catalog resource policy for cross-account use;
• and every engine separately: EMR, Redshift Spectrum, SageMaker.

Worse, S3 permissions work on objects, not columns or rows. If the customers Parquet file holds a national ID column, anyone who can read the object can read every column and every row of it. To hide a column, teams copied the data into a second, trimmed dataset, and then had two copies to keep in sync.

Lake Formation replaces this with one permission model, managed in one place:

1. You register the S3 location with Lake Formation, using a role that Lake Formation itself uses to read it.
2. You grant Lake Formation permissions (SELECT, INSERT, DESCRIBE, ALTER…) on databases, tables and columns to IAM users and roles, Identity Center users and groups, SAML users, or other AWS accounts.
3. For rows and cells you create data filters: a row filter expression (branch = 'Dhaka') plus a column list; together they give cell-level security.
4. When a user queries through an integrated engine (Athena, Redshift Spectrum, EMR, Glue, QuickSight through Athena), the engine asks Lake Formation what the user may see. Lake Formation checks the grants and hands the engine temporary credentials for the S3 data (credential vending). The engine returns only the permitted columns and rows.

The slide’s centralised-permissions example shows exactly this: data ingested from S3, RDS and Aurora into the lake, Lake Formation applying column-level security, and users of Athena and QuickSight seeing only what they were granted.

Try the same table yourself. Pick a role and an access path, then run the query:

#LF-Tags: permissions that scale

Granting table by table does not scale to thousands of tables. LF-Tags (Lake Formation tag-based access control) let you attach tags such as domain = sales or sensitivity = pii to databases, tables and columns, and grant permissions on tag expressions: “the marketing role may SELECT everything tagged domain = sales and sensitivity = public”. A new table that gets the same tags is covered automatically; nobody has to add a grant.

#Sharing across accounts

Many companies keep the lake in one producer account and let other accounts (the finance account, a subsidiary) query it. Lake Formation can grant a table, or an LF-Tag expression, to another AWS account or an organisation unit. The sharing happens through AWS Resource Access Manager; the consumer account’s data lake admin then re-grants to its own users, who query with their own Athena or Redshift. The data never leaves the producer’s bucket and is never copied.

#What goes wrong

#Glue and Lake Formation in system design

• Transform where it is cheapest. A one-off conversion: Athena CREATE TABLE AS SELECT. A repeatable, scheduled pipeline: a Glue job (serverless, pay per second). Huge, long-running or custom big-data frameworks (Hive, Presto or Flink on your own cluster, Spot pricing for hours of work): EMR (lesson 17.4). Simple record-by-record transforms inside a stream: a Firehose Lambda transform (lesson 13.6).
• Moving a database, not analysing it? That is DMS (chapter 23). Glue is for analytics pipelines; DMS keeps a target database in sync.
• Analysts who do not code prepare data in DataBrew; engineers write Glue jobs or build them visually in Glue Studio.
• One catalog for every engine. Point Athena, Redshift Spectrum and EMR at the same Glue Data Catalog, so a table defined once is queryable everywhere.
• Raw, curated, reporting zones. Keep raw files untouched in one prefix or bucket, Glue writes cleaned Parquet to a curated zone, aggregates go to a reporting zone. A bug in a job can always be fixed by reprocessing raw data.
• Govern at the lake, not in every app. Lake Formation grants plus data filters replace trimmed copies of tables; QuickSight users then inherit those rules through Athena (lesson 17.5).
• Lesson 17.8 puts these pieces into one serverless big-data pipeline.

#Try it yourself

These commands use the AWS CLI with an account that is its own data lake admin. Bucket names end in -example; replace them with yours.

1. 1. Create a catalog database and a crawler

   aws glue create-database --database-input '{"Name":"shop_raw"}'
   
   aws glue create-crawler \
     --name orders-raw-crawler \
     --role arn:aws:iam::111122223333:role/GlueCrawlerRole \
     --database-name shop_raw \
     --targets '{"S3Targets":[{"Path":"s3://shop-raw-example/orders/"}]}' \
     --schedule "cron(15 * * * ? *)" \
     --schema-change-policy UpdateBehavior=UPDATE_IN_DATABASE,DeleteBehavior=LOG
   
   aws glue start-crawler --name orders-raw-crawler
   

2. 2. Look at what the crawler found

   aws glue get-tables --database-name shop_raw \
     --query 'TableList[].{name:Name,cols:StorageDescriptor.Columns[].Name,parts:PartitionKeys[].Name}'
   # [
   #   {
   #     "name": "orders",
   #     "cols": ["order_id", "customer_id", "district", "amount", "created_at"],
   #     "parts": ["dt"]
   #   }
   # ]
   

3. 3. Create a CSV-to-Parquet job with bookmarks on

   aws glue create-job \
     --name orders-to-parquet \
     --role arn:aws:iam::111122223333:role/GlueJobRole \
     --command Name=glueetl,ScriptLocation=s3://shop-scripts-example/orders_to_parquet.py,PythonVersion=3 \
     --glue-version 5.0 --worker-type G.1X --number-of-workers 10 \
     --default-arguments '{"--job-bookmark-option":"job-bookmark-enable","--enable-auto-scaling":"true"}'
   

   The heart of the script (PySpark): read through the catalog with a transformation_ctx, write partitioned Parquet, commit the bookmark.

   orders = glueContext.create_dynamic_frame.from_catalog(
       database="shop_raw", table_name="orders", transformation_ctx="orders_src")
   clean = orders.filter(lambda r: r["amount"] is not None and r["amount"] >= 0)
   glueContext.write_dynamic_frame.from_options(
       frame=clean, connection_type="s3", format="parquet",
       connection_options={"path": "s3://shop-curated-example/orders/", "partitionKeys": ["dt"]},
       transformation_ctx="orders_out")
   job.commit()   # without this, the bookmark is not saved
   

4. 4. Run it and watch the run

   aws glue start-job-run --job-name orders-to-parquet
   # { "JobRunId": "jr_3f1c0e9a5b..." }
   
   aws glue get-job-run --job-name orders-to-parquet --run-id jr_3f1c0e9a5b... \
     --query 'JobRun.{state:JobRunState,secs:ExecutionTime,dpuSecs:DPUSeconds}'
   # { "state": "SUCCEEDED", "secs": 412, "dpuSecs": 3610.0 }
   

   Run it again straight away: with the bookmark on, the second run finds no new files and finishes in under a minute. To reprocess everything after a bug fix: aws glue reset-job-bookmark --job-name orders-to-parquet.

5. 5. Put the lake under Lake Formation

   # Register the curated bucket; Lake Formation reads it with its service-linked role
   aws lakeformation register-resource \
     --resource-arn arn:aws:s3:::shop-curated-example --use-service-linked-role
   
   # Stop the IAM-only default for new databases and tables
   aws lakeformation put-data-lake-settings --data-lake-settings '{
     "DataLakeAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::111122223333:role/DataLakeAdmin"}],
     "CreateDatabaseDefaultPermissions":[], "CreateTableDefaultPermissions":[] }'
   

6. 6. Grant columns, then rows

   # Marketing: only branch and amount
   aws lakeformation grant-permissions \
     --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/Marketing \
     --permissions SELECT \
     --resource '{"TableWithColumns":{"DatabaseName":"bank","Name":"branch_txn","ColumnNames":["branch","amount"]}}'
   
   # Dhaka manager: Dhaka rows, every column except nid
   aws lakeformation create-data-cells-filter --table-data '{
     "TableCatalogId":"111122223333","DatabaseName":"bank","TableName":"branch_txn",
     "Name":"dhaka_no_nid","RowFilter":{"FilterExpression":"branch = '\''Dhaka'\''"},
     "ColumnWildcard":{"ExcludedColumnNames":["nid"]} }'
   
   aws lakeformation grant-permissions \
     --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/DhakaManager \
     --permissions SELECT \
     --resource '{"DataCellsFilter":{"TableCatalogId":"111122223333","DatabaseName":"bank","TableName":"branch_txn","Name":"dhaka_no_nid"}}'
   

   The Dhaka manager now runs SELECT * FROM bank.branch_txn in Athena and gets only Dhaka rows, without the nid column.

7. 7. Tag-based access, then clean up

   aws lakeformation create-lf-tag --tag-key sensitivity --tag-values public pii
   aws lakeformation add-lf-tags-to-resource \
     --resource '{"Table":{"DatabaseName":"bank","Name":"branch_txn"}}' \
     --lf-tags TagKey=sensitivity,TagValues=public
   
   # Remove the demo
   aws glue delete-job --job-name orders-to-parquet
   aws glue delete-crawler --name orders-raw-crawler
   aws lakeformation deregister-resource --resource-arn arn:aws:s3:::shop-curated-example
   

#Summary

Key takeaways

• Glue is serverless, managed ETL on Apache Spark: extract from S3, JDBC databases, DynamoDB and streams, transform with PySpark or Scala, load to S3 or Redshift. Billed per DPU-hour (4 vCPU, 16 GB), per second, 1-minute minimum.
• Glue is not event-driven by itself: jobs run on demand, on a schedule, after another job, or when Lambda or EventBridge starts them from an S3 event.
• The classic pattern: CSV lands in S3 → event → Lambda or EventBridge → Glue job → Parquet in another bucket → Athena scans far less. Never write the output where it triggers the job again.
• The Glue Data Catalog is the central metadata repository (location, schema, format, partitions) used by Athena, Redshift Spectrum, EMR, Glue and Lake Formation. Crawlers infer schemas and partitions; a new partition is invisible until it is added to the catalog.
• Job bookmarks stop reprocessing; they need transformation_ctx and job.commit(), and JDBC bookmarks only see new rows. Studio is the visual editor, DataBrew the no-code cleaner, Streaming ETL reads Kinesis, Kafka and MSK, Data Quality checks rules. Elastic Views and governed tables are retired.
• Lake Formation sets up a data lake in days on top of Glue: register S3 locations, blueprints (database snapshot, incremental database, log file), deduplication, and centralised, fine-grained permissions: columns, rows (data filters) and cells, LF-Tags at scale, cross-account sharing without copies.
• Its rules hold only for integrated engines: remove direct S3 access and the IAMAllowedPrincipals default, or the fine-grained grants are bypassed.

Self-check: AWS Glue and Lake Formation31 questions