Streaming Insert Example

Length: 00:08:38

Lesson Summary:

We are going to revisit our streaming Dataflow/PubSub pipeline and take a closer look at the end result, which is a streaming insert into BigQuery. For easy setup, the below commands and scripts will quickly create your pipeline so we can focus on BigQuery, as well as cleanup when finished.

Quick setup:

cd

gsutil cp -r gs://gcp-course-exercise-scripts/data-engineer/* .

bash streaming-insert.sh

Clean up:

bash streaming-cleanup.sh

Manually stop the Dataflow job.


This lesson is only available to Linux Academy members.

Sign Up To View This Lesson
Or Log In

Looking For Team Training?

Learn More