How you can offer "data pipelines as a service" on Google BigQuery, following our end-to-end process
What You'll Learn
Down with chaos!
At CIFL, we've built data pipelines (mostly in Bigquery) for a wide array of businesses.
What we've found, is that coding is the easy part.
The hard part, as usual, is getting your team (or your client's teams) on the same page throughout the development process - so that what gets built actually gets used when it's done.
This course focuses on the processes that can ensure a good result - whether you're building a data pipeline internally for your company, or offering data pipelining as a service to your clients.
We'll dive into:
At CIFL, we've built data pipelines (mostly in Bigquery) for a wide array of businesses.
What we've found, is that coding is the easy part.
The hard part, as usual, is getting your team (or your client's teams) on the same page throughout the development process - so that what gets built actually gets used when it's done.
This course focuses on the processes that can ensure a good result - whether you're building a data pipeline internally for your company, or offering data pipelining as a service to your clients.
We'll dive into:
- How to decide if you even need a data pipeline at all
- How to roadmap a pipeline, to get buy-in + validation upfront
- How to break a pipeline roadmap into bite-sized development sprints to deliver small wins
- What roles are required to actually build a reporting pipeline, and how to hire for them
- How to validate data to build trust in the outcome
This course is a labor of love - these are the lessons learned over a few years deep in the weeds of building data pipelines for clients.
Hope you enjoy it as much as we enjoy sharing it.
Many thanks to Supermetrics and Stitch for providing demo accounts for use in building the course.
What's included?
57 videos
3 files
6 text files
Contents
Getting Started + FAQs
Additional (FREE) CIFL Courses
0.1 The Sales Flow
0.2 Staffing & Resourcing
1.1 Planning Development Sprints
2.1 Data Feeds - Getting Started
2.2 Data Feeds - Stitch
3.1 Intro to dbt
3.2 Data modeling with dbt
3.3 Productionalizing your dbt project
4.1 Visualizations in Data Studio
5.1 Sprint wrapup + review
Congrats!
FAQ
How much does the toolbelt cost to build a pipeline?
The tools required to build data pipelines are:
- Stitch ($100 / month for base plan) or Supermetrics ($89 / month) for pulling data from APIs
- Google Sheets + Apps Script (Free)
- Google BigQuery (or similar database) for data warehousing (free 12month trial + $300 credit, cost varies by usage after that)
- dbt to model data using SQL (Free + open-source)
- dbt cloud to run dbt models on a schedule (free for one seat, $50/seat after that)
So the bottom line is - the cost of your pipeline depends on the size of your pipeline. If you're pulling + storing a small amount of data, it may be completely free.
If you're pulling a large amount of data, it will be more expensive. The raw tool cost for most of our data pipeline clients is around $400-500 per month.