Build your Agency Data Pipeline

Bring your digital agency into the 22nd century

Overview

Wrestle your agency's data analysis workflow completely to the ground, by building a BigQuery data pipeline to house + analyze all of your critical data.  Down with chaos!

At CIFL, we operate the Agency Data Pipeline service for a number of digital agencies - ranging from SEO to eCommerce, and in size from 5 clients to 50.  

This course completely opens the kimono on how we operate that service, and allows you to build + operate your own Agency Data Pipeline.

The course includes: 

  • A detailed walkthrough of how we built our own internal data pipeline at CIFL, pulling data from 8 APIs (Google Analytics, FB Ads, and more) to answer the question: which is the highest performing marketing channel for the CIFL blog?
  • Roadmapping exercises and instructions for building your own data pipeline, specific to the challenges of your work. 
  • A one-hour coaching session with David, commissioner of Coding is for Losers, to make sure your data pipeline tracking plan gets put into action.

Many thanks to Supermetrics and Stitch for providing demo accounts for use in building the course.

With love,
David
Commissioner, Coding is for Losers

What's included?

Video Icon 38 videos File Icon 5 files

Contents

Intro to the Course Pipeline Project
WTF is ADP?
Copy the Tracking Plan Template
Check out the Demo Pipeline Tracking Plan
Check out the Demo Pipeline SQL Models on Github
Check out the Demo Pipeline Data Studio dashboard
Start your Pipeline with a Question
Intro to question asking
2 mins
Demo questions for the CIFL pipeline
7 mins
Translating your Question into a Tracking Plan
Meet the Tracking Plan
5 mins
Mapping out your Data Sources
6 mins
Planning your Data Models
7 mins
Scoping out each Site or Client
5 mins
Firing up your Data Feeds
Intro to data feeds
2 mins
BigQuery initial setup
4 mins
Setting up your BigQuery tables
7 mins
Pushing data from Sheets to BigQuery
6 mins
Supermetrics quickstart for beginners
3 mins
Pulling data from unsupported APIs into the Tracking Plan
9 mins
Using Stitch for Automated Data Feeding
Stitch initial setup
3 mins
Pulling GA data using Stitch
7 mins
Pulling Adwords data using Stitch
4 mins
Pulling FB Ads data using Stitch
4 mins
Writing SQL queries to answer your questions
Intro to DBT for SQL data modeling
4 mins
Creating your DBT project
8 mins
Connect your BigQuery database to DBT
8 mins
Writing your 'processing' level SQL queries
9 mins
Writing your 'join' level SQL models
6 mins
Sidenote: on debugging DBT models
7 mins
Pro tip: standardizing URL structure
3 mins
Pro tip: using DBT macros
6 mins
Writing your 'admin' level SQL models
3 mins
Writing your 'math' and 'visualization' level SQL models
8 mins
Visualizing your analysis in Data Studio
Intro to visualization with Data Studio
3 mins
QCing your raw data in Data Studio
6 mins
Reaching the summit: answering your question
11 mins
Answering ancillary questions
4 mins
Pulling data from BigQuery into Sheets
4 mins
Putting your Pipeline into production
Intro to productionalizing your pipeline
2 mins
Pushing your SQL models to Github
3 mins
Using Sinter to run your SQL models on a schedule
5 mins
Scheduling your data pipeline
5 mins
Testing changes to your data pipeline
3 mins
QCing data using Supermetrics as a check
5 mins
Congrats!
Wapow! You made it.
2 mins

FAQs

Is the course only for digital agency owners?

Nope - if you work with sales + marketing data in any way (Google Analytics, FB Ads, SEMrush, a CRM, Shopify, etc) in your work, you can benefit from data pipelining.

If you don't analyze digital marketing + sales data in your day-to-day work, I'd recommend not taking this course.

How much do the tools cost to build my pipeline?

The tools required to build your pipeline are:

  1. Stitch (Free up to 5mm rows pushed to a database, $100 / month and up after that) or Supermetrics ($50 / month) for pulling data from APIs
  2. Google Sheets + Apps Script (Free)
  3. Google BigQuery (or similar database) for data warehousing (free 12month trial + $300 credit, cost varies by usage after that)
  4. DBT to model data using SQL (Free + open-source)
  5. Sinter to run DBT models on a schedule (Free for up to 6 runs a day, $100 / month for unlimited runs)

So the bottom line is - the cost of your pipeline depends on the size of your pipeline.  If you're pulling + storing a small amount of data, it may be completely free.  

If you're pulling a large amount of data, it will be more expensive.  The raw tool cost for most of our large Agency Data Pipeline implementations is around $500-$600 a month.

Making Spreadsheets Great Again

Helping you wrangle your data + automate your work, without (hardly ever) leaving Google Sheets.