Exploring Data With dbt: Renormalizing Data

Exploring Data With dbt: Renormalizing Data

Sometimes, we get data in its natural, normalized form. More frequently, however, we get denormalized data, where some attribute or identity that exists in its own right has been embedded in related data. When modeling for enterprise concepts (nouns, not verbs) we...
Fivetran FTP: Picking up files from FTP sites

Fivetran FTP: Picking up files from FTP sites

When we talk about Fivetran, we normally focus on their SaaS connectors, like Zoho CRM and Microsoft Dynamics 365. But did you know that you can use the Fivetran FTP connector to pull data out of FTP drops and automatically sync the data into your data cloud? In this...
Exploring Data With dbt: Anticipating the Future

Exploring Data With dbt: Anticipating the Future

So far, we’ve spun up a lab environment, landed and de-duplicated our data, used Postgres’ JSONB capabilities to extract fields, and staged all of it into an emerging model with dbt. Our understanding of the current Scryfall data schema is taking shape, and we’re...
Exploring Data With dbt: Extracting JSON Fields

Exploring Data With dbt: Extracting JSON Fields

In the last essay, we got our lab environment spun up (thanks to the wonders of Docker Compose), set up our dbt project, and wrote our first model to de-duplicate landed JSON blobs. Today, we’re going to roll up our sleeves and dive right into the JSON objects we’re...
Exploring Data With dbt: Loading & Landing

Exploring Data With dbt: Loading & Landing

dbt is fantastic. It’s mostly SQL, with enough convention and some templating thrown in to cover the gaps for things like dynamic column families, open-ended data-driven joins, etc. To show off the sheer power and flexibility that dbt brings the data modeler, I want...
Running SQL in a Jupyter Notebook

Running SQL in a Jupyter Notebook

Who doesn't love a good interactive data session? Usually the people who weren't there to watch the ideas unfold. That includes future versions of our present selves, who often don't remember the particulars of decisions made in the moment. Why is this...