Sometimes, we get data in its natural, normalized form. More frequently, however, we get denormalized data, where some attribute or identity that exists in its own right has been embedded in related data. When modeling for enterprise concepts (nouns, not verbs) we...
When we talk about Fivetran, we normally focus on their SaaS connectors, like Zoho CRM and Microsoft Dynamics 365. But did you know that you can use the Fivetran FTP connector to pull data out of FTP drops and automatically sync the data into your data cloud? In this...
So far, we’ve spun up a lab environment, landed and de-duplicated our data, used Postgres’ JSONB capabilities to extract fields, and staged all of it into an emerging model with dbt. Our understanding of the current Scryfall data schema is taking shape, and we’re...
In the last essay, we got our lab environment spun up (thanks to the wonders of Docker Compose), set up our dbt project, and wrote our first model to de-duplicate landed JSON blobs. Today, we’re going to roll up our sleeves and dive right into the JSON objects we’re...
dbt is fantastic. It’s mostly SQL, with enough convention and some templating thrown in to cover the gaps for things like dynamic column families, open-ended data-driven joins, etc. To show off the sheer power and flexibility that dbt brings the data modeler, I want...
Who doesn't love a good interactive data session? Usually the people who weren't there to watch the ideas unfold. That includes future versions of our present selves, who often don't remember the particulars of decisions made in the moment. Why is this...
Recent Comments