Fivetran ELT Platform

The best connectors for your SaaS and on-premise data.

Before we can talk about key performance indicators, or brainstorm how to model the facts and dimensions so we can answer business questions, we need data.  A lot of it.  In modern enterprises, that data is all over the place: in multiple cloud SaaS offerings, on-premise databases, bespoke spreadsheets, and in files dropped off by data vendors.

We have to get that data into the platform.

Enter Fivetran.  With more than 190 opinionated systems connectors available out of the box, Fivetran lets you regularly pull new data into your Snowflake instances from almost any system you’re using today.  If there isn’t a pre-built connector, Vivanti can build you one via Fivetran Functions, and even run it for you if you don’t want the operational responsibilities.

Fivetran gets your data into your data platform.

What’s an ELT Platform?

No, we didn’t misspell E-T-L; we just rearranged the letters a bit.

In the data world, ETL stands for Extract – Transform – Load, and it refers to a disciplined approach to data integration across systems that involves three discrete actions:

  • Extract the data from the source (upstream) system.
  • Transform the data according to business requirements.
  • Load the transformed data into the destination (downstream) system.

The problem with ETL is that most of the T process takes place outside of view, outside of database systems.  Transformation is the most complicated and complex part of data integration. The fact that we don’t traditionally use SQL and relational concepts like tables and views to implement causes a massive amount of operational support overhead.

ELT shuffles the phases around so that data is extracted (mostly mechanically) from upstream systems, and then immediately loaded into the downstream data platform – i.e. Snowflake – where the transformation (carried out in SQL by something like dbt) takes place.

Fivetran handles the E and L parts of this equation.  Through its opinionated and consistent manner of normalizing upstream data into downstream landing tables, Fivetran eliminates a major source of headaches and wasted work.  Let Fivetran handle the landing, and you can focus on modeling and asking questions.

In Vivanti’s opinionated architecture for data warehousing, Fivetran is how we get the data in the first place.

Accelerate your Fivetran initiative

Vivanti’s Accelerator programs give you the templates to get going faster.

Sure, you could start from nothing and build up your connector configurations, sync scheduling, and on-premise deployments piece by piece. But wouldn’t you rather start with some templates propel you towards your data goals much faster?

With Vivant’s Accelerator Programs, you start with our cultivated set of Terraform automation recipes, HVR agent packaging scripts and containers, and more.  Don’t waste time reinventing the wheel; let Vivanti do the heavy lifting for you.

The 3 best things about Fivetran

We could have picked more, but we ran out of word count.

Fivetran is awesome at landing data.  Here are our three favorite features:

So. Many. Connectors

Fivetran integrates with so many different systems.  Looking to pull CRM data into your warehouse for marketing analytics?  Salesforce, Marketo, and Business Dynamic all have connectors.  Want to pull in advertising metrics?  Fivetran can talk to Google, Amazon, Apple Search, TikTok ads and more.

The best part about all these connectors is that you don’t have to write them yourself.  The talented folks over at Fivetran obsess over these connectors, keeping them patched and updated as upstream platforms change and evolve.

That’s not just one less thing for your team to worry about – it’s one less thing per data source.

HVR for on-premise data

Cloud is great, but if you still have sensitive, proprietary on-premise systems – with data hosted in more traditional relational databases like Oracle, IBM DB2, Microsoft SQL Server, or SAP HANA – you might be wondering what Fivetran can do for you..

With HVR, you load a small software agent on – or near – the database server installation, which reads transactional logs streaming off the database itself.  This agent performs some pretty impressive algorithmic change detection and compression, and then ships that data off to your Snowflake or other data platform.  It’s a bit more hands-on than the SaaS connectors Fivetran offers, but it is every bit as useful.

Fivetran Functions

If your data is not on-premise, and there’s no pre-made connector for it, you’re still not out of luck!  Fivetran offers a custom integration that lets us provide the system-specific bits pertaining to data extraction, while still relying on the Fivetran ELT platforms scheduling, error handling, and retry facilities.

With a Fivetran Function, we build a small serverless application for something like Amazon Lambda, Google Cloud Functions, or Microsoft Azure Functions.  You choose the cloud and the language to use, and we do the rest.  We can even run the function in production for you, and take care of things like monitoring and error handling.

Connect Fivetran to SaaS and on-premise data sources

With HVR – High Volume Replication – you can pull data from traditional databases, too.

On-premise data systems – the Oracles, SAP HANAs, and SQL Servers of the world – are some of the most difficult to integrate with centralized data platforms like Snowflake.  Schemas are usually proprietary and unique, access control is often a mess, and network security policies provide more than a few obstacles to be overcome.

Vivanti can help you navigate these complexities with Fivetran’s HVR offering; a high-volume replication agent that streams data from locally-accessible data systems to your connected data destinations.

With HVR, you’ll have a single multi-faceted pipeline of data coming in from all of your disparate systems; both cloud and on-premise.

Powered By Fivetran – data ingestion for app developers

Let your customers onboard their data into your platform, hassle-free.

So far, we’ve talked about Fivetran in the context of a single organization’s data.  Powered By Fivetran is a different take on data integration: coordinating data integration between an application (as the destination) and its users (the source).

If you’re developing an application that would benefit from a seamless data onboarding workflow, to let customers regularly synchronize data from their data sources, Powered by Fivetran is definitely worth a look.

Connect Cards – letting Fivetran handle the user experience

The simplest and quickest way to get started with Powered by Fivetran is to integrate Connect Cards into your application.  This web-based flow involves Fivetran’s infrastructure to handle the configuration and setup of inbound data sync sources, with appropriate redirect hooks for continuing flow through your application when appropriate.

Vivanti can help you integrate this workflow into new and existing web-based applications, whether you’re writing them in Vue (or Nuxt), Svelte, React (or Next.js), Angular, or something else entirely.

PBF Headless – completely own the user experience

For a more curated user experience, you can opt to integrate Powered by Fivetran in a headless mode, where Fivetran provides backend API access, allowing your application to drive the whole onboarding process.  This requires a bit more work on your part, since you have to provide the forms and interaction handling to get data system connectivity and configuration parameters from the end user, but it’s worth it for a fully-branded onboarding.

If you’re looking at headless mode, but aren’t sure you have the time or resources to devote to the integration effort, Vivanti can help with talented application software engineering consultants.

Custom API?  Functions to the rescue!

Don’t let a little thing like connector availability get in the way…

When there’s a pre-built connector in the Fivetran wheelhouse, life is great.  But what do you do when you are dealing with a third-party API that isn’t supported?  Or a private solution behind a VPN that isn’t even on the radar for Fivetran support?

Vivanti can help you implement what’s called a Fivetran Function; a small serverless bit of code that speaks Fivetran’s language and communicates with your upstream API to synchronize data.

Targeted Proof-of-Concept

Sometimes, you just need to show that the whole thing works as advertised.

The promises of the cloud data platform – especially enabled by the power of dbt – are compelling. But, it can be difficult to see just how you get from where you are today, to where you want to go. With a Vivanti targeted Proof-of-Concept, we present a time-boxed, scope- and cost-controlled engagement.The focus? Delivering an executive business use case for a larger, more comprehensive, and more impactful project.

 

If you’re unsure about how well Fivetran will unify your data onboarding story, get in touch and we can discuss a targeted PoC to help reduce that doubt.

We can tend to your Fivetran Functions

Don’t worry about your data ingestion integrations.

We get it; serverless functions aren’t everyone’s cup of tea.  If you’re not keen on maintaining and monitoring the infrastructure to run lambdas on AWS, GCP, or Azure, let us take over.  We have been in the cloud game for a long time and we love the FaaS offerings from all the major cloud providers.

We can take on the day-to-day operational duties of tending to your Fivetran custom functions.  Our managed service offering includes everything from deployment and configuration to monitoring and optimization.

We watch:

  • Function error streams
    The Internet is a chaotic place, and sometimes things don’t go according to plan.  We keep an eye on your sync functions, and when they throw errors, we get to the bottom of what’s going on.
  • Cost and consumption
    The nice thing about serverless architectures is that they cost you nothing if you don’t use them.  They also have a tendency to run up bills under heavy load.  We keep an eye on the budgets and forecast usage for you, so you don’t get surprised.
  • Performance metrics
    Are your functions running at their most efficient?  Is per-row processing time staying constant, or creeping up?  We watch memory, CPU, and throughput metrics to make sure you are getting the most out of your Fivetran custom sync.

We’ve always been cloud people

Automation is in our bones. We’ve automated everything from code review tests to entire infrastructure deployments and platform scale-outs – managing thousands of machines and hundreds of thousands of applications.

Your Fivetran functions are in expert hands, under watchful eyes.