Google Cloud just added Shopify and Mailchimp to their list of data collection services
deffo a case of slowly slowly catchy monkey
Google Cloud just added Shopify and Mailchimp to their list of data collection services, deffo a case of slowly slowly catchy monkey.
Just under seven years ago Nigel Vining and I onboarded our first customer, The Good Registry, to give our MVP AgileData.Cloud platform a run for its money, to test what survived and what didn’t.
One of the first requirements was to collect data from Shopify.
We had a principle back then (a principle we still have today) to use the Google Cloud Data Transfer Services as our first cab off the rank for data collection.
Alas at that time there was no Google Cloud data collection service for Shopify, so Nigel crafted a set of patterns using the Meltano open source framework to automate the collection of this data.
Part of this work was to refactor Meltano so we could operate it as a “serverless” pattern on Google Cloud, rather than it operating under a 24/7 always on container pattern, that would cost more money.
We have reused this data collection pattern many many times over the last 7 years, another great example of our DORO (Define Once, Rese Often) principle.
I saw last month an announcement that Google Cloud have added Shopify and Mailchimp to their ever expanding list of data collection services or what they call “Data Transfer Services”.
If you haven’t noticed Google have been quietly expanding out the list of systems of capture that this service can collect data from.
To me it looks like they are starting to ramp up in this space and will become a serious competitor to tools like Fivetran.
Time will tell.
One of the many things I have learnt about Google Cloud over the last 7 years, is they may not be first to market, but when they go after a market, they have the firepower to do it at scale.
Google Gemini showed us that yet again.
You can check out the doco for the new Data Transfer Services here:
https://docs.cloud.google.com/bigquery/docs/shopify-transfer
https://docs.cloud.google.com/bigquery/docs/mailchimp-transfer