Today, Vantage announces support for Snowflake costs in the Vantage console. Vantage customers can now see their overall Snowflake costs as well as costs per query, costs per warehouse, costs per Snowflake user, and more. Customers can add any number of Snowflake integrations from their integrations page and Vantage will automatically ingest and visualize Snowflake costs accordingly.
Customers are increasingly using one or more cloud service providers alongside their primary cloud of AWS, GCP, or Azure. For some customers, Snowflake is core to their business and can be a significant driver to overall cloud costs as a result. Despite this, it was difficult for customers to understand which applications or users were driving their Snowflake costs. As organizations increase their Snowflake usage, it can become cumbersome to understand cost efficiency and which queries, users or warehouses were ultimately driving costs.
Now, customers can grant Vantage read-only access to their Snowflake usage views. Upon granting access, Vantage will ingest sanitized query history and accompanying billing history. This allows Vantage customers to see their Snowflake costs broken down by cost category (compute, storage, and cloud services), as well as see costs for each unique Snowflake query. Filter Sets in Vantage have been expanded to allow for Cost Reports to report on Snowflake costs with dimensions of a specific Snowflake username, warehouse, organization, region, or query tag.
“Snowflake always looks for best in class tools to help companies scale the Data Cloud efficiently. That’s why we’re excited to have Vantage as a partner for Snowflake customers to get visibility on cloud costs. Vantage provides a self-serve tool for data teams to see cost per query, cost per user, warehouse details, and much more. This visibility means users can centralize all their data in Snowflake, cover every analytics workload, and build any end-user data application while knowing their costs down to the individual query. With Snowflake and Vantage, companies can achieve huge productivity gains with zero maintenance and low costs,” said Tarik Dwiek, Head of Technology Alliances, Snowflake.
Snowflake support is available to all Vantage users, including users in the free tier. Note that Snowflake costs will contribute to monthly tracked costs tier enforcement quotas. In the event that Snowflake costs push you over your current tier monthly tracked costs, you may be required to upgrade your plan.
“Snowflake is core to our business helping operations teams monitor events and centralize alerting from their data warehouses. Before, we built custom tooling to keep track of Snowflake costs, and needed to write SQL to investigate our spend. Now Vantage tracks and surfaces all of this for us, and we can get notifications and reporting in a great UI without using precious engineering cycles,” said Jeff Barg, CTO, Avenue.
To begin viewing Snowflake costs in Vantage, head to the integrations page to provide Vantage with warehouse connection and authentication details. To read more about the technical and security details of this integration, please see the Vantage docs on Snowflake.
Frequently Asked Questions
1. What is being launched today?
Today, Vantage is announcing general availability support for Snowflake. Vantage users can now provide Vantage with a read-only Snowflake user and corresponding costs will automatically begin to be ingested and visualized within Vantage from the corresponding data warehouse. Vantage will refresh cost data from Snowflake on a daily basis to ensure data is always up-to-date.
2. Who is the customer?
This feature is available to all Vantage users, including users in the free tier. You must have a Snowflake warehouse deployed on Azure, GCP, or AWS.
3. What is Snowflake?
Snowflake delivers the Data Cloud—a global network where thousands of organizations mobilize data with near-unlimited scale, concurrency, and performance. Inside the Data Cloud, organizations unite their siloed data, easily discover and securely share governed data, and execute diverse analytic workloads. Wherever data or users live, Snowflake delivers a single and seamless experience across multiple public clouds. Snowflake’s platform is the engine that powers and provides access to the Data Cloud, creating a solution for data warehousing, data lakes, data engineering, data science, application development, and collaboration.
3. How much does this cost?
There is no additional cost to Snowflake support. However, Snowflake costs will be included in quota tier enforcement. In the event that your Snowflake costs push you over your current tier limit, you may be prompted to upgrade. To see more details on pricing, please refer to the pricing page here: https://www.vantage.sh/pricing
4. How does Vantage technically integrate with Snowflake?
Vantage requires a read-only user with access to the following views:
Returns hourly credit usage for both virtual warehouse credit usage and cloud service credit usage per warehouse for all warehouses in your account. Data retained for 1 year.
Query history with various dimensions, including total elapsed time, warehouse used, data bytes scanned, etc. Data retained for 1 year.
Returns the daily credit usage and usage in currency for an organization.
By reading from these views Vantage will ingest the cost of the warehouse as well as calculate the cost for a specific query. Unique queries will be identified by user, structure, and warehouse of the specific query.
5. Does Vantage have access to data stored in Snowflake?
No. Vantage will only have access to the views or tables that the user is granted. None of the permissions Vantage requires will give it access to data stored outside of the three metadata views mentioned above.
6. Does Vantage have access to values used in Snowflake queries?
Vantage sanitizes queries of all values on ingestion and does not store any values that are used in an SQL query. For more information on query sanitation and the security of this connection, see the Snowflake docs page.
For example:
SELECT 'value' as sensitive FROM users WHERE a = 2 AND b = 'very-sensitive'")
will be stored and viewable in Vantage as:
SELECT $1 as sensitive FROM users WHERE a = $2 AND b = $3
7. What dimensions can Snowflake costs be filtered by?
Snowflake costs can be filtered by Region, Organization, User, Warehouse, and Query Tag.
8. Can I see costs on a per-resource basis from Snowflake?
Snowflake queries are available as resources in Vantage. A Snowflake query is a unique permutation of the user that performed the query, the warehouse in which the query was performed, and the structure of the query excluding specific filter values. This allows Vantage users to easily identify the source of the query which is driving the cost. These queries are available when clicking “Costs by Resource” on a Cost Report.
9. Can I see Snowflake queries in my Active Resources?
Yes. Any Snowflake query that was executed in the previous 30 days will be available in the active resource list. When viewing a specific query you will be able to see a query’s user, structure, organization, warehouse, and the last 10 invocations of that query. The invocation will contain a unique query ID which can be viewed in your Snowflake QUERY_HISTORY view and a timestamp.
10. I am a Vantage customer who has already connected my AWS and/or GCP account, can I see my Snowflake costs alongside existing AWS and GCP costs?
Yes. After adding a Snowflake integration, Snowflake will be available as a provider under Cost Report filters. Snowflake costs will also automatically be included under the “All Resources” cost report.
11. What Cost Report groupings are available for Snowflake costs?
You can group Snowflake costs by organization, user, warehouse, and query tag. When viewing costs by category you can group the costs by category. When viewing Costs by Resource you can group the costs by query.
12. How often does Snowflake data refresh in the Vantage console?
Vantage will refresh at least once daily at 2 AM UTC for Snowflake.
13. What happens if I remove a Snowflake integration?
If you decide to remove your Snowflake integration from Vantage, all associated costs will be removed from the Vantage console associated with that Snowflake cluster.
14. Can I have multiple Snowflake warehouse integrations?
Yes. There is no limit to the number of Snowflake integrations you can add to Vantage.
15. Will Snowflake costs be represented on the Overview page?
Yes. Snowflake costs will be summarized in the “Provider Summary” section of the Overview page. Providers are sorted in descending order of accrued costs for the current month.
There will also be a top queries widget which will display the top 10 queries for the current period compared against their costs from the previous period with links to see the active resource page for that query.
16. How is cost per query calculated?
Snowflake charges for compute dependent upon the number of credits used. Credits are based on the size of the warehouse and the amount of time the warehouse is performing compute tasks. If three queries are running simultaneously and all take the amount of time to complete costs are only incurred for that time period once. Thus it can be difficult to understand whether a specific query is expensive or not because costs can be hidden by overlap with other queries.
Vantage looks at the exact run time of a query and uses a formula based on the total credits used for that day. From here Vantage applies a weighted cost to each query based on the overall daily compute cost.
Vantage will also normalize queries in order to group them together.
SELECT id FROM products WHERE type = ‘car’ AND color = ‘red’
will be grouped with SELECT id FROM products WHERE color = ‘red’ AND type = ‘car’
17. Is there any additional documentation on this integration?
Yes. Documentation on the Snowflake integration can be found here: https://docs.vantage.sh/connecting_snowflake
18. I’ve just added my Snowflake account, how long will it take for my cost data to be present in the Vantage console?
Costs will be ingested and processed immediately, but may take several hours to populate all Vantage tools depending on the query volume of your warehouse.
19. Do you support Virtual Private Snowflake?
Maybe. Please email support@vantage.sh if you are looking for Virtual Private Snowflake support.
20. Snowflake can be deployed on Azure, GCP, or AWS. Does Vantage support all clouds?
Yes. Vantage supports pulling data from a Snowflake cluster deployed in any of the cloud providers.
21. Will historical cost data for Snowflake be available?
Yes. By default, Vantage imports the current month-to-date costs as well as the full previous 3 months of cost data. Vantage has the ability to ingest data as far back as 12 months. In the event you’d like to see more than 3 months of data, please request it by emailing support@vantage.sh or contacting the support team via Intercom.
22. Is there any delay in importing Snowflake Cost Data?
Snowflake may take up to 72 hours to populate billing data in your warehouse.
23. What comprises Snowflake costs?
- Storage
- Compute used to execute queries
- Cloud Services - compute operations that Snowflake has to perform outside of query results.
24. What other cloud service providers is Vantage adding?
Vantage is planning on adding support for Databricks and MongoDB Atlas. Please contact support@vantage.sh with other providers you’d like to see added.
25. I have a request for a cloud service provider who is not on the current roadmap - can I get it supported?
Maybe. Please email support@vantage.sh as we are prioritizing the next set of cloud service providers.