Get to know Steampipe: A New Way to Talk to AWS!
Have you ever wished you could chat with your cloud resources just like you do with databases? Meet Steampipe, the cool open-source tool that lets you do just that using SQL! It’s a game-changer for the DevOps and cloud engineers bond with cloud infrastructure.
In this article, I’ll walk you through what Steampipe is all about, how to set it up, and why it’s a must-have for DevOps and cloud engineers.
So, What’s Steampipe?
Think of Steampipe as the bridge connecting traditional databases and the vast cloud universe. In simple terms, it turns those complex cloud APIs into friendly database tables.
With Steampipe in your corner, you can:
→ Fetch real-time data from cloud resources without diving deep into complex scripts or SDKs.
→ Get a bird’s-eye view of multiple cloud platforms.
→ Make those compliance and audit tasks a breeze with SQL-based checks.
Setting Up Steampipe:
First,download and install the Steampipe CLI is straightforward, If you are on MacOS run :
brew install turbot/tap/steampipeConfirm the installation by running:
steampipe --versionNow install the aws Plugin:
steampipe plugin install awsOne the plugin is installed it should create a config file ~/.steampipe/config/aws.spc open the file and configure your aws connection for steampipe:
connection "aws_morocco" {
plugin = "aws"
profile = "aws-morocco-admin"
default_region = "us-east-1"
}You can define multiple connection configurations to interact with various AWS environments. For example, you can set up connections like aws_morocco, aws_dev, and aws_prod, each with its own AWS profile and default region.
And, you can also create an aggregator connection, such as aws_all, to combine multiple AWS connections for more comprehensive querying.
connection "aws_morocco" {
plugin = "aws"
profile = "aws-morocco-admin"
default_region = "us-east-1"
}
connection "aws_dev" {
plugin = "aws"
profile = "aws-prod-admin"
default_region = "us-east-1"
}
connection "aws_prod" {
plugin = "aws"
profile = "aws-prod-admin"
default_region = "us-east-2"
}
connection "aws_all" {
plugin = "aws"
type = "aggregator"
connections = ["aws_dev", "aws_prod", "aws_demo"]
}Here, I am configuring the aws connection using aws profile, but you can use other methods like AWS SSO or profile with AssumeRole (with or without MFA) or even IAM Access Key Pair Credentials.
Query Away:
Now that various AWS connections in Steampipe are set up, the real fun begins: querying your cloud resources.
These connections essentially act as named schemas in a PostgreSQL database, allowing you to target specific environments or configurations. For instance, running the query select name, region, account_id, bucket_policy_is_public from aws_morocco.aws_s3_bucket; would retrieve S3 bucket information specifically from the aws_morocco connection.
One of the most powerful features of Steampipe is its ability to represent cloud resources as tables within these named connections. Take, for example, the aws_s3_bucket table, which allows you to query all the S3 buckets within a specific AWS environment like aws_morocco. As of the writing of this tutorial, Steampipe offers an impressive 454 tables for querying. These tables span a wide range of AWS services, from EC2 instances (aws_ec2_instance) to IAM roles (aws_iam_role), and much more.
This extensive list of tables equips you with the flexibility to query almost any aspect of your AWS infrastructure. The sheer number of available tables opens up endless possibilities for auditing, monitoring, and data analysis. Whether you're looking to enforce compliance standards or delve into your cloud architecture, Steampipe's expansive list of tables has you covered.
steampipe query
> select * from aws_morocco.aws_s3_bucket;
> select
name as user_name,
split_part(attachments, '/', 2) as attached_policies
from
aws_morocco.aws_iam_user
cross join jsonb_array_elements_text(attached_policy_arns) as attachments;
This feature enables you to perform granular queries across different AWS environments, making it incredibly powerful for tasks ranging from routine audits to complex data analysis.
The SQL way:
One more cool feature of Steampipe is its broad compatibility with various database management tools, making it an incredibly versatile asset for cloud resource querying.
For instance, DBeaver, a popular database management solution, integrates effortlessly with Steampipe. This integration allows you to tap into DBeaver’s intuitive interface and robust SQL editing capabilities while running your Steampipe queries.
The connection settings :
Host: localhost
Port: 9193
Database: steampipe
Username: steampipe
Password: Leave this empty
The under schema on aws_morocco > Foreign Tables :
The Steampipe Advantage: Why It’s a Game-Changer
- Unified Dashboard: Steampipe aggregates data from multiple cloud providers, offering a unified dashboard for cloud management.
- Granular Insights: Use SQL queries to unearth deep-seated insights into your infrastructure, facilitating better governance.
- CI/CD Integration: Seamlessly incorporate Steampipe into your CI/CD pipelines, enabling real-time checks during deployment phases.
- Automated Compliance: Create SQL-based checks to automate your compliance workflows, ensuring your infrastructure is always audit-ready. And we will make sure to cover this part in details in a coming tutorial 🤞
Conclusion
Steampipe is revolutionizing our interaction with cloud resources. By humanizing cloud configurations through SQL, it provides a unique blend of control and ease. Whether you’re a cloud manager, an optimizer, or simply cloud-curious, Steampipe is poised to enhance your cloud management experience.
For a deeper dive into the synergy between Steampipe and AWS:
Discover how Steampipe’s open-source graph capabilities enhance AWS resource navigation. Its intuitive visual representations promptly frame cloud contexts and spotlight crucial data about your assets:
