Skip to main content

7 posts tagged with "cloud"

View All Tags

· 5 min read
Lukas Lösche

Cloud2SQL is a tool based on Resoto's collector plugins that allows you to collect data from various cloud infrastructure sources and export it to a database (like Snowflake, PostgreSQL, MariaDB, or MySQL) or write it as Parquet, SQLite, or CSV files for ingestion in your data lake.

In this post, I will guide you through the process of installing Cloud2SQL and demonstrate how to export data from AWS to a local SQLite database file.

Whether you are looking to integrate cloud data into your existing SQL workflows or simply want an easy way to access and analyze data from multiple cloud sources, Cloud2SQL is an excellent tool to consider.

· 7 min read
Matthias Veit

"A generation which ignores history has no past—and no future."
— Robert A. Heinlein

While Heinlein's words refer to human history, they also apply to cloud infrastructure. Most of the time, we care about the current state of resources; but sometimes, we want to know the origin of a resource, when a resource was deleted, or when/how a resource was updated.

Such knowledge is necessary in situations where you need to understand the timeline to investigate a specific system behaviour:

  • To perform the post-mortem analysis of an outage, we need to know which cloud resources changed and how they changed to yield the behaviour that we saw in our application. Without the ability to review a change log this becomes impossible.
  • To understand cost spikes in your cloud billing dashboard, you need to understand what resources were created, when they were created, and by whom they were created. Not only do you need a list of changes, but also the ability to filter, group, sort, and aggregate the data to see the big picture.
  • To check for security issues or compliance violations, you may need to reduce the scope to verify only those resources that were created or updated since the previous scan. Even complex checks can be performed on large infrastructures if they are only run against changed resources.

History is a log of events defining your infrastructure. This event log is important, as it will enable you to answer future questions about the state of your infrastructure retrospectively, including tomorrow's questions that have not yet crossed your mind.

· 6 min read
Anja Freihube

Cloud tagging strategies and policies are hailed as one of the most efficient ways to keep your cloud infrastructure controllable. But are they really?

Generally, the idea is that every piece of cloud service gets tagged (or labeled, in case of GCP) by the developers or maintainers who work with it. This could be accomplished with infrastructure-as-code (IaC) tools (such as Terraform), with a command-line interface (CLI), or in the cloud UI.

Tagging policies could require that each resource needs tags identifying the owner, cost center, product, project, and/or any other metadata. By being diligent about tagging, resources can be managed via their tags and nothing gets overlooked.

In theory, this is the correct way to manage resources; in practice, however, this hardly ever works as intended. Each tag created is a tag that requires maintenance. Tagging policies may change over time and people can make mistakes (in AWS, for example, tag keys are case sensitive). And, to properly use tagging on a greenfield cloud account is one thing; to retroactively apply tags to sprawling cloud infrastructure is quite another (especially when utilizing a multi-cloud strategy, where you'd need to repeat any operation over multiple interfaces).

· 7 min read
Anja Freihube

"It [is] the best of times, it [is] the worst of times." Software engineers working with AWS have any cloud service imaginable at their fingertips and developer velocity could hardly be higher. But, even the most shiny of coins has two sides.

While developers can spin up compute instances, databases, and less tangible things like Lambda functions or virtual identities as they wish—at some point, someone will ask, "What is all of this?" And as they hack away in the CLI trying to get an overview of the resources in all of their AWS accounts, they will inevitably get frustrated. While Amazon has been a pioneer in cloud computing and offers the largest array of services, there are some things that aren't so ideal. Namely, API consistency.

In this post, I describe a few of the challenges and quirks with the AWS API and why we're building Resoto. (Spoiler alert: It is so that you don't have to!)

· 10 min read
Matthias Veit

Today's world of cloud computing is complex. There are many cloud providers, each with their own set of services. Getting insights out of your infrastructure requires specialized understanding of the data from each service.

Properties in different services may have different names but the same meaning, or vice versa. To interpret properties, we need to ensure that values have a defined unit of measurement and one base unit. You can see the challenge if you imagine the many ways you can specify the size of a volume, the number of CPU cores, or even timestamps.

Banner

· 12 min read
Matthias Veit

Kubernetes has dramatically improved the way we manage our workloads. It has become the de-facto standard for deploying and managing containerized applications, and is available in all major cloud providers.

A typical setup consists of distinct Kubernetes clusters for each application stage (e.g., dev, test, prod) or a cluster per tenant, and Kubernetes clusters shared between different users and teams often utilize namespaces and roles to control access. Deploying a single application to a Kubernetes cluster usually consists of tens to hundreds of resources (e.g., deployments, services, ConfigMaps, secrets, ingresses, etc.).

Even a relatively simple setup quickly becomes tedious to manage as the resource count grows. It is difficult for a human to keep track of resources, especially with user access limited to certain clusters in select namespaces.