Skip to main content

Environments

An environment is the context in which your app will be running. It includes the secrets that the app uses to connect to datasets, and the Iceberg catalogs it uses to access Iceberg tables. In Tower, you will always have at least one environment called default. As your data platform grows, you will potentially add environments for integration testing, for production etc.

The default environment​

The default environment is special, because its secrets and catalogs are inherited by all other environments. You can override the values specified in the default environment in other environments by creating a secret or catalog with the same name.

Secrets​

Database passwords, S3 bucket locations, and encryption keys are all examples of secrets that you might want to inject into your apps running either locally or in the Tower cloud at runtime.

Tower provides a secure secrets management system that ensures that your secrets are not only available where you need them, but they're available in a secure way, too. You can learn more about our standard security practices in the architecture section.

Secrets can be thought of as name-value pairs.

Secrets in the default environment​

When you create a secret with the tower secrets create command and don't specify an environment, the secret will be created in the default environment. The secret and its value will be available in all your environments, unless you override it.

tower secrets create --name=snowflake_url \
    --value=https://abc123.snowflake.com
✔ Creating secret... Done!
Success! Secret "snowflake_url" was created

Use the tower secrets list command to determine which secrets you have configured. If you don't specify the environment, it will show you only the secrets in the default environment.

tower secrets list
 Secret         Environment  Preview
----------------------------------------
 s3_bucket      default      XXXXXXcket
 snowflake_url  default      XXXXXX.com

Secrets in other environments​

Sometimes you will want to use different secret values for different environments that you run you apps in. For example, imagine you have an app that reads data from S3 and writes it to a Snowflake database. It's likely that you use a different Snowflake database in development, staging, and production environments. Therefore, you would want to have the same secret name (e.g. snowflake_url) in all your environments, but use different secret values depending on the environment.

To create a secret in an environment different from default, use the --environment parameter.

tower secrets create --name=snowflake_url \
    --value=https://abc123-prod.snowflake.com \
    --environment=production
✔ Creating secret... Done!
Success! Secret "snowflake_url" was created

When you run both of these commands, you will have a secret with key snowflake_url in both the default and the production environments, but with different values.

Use the --environment parameter with the tower secrets list command to show what secrets will be used in a given environment.

tower secrets list --environment=production
 Secret         Environment  Preview
----------------------------------------
 s3_bucket      default      XXXXXXcket
 snowflake_url  production   XXXXXX.com

Use the -a parameter to list all secrets in all environments.

tower secrets list -a
 Secret         Environment  Preview
----------------------------------------
 s3_bucket      default      XXXXXXcket
 snowflake_url  default      XXXXXX.com
 snowflake_url  production   XXXXXX.com

Managing secrets in the Tower UI​

You can use the Tower UI to create, list and delete secrets.

Navigate to (> Secrets ) and click on "Create New Secret". You can add a secret to the default environment or select "+New Environment" in the drop-down and type in e.g. production.

New secret

You can later delete secrets from the secrets list.

Catalogs​

Apache Iceberg catalogs simplify the use of Iceberg tables inside Tower apps.

Catalogs are defined per each Tower environment. As with secrets, catalogs defined in the default environment are inherited by all other environments.

Creating catalogs​

To create a catalog, use the Tower UI.

Catalog Settings

When creating a catalog in Tower, specify its slug, and the 4 properties that Tower (and the pyiceberg library that Tower uses internally) need to know to connect to the catalog service: URI, Credential, Scope, and Warehouse. You can get these values during the set-up of the catalog service.

Set the catalog slug to the catalog identifier that you are planning to use inside your Tower app, e.g. 'mycatalog'. Pyiceberg calls this identifier the catalog name.

If you set the slug to the string default, you won't need to explicitely specify the catalog identifier in your code.

Using catalogs​

To work with Iceberg tables inside your Tower apps you should use the Tower tables SDK. Tower tables use the pyiceberg library internally. You can use pyiceberg to access Iceberg tables directly, but the tables SDK will typically be simpler to use, as it automatically loads all catalogs defined in the environment the app is running in.

import pyarrow as pa
import tower
SCHEMA = pa.schema([...])
mytable = tower.tables('mytable', catalog='mycatalog').create_if_not_exists(SCHEMA)

In the above example, when checking for the existance of the table, Tower will look in its metadata for the catalog that has the slug 'mycatalog'.

When you set the slug of your catalog to 'default', the catalog identifier can be dropped and the above example can be simplified to:

import pyarrow as pa
import tower
SCHEMA = pa.schema([...])
mytable = tower.tables('mytable').create_if_not_exists(SCHEMA)

When using pyiceberg to access Iceberg tables, do something like this.

import pyarrow as pa
from pyiceberg.catalog import load_catalog
SCHEMA = pa.schema([...])
mycatalog = load_catalog('mycatalog')
mytable = mycatalog.create_table_if_not_exists('mytable', SCHEMA)

Supported catalog types​

Tower currently supports Apache Polaris and Snowflake Open Catalog. More catalog types will be supported over time.

Advanced use cases​

Programmatically determining the environment​

Sometimes users need to programmatically determine which environment the app is running in, e.g. to use different data source types (DuckDB vs Snowflake) depending on the environment. Tower allows you to retrieve the environment your code is running in through the special TOWER_ENVIRONMENT environment variable. Review this section to understand how to use this environment variable.