Skip to main content

Towerfile

A Towerfile is a text file that contains Tower instructions for how to package an application and how to run it in a Tower runner. It contains information such as the app name, the main script, files that need to be packaged together, the schedule of the app, and the app parameters. Each Towerfile starts with the [app] section declaration.

[app]

  • name : string, mandatory
  • script : string, mandatory
  • source : list of strings, optional
  • schedule : string, optional
  • import_paths: list of strings, optional
  • parameters : list of structs, optional
    • name : string, mandatory
    • description : string, optional
    • default : string, optional

Here is an example of a Towerfile with all sections

[app]
name = "hello-world"
script = "./task.py" # or "./task.sh"
source = [
    "./.dlt/config.toml",
    "./**/*.py",
    "./*.py",
    "requirements.txt",
    "./_data/*",
    "./task.sh",
    "./dlt_project.yml"
]
schedule = "every 5 minutes" # or "0 */5 * * * *"

[[parameters]]
name = "param1"
description = "First parameter"
default = "some value"

[[parameters]]
name = "param2"
description = "Second parameter"
default = "another value"

name​

  • Required: Yes
  • Description: App name. Can be used to refer to the app on command line and in source code.

script​

  • Required: Yes
  • Description: Main script of the application. Will be launched by the Tower runner when the app is run.
  • Values: Tower currently supports .py Python scripts and .sh shell scripts. Specify paths that terminate in .py or .sh extensions.

source​

  • Required: No
  • Description: A list of strings, representing globs of files you want to include in the app. If blank, then all files under your current directory will be packaged.
  • Values: Paths can be glob patterns, and they are always relative to the folder containing the Towerfile.
  • Important: If you choose to be explicit about what code you package, you must include the path to your script, otherwise the script won't be made part of the app package and the app run will fail.

import_paths​

  • Required: No
  • Description: A list of strings, which are relative paths to other Python code you want imported
  • Values: Relative paths to directories containing Python modules and shared code you want to import at runtime. These paths will be added to your PYTHONPATH such that you can import from them at runtime.

schedule​

  • Required: No
  • Description: A schedule at which to run the app regularly. Tower currently supports only one schedule per app.
  • Values: Can be in natural language (English) e.g. "every N minutes", "every Monday", or in cron syntax, e.g. "0 */5 * * * *"

parameters​

  • Required: No

  • Description: A list of sections, one section per parameter, all separated by [[parameters]]. In each [[parameters]] section there should be the following 3 fields

    • name: name of the parameter
    • description: short description of the parameter
    • default: default value to use when the parameter is omitted

  • Example:

    Assume an app with a Towerfile that defines two parameters: iceberg_table and AWS_REGION.


[[parameters]]
name = "iceberg_table"
description = "The name of the input iceberg table"
default = "invalid_table"

[[parameters]]
name = "AWS_REGION"
description = "The region of S3 endpoint"
default = "us-east-1"

  To run this app from the command line, add both parameters to the command:

tower run\
  --parameter=AWS_REGION='eu-central-1'\
  --parameter=iceberg_table='default.japan_trade_stats_2017_2020'

  • Important: When you add a schedule to an app with parameters, Tower will currently use the default values of the parameters to run the app.