Skip to main content

Towerfile

A Towerfile is a text file that contains Tower instructions for how to package an application and how to run it in a Tower runner. It contains information such as the app name, the main script, files that need to be packaged together, the schedule of the app, and the app parameters. Each Towerfile starts with the [app] section declaration.

[app]

  • name : string, mandatory
  • script : string, mandatory
  • source : list of strings, mandatory
  • schedule : string, optional
  • parameters : list of structs, optional
    • name : string, mandatory
    • description : string, optional
    • default : string, optional

Here is an example of a Towerfile with all sections

[app]
name = "hello-world"
script = "./task.py" # or "./task.sh"
source = [
    "./.dlt/config.toml",
    "./**/*.py",
    "./*.py",
    "requirements.txt",
    "./_data/*",
    "./task.sh",
    "./dlt_project.yml"
]
schedule = "every 5 minutes" # or "0 */5 * * * *"

[[parameters]]
name = "param1"
description = "First parameter"
default = "some value"

[[parameters]]
name = "param2"
description = "Second parameter"
default = "another value"

name​

  • Required: Yes
  • Description: App name. Can be used to refer to the app on command line and in source code.

script​

  • Required: Yes
  • Description: Main script of the application. Will be launched by the Tower runner when the app is run.
  • Values: Tower currently supports .py Python scripts and .sh shell scripts. Specify paths that terminate in .py or .sh extensions.

source​

  • Required: Yes
  • Description: A list of strings, representing paths to files.
  • Values: Paths can be glob patterns, and they are always relative to the folder containing the Towerfile.
  • Important: You have to explicitely include the path to your script, otherwise the script won't be made part of the app package and the app run will fail.

schedule​

  • Required: No
  • Description: A schedule at which to run the app regularly. Tower currently supports only one schedule per app.
  • Values: Can be in natural language (English) e.g. "every N minutes", "every Monday", or in cron syntax, e.g. "0 */5 * * * *"

parameters​

  • Required: No

  • Description: A list of sections, one section per parameter, all separated by [[parameters]]. In each [[parameters]] section there should be the following 3 fields

    • name: name of the parameter
    • description: short description of the parameter
    • default: default value to use when the parameter is omitted

  • Example:

    Assume an app with a Towerfile that defines two parameters: iceberg_table and AWS_REGION.


[[parameters]]
name = "iceberg_table"
description = "The name of the input iceberg table"
default = "invalid_table"

[[parameters]]
name = "AWS_REGION"
description = "The region of S3 endpoint"
default = "us-east-1"

  To run this app from the command line, add both parameters to the command:

tower run\
  --parameter=AWS_REGION='eu-central-1'\
  --parameter=iceberg_table='default.japan_trade_stats_2017_2020'

  • Important: When you add a schedule to an app with parameters, Tower will currently use the default values of the parameters to run the app.