Orchestrate apps

Orchestration in Tower gives you control over when and how your apps execute, allowing you to automate and compose more complex flows.

Tower's orchestration capabilities help you:

• Run apps on demand
• Define a control flow and run several apps in parallel or in a sequence
• Schedule regular runs at specific intervals

Run Apps On Demand

Run Apps in CLI

Use the Tower CLI when you need to run an app on demand or as part of your own automation scripts:

tower run my-app

For additional options like passing parameters, see the tower run command documentation.

Run Apps in Tower UI

Navigate to the App Details page, click on the "Create Run" button on the top right of the screen and follow the prompts to launch a Run.

Run Apps via Tower API

For programmatic control or integration with external systems, use the Tower API:

curl -X POST https://api.tower.dev/v1/apps/<MY_APP>/runs \
  -H "Authorization: Bearer <MY_API_TOKEN>"

See the Run App API documentation for complete details.

Compose Flows

Tower apps can call other Tower apps and wait for their completion. The run and wait helper functions allow you to define flows of any complexity.

The control flows can be as simple as running two or three apps in a sequence. They can also be as complex as creating a list of app runs depending on the input parameters or the data your received, launching these apps in parallel execution, waiting for the completion of all or some of these apps, and then doing some post-processing at the end.

Run Apps

To run an app, use the run_app helper function and specify the app slug. You can also pass parameters to the app the same way you pass parameters via the CLI or API.

from tower import run_app

# Run an app with default environment
run = run_app("my-app")

# Run an app with custom environment and parameters
params = {"input_file": "data.csv", "output_dir": "results"}
run = run_app("my-app", parameters=params)

To run multiple apps in parallel, call run_app multiple times. You can even launch apps dependent on the input data, and you can do it in loops or conditionals. This allows you creating dynamic control flows. Tower example fan-out-ticker-runs shows how to:

tickers_str="AAPL,MSFT,GOOGL"

ticker_list = [ticker.strip() for ticker in tickers_str.split(",")]
    
# Process each ticker's data
child_runs = []
for ticker in ticker_list:
    params = {
        "PULL_DATE": f"{pull_date_str}",
        "TICKERS": f"{ticker}"
    }
    run = tower.run_app("write-ticker-data-to-iceberg", parameters=params)
    child_runs.append(run)

In the above example, multiple runs of the same app "write-ticker-data-to-iceberg" will be started, one per ticker. Each run will have a different value for the ticker parameter.

Wait for App Run Completions

In a typical control flow you will want to wait for the completion of a run or set of runs before proceeding to the next step. You might also want to deal with failure of one of the runs.

The wait_for_run() helper function will wait for the completion of the run that you pass as parameter. It will return a Run object that you can inspect for run status.

run = tower.run_app("my-app")
# Wait for a run to complete
final_run = tower.wait_for_run(run)

Its multi-run sibling wait_for_runs() waits for the completion of multiple runs that you started. It will return a tuple containing two lists on Run objects: successful_runs and failed_runs. You can inspect these lists to define what to do next, e.g. retry one of the apps, log the errors and terminate etc.

In our Tower example fan-out-ticker-runs, we use wait_for_runs and pass the child_runs list that we created by launching multiple apps. In return we get the runs that were successful and not.

    successful, unsuccessful = tower.wait_for_runs(child_runs)

    print(f"Successful ticker downloads: {len(successful)}")
    print(f"Unsuccessful ticker downloads: {len(unsuccessful)}")

If, instead of waiting for the completion of every run you want to break on any failure, use the raise_on_failure=True parameter.

# Wait for multiple runs with custom settings
try:
    successful, unsuccessful = tower.wait_for_runs(child_runs, raise_on_failure=True)
except RunFailedError as e:
    print(f"One or more runs failed: {e}")

Schedule App Runs

Schedules allow your apps to run automatically at predetermined times.

Adding a Schedule

To add a schedule, include the schedule setting in your app's Towerfile:

[app]
name = "data-processing-app"
# Other app settings...

schedule = "every 30 minutes"

Schedule Syntax Options

Tower supports two schedule syntax formats:

Natural Language Format: Some examples of acceptable schedules.

schedule = "every 15 minutes"
schedule = "every hour"
schedule = "every day at 2am"
schedule = "every Monday at 9:00"
schedule = "first day of every month at 00:00"

Cron Syntax: For more complex scheduling needs, use standard cron syntax:

schedule = "*/15 * * * *"  # Every 15 minutes
schedule = "0 * * * *"     # Top of every hour
schedule = "0 2 * * *"     # 2am daily
schedule = "0 9 * * 1"     # 9am every Monday
schedule = "0 0 1 * *"     # Midnight on first of every month

Schedule Limitations

• Only one schedule is supported per app
• Minimum interval is 1 minute
• Schedules are executed in UTC time zone

Managing Schedules

Updating a Schedule

To modify an existing schedule:

1. Edit the schedule value in your Towerfile
2. Redeploy your app using the Tower CLI:

tower deploy

3. Verify the updated schedule in the Tower UI

Removing a Schedule

To remove a schedule completely:

1. Delete the schedule line from your Towerfile
2. Redeploy your app

Viewing Scheduled Runs

Monitor your scheduled runs in the Tower UI:

1. Navigate to your app's details page
2. The next scheduled run shows you the next scheduled execution
3. The history of scheduled runs and their status are also found on this page

Orchestration Best Practices

For effective app orchestration in Tower:

Schedule Frequency

• Consider data volumes: Schedule frequency should align with expected data processing volumes
• Resource efficiency: Avoid scheduling too frequently if data doesn't change often
• Dependency awareness: Schedule dependent apps after their upstream data sources or dependencies get updated. Or use control flows to orchestrate multiple apps.

Monitoring

• Regularly check the Tower dashboard for failed scheduled runs
• Set up notifications for scheduled run failures (Coming soon)

Troubleshooting

If scheduled runs aren't executing as expected:

• Verify the app has been successfully deployed with the schedule
• Check for any error messages in the Tower logs
• Ensure your app has the necessary permissions and that all Secrets are up to date
• Confirm your schedule syntax is valid