.. _environment_variables: Environment Variables ===================== LumiBot supports configuring many behaviors via environment variables. This page documents the variables most commonly used for **backtesting**, **ThetaData**, and **remote caching**. .. important:: **Never commit secrets** (API keys, passwords, AWS secret keys) into any repo or docs. Document variable names and semantics only. Backtesting configuration ------------------------- LUMIBOT_DISABLE_DOTENV ^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Disable recursive ``.env`` discovery (directory scanning) at startup. - Values: truthy enables (``1``, ``true``, ``yes``); unset/``0`` disables. - Default: disabled. - Notes: - Recursive ``.env`` scanning can add startup latency and can accidentally load the wrong ``.env`` when running in a directory with nested repos. - In production/BotManager backtests we rely on injected environment variables, so ``.env`` discovery should be off. IS_BACKTESTING ^^^^^^^^^^^^^^ - Purpose: Signals backtesting mode for certain code paths. - Values: ``True`` / ``False`` (string). BACKTESTING_START / BACKTESTING_END ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Default date range used when dates are not passed in code. - Format: ``YYYY-MM-DD`` BACKTESTING_DATA_SOURCE ^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Select the backtesting datasource **even if your code passes a `datasource_class`**. - Values (case-insensitive): - ``thetadata``, ``yahoo``, ``polygon``, ``alpaca``, ``ccxt``, ``databento`` - ``none`` to disable the env override and rely on code. Testing / CI guardrails ----------------------- LUMIBOT_ACCEPTANCE_TRIPWIRE ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: **Acceptance backtests only** — when truthy, a Python startup hook aborts the subprocess the moment it attempts to call the remote Data Downloader. - Values: truthy enables (``1``, ``true``, ``yes``); unset/``0`` disables. - Notes: - This is an engineering/CI guardrail to enforce “warm-cache” acceptance backtests. It should not be used for normal production backtests. - When triggered, it prints a marker and exits the subprocess with a non-zero code so the test fails reliably. Backtest artifacts + UX flags ----------------------------- SHOW_PLOT / SHOW_INDICATORS / SHOW_TEARSHEET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Enable/disable artifact generation. - Values: ``True`` / ``False`` (string). BACKTESTING_QUIET_LOGS ^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Reduce log noise during backtests. - Values: ``true`` / ``false`` (string). BACKTESTING_SHOW_PROGRESS_BAR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Enable progress bar updates. - Values: ``true`` / ``false`` (string). Backtest progress file (BotSpot/BotManager UI) ---------------------------------------------- LOG_BACKTEST_PROGRESS_TO_FILE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: When truthy, write ``logs/progress.csv`` during backtests so BotManager/BotSpot can show live progress. - Values: truthy enables (``1``, ``true``, ``yes``); unset/``0`` disables. - Notes: - On startup, LumiBot writes an initial ``progress.csv`` row immediately to reduce “time-to-first-progress” latency for short backtests. - In BotManager, a background thread watches ``/app/logs/*progress.csv`` and uploads the most recent row to DynamoDB. BACKTESTING_PROGRESS_HEARTBEAT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Enable periodic ``progress.csv`` updates while a ThetaData download is active (prevents the UI appearing stuck when simulation datetime is not advancing). - Values: ``true`` / ``false`` (string). - Default: enabled (``true``). BACKTESTING_PROGRESS_HEARTBEAT_SECONDS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Heartbeat interval (seconds) for writing ``progress.csv`` while downloading. - Values: float seconds (string). - Default: ``2.0`` Trade audit telemetry (accuracy investigations) ----------------------------------------------- LUMIBOT_BACKTEST_AUDIT ^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Emit **per-fill audit telemetry** into the trade-event CSV as ``audit.*`` columns. - Values: ``1`` enables (any truthy value); unset/``0`` disables. - Output: - Writes a full trade-event export ``*_trade_events.csv`` with ``audit.*`` columns (for example, quote bid/ask snapshots, bar OHLC, SMART_LIMIT inputs, and multileg linkage). - Notes: - This increases CSV width and can add overhead; keep it enabled only when you need a full audit trail. Profiling (performance + parity investigations) ------------------------------------------------ BACKTESTING_PROFILE ^^^^^^^^^^^^^^^^^^^ - Purpose: Enable profiling during backtests to attribute runtime (S3 IO vs compute vs artifacts). - Values: - ``yappi`` (supported) - Output: - Produces a ``*_profile_yappi.csv`` artifact alongside other backtest artifacts. ThetaData via remote downloader ------------------------------- DATADOWNLOADER_BASE_URL ^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Points LumiBot at a remote downloader service that can fetch ThetaData on demand. - Example: ``http://data-downloader.lumiwealth.com:8080`` DATADOWNLOADER_API_KEY / DATADOWNLOADER_API_KEY_HEADER ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Authentication for the downloader service. - Values: provided by your runtime environment (**do not hardcode**). DATADOWNLOADER_SKIP_LOCAL_START ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Prevent any local downloader/ThetaTerminal bootstrap logic from running (production backtests must use the remote downloader). - Values: ``1`` / ``true`` enable; unset/``0`` disable. ThetaData option-chain building (performance) --------------------------------------------- THETADATA_CHAIN_DEFAULT_MAX_DAYS_OUT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Bounds the default option-chain expiration window for equity underlyings to reduce strike-list fanout in cold caches/backtests. - Values: integer days. - Default: ``730`` (2 years). - Notes: set to ``0`` to disable the default bound (fetch all expirations). THETADATA_CHAIN_DEFAULT_MAX_DAYS_OUT_INDEX ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Same as ``THETADATA_CHAIN_DEFAULT_MAX_DAYS_OUT``, but for index-like underlyings (SPX/NDX/VIX/etc) with dense expiration schedules. - Values: integer days. - Default: ``180``. - Notes: set to ``0`` to disable the default bound. THETADATA_CHAIN_RECENT_FILE_TOLERANCE_DAYS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Local chain cache file reuse window (equities) when no chain hints are in effect. - Values: integer days. - Default: ``7``. THETADATA_CHAIN_STRIKES_TIMEOUT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Downloader wait timeout per strike-list request when building chains. - Values: seconds (float). - Default: ``300``. THETADATA_CHAIN_STRIKES_BATCH_SIZE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Number of in-flight strike-list requests when building chains. - Values: integer. - Default: ``0`` (use queue client concurrency). ThetaData corporate action normalization (accuracy) ------------------------------------------------------------ THETADATA_APPLY_CORPORATE_ACTIONS_INTRADAY ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Apply split/dividend adjustments to **intraday** frames (minute/second/hour) in backtests so intraday prices match daily split-adjusted prices and option-chain strike normalization stays consistent. - Values: ``1`` / ``true`` enable; ``0`` / ``false`` disable. - Default: - enabled when ``IS_BACKTESTING`` is truthy - disabled otherwise - Notes: - Disabling can break options strike selection around splits (example: NVDA 10-for-1 split on 2024-06-10). Remote cache (S3) ----------------- LUMIBOT_CACHE_BACKEND / LUMIBOT_CACHE_MODE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Enable remote cache mirroring (for example, mirroring parquet cache files to S3). - Common values: - ``LUMIBOT_CACHE_BACKEND=s3`` - ``LUMIBOT_CACHE_MODE=readwrite`` (or ``readonly``) LUMIBOT_CACHE_FOLDER ^^^^^^^^^^^^^^^^^^^^ - Purpose: Override the local cache folder (useful to simulate a fresh container/task). LUMIBOT_CACHE_S3_BUCKET / LUMIBOT_CACHE_S3_PREFIX / LUMIBOT_CACHE_S3_REGION ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: S3 target configuration. LUMIBOT_CACHE_S3_VERSION ^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Namespace/version the remote cache without deleting anything. - Practical use: set a unique version to simulate a “cold S3” run safely. LUMIBOT_CACHE_S3_ACCESS_KEY_ID / LUMIBOT_CACHE_S3_SECRET_ACCESS_KEY / LUMIBOT_CACHE_S3_SESSION_TOKEN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Credentials for S3 access when not using an instance/task role. - Values: provided by your runtime environment (**do not hardcode**). For cache key layout and validation workflow, see :doc:`Backtesting ` and the engineering notes in ``docs/remote_cache.md``. Strategy configuration ---------------------- STRATEGY_NAME ^^^^^^^^^^^^^ - Purpose: Name for the strategy to be used in database logging and identification. - Values: Any string. MARKET ^^^^^^ - Purpose: Market to be traded (used for market calendar selection). - Values: ``NYSE``, ``NASDAQ``, ``24/7`` (crypto), etc. HIDE_TRADES / HIDE_POSITIONS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Hide trade and position information in logs/output. - Values: ``true`` / ``false`` (string). - Default: ``false``. DISCORD_WEBHOOK_URL ^^^^^^^^^^^^^^^^^^^ - Purpose: Discord webhook URL for notifications. - Values: Full Discord webhook URL (**do not hardcode in public repos**). Database configuration ---------------------- DB_CONNECTION_STR ^^^^^^^^^^^^^^^^^ - Purpose: PostgreSQL connection string for account history and strategy persistence. - Values: ``postgresql://user:password@host:port/database`` (**do not hardcode**). - Note: Replaces deprecated ``ACCOUNT_HISTORY_DB_CONNECTION_STR``. LOG_BACKTEST_PROGRESS_TO_FILE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Log backtest progress to a file instead of stdout. - Values: ``true`` / ``false`` (string). Broker selection ---------------- TRADING_BROKER ^^^^^^^^^^^^^^ - Purpose: Explicitly specify which broker to use for live trading. - Values (case-insensitive): - ``alpaca``, ``tradier``, ``ccxt``, ``coinbase``, ``kraken`` - ``ib``, ``interactivebrokers``, ``ibrest``, ``interactivebrokersrest`` - ``tradovate``, ``schwab``, ``bitunix`` - ``projectx``, ``projectx-topstepx``, ``projectx-topone``, etc. - Note: If not set, broker is auto-detected based on available credentials. DATA_SOURCE ^^^^^^^^^^^ - Purpose: Explicitly specify which data source to use. - Values (case-insensitive): - ``alpaca``, ``tradier``, ``polygon``, ``yahoo``, ``thetadata``, ``databento`` - ``ccxt``, ``coinbase``, ``kraken``, ``schwab``, ``bitunix``, ``projectx`` - Note: If not set, uses broker's default data source. Alpaca broker ------------- ALPACA_API_KEY / ALPACA_API_SECRET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Alpaca API credentials for trading. - Values: Obtain from Alpaca dashboard (**do not hardcode**). ALPACA_OAUTH_TOKEN ^^^^^^^^^^^^^^^^^^ - Purpose: OAuth token (alternative to API key/secret). - Values: OAuth token (**do not hardcode**). - Note: Either OAuth token OR API key/secret must be provided, not both. ALPACA_IS_PAPER ^^^^^^^^^^^^^^^ - Purpose: Toggle between paper and live trading. - Values: ``true`` (paper) / ``false`` (live). - Default: ``true`` (paper trading). Tradier broker -------------- TRADIER_ACCESS_TOKEN ^^^^^^^^^^^^^^^^^^^^ - Purpose: Tradier API access token. - Values: Obtain from Tradier dashboard (**do not hardcode**). TRADIER_ACCOUNT_NUMBER ^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Tradier account number for trading. - Values: Your Tradier account number. TRADIER_IS_PAPER ^^^^^^^^^^^^^^^^ - Purpose: Toggle between paper and live trading. - Values: ``true`` (paper) / ``false`` (live). - Default: ``true`` (paper trading). Interactive Brokers ------------------- INTERACTIVE_BROKERS_PORT ^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Socket port for IB Gateway/TWS connection. - Values: Integer (e.g., ``7497`` for paper, ``7496`` for live). INTERACTIVE_BROKERS_CLIENT_ID ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Client ID for IB connection (must be unique per connection). - Values: Integer. INTERACTIVE_BROKERS_IP ^^^^^^^^^^^^^^^^^^^^^^ - Purpose: IP address of IB Gateway/TWS. - Values: IP address string. - Default: ``127.0.0.1``. IB_SUBACCOUNT ^^^^^^^^^^^^^ - Purpose: Sub-account identifier for IB multi-account setups. - Values: Account identifier string. Interactive Brokers REST ------------------------ IB_USERNAME / IB_PASSWORD ^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Credentials for IB REST API authentication. - Values: IB credentials (**do not hardcode**). IB_ACCOUNT_ID ^^^^^^^^^^^^^ - Purpose: Account ID for IB REST API. - Values: Account identifier string. IB_API_URL ^^^^^^^^^^ - Purpose: Base URL for IB REST API endpoint. - Values: URL string. Schwab broker ------------- SCHWAB_ACCOUNT_NUMBER ^^^^^^^^^^^^^^^^^^^^^ - Purpose: Schwab account number (required). - Values: Account number string. SCHWAB_APP_KEY / SCHWAB_APP_SECRET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Schwab API application credentials. - Values: Obtain from Schwab developer portal (**do not hardcode**). SCHWAB_TOKEN ^^^^^^^^^^^^ - Purpose: Optional pre-existing OAuth token. - Values: Token string (**do not hardcode**). SCHWAB_BACKEND_CALLBACK_URL ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: OAuth callback URL for authentication flow. - Values: URL string. Tradovate broker ---------------- TRADOVATE_USERNAME / TRADOVATE_DEDICATED_PASSWORD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Tradovate credentials. - Values: Tradovate credentials (**do not hardcode**). TRADOVATE_APP_ID / TRADOVATE_APP_VERSION ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Application identification for Tradovate API. - Values: String identifiers. - Default: ``Lumibot`` / ``1.0``. TRADOVATE_CID / TRADOVATE_SECRET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Client credentials for Tradovate OAuth. - Values: Obtain from Tradovate (**do not hardcode**). TRADOVATE_IS_PAPER ^^^^^^^^^^^^^^^^^^ - Purpose: Toggle between paper and live trading. - Values: ``true`` (paper) / ``false`` (live). - Default: ``true``. TRADOVATE_MD_URL ^^^^^^^^^^^^^^^^ - Purpose: Market data URL override. - Values: URL string. - Default: ``https://md.tradovateapi.com/v1``. Crypto brokers (CCXT) --------------------- KRAKEN_API_KEY / KRAKEN_API_SECRET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Kraken exchange API credentials. - Values: Obtain from Kraken (**do not hardcode**). COINBASE_API_KEY_NAME / COINBASE_PRIVATE_KEY ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Coinbase Advanced Trade API credentials. - Values: Obtain from Coinbase (**do not hardcode**). COINBASE_API_PASSPHRASE ^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: API passphrase (if required by Coinbase). - Values: Passphrase string (**do not hardcode**). COINBASE_SANDBOX ^^^^^^^^^^^^^^^^ - Purpose: Use Coinbase sandbox environment. - Values: ``true`` / ``false``. - Default: ``false``. Bitunix broker -------------- BITUNIX_API_KEY / BITUNIX_API_SECRET ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Bitunix exchange API credentials. - Values: Obtain from Bitunix (**do not hardcode**). BITUNIX_TRADING_MODE ^^^^^^^^^^^^^^^^^^^^ - Purpose: Trading mode selection. - Values: ``FUTURES`` / ``SPOT``. - Default: ``FUTURES``. ProjectX brokers ---------------- ProjectX supports multiple prop trading firms. Each firm uses a unique prefix pattern. PROJECTX_FIRM ^^^^^^^^^^^^^ - Purpose: Select which ProjectX firm to use. - Values: ``TOPSTEPX``, ``TOPONE``, ``TICKTICKTRADER``, ``BULENOX``, ``E8X``, etc. PROJECTX_{FIRM}_API_KEY ^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: API key for the specified firm. - Example: ``PROJECTX_TOPSTEPX_API_KEY``, ``PROJECTX_TOPONE_API_KEY`` - Values: Obtain from firm's platform (**do not hardcode**). PROJECTX_{FIRM}_USERNAME ^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Username for the specified firm. - Example: ``PROJECTX_TOPSTEPX_USERNAME`` - Values: Your username on the firm's platform. PROJECTX_{FIRM}_PREFERRED_ACCOUNT_NAME ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Preferred account name when multiple accounts exist. - Example: ``PROJECTX_TOPSTEPX_PREFERRED_ACCOUNT_NAME`` - Values: Account name string. Data source credentials ----------------------- POLYGON_API_KEY ^^^^^^^^^^^^^^^ - Purpose: Polygon.io API key for market data. - Values: Obtain from Polygon.io (**do not hardcode**). POLYGON_MAX_MEMORY_BYTES ^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: Hard limit on memory Polygon can use for caching. - Values: Integer (bytes). THETADATA_USERNAME / THETADATA_PASSWORD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Purpose: ThetaData API credentials. - Values: Obtain from ThetaData (**do not hardcode**). - Note: Required for ThetaData backtesting and live data. DATABENTO_API_KEY ^^^^^^^^^^^^^^^^^ - Purpose: DataBento API key for market data. - Values: Obtain from DataBento (**do not hardcode**). DATABENTO_TIMEOUT ^^^^^^^^^^^^^^^^^ - Purpose: Request timeout for DataBento API calls. - Values: Integer (seconds). - Default: ``30``. DATABENTO_MAX_RETRIES ^^^^^^^^^^^^^^^^^^^^^ - Purpose: Maximum retry attempts for failed DataBento requests. - Values: Integer. - Default: ``3``. LUMIWEALTH_API_KEY ^^^^^^^^^^^^^^^^^^ - Purpose: LumiWealth platform API key (for enterprise features). - Values: Obtain from LumiWealth (**do not hardcode**).