feat: Allow configuration with a config file

docs/en/esptool/configuration-file.rst

@@ -0,0 +1,109 @@
+.. _config:
+
+Configuration File
+==================
+
+``esptool.py`` relies on serial communication when connecting to, reading from, or writing to an ESP chip.
+To ensure this two-way serial connection works properly, ``esptool.py`` is tuned with several pre-defined
+variables describing the timings and other nuances when sending or receiving data.
+These variables have been finely tuned to work in absolute majority of environments.
+However, it is impossible to cover all of the existing combinations of hardware, OS, and drivers.
+Sometimes little tweaking is necessary to cover even the most extreme edge cases.
+
+These options can be specified in a configuration file. This makes it easy to run
+``esptool.py`` with custom settings, and also allows for specification of options
+that are otherwise not available to a user without having to tamper with the source code.
+
+File Location
+-------------
+
+The default name for a configuration file is ``esptool.cfg``. First, the same
+directory ``esptool.py`` is being run in is inspected.
+
+If a configuration file is not found here, the current user's OS configuration directory is inspected next:
+
+ - Linux: ``/home/<user>/.config/esptool/``
+ - MacOS ``/Users/<user>/.config/esptool/``
+ - Windows: ``c:\Users\<user>\AppData\Local\esptool\``
+
+If a configuration file is still not found, the last inspected location is the home directory:
+
+ - Linux: ``/home/<user>/``
+ - MacOS ``/Users/<user>/``
+ - Windows: ``c:\Users\<user>\``
+
+On Windows, the home directory can be set with the ``HOME`` or ``USERPROFILE`` environment variables.
+Therefore, the Windows configuration directory location also depends on these.
+
+A different location for the configuration file can be specified with the ``ESPTOOL_CFGFILE``
+environment variable, e.g. ``ESPTOOL_CFGFILE = ~/custom_config.cfg``.
+This overrides the search priorities described above.
+
+``esptool.py`` will read settings from other usual configuration files if no other
+configuration file is used.  It will automatically read from ``setup.cfg`` or
+``tox.ini`` if they exist.
+
+As a result, the order of priority of inspected configuration files is:
+
+#. a file specified with the ``ESPTOOL_CFGFILE`` environment variable
+#. ``esptool.cfg``
+#. ``setup.cfg``
+#. ``tox.ini``
+
+Syntax
+------
+
+An ``esptool.py`` configuration file is in .ini file format: it must be
+introduced by an ``[esptool]`` header to be recognized as valid.
+This section then contains ``name = value`` entries.
+Lines beginning with ``#`` or ``;`` are ignored as comments.
+
+Delay and timeout options accept float values,
+other numeric options are integers. Strings don't need quotes.
+
+Sample configuration file:
+
+.. code-block:: text
+
+    # esptool.cfg file to configure internal settings of esptool
+    [esptool]
+    chip_erase_timeout = 140
+    serial_write_timeout = 8.5
+    connect_attempts = 7
+    write_block_attempts = 2
+    reset_delay = 0.75
+    # Overriding the default reset sequence to work in an abnormal environment
+    custom_reset_sequence = D0|R1|W0.1|D1|R0|W0.5|D0
+
+Options
+-------
+
+Complete list configurable options:
+
++------------------------------+-----------------------------------------------------------+----------+
+| Option                       | Description                                               | Default  |
++==============================+===========================================================+==========+
+| timeout                      | Timeout for most flash operations                         | 3 s      |
++------------------------------+-----------------------------------------------------------+----------+
+| chip_erase_timeout           | Timeout for a full chip erase                             | 120 s    |
++------------------------------+-----------------------------------------------------------+----------+
+| max_timeout                  | The longest any command can run                           | 240 s    |
++------------------------------+-----------------------------------------------------------+----------+
+| sync_timeout                 | Timeout for syncing with the bootloader                   | 0.1 s    |
++------------------------------+-----------------------------------------------------------+----------+
+| md5_timeout_per_mb           | Timeout (per megabyte) for calculating md5sum             | 8 s      |
++------------------------------+-----------------------------------------------------------+----------+
+| erase_region_timeout_per_mb  | Timeout (per megabyte) for erasing a region               | 30 s     |
++------------------------------+-----------------------------------------------------------+----------+
+| erase_write_timeout_per_mb   | Timeout (per megabyte) for erasing and writing data       | 40 s     |
++------------------------------+-----------------------------------------------------------+----------+
+| mem_end_rom_timeout          | Short timeout for ESP_MEM_END                             | 0.05 s   |
++------------------------------+-----------------------------------------------------------+----------+
+| serial_write_timeout         | Timeout for serial port write                             | 10 s     |
++------------------------------+-----------------------------------------------------------+----------+
+| connect_attempts             | Default number of times to try connection                 | 7        |
++------------------------------+-----------------------------------------------------------+----------+
+| write_block_attempts         | Number of times to try writing a data block               | 3        |
++------------------------------+-----------------------------------------------------------+----------+
+| reset_delay                  | Time to wait before the boot pin is released after reset  | 0.05 s   |
++------------------------------+-----------------------------------------------------------+----------+

docs/en/esptool/index.rst

@@ -18,6 +18,7 @@ To see all options for a particular command, append ``-h`` to the command name.
    Flash Modes <flash-modes>
    Entering the Bootloader <entering-bootloader>
    Serial Connection <serial-connection>
+   Configuration File <configuration-file>
    Remote Serial Ports <remote-serial-ports>
    Flashing Firmware <flashing-firmware>
    Scripting <scripting>

docs/en/troubleshooting.rst

@@ -119,6 +119,13 @@ Running ``esptool.py --trace`` will dump all serial interactions to the standard
 
 See :ref:`the related Advanced Topics page <tracing-communications>` for more information.
 
+Configuration File
+------------------
+
+Although ``esptool.py`` has been tuned to work in the widest possible range of environments, an incompatible combination of hardware, OS, and drivers might cause it to fail. If you suspect this is the case, a custom configuration of internal variables might be necessary.
+
+These variables and options can be specified in a configuration file. See :ref:`the related Configuration File page <config>` for more information.
+
 Common Errors
 -------------
 

esptool/__init__.py

@@ -62,6 +62,7 @@ from esptool.cmds import (
     write_flash_status,
     write_mem,
 )
+from esptool.config import load_config_file
 from esptool.loader import DEFAULT_CONNECT_ATTEMPTS, ESPLoader, list_ports
 from esptool.targets import CHIP_DEFS, CHIP_LIST, ESP32ROM
 from esptool.util import (
@@ -627,6 +628,7 @@ def main(argv=None, esp=None):
 
     args = parser.parse_args(argv)
     print("esptool.py v%s" % __version__)
+    load_config_file(verbose=True)
 
     # operation function can take 1 arg (args), 2 args (esp, arg)
     # or be a member function of the ESPLoader class.

esptool/config.py

@@ -0,0 +1,88 @@
+# SPDX-FileCopyrightText: 2014-2023 Espressif Systems (Shanghai) CO LTD,
+# other contributors as noted.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import configparser
+import os
+
+CONFIG_OPTIONS = [
+    "timeout",
+    "chip_erase_timeout",
+    "max_timeout",
+    "sync_timeout",
+    "md5_timeout_per_mb",
+    "erase_region_timeout_per_mb",
+    "erase_write_timeout_per_mb",
+    "mem_end_rom_timeout",
+    "serial_write_timeout",
+    "connect_attempts",
+    "write_block_attempts",
+    "reset_delay",
+]
+
+
+def _validate_config_file(file_path, verbose=False):
+    if not os.path.exists(file_path):
+        return False
+
+    cfg = configparser.RawConfigParser()
+    try:
+        cfg.read(file_path, encoding="UTF-8")
+        # Only consider it a valid config file if it contains [esptool] section
+        if cfg.has_section("esptool"):
+            if verbose:
+                unknown_opts = list(set(cfg.options("esptool")) - set(CONFIG_OPTIONS))
+                suffix = "s" if len(unknown_opts) > 1 else ""
+                print(
+                    "Ignoring unknown config file option{}: {}".format(
+                        suffix, ", ".join(unknown_opts)
+                    )
+                )
+            return True
+    except (UnicodeDecodeError, configparser.Error) as e:
+        if verbose:
+            print(f"Ignoring invalid config file {file_path}: {e}")
+    return False
+
+
+def _find_config_file(dir_path, verbose=False):
+    for candidate in ("esptool.cfg", "setup.cfg", "tox.ini"):
+        cfg_path = os.path.join(dir_path, candidate)
+        if _validate_config_file(cfg_path, verbose):
+            return cfg_path
+    return None
+
+
+def load_config_file(verbose=False):
+    set_with_env_var = False
+    env_var_path = os.environ.get("ESPTOOL_CFGFILE")
+    if env_var_path is not None and _validate_config_file(env_var_path):
+        cfg_file_path = env_var_path
+        set_with_env_var = True
+    else:
+        home_dir = os.path.expanduser("~")
+        os_config_dir = (
+            f"{home_dir}/.config/esptool"
+            if os.name == "posix"
+            else f"{home_dir}/AppData/Local/esptool/"
+        )
+        # Search priority: 1) current dir, 2) OS specific config dir, 3) home dir
+        for dir_path in (os.getcwd(), os_config_dir, home_dir):
+            cfg_file_path = _find_config_file(dir_path, verbose)
+            if cfg_file_path:
+                break
+
+    cfg = configparser.ConfigParser()
+    cfg["esptool"] = {}  # Create an empty esptool config for when no file is found
+
+    if cfg_file_path is not None:
+        # If config file is found and validated, read and parse it
+        cfg.read(cfg_file_path)
+        if verbose:
+            msg = " (set with ESPTOOL_CFGFILE)" if set_with_env_var else ""
+            print(
+                f"Loaded custom configuration from "
+                f"{os.path.abspath(cfg_file_path)}{msg}"
+            )
+    return cfg, cfg_file_path

esptool/loader.py

@@ -14,6 +14,7 @@ import struct
 import sys
 import time
 
+from .config import load_config_file
 from .reset import (
     ClassicReset,
     DEFAULT_RESET_DELAY,
@@ -69,18 +70,31 @@ except Exception:
         raise
 
 
-DEFAULT_TIMEOUT = 3  # timeout for most flash operations
-START_FLASH_TIMEOUT = 20  # timeout for starting flash (may perform erase)
-CHIP_ERASE_TIMEOUT = 120  # timeout for full chip erase
-MAX_TIMEOUT = CHIP_ERASE_TIMEOUT * 2  # longest any command can run
-SYNC_TIMEOUT = 0.1  # timeout for syncing with bootloader
-MD5_TIMEOUT_PER_MB = 8  # timeout (per megabyte) for calculating md5sum
-ERASE_REGION_TIMEOUT_PER_MB = 30  # timeout (per megabyte) for erasing a region
-ERASE_WRITE_TIMEOUT_PER_MB = 40  # timeout (per megabyte) for erasing and writing data
-MEM_END_ROM_TIMEOUT = 0.05  # short timeout for ESP_MEM_END, as it may never respond
-DEFAULT_SERIAL_WRITE_TIMEOUT = 10  # timeout for serial port write
-DEFAULT_CONNECT_ATTEMPTS = 7  # default number of times to try connection
-WRITE_BLOCK_ATTEMPTS = 3  # number of times to try writing a data block
+cfg, _ = load_config_file()
+cfg = cfg["esptool"]
+
+# Timeout for most flash operations
+DEFAULT_TIMEOUT = cfg.getfloat("timeout", 3)
+# Timeout for full chip erase
+CHIP_ERASE_TIMEOUT = cfg.getfloat("chip_erase_timeout", 120)
+# Longest any command can run
+MAX_TIMEOUT = cfg.getfloat("max_timeout", CHIP_ERASE_TIMEOUT * 2)
+# Timeout for syncing with bootloader
+SYNC_TIMEOUT = cfg.getfloat("sync_timeout", 0.1)
+# Timeout (per megabyte) for calculating md5sum
+MD5_TIMEOUT_PER_MB = cfg.getfloat("md5_timeout_per_mb", 8)
+# Timeout (per megabyte) for erasing a region
+ERASE_REGION_TIMEOUT_PER_MB = cfg.getfloat("erase_region_timeout_per_mb", 30)
+# Timeout (per megabyte) for erasing and writing data
+ERASE_WRITE_TIMEOUT_PER_MB = cfg.getfloat("erase_write_timeout_per_mb", 40)
+# Short timeout for ESP_MEM_END, as it may never respond
+MEM_END_ROM_TIMEOUT = cfg.getfloat("mem_end_rom_timeout", 0.05)
+# Timeout for serial port write
+DEFAULT_SERIAL_WRITE_TIMEOUT = cfg.getfloat("serial_write_timeout", 10)
+# Default number of times to try connection
+DEFAULT_CONNECT_ATTEMPTS = cfg.getint("connect_attempts", 7)
+# Number of times to try writing a data block
+WRITE_BLOCK_ATTEMPTS = cfg.getint("write_block_attempts", 3)
 
 STUBS_DIR = os.path.join(os.path.dirname(__file__), "./targets/stub_flasher/")
 
@@ -558,9 +572,12 @@ class ESPLoader(object):
         """
         # TODO: If config file defines custom reset sequence, parse it and return it
 
-        delay = DEFAULT_RESET_DELAY
-        extra_delay = DEFAULT_RESET_DELAY + 0.5
-        # TODO: If config file defines custom delay, parse it and set it
+        cfg_reset_delay = cfg.getfloat("reset_delay")
+        if cfg_reset_delay is not None:
+            delay = extra_delay = cfg_reset_delay
+        else:
+            delay = DEFAULT_RESET_DELAY
+            extra_delay = DEFAULT_RESET_DELAY + 0.5
 
         # This FPGA delay is for Espressif internal use
         if (

test/test_esptool.py

@@ -1046,3 +1046,81 @@ class TestMakeImage(EsptoolTestCase):
             )
         finally:
             os.remove("test0x00000.bin")
+
+
+@pytest.mark.skipif(arg_chip != "esp32", reason="Don't need to test multiple times")
+class TestConfigFile(EsptoolTestCase):
+    class ConfigFile:
+        """
+        A class-based context manager to create
+        a custom config file and delete it after usage.
+        """
+
+        def __init__(self, file_path, file_content):
+            self.file_path = file_path
+            self.file_content = file_content
+
+        def __enter__(self):
+            with open(self.file_path, "w") as cfg_file:
+                cfg_file.write(self.file_content)
+                return cfg_file
+
+        def __exit__(self, exc_type, exc_value, exc_tb):
+            os.unlink(self.file_path)
+            assert not os.path.exists(self.file_path)
+
+    dummy_config = (
+        "[esptool]\n"
+        "connect_attempts = 5\n"
+        "reset_delay = 1\n"
+        "serial_write_timeout = 12"
+    )
+
+    def test_load_config_file(self):
+        # Test a valid file is loaded
+        config_file_path = os.path.join(os.getcwd(), "esptool.cfg")
+        with self.ConfigFile(config_file_path, self.dummy_config):
+            output = self.run_esptool("version")
+            assert f"Loaded custom configuration from {config_file_path}" in output
+
+        # Test invalid files are ignored
+        # Wrong section header, no config gets loaded
+        with self.ConfigFile(config_file_path, "[wrong section name]"):
+            output = self.run_esptool("version")
+            assert f"Loaded custom configuration from {config_file_path}" not in output
+
+        # Correct header, but options are unparseable
+        faulty_config = "[esptool]\n" "connect_attempts = 5\n" "connect_attempts = 9\n"
+        with self.ConfigFile(config_file_path, faulty_config):
+            output = self.run_esptool("version")
+            assert f"Ignoring invalid config file {config_file_path}" in output
+            assert (
+                "option 'connect_attempts' in section 'esptool' already exists"
+                in output
+            )
+
+        # Test other config files (setup.cfg, tox.ini) are loaded
+        config_file_path = os.path.join(os.getcwd(), "tox.ini")
+        with self.ConfigFile(config_file_path, self.dummy_config):
+            output = self.run_esptool("version")
+            assert f"Loaded custom configuration from {config_file_path}" in output
+
+    def test_load_config_file_with_env_var(self):
+        config_file_path = os.path.join(TEST_DIR, "custom_file.ini")
+        with self.ConfigFile(config_file_path, self.dummy_config):
+            # Try first without setting the env var, check that no config gets loaded
+            output = self.run_esptool("version")
+            assert f"Loaded custom configuration from {config_file_path}" not in output
+
+            # Set the env var and try again, check that config was loaded
+            tmp = os.environ.get("ESPTOOL_CFGFILE")  # Save the env var if it is set
+
+            os.environ["ESPTOOL_CFGFILE"] = config_file_path
+            output = self.run_esptool("version")
+            assert f"Loaded custom configuration from {config_file_path}" in output
+            assert "(set with ESPTOOL_CFGFILE)" in output
+
+            if tmp is not None:  # Restore the env var or unset it
+                os.environ["ESPTOOL_CFGFILE"] = tmp
+            else:
+                os.environ.pop("ESPTOOL_CFGFILE", None)