add specification, rationale section; doc why env vars arent an option

python · rickeylev · Jun 27, 2025 · Jun 28, 2025 · Jun 28, 2025 · Jun 28, 2025
commit 55a5e2857487d435a82772717f6a6d5083f94cd8
@@ -50,13 +50,14 @@ requirement should be relaxed and relative path behavior allowed and defined.
 Second, virtual environments are the standard way for running Python
 applications. This means ...
 
-* The closer the dev environment is to the non-dev environment, the more reliable
-  software one can get.
-* The closer the dev and non-dev envs are, the easier it is to reproduce issues.
+* The closer the dev environment is to the non-dev environment, the more
+  reliable software one can get.
+* The closer the dev and non-dev envs are, the easier it is to reproduce
+  issues.
 * The simpler the process is to re-create the environment, the more reliable
   software one can get.
-* The simpler the process is to re-create the environment, the faster the process
-  can be.
+* The simpler the process is to re-create the environment, the faster the
+  process can be.
 
 By making it trivial to copy a virtual environment from one host to another, it
 avoids these categories of problems. Additionally, the development tools to
@@ -70,6 +71,36 @@ caching of artifacts.
 Rationale
 =========
 
+The reason support for relative virtual environments needs to be
+in the interpreter itself is because locating ``PYTHONHOME`` happens
+very early in the interpreter startup process, which limits the options for
+customizing how it's computed. Without the ability to specify where the
+supporting Python runtime files are, the interpreter can't finish startup,
+so other hook points (e.g. ``site`` initialization) never trigger.
+
+Specification
+=============
+
+The ``home`` value in ``pyvenv.cfg`` is permitted to use a relative path value.
+These may contain up-references outside of the virtual environnment root
+directory.  Examples:
+
+* ``subdir/whatever/bin`` (a directory within the virtual environment).
+* ``../../../../../elsewhere/runtime/bin`` (a directory outside the virtual
+  environment).
+
+Relative paths are relative to the directory containing ``pyvenv.cfg``. During
+interpreter startup (i.e. ``getpath.py``), the relative path is joined to the
+directory to form an absolute path. Up-references are resolved syntactically
+(i.e. not resolving symlinks). Symlinks are *not* resolved prior to
+construction of the absolute path to ensure semantics between a relative path
+and absolute path remain the same.
+
+For example, given
+``/home/user/venv/bin/pyvenv.cfg`` with
+``home = ../../runtime/bin``, the result is ``home = /home/user/runtime/bin``,
+i.e. it's equivalent to using that value verbatim in ``pyvenv.cfg``.
+
 
 Python Runtime Changes
 ======================
@@ -82,18 +113,18 @@ Today, relative paths resolve relative to the process's current working
 directory. Because CWD isn't knowable in advance, it makes relative paths today
 effective impossible.
 
-Instead, the paths should be relative to the location of the ``pyvenv.cfg`` file.
-This file is chosen as the anchor point because the tool that creates the file
-also has to know where the Python runtime is, so can easily calculate the
+Instead, the paths should be relative to the location of the ``pyvenv.cfg``
+file. This file is chosen as the anchor point because the tool that creates the
+file also has to know where the Python runtime is, so can easily calculate the
 correct relative path. For tools that read the ``pyvenv.cfg``, it is also easy
 to simple join the directory name of where ``pyvenv.cfg`` was found with the
 path in the config file. When a person reads the config file, they can do
 similar, which is a lower cognitive burden and helps avoids the question of
 "relative to what?"
 
 This change is only a couple of lines in the start up code. Specifically, when
-parsing the ``pyvenv.cfg`` file and finding the ``home`` value, it just needs to
-be checked if it's already absolute. If not, then join to the directory name
+parsing the ``pyvenv.cfg`` file and finding the ``home`` value, it just needs
+to be checked if it's already absolute. If not, then join to the directory name
 of the ``pyvenv.cfg`` file. The code alredy knows the directory and has helpers
 already present for checking if a path is absolute and joining two paths.
 
@@ -103,12 +134,12 @@ A proof-of-concept of this is implemented in
 Virtual Environment Tool Changes
 ================================
 
-Virtual environment management tools should support a mechanism to generate
-a ``pyvenv.cfg`` file with a ``home`` key that is a relative path to the relevant
+Virtual environment management tools should support a mechanism to generate a
+``pyvenv.cfg`` file with a ``home`` key that is a relative path to the relevant
 Python installation.
 
-For the standard library's ``venv`` module, the flag ``--relative`` will be added
-to trigger this behavior.
+For the standard library's ``venv`` module, the flag ``--relative`` will be
+added to trigger this behavior.
 
 Package Installer Changes
 =========================
@@ -120,20 +151,29 @@ absolute paths occur today are:
 1. ``bin/`` scripts, where shebangs are rewritten.
 2. ``site-packages`` symlinks.
 
-Instead, they should create relative symlinks pointing to the actual
+For symlinks, they should create relative symlinks pointing to the actual
 installation directory for packages.
 
+For ``bin/`` scripts, installers should generate executables that are able to
+locate the venv's interpreter at runtime. How to do this is implementation
+defined. Typically it means having shell code use ``$0`` to locate the
+appropriate ``bin/python3`` executable, but installers are free to use whatever
+mechanism they want (e.g. a native polyglot executable).
+
+todo: copy/paste example shell?
 
 Copying a Venv to Another Host
 =================================
 
-Copying the venv to another host is simple: copy the venv directory itself, the
-relative location ``home`` points to, and preserve the relative directory
-structures.
+Copying the venv to another host is simple: copy the venv directory itself, and
+any directories outside the venv it needs, while preserving the relative
+directory structures.
+
+How to do this, exactly, is implementation-defined.
 
 In practice, this typically means copying a parent directory of the virtual
-environment, which contains the runtime, installations of dependencies, and
-the group of virtual environments to relocate.
+environment, which contains the runtime, one or more venvs, installations
+of dependencies, and any other supporting files.
 
 Backwards Compatibility
 =======================
@@ -157,8 +197,7 @@ Reference Implementation
 A reference implementation is available by using the combination of:
 
 * Python runtime from `rickeylev/feat.relative.pyvenv.home <https://github.com/python/cpython/compare/main...rickeylev:cpython:feat.relative.pyvenv.home>`__
-* rules_python with (todo: alterations to use cpython build as in-build
-  runtime)
+* rules_python with (todo: link to branch that uses above)
 
 Open Issues
 ===========
@@ -178,7 +217,7 @@ Footnotes
   implements same-host relocatable virtual environments.
 * `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__:
-* `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__:
+* `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__
-* `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__:
+* `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__
   A relocatable Python runtime.
-
+* `PoC for relative home in Python startup <https://github.com/python/cpython/compare/main...rickeylev:cpython:feat.relative.pyvenv.home>`__
 
 Rejected Ideas
 =====================
@@ -223,10 +262,22 @@ Second, when using relative paths that point outside the venv, the venv is only
 relocatable insofar as those external artifacts are also relocated. This is an
 additional nuance that requires qualification of the term.
 
-To better avoid this confusiong, "relative" is chosen, which more naturally
+To better avoid this confusion, "relative" is chosen, which more naturally
 invites the question *"Relative to what?"*.
 
 
+Using PYTHONHOME at runtime to specify home
+-------------------------------------------
+
+Using the ``PYTHONHOME`` environment variable (or any environment variable) is
+problematic because it's difficult to know and control when an environment
+variable should or shouldn't be inherited by subprocesses. In some cases, it's
+not feasible because of how layers of programs calling programs interact.
+
+Code generally assumes that any virtual environment will be
+automatically detected and activated by the presence of ``pyvenv.cfg``, so
+things work better when alterations to the environemtn aren't a concern.
+
 Copyright
 =========