Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] _compture_modules can't import modules from code_paths #14145

Open
2 of 23 tasks
xDoomi opened this issue Dec 21, 2024 · 1 comment
Open
2 of 23 tasks

[BUG] _compture_modules can't import modules from code_paths #14145

xDoomi opened this issue Dec 21, 2024 · 1 comment
Labels
area/models MLmodel format, model serialization/deserialization, flavors bug Something isn't working

Comments

@xDoomi
Copy link

xDoomi commented Dec 21, 2024

Issues Policy acknowledgement

  • I have read and agree to submit bug reports in accordance with the issues policy

Where did you encounter this bug?

Local machine

MLflow version

  • Client: 2.19.0
  • Tracking server: 2.19.0

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 22.04
  • Python version: Python 3.11.11
  • yarn version, if running the dev UI:

Describe the problem

I log pyfunc with loader_module and code_paths:
mlflow.pyfunc.log_model(artifact_path="scripted_model", loader_module="wrapper", data_path="scripted_model/model.pth", code_paths=["inference/wrapper.py"])
MLflow try to determine which packages are imported and run _capture_modules.py. However, it executes with an error:
ModuleNotFoundError: No module named 'wrapper'
I thought this was a strange behavior, because mlflow.pyfunc.load_model could import wrapper. Then I noticed that before importing wrapper it adds code_paths to system_path _add_code_from_conf_to_system_path(local_path, conf, code_key=CODE).
So the import error in _capture_modules.py can be solved by adding code_paths to system_path.

Tracking information

System information: Linux #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov  6 17:42:15 UTC 2
Python version: 3.11.11
MLflow version: 2.19.0
MLflow module location: /home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/__init__.py
Tracking URI: http://localhost:8000
Registry URI: http://localhost:8000
MLflow environment variables: 
  MLFLOW_TRACKING_URI: http://localhost:8000
MLflow dependencies: 
  Flask: 3.1.0
  Jinja2: 3.1.5
  alembic: 1.14.0
  docker: 7.1.0
  graphene: 3.4.3
  gunicorn: 23.0.0
  markdown: 3.7
  matplotlib: 3.10.0
  mlflow-skinny: 2.19.0
  numpy: 2.2.0
  pandas: 2.2.3
  pyarrow: 18.1.0
  scikit-learn: 1.6.0
  scipy: 1.14.1
  sqlalchemy: 2.0.36

Code to reproduce issue

# inference/wrapper.py
import torch
from torch import Tensor
import numpy as np
from mlflow.pytorch import _load_model
from typing import Optional, Dict, Any, Union


def _load_pyfunc(path, **kwargs):
    """
    Load PyFunc implementation. Called by ``pyfunc.load_model``.

    :param path: Local filesystem path to the MLflow Model with the ``pytorch`` flavor.
    """
    device = kwargs.get("map_location")
    return SegmentationTorchWrapper(_load_model(path, **kwargs), device)


class SegmentationTorchWrapper():

    def __init__(self, pytorch_model, device = "cpu"):
        self.python_model = pytorch_model
        self.python_model.to(device)
        self.python_model.eval()

    def predict(self, data: Union[np.ndarray, Tensor], params: Optional[Dict[str, Any]] = None) -> Tensor:
        """ do something """
        pass


# log_pyfunc.py
import os
import mlflow
os.environ["MLFLOW_REQUIREMENTS_INFERENCE_RAISE_ERRORS"] = "true"

mlflow.pyfunc.log_model(artifact_path="scripted_model", loader_module="wrapper", data_path="scripted_model/model.pth", code_paths=["inference/wrapper.py"])

Stack trace

Traceback (most recent call last):
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/get_pip_requirements.py", line 10, in <module>
    pip_list = infer_pip_requirements(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/environment.py", line 430, in infer_pip_requirements
    return _infer_requirements(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/requirements_utils.py", line 511, in _infer_requirements
    modules = _capture_imported_modules(model_uri, flavor, extra_env_vars=extra_env_vars)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/requirements_utils.py", line 385, in _capture_imported_modules
    _run_command(
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/requirements_utils.py", line 264, in _run_command
    raise MlflowException(msg)
mlflow.exceptions.MlflowException: Encountered an unexpected error while running ['/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/bin/python3', '/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/_capture_modules.py', '--model-path', '/home/xdoomi/work/notebooks/investigate_pyfunc/models/scripted_model', '--flavor', 'python_function', '--output-file', '/tmp/tmpe9j8arqk/imported_modules.txt', '--error-file', '/tmp/tmpe9j8arqk/error.txt', '--sys-path', '["/home/xdoomi/work/notebooks/investigate_pyfunc", "/usr/lib/python311.zip", "/usr/lib/python3.11", "/usr/lib/python3.11/lib-dynload", "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages"]']
exit status: 1
stdout: 
stderr: Traceback (most recent call last):
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/_capture_modules.py", line 256, in <module>
    main()
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/_capture_modules.py", line 233, in main
    store_imported_modules(
  File "/home/xdoomi/work/notebooks/investigate_pyfunc/.venv/lib/python3.11/site-packages/mlflow/utils/_capture_modules.py", line 191, in store_imported_modules
    loader_module = importlib.import_module(pyfunc_conf[MAIN])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'wrapper'

Other info / logs

REPLACE_ME

What component(s) does this bug affect?

  • area/artifacts: Artifact stores and artifact logging
  • area/build: Build and test infrastructure for MLflow
  • area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
  • area/docs: MLflow documentation pages
  • area/examples: Example code
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
  • area/projects: MLproject format, project running backends
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/server-infra: MLflow Tracking server backend
  • area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
  • area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
  • area/windows: Windows support

What language(s) does this bug affect?

  • language/r: R APIs and clients
  • language/java: Java APIs and clients
  • language/new: Proposals for new client languages

What integration(s) does this bug affect?

  • integrations/azure: Azure and Azure ML integrations
  • integrations/sagemaker: SageMaker integrations
  • integrations/databricks: Databricks integrations
@xDoomi xDoomi added the bug Something isn't working label Dec 21, 2024
@github-actions github-actions bot added the area/models MLmodel format, model serialization/deserialization, flavors label Dec 21, 2024
@serena-ruan
Copy link
Collaborator

Thanks for reporting this! We should probably call _add_code_from_conf_to_system_path inside _capture_modules to make sure the model can be loaded successfully.
I'll convert this issue into an internal ticket and track the progress, as we needs some extra work to make sure adding that won't introduce unexpected dependencies as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/models MLmodel format, model serialization/deserialization, flavors bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xDoomi @serena-ruan
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy