DataJoint import error due to missing `pyarrow` (a `pandas` dependency)

[ttngu207]

Bug Report

Description

A fresh datajoint installation on python 3.10 is successful
However, upon import (import datajoint as dj), the following error is raised

Traceback (most recent call last):
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\IPython\core\interactiveshell.py", line 3579, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-0b6eed5a3415>", line 1, in <module>
    import datajoint
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\datajoint\__init__.py", line 62, in <module>
    from .schemas import Schema
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\datajoint\schemas.py", line 10, in <module>
    from .jobs import JobTable
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\datajoint\jobs.py", line 4, in <module>
    from .table import Table
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\datajoint\table.py", line 6, in <module>
    import pandas
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\pandas\__init__.py", line 39, in <module>
    from pandas.compat import (
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\pandas\compat\__init__.py", line 27, in <module>
    from pandas.compat.pyarrow import (
  File "C:\Program Files\JetBrains\PyCharm 2023.3.4\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\thinh\.conda\envs\microns_phase3\lib\site-packages\pandas\compat\pyarrow.py", line 10, in <module>
    _palv = Version(Version(pa.__version__).base_version)
AttributeError: module 'pyarrow' has no attribute '__version__'

Upon further investigation, it looks like pandas>2.2 requires pyarrow as its dependency, however, pyarrow is not explicitly specified as a requirement for pandas (for good reasons, lots of things to consider, see this discussion), thus not installed when pandas is installed.

For datajoint, we can either

1. pin pandas<2
2. install pandas[pyarrow]
3. set pyarrow as one of the dependency in pyproject.toml

Reproducibility

Include:

• OS (WIN)
• Python Version: 3.10
• DataJoint Version: 0.14.3

[dimitri-yatsenko]

Why do we need pyarrow if pandas does not include it?

[dimitri-yatsenko]

Analysis

The reported error:

AttributeError: module 'pyarrow' has no attribute '__version__'

This indicates a corrupted or incompatible pyarrow installation, not a missing pyarrow. The pyarrow module exists but is broken.

Root Cause

This is an environment issue where:

1. pandas 2.2+ attempts to check pyarrow compatibility at import time
2. pyarrow is partially installed or incompatible with the Python version
3. The pyarrow module exists but lacks proper attributes

Resolution

Fix for affected users:

# Reinstall pyarrow cleanly
pip uninstall pyarrow
pip install pyarrow

Or use conda which handles binary dependencies better:

conda install pyarrow pandas

DataJoint 2.0 Status

DataJoint 2.0 includes pyarrow in test/dev dependencies:

[project.optional-dependencies]
arrow = ["pyarrow>=14.0.0"]
test = [
  ...
  "pyarrow>=14.0.0",
]

The pandas import remains at module level because:

1. pandas is a core dependency used throughout DataJoint
2. Most DataJoint operations need pandas (previews, data manipulation)
3. Making it lazy would complicate the codebase significantly

Recommendation

This is an environment/installation issue rather than a DataJoint bug. Users experiencing this should:

1. Reinstall pyarrow: pip install --force-reinstall pyarrow
2. Ensure pandas and pyarrow versions are compatible
3. Consider using conda for better binary dependency management

Closing as this is an environment issue, not a DataJoint code bug.

[dimitri-yatsenko]

Closing as this is an environment/installation issue (corrupted pyarrow) rather than a DataJoint code bug. See above comment for resolution steps.