Defer pandas/omegaconf imports#207
Merged
Merged
Conversation
9f8bad1 to
1dc6546
Compare
Contributor
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #207 +/- ##
==========================================
+ Coverage 95.22% 95.33% +0.10%
==========================================
Files 140 142 +2
Lines 10354 10714 +360
Branches 599 613 +14
==========================================
+ Hits 9860 10214 +354
- Misses 369 374 +5
- Partials 125 126 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Move heavy module-level imports out of the import chain triggered by `import ccflow`. Only the import sites are deferred; runtime behaviour is unchanged. * exttypes/frequency.py: drop module-level `import pandas as pd` and `from pandas.tseries.frequencies import to_offset`. Import them locally inside `Frequency.offset`, `Frequency.timedelta`, and `Frequency._validate`. Replace the module-level `_PD_GE_22` constant (which read `pd.__version__` at import time) with a `@cache`d `_pd_ge_22()` function called from `_normalize_frequency_alias`. * utils/chunker.py: move `import pandas as pd` into `dates_to_chunks`. * validators.py: move `import pandas as pd` into `normalize_date` and `normalize_datetime`. * base.py: drop module-level `import omegaconf` and `from omegaconf import DictConfig, OmegaConf`. Add a `TYPE_CHECKING` import for `DictConfig` (so static type-checkers still resolve the annotations) and quote the runtime `DictConfig` annotations on `ModelRegistry.load_config`, `ModelRegistry.create_config_from_path`, and the `_ModelRegistryLoader` methods. Add local imports of `DictConfig`/`OmegaConf`/`omegaconf` inside `_is_config_subregistry`, `_make_subregistries`, and `_ModelRegistryLoader.load_config` where they are actually used at runtime. Effect on `import ccflow`: pandas, omegaconf, and antlr4 are no longer in `sys.modules` after a bare import. `-X importtime` total drops from ~525 ms to ~320 ms (~38% faster) on Python 3.11. Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
1dc6546 to
22e4ca4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Defer eager imports of
pandasandomegaconfso they no longer load on a bareimport ccflow. Pure import-site relocation; runtime behaviour unchanged.Changes
ccflow/exttypes/frequency.py: drop module-levelimport pandas as pdandfrom pandas.tseries.frequencies import to_offset. Re-import locally inFrequency.offset,Frequency.timedelta, andFrequency._validate. Replace the module-level_PD_GE_22constant (which readpd.__version__at import) with a@cached_pd_ge_22()function called from_normalize_frequency_alias.ccflow/utils/chunker.py: moveimport pandas as pdintodates_to_chunks.ccflow/validators.py: moveimport pandas as pdintonormalize_dateandnormalize_datetime.ccflow/base.py: drop module-levelimport omegaconfandfrom omegaconf import DictConfig, OmegaConf. Useif TYPE_CHECKING: from omegaconf import DictConfigso static type-checkers still resolve the annotations, and quote the runtimeDictConfigannotations onModelRegistry.load_config,ModelRegistry.create_config_from_path, and the_ModelRegistryLoadermethods. Add local imports ofDictConfig/OmegaConf/omegaconfinside_is_config_subregistry,_make_subregistries, and_ModelRegistryLoader.load_configwhere they are actually used at runtime.Measurements
Profiled in a clean Python 3.11 venv with only ccflow's base deps installed.
import ccflowtotalpandasinsys.modulesafter importomegaconfinsys.modulesafter importantlr4insys.modulesafter importCompatibility
ccflow(e.g.from ccflow import Frequency) still work."DictConfig"annotations plus theTYPE_CHECKINGimport keep static type information intact.Frequencyvalidation (including legacy alias path that exercises_pd_ge_22()),normalize_date,normalize_datetime, anddates_to_chunksall still work.