Skip to content

Enable doc tests in local and CI testing #1409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 3 commits into from
Mar 5, 2026
+14 −7
Merged

Enable doc tests in local and CI testing #1409

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
088482d
Turn on doctests
ntjohnson1 Feb 26, 2026
8639eaa
Fix existing doc examples
ntjohnson1 Feb 26, 2026
a5448dc
Remove stale referenece to rust-toolchain removed in #1383, surpised …
ntjohnson1 Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.cargo
key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
key: cargo-cache-${{ matrix.toolchain }}-${{ hashFiles('Cargo.lock') }}

- name: Install dependencies
uses: astral-sh/setup-uv@v7
Expand Down Expand Up @@ -106,7 +106,7 @@ jobs:
RUST_BACKTRACE: 1
run: |
git submodule update --init
uv run --no-project pytest -v . --import-mode=importlib
uv run --no-project pytest -v --import-mode=importlib

- name: FFI unit tests
run: |
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ features = ["substrait"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"
addopts = "--doctest-modules"
Copy link
Contributor

@alamb alamb Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

matches https://docs.pytest.org/en/stable/how-to/doctest.html 👍

doctest_optionflags = ["NORMALIZE_WHITESPACE", "ELLIPSIS"]
testpaths = ["python/tests", "python/datafusion"]

# Enable docstring linting using the google style guide
[tool.ruff.lint]
Expand Down
14 changes: 9 additions & 5 deletions python/datafusion/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,8 +327,9 @@ def into_view(self, temporary: bool = False) -> Table:
>>> df = ctx.sql("SELECT 1 AS value")
>>> view = df.into_view()
>>> ctx.register_table("values_view", view)
>>> df.collect() # The DataFrame is still usable
>>> ctx.sql("SELECT value FROM values_view").collect()
>>> result = ctx.sql("SELECT value FROM values_view").collect()
>>> result[0].column("value").to_pylist()
[1]
"""
from datafusion.catalog import Table as _Table

Expand Down Expand Up @@ -1389,9 +1390,12 @@ def fill_null(self, value: Any, subset: list[str] | None = None) -> DataFrame:
DataFrame with null values replaced where type casting is possible

Examples:
>>> df = df.fill_null(0) # Fill all nulls with 0 where possible
>>> # Fill nulls in specific string columns
>>> df = df.fill_null("missing", subset=["name", "category"])
>>> from datafusion import SessionContext, col
Copy link
Contributor

@alamb alamb Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ntjohnson1

Did you verify from the CI run logs that these tests are now run? I took a brief look and couldn't find anything that was new http://github.com/apache/datafusion-python/actions/runs/22687531822/job/65776433980?pr=1409

🤔

Copy link
Contributor Author

@ntjohnson1 ntjohnson1 Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doctests run as a part of pytest so if you click on Run Tests you can see the new doctest entries.

There aren't that many examples in the code at the moment.
If you scroll all the way to the bottom you can now see source code showing up in addition to test code

python/datafusion/dataframe.py::datafusion.dataframe.DataFrame.fill_null PASSED [ 99%]
python/datafusion/dataframe.py::datafusion.dataframe.DataFrame.into_view PASSED [ 99%]
python/datafusion/dataframe_formatter.py::datafusion.dataframe_formatter.configure_formatter PASSED [ 99%]
python/datafusion/dataframe_formatter.py::datafusion.dataframe_formatter.get_formatter PASSED [ 99%]
python/datafusion/dataframe_formatter.py::datafusion.dataframe_formatter.reset_formatter PASSED [ 99%]
python/datafusion/dataframe_formatter.py::datafusion.dataframe_formatter.set_formatter PASSED [100%]

Copy link
Contributor

@alamb alamb Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice -- I checked http://github.com/apache/datafusion-python/actions/runs/22676065624/job/65741468321 (a recent run from main) and indeed those tests are not there


python/tests/test_view.py::test_register_filtered_dataframe PASSED       [ 99%]
python/tests/test_wrapper_coverage.py::test_datafusion_missing_exports PASSED [100%]

>>> ctx = SessionContext()
>>> df = ctx.from_pydict({"a": [1, None, 3], "b": [None, 5, 6]})
>>> filled = df.fill_null(0)
>>> filled.sort(col("a")).collect()[0].column("a").to_pylist()
[0, 1, 3]

Notes:
- Only fills nulls in columns where the value can be cast to the column type
Expand Down