Stub QA testing (Validation Snippets)¶
This folder contains a collection of code snippets to help validate the quality of the stubs by providing some code to validate. Think of this as ‘End to End’ tests or integration tests for the stubs.
Please read : https://typing.readthedocs.io/en/latest/source/quality.html#testing-using-assert-type-and-warn-unused-ignores
Usage¶
Note: In order to get the correct typechecking for each of the folders/mcu architectures,
you should open/add this folder to a VSCode workspace workspace or open it in a seperate VSCode window
You can update / install the type-stubs in the various typings folders by running the following command:
# Update the type stubs
# , "v1.18.0", "v1.17.0"
foreach ($version in @( "latest","v1.21.0", "v1.20.0", "v1.19.1" )) {
stubber get-docstubs --version $version
stubber get-frozen --version $version
stubber merge --version $version --port auto --board auto
stubber build --version $version --port auto --board auto
}
stubber switch latest stubber get-docstubs stubber merge –version latest stubber build –version latest
.\snippets\install-stubs.ps1
## Test with pytest
There is a custom pytest configuration in `conftests.py` that will automatically download and copy the relevant stubs to the `typings` folder in the various `check_xxxx` and `feat_yyyy` folders.
The configuration of these test is part of the `test_snippets.py` file in the `snippets` folder.
Example of running the tests:
- run all tests (using any cached stubs - Max lifetime = 24 Hr) :
`pytest ./snippets`
- run all tests - but clear the cache first : :
`pytest ./snippets --cache-clear`
- run a single test :
`pytest snippets\test_snippets.py::test_pyright[local-v1.20.0-stm32-stdlib]`
- run a single test but clear the cache first : :
`pytest --cache-clear snippets\test_snippets.py::test_pyright[local-v1.20.0-stm32-stdlib]`
## Test with pyright (used by the Pylance VSCode extension)
```powershell
.\snippets\check-stubs.ps1
Naming convention¶
Use the same top-level name for the module / package you would like to test.
Use the check_${thing}.py
naming pattern for individual test files.
By default, test cases go into a file with the same name as the stub file, prefixed with check_
.
For example: stdlib/check_contextlib.py
.
If that file becomes too big, we instead create a directory with files named after individual objects being tested.
For example: stdlib/builtins/check_dict.py
.
How the tests work¶
Below is a relevant section from pypy’s testing readme.md
The code in this directory is not intended to be directly executed. Instead, type checkers are run on the code, to check that typing errors are emitted at the correct places.
Some files in this directory simply contain samples of idiomatic Python, which should not (if the stubs are correct) cause a type checker to emit any errors.
Many test cases also make use of
assert_type
,
a function which allows us to test whether a type checker’s inferred type of an
expression is what we’d like it be.
Finally, some tests make use of # type: ignore
comments (in combination with
mypy’s
--warn-unused-ignores
setting and pyright’s
reportUnnecessaryTypeIgnoreComment
setting) to test instances where a type checker should emit some kind of
error, if the stubs are correct. Both settings are enabled by default for the entire
subdirectory.
For more information on using assert_type
and
--warn-unused-ignores
/reportUnnecessaryTypeIgnoreComment
to test type
annotations,
this page
provides a useful guide.
Caching of packages¶
In order to reduce the time needed to run the tests of the snippets, we cache the packages in the .pytest_cache
folder.
this makes use of the pytest_cache
plugin : https://pypi.org/project/pytest-cache/
The cache lifetime for each package is 24 hours, after which it will be re-downloaded. The cache can be cleared by:
running the following command:
pytest --clearcache
deleting the
.pytest_cache
folder