General Guide: Python Versions

 

Avoid working on and suggesting migrations to any project developed in Python 2 due to encountering EOL. Starting with Python 3, the changes introduced do not usually break implementations, although some caution should be taken if the difference in versions is very high.

 

It is also important to stay informed of the status and timeline of each version by visiting https://devguide.python.org/versions/. At the time of this guideline Python 3.11 is a good option for new projects.

 

Python style guide

 

PSF (Python Foundation) suggests following a style guide known as PEP 8 (https://peps.python.org/pep-0008/). It is a long but recommended read; However, there is also PEP 20 or "Zen of Python" (https://peps.python.org/pep-0020/#the-zen-of-python) that helps to understand the rules of the language at a higher level.

 

These two PEPs greatly influence how the code in Python is written and structured. It is not necessary to learn these standards completely, there is a package called pycodestyle that helps to follow PEP 8. There are also others that extend the functionality of pycodestyle, although with various "flavors" or rules (flake8, black) and others for cleanliness of code and maintain order ( isort ). A combination of these packages serves as a good basis for the development of CI pipelines.

 

In the case of Rootstack, the default package to use for code formatting will be black, which validates with PEP8 and offers to view the recommended changes (It should not be used for automatic changes, but rather as a guide). It is necessary to set the -l 79 option to black so that it follows the PEP 8 standard in terms of line length.

 

Example:

 

black -l 79 --check --diff

 

 

Using static types

 

Use static types in the definition of functions or methods if the technology/framework allows it. In the case of variables, it is not necessary to note the type if it is used only locally. Attributes of classes and objects must remain with their annotated types. To validate the static types noted in the code we use mypy.

 

Configuration example (mypy.ini):

 

[mypy]
python_version = "3.10"
# Do not return Any
warn_return_any = True
warn_unused_configs = True
# Functions must have a return type
disallow_untyped_refs = True
disallow_untyped_calls = True
# Necessary in some cases for libraries that do not include types
ignore_missing_imports = True

 

The code should be expressive enough not to need comments; However, it is acceptable in case of clarifications of "non-standard" implementations, linked to a specific need (for example, a task requires it). Methods and functions must be commented with a brief description of the function, its parameters and return value.

 

 

Use a virtual environment (virtualenv)

 

A virtual environment provides an isolated environment, where Python packages and programs are installed apart from the installations performed globally at the system level; This makes it easier to better control requirements per project.

 

One of the most used packages to manage virtual environments is virtualenv and can be acquired through pip. Assuming that pip is already installed on the system (python3-pip in the case of Ubuntu), the following are instructions for basic use of the virtualenv package.

 

# Install the package globally
pip install virtualenv
# Create and "activate" the virtual environment, adding the binaries (`.venv/bin`)
to the PATH and setting other Python variables.
# The "-p" option selects the Python3 executable as the
Python for this virtual environment.
# The virtual environment will save its files in `.venv`
virtualenv -p /usr/bin/python3 .venv
source .venv/bin/activate

 

After this, any calls to python or pip will install and use the packages in the active virtual environment. That is, performing a pip install -r requirements.txt will only download the dependencies in this environment; making it easier to manage multiple projects with different dependencies.

 

# Stop using the isolated environment.
.venv/bin/deactivate

 

Normally each project would use its own virtual environment, but it may also be the case that projects with similar dependencies or that depend on each other are worked on. In these cases there is no problem in maintaining the same active environment while working on both projects.

 

Define a dependencies file

 

The main rule is to maintain a requirements.txt and a requirements-dev.txt for the functional and development dependencies of the project.

 

This file should list the dependencies and versions used in the project. For new projects, these can be obtained by running pip freeze > requirements.txt within the project's virtual environment. From here you can separate into development dependencies and update versions.

 

Note: Running this command outside of a virtual environment may list dependencies on other projects, even installed by the system package manager, and is not the optimal way to obtain project dependencies.

 

For development dependencies such as pytest or black , a separate requirements-dev.txt file must be created. This way we can install the packages necessary for execution, without having to include other test packages.

 

It is common practice to also include the requirements.txt file within the development dependencies, especially if you are including tests that depend on the functionality of the application.

 

-r requirements.txt
pytest>=7.4.0

 

Code Structure

 

For this section it is important to be clear about the concept of module vs package. A module is each of the .py files in the project, they can contain multiple definitions of classes or functions. Packages are used to structure/group modules within an application.

 

As a general rule, packages are used to separate modules according to the purpose (or domain) they have within the application (controllers, config, services...), while the modules of each one indicate the specificity (user service , publishing service). How they are grouped at the directory level and the nomenclature may vary between frameworks, but it usually follows the same principle.

 

With these clear concepts, namespaces will define the interface of the packages, what classes and functions are available and how other modules will access them.

 

Example:

├── README.md
├── package
│ ├── __init__.py
│ ├── app.py
│ ├── subpackage1
│ │ ├── __init__.py
│ │ └── config.py
│ └── subpackage2
│ ├── __init__.py
│ └── user_service.py
├── requirements-dev.txt
├── requirements.txt
└── tests
└── __init__.py
5 directories, 10 files

 

 

Common Bad Practices

 

Mutable arguments in method definitions

 

Method arguments get their value at run time, causing multiple calls to the same method or function to share the value of the variable. Example:

 

def test(mutable_argument: dict = {}) -> None:
pass

 

In this case, if the mutable_argument dictionary is modified within the function, the value will change in different executions since mutable_argument is defined at a higher scope. The correct version would be to define it as None or for a better understanding of the IDEs, the use of typing.Optional to indicate the correct type would be indicated.

 

Another option starting with Python 3.10+ is to use the Union type, now included in the language standard.

 

 

Import complete packages

 

Taking the previous error as an example, it is also important to highlight the import made of the typing package. The "Zen of Python" says it ( import this ) "Explicit is better than implicit."

 

The Optional module is specifically imported, rather than just using import typing or from typing import * . This is good practice to avoid conflicts and be clear about the dependencies of our project. Thus, if another package has the Optional module, it is explicit that the typing module is being imported and if both are required, one can act accordingly.

 

In the latter case, you must make sure to always use the namespace where Optional is located. Having to be called typing.Optional within the code.

 

Common Tasks/Packages

 

Logging

 

Logging in Python is built in as part of the language standard in the logging module. In general, all other modules are developed with this package in mind and expect the main application to use it as well.

 

Example of format configuration in a specific logger:

 

import logging
def main():
logger = logging.getLogger()
handler = logging.StreamHandler() # Stream for Unix streams, too
can be configured to write files.
formatter = logging.Formatter("%(asctime)s:%(levelname)s:" "%(name)-0s:
%(message)s")
logger.setLevel(logging.INFO)
handler.setFormatter(formatter) # The formatter is tied to the handler.
logger.addHandler(handler)
logging.info("TEST")
logging.info(['this', 'is', 'a list'])

 

Testing

 

The tests are created in the tests/ directory of the project and executed with pytest . This allows direct integration with tests that use the unittest library, taking into account that the final evaluations will be done mostly with assert.

 

The exception to this rule is the case of test objects, created using unittest.mock , where evaluations with mock_func.assert_* ( assert_called , assert_called_with , etc...) apply.

 

Coverage

 

To review code coverage in relation to testing, we use a Pytest plugin that is installed under the pytest-cov package. Report generation is enabled by passing the --cov=[path to code] option in the Pytest command as shown below.

 

The other options for the report format or acceptable percentage of coverage, which will depend on the project, can be found in the reference of this section.

 

We recommend you on video