[WIP] How to create a modern pytest dev environment with vscode

python Jun 10, 2022

I started in my new position 1 year ago as a Quality Engineer at Red Hat. I was struggling to find a better way to design, create, and run tests, so I decided to create my own setup from scratch. Here you'll check my own development workflow for pytest on Visual Studio Code.

Before we start, a big thank you to Lucas Aoki, who teaches me most of what you'll see in this post.

Why pytest and vscode?

I'm working on a project that has a big amount of code created using the pytest framework, so that's the reason why I'm using pytest. I don't have experience yet to define a testing framework or think about migrating to another one.

Now, about the vscode, it's mostly because is the most flexible development tool that I ever used, allowing me to mix and match my extensions to work with Vagrant, Podman, Docker, Kubernetes, etc. Unfortunately is not entirely open source and I have hopes that we will have some alternative in the future, but in the end, I don't think this matters too much, considering the alternative I can imagine would be JetBrain's PyCharm, an even closer and harder to extend IDE.

Let's create an application example

You can check the code that I created here (if you want to just clone it):
- https://github.com/thenets/study-pytest

.
├── src
│   ├── __init__.py
│   ├── main.py

The __init__.py file will be e mpty. This file is required anyway. It will be used later on to allow the FastAPI app object to be imported by the pytest because this empty file makes the src/ dir be interpreted as a class.

The main.py file is our application. It's a simple web application created using the FastAPI framework.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


@app.get("/")
async def root():
    return {"message": "Hello World"}


@app.get("/contact")
async def contact():
    return {"message": "Página de contato"}


# Create user model from BaseModel
class User(BaseModel):
    username: str
    password: str


# Create user from User model
@app.post("/user/", response_model=User)
async def create_user(user: User):
    return {"username": user.username, "password": user.password}


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)

./src/main.py

Our dependency file and virtual env

I created two dependency files, one that contains the application dependencies and the other that contains all the development and testing dependencies, but also import the first one as well.

.
├── requirements.txt
├── requirements-dev.txt

pip
fastapi
uvicorn

requirements.txt

-r requirements.txt
pytest
black
tox
requests

requirements-dev.txt

The pytest files

Maybe you are new in the QA world just like, so I'll try to give you a little more explanation about the test files.

.
├── pytest.ini
├── tests
│   ├── __init__.py
│   ├── conftest.py

The pytest.ini is the main file for pytest settings. You have different ways to configure a pytest run, you can check all those options and priorities over each other at official docs:
- https://docs.pytest.org/en/6.2.x/customize.html

I already comment below on the options that I used, but I think it's important to mention the addopts option to avoid pass all the pytest arguments for each run. For example the --strict-markers that will make sure you and all your team will be forced to use only the markers defined in this config file.

[pytest]
; directories containing tests
testpaths =
    tests

; force pattern for test content
python_files = test_*.py
python_functions = test_*
python_classes = Test*
python_methods = test_*

; equivalent to pass the argument to pytest CLI
addopts =
    ; increase verbosity
    --verbose
    ; same as -s. per-test capturing method: one of fd|sys|no|tee-sys.
    --capture=no
    ; fail if there are markers not registered in this file
    --strict-markers

; define all the possible markers
markers =
    get
    post

./pytest.ini

Now, the conftest.py is a dir-scoped file for your fixtures, allows you to load external plugins and specify hooks (e.g. setup and teardown hooks).

You can learn more about conftest.py at this Stack Overflow thread:
- https://stackoverflow.com/a/34520971/6530336

Everything defined in this file will be available for the test_*.py files in this directory and all subdirectories.

from fastapi.testclient import TestClient

import pytest

from src import main


@pytest.fixture(scope="session")
def app():
    """Fixture to get FastAPI app"""
    return main.app


@pytest.fixture(scope="session")
def client(app):
    """Fixture to get FastAPI client"""
    return TestClient(app)

./tests/conftest.py

My test files

My application is a REST API server, so I have two major types of endpoint: static and resource based. The static is a simple returned text (e.g. version endpoint). The resource-based endpoint is the REST API itself, you have all the possible interactions with the resource. For example, considering we have a resource type called user, then we can have an endpoint to add, remove, list, and change a user.

This Thoghtworks post is a good introduction
https://www.thoughtworks.com/insights/blog/rest-api-design-resource-modeling

I'm talking about REST API in this pytest post because the test files and how it'll be structured is tightly related to the application itself. That means you will need to get some abstractions from the application architecture to architect your test suite.

.
├── tests
│   ├── test_static_endpoint.py
│   ├── test_user.py

import pytest


@pytest.mark.get
def test_static_root(client):
    response = client.get("/")
    assert response.status_code == 200
    assert response.json() == {"message": "Hello World"}


@pytest.mark.get
def test_static_contact(client):
    response = client.get("/contact")
    assert response.status_code == 200
    assert response.json() == {
        "message": "Hi, I'm Luiz. You can reach me at https://thenets.org"
    }

./tests/test_static_endpoint.py

In this test_user.py file there's an important pytest feature, the parametrize. This allows you to run the same function multiple times using different parameters. A simple and easy to ready approach for input validation, for example.

import pytest


@pytest.mark.post
@pytest.mark.parametrize(
    "username, password, expected_value, expected_bool",
    [
        ("zezinho", "xuxa", {"username": "zezinho", "password": "xuxa"}, True),
        ("kratos", "freya", {"username": "kratos", "password": "freya"}, True),
        ("admin", "admin", {"username": "admin", "password": "xuxa"}, False),
    ],
)
def test_user_create(
    client,
    username: str,
    password: str,
    expected_value: dict,
    expected_bool: bool,
):
    """Test user creation

    Steps:
        1. Send a POST with username and password
        2. Check if the response is the expected one
    """
    response = client.post(
        "/user/", json={"username": username, "password": password}
    )
    assert response.status_code == 200
    assert (response.json() == expected_value) == expected_bool

./tests/test_user.py

Visual Studio Code files

Now, we will finally check the vscode features. You'll need only a single configuration file. This one, depending on how your team works, could even be committed to a Git repository and shared across the team.

Considering the .json file doesn't support comments, I'll describe each option in the following list:

python.testing.pytestEnabled: enable all the pytest features if the python plugin is installed.
python.defaultInterpreterPath: default Python interpreter to be used. Very useful to define a Python binary from a virtualenv.
python.linting.pycodestyleEnabled: enables the code style features. It shows alerts when you have something that does not respect the defined code style.
python.linting.pycodestyleArgs: define a code style to be respected by the vscode. The arg --max-line-lenght 80 it's used by the linter whenever you have lines longer than 80 characters.
editor.rulers: Creates a vertical line after each number in the list. Guides you to not create longer lines.

.
├── .vscode
│   ├── settings.json

{
    "python.testing.pytestEnabled": true,
    "python.defaultInterpreterPath": "~/venvs/learn/bin/python",
    "python.linting.pycodestyleEnabled": true,
    "python.linting.pycodestyleArgs": [
        "--max-line-lenght 80"
    ],
    "editor.rulers": [
        80
    ]
}

./.vscode/settings.json

Visual Studio Code dev tools

There is one plugin that I enjoy too much about the vscode, the Remote development using the SSH plugin. This is always my main approach to create a controlled environment to do my development.

I create a VM using Vagrant, install everything that I need using the Ansible provisioner, and jump into the VM using this plugin. Doesn't even matter my host OS, I always have my fresh new Linux waiting for me.

With that plugin, I'm even able to forward the development machine to my localhost port, allowing me to easily reach web development servers, for example, using a secure and encrypted channel.

WIP PUT A VIDEO HERE