Discover some way to see which files are being tested in runtime

[Igorxp5]

See if we can do the same as original Ekstazi do in Java, observing the files that are being used in runtime. If we can't do that, our next approach would be read the test files before running the test suite, and check import files and their usage in the test cases.

The next steps depends on the success of the research, we would create a function to get dependecies of each test case in a test file, or create a wrapper to execute the tests and collect their dependencies.

[Igorxp5]

Observing the files that are being used in runtime may be possible using Python's trace module.

Parse Python files for checking its depedencies cabe done by using Python's ast module.

[Igorxp5]

It's possible observe the functions call in runtime by using trace module. See below a brief usage of the trace module in a pytest conftest.py file.

Tests file:

from dependency import method_1
from dependency_2 import method_2
from dependency_3 import method_3

def test_method_1():
    assert method_1() == 'I am the method 1'

def test_method_2():
    assert method_1() == 'I am the method 1'
    assert method_2() == 'I am the method 2'

def test_method_3():
    assert False

conftest:

import sys
import trace
import functools

tracers = []


def pytest_pyfunc_call(pyfuncitem):
    test_function = pyfuncitem.obj

    # Trace each test individually 
    tracer = trace.Trace(trace=0, count=1, countfuncs=1, ignoredirs=[sys.prefix, sys.exec_prefix])
    tracers.append(tracer)

    @functools.wraps(test_function)
    def tracer_wrapper(*args, **kwargs):
        tracer.runfunc(test_function, *args, **kwargs)
    
    pyfuncitem.obj = tracer_wrapper


def pytest_sessionfinish(session, exitstatus):
    for i, tracer in enumerate(tracers):
        results = tracer.results()
        calls = sorted(results.calledfuncs)[:-1]  # ignore the test function call itself
        for filename, modulename, funcname in calls:    
            # ignore Python internal calls
            if not filename.startswith(sys.prefix) and not filename.startswith(sys.exec_prefix):
                print(f'Test #{i + 1} file dependency:', filename)

Output:

Test #1 file dependency: C:\Users\Igorxp5\Desktop\trace\dependency.py
Test #2 file dependency: C:\Users\Igorxp5\Desktop\trace\dependency.py
Test #2 file dependency: C:\Users\Igorxp5\Desktop\trace\dependency_2.py

[Igorxp5]

By the results we can implement Ekstazi as the original. Let's create a Pytest Plugin for Ekstazi algorithm.