Plugins

This page describes how to use and write plugins for checker pipelines.

You can refer to the course-template repository for examples of plugins usage and custom plugins development.

What is the Plugin

Plugin is a single stage of the pipeline, have arguments, return exclusion result. In a nutshell, it is a Python class overriding abstract class checker.plugins.PluginABC:

Bases: ABC

Abstract base class for plugins. :ivar name: str plugin name, searchable by this name

Args

Bases: BaseModel

Base class for plugin arguments. You have to subclass this class in your plugin.

run(args, *, verbose=False)

Run the plugin. :param args: dict plugin arguments to pass to subclass Args :param verbose: if True should print teachers debug info, if False student mode :raises BadConfig: if plugin arguments are invalid :raises ExecutionFailedError: if plugin failed :return: PluginOutput with stdout/stderr and percentage

validate(args) classmethod

Validate the plugin arguments. :param args: dict plugin arguments to pass to subclass Args :raises BadConfig: if plugin arguments are invalid :raises BadStructure: if _run method is not implemented

Note that each plugin should override checker.plugins.PluginABC.Args class to provide arguments validation. Otherwise, empty arguments will be passed to run method.

Bases: BaseModel

Base class for plugin arguments. You have to subclass this class in your plugin.

Each plugin output checker.plugins.PluginOutput class when executed successfully.

Plugin output dataclass. :ivar output: str plugin output :ivar percentage: float plugin percentage

In case of error, checker.exceptions.PluginExecutionFailed have to be raised.

Bases: TestingError

Exception raised when plugin execution failed

Note

Base Plugin class will handle all ValidationErrors of Args and raise error by itself.
So try to move all arguments validation to Args class in pydantic way.

How to use plugins

Plugins are used in the pipelines described in .checker.yml file. When running a pipeline the checker will validate plugin arguments and run it.

The following plugins are available out of the box:

TBA

How to write a custom plugin

To write a custom plugin you need to create a class inheriting from checker.plugins.PluginABC and override _run method, Args inner class and set name class attribute.

from random import randint
from checker.plugins import PluginABC, PluginOutput
from checker.exceptions import PluginExecutionFailed
from pydantic import AnyUrl

class PrintUrlPlugin(PluginABC):
    """Plugin to print url"""

    name = "print_url"

    class Args(PluginABC.Args):
        url: AnyUrl

    def _run(self, args: Args, *, verbose: bool = False) -> PluginOutput:
        if randint(0, 1):
            if verbose:
                raise PluginExecutionFailed("Verbose error, we got randint=1")
            else:
                raise PluginExecutionFailed("Random error")

        return PluginOutput(
            output=f"Url is {args.url}",
            percentage=1.0,  # optional, default 1.0 on success
        )

Important

The Plugin must implement verbose functionality!
If verbose is True the plugin should provide all info and possible debug info.
If verbose is False the plugin should provide only public-friendly info, e.g. excluding private test output.

Note

It is a nice practice to write a small tests for your custom plugins to be sure that it works as expected.