Skip to main content

Testing Environment Options for Tagion tools

This document explores various options for setting up a testing environment for Tagion tools, considering the programming languages already used in the project (D-lang, Bash, Python, and Jest with NodeJS+TypeScript). Each option is evaluated based on convenience to write tests, simplicity of setup, organization for many tests, and reporting capabilities.

Testing options​

Bash Scripting​

Pros:

  • Direct execution on Linux, no extra setup.
  • Simple to invoke binaries and compare outputs.

Cons:

  • Limited flexibility for complex parsing/comparison.
  • Basic error handling and reporting.

Evaluation:

  • Convenience to write test: 7
  • Simplicity of setup: 10
  • Organization for many tests: 5
  • Reporting: 3
  • Average: 6.25

Minimal Example:

#!/bin/bash
expected_output="expected.txt"
actual_output="actual.txt"
./your_binary inputfile > "$actual_output"
if diff "$expected_output" "$actual_output"; then
    echo "Test passed"
else
    echo "Test failed"
fi

D-lang with std.process​

Pros:

  • Seamless integration with D-lang projects.
  • Powerful language features for complex tests.

Cons:

  • Manual organization for tests required.

Evaluation:

  • Convenience to write test: 7
  • Simplicity of setup: 8
  • Organization for many tests: 6
  • Reporting: 5
  • Average: 6.5

Minimal Example:

import std.process;
import std.stdio;
import std.file;

void main() {
    auto expectedOutput = readText("expected.txt");
    auto actualOutput = executeShell("./your_binary inputfile");
    assert(actualOutput.output == expectedOutput, "Test failed");
}

pytest with Python​

Pytest is a mature full-featured Python testing tool that helps you write better programs. It simplifies the creation, organization, and execution of tests, including complex functional testing.

Pros:

  • Easy to start with due to its simple syntax for writing tests.
  • Powerful fixture system for setup and teardown, which is particularly useful for pre-running processes or configurations.
  • Supports parameterized tests and can run tests in parallel.
  • Rich plugin architecture for extending functionality.
  • Excellent support for different types of tests, from unit to integration and end-to-end tests.
  • Automatic test discovery.
  • Detailed and customizable reports, outputting to both console and files.

Cons:

  • Requires familiarity with Python.
  • Environment setup involves creating a Python virtual environment and installing dependencies.

Evaluation:

  • Convenience to write test: 9
  • Simplicity of setup: 7
  • Organization for many tests: 9
  • Reporting: 9
  • Average: 8.5

Minimal Example (Test File Skeleton):

import pytest

def test_feature_1():
    assert True

def test_feature_2():
    assert True

Jest with NodeJS and TypeScript​

Jest is a delightful JavaScript Testing Framework with a focus on simplicity. It works with projects using: Babel, TypeScript, Node, React, Angular, Vue, and more. It's well-suited for JavaScript and TypeScript projects, making it a popular choice for frontend and backend testing.

Pros:

  • Zero configuration for many projects, with automatic discovery of test files.
  • Built-in code coverage reports, with support for console and file outputs.
  • Rich mocking, spying, and test isolation features.
  • Supports testing asynchronous code out of the box.
  • Integrated with modern JavaScript ecosystems.

Cons:

  • Primarily focused on the JavaScript/TypeScript ecosystem, might not be ideal for non-JS projects.
  • Can become slow in large projects without proper configuration.

Evaluation:

  • Convenience to write test: 9
  • Simplicity of setup: 8
  • Organization for many tests: 9
  • Reporting: 9
  • Average: 8.75

Minimal Example (Test File Skeleton):

Creating two test files for different tools, toolA and toolB, with two example tests in each:

tests/toolATests.test.ts

describe('Tool A Tests', () => {
  test('Feature 1 should work', () => {
    // Test implementation
  });

  test('Feature 2 should work', () => {
    // Test implementation
  });
});

Summary​

When scaling up to about 10 tests for each command-line tool, organization and maintenance become crucial. Scripting solutions like Bash and Makefiles might start simple but can quickly become unwieldy as complexity grows. Python and Jest, with their structured testing frameworks, offer more scalability and maintainability, making them suitable for larger test suites. D-lang provides a middle ground, with strong language features but potentially requiring more manual organization.

Each option's ability to handle multiple tests effectively varies, with Jest and Python offering more structured approaches that scale better as the number of tests increases. Bash and D-lang, while capable, may require more manual effort to maintain clarity and organization as the suite expands.

Pytest vs. Jest Comparison​

When comparing Pytest and Jest for a project with about 5-10 tools and up to 10 tests for each tool, several factors are crucial, including the complexity of organization, reporting capabilities, environment setup, and the ability to pre-run processes or configurations for tests.

Organization​

Pytest:

  • Test files and functions are automatically discovered based on naming conventions.
  • Supports structuring tests in a modular way using directories and files.
  • The fixture system provides a powerful way to set up and tear down configurations or dependencies.

Jest:

  • Similar to Pytest, Jest discovers tests based on naming conventions and supports organization using directories and files.
  • Jest's setup and teardown mechanisms are managed through global or individual test lifecycle hooks.

Both frameworks support a clean and scalable organization of tests, but Pytest's fixture system is exceptionally versatile for managing dependencies and state.

Reporting​

Pytest:

  • Offers detailed reports in the console, highlighting failed tests with specific error messages.
  • Supports generating reports in various formats, including HTML, through plugins.

Jest:

  • Provides an interactive watch mode with clear output in the console, including a summary of test suites and individual tests.
  • Capable of outputting coverage reports in various formats directly.

Both Pytest and Jest offer excellent reporting capabilities, with both console and file outputs. Pytest's plugin system and Jest's built-in coverage tool are highlights.

Environment Setup​

Pytest Setup on a fresh Ubuntu:

# Update packages and install Python and pip
sudo apt update
sudo apt install python3 python3-pip -y

# (Optional but recommended) Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies from requirements.txt
pip install -r requirements.txt

# Run tests with Pytest
pytest

Jest Setup on a fresh Ubuntu:

# Update packages and install Node.js and npm
sudo apt update
sudo apt install nodejs npm -y

# Install project dependencies including Jest
npm install

# Run tests with Jest
npm test

Pytest requires Python-specific setup, while Jest requires Node.js ecosystem setup. The complexity is similar, but the familiarity with the respective language's environment might sway the preference.

Handling Pre-run Processes​

Pytest:

  • Can use fixtures to start and stop background processes or perform setup tasks before running tests.

Jest:

  • Utilizes global setup/teardown files or beforeEach/afterEach hooks for similar purposes.

Both frameworks provide mechanisms to manage pre-run processes, but Pytest's fixtures offer more granularity and control.

Summary​

Choosing between Pytest and Jest largely depends on the primary technology stack of the project and the team's familiarity with Python or JavaScript/TypeScript. For Python-centric projects or when testing requires intricate setup and teardown, Pytest is exceptionally powerful. Jest, being part of the JavaScript ecosystem, is ideal for projects already using Node.js, particularly when uniformity across frontend and backend testing is desired.