Skip to content

Code Quality Infrastructure

This document describes the full code quality infrastructure implemented in the Opifex framework, including pre-commit hooks, static analysis, testing, and enterprise-grade standards.

🎯 Overview

The Opifex framework maintains enterprise-grade code quality through a full infrastructure that ensures:

  • Perfect Compliance: 19/19 pre-commit hooks passing with zero errors/warnings
  • Static Analysis: Complete type safety with pyright and code quality with ruff
  • Security: Automated security scanning with bandit
  • Documentation: Consistent documentation standards with pydocstyle
  • Testing: Extensive test suite with high pass rate

🔧 Pre-commit Infrastructure

Pre-commit Hooks Configuration

The framework uses a full .pre-commit-config.yaml with the following hooks:

1. File Management & Formatting

# Trim trailing whitespace
- repo: https://github.com/pre-commit/pre-commit-hooks
  hooks:
    - id: trailing-whitespace
    - id: end-of-file-fixer
    - id: check-merge-conflicts
    - id: check-added-large-files

2. Configuration Validation

# YAML, TOML, and JSON validation
- repo: https://github.com/pre-commit/pre-commit-hooks
  hooks:
    - id: check-yaml
    - id: check-toml
    - id: check-json

3. Python Code Quality

# Ruff for linting and formatting
- repo: https://github.com/astral-sh/ruff-pre-commit
  hooks:
    - id: ruff
      args: [--fix, --exit-non-zero-on-fix]
    - id: ruff-format

# Pyright for type checking
- repo: https://github.com/RobertCraigie/pyright-python
  hooks:
    - id: pyright

4. Security Analysis

# Bandit for security scanning
- repo: https://github.com/PyCQA/bandit
  hooks:
    - id: bandit
      args: ['-c', 'pyproject.toml']

5. Documentation Standards

# pydocstyle for documentation compliance
- repo: https://github.com/PyCQA/pydocstyle
  hooks:
    - id: pydocstyle

6. Jupyter Notebook Quality

# nbqa-ruff for notebook linting
- repo: https://github.com/nbQA-dev/nbQA
  hooks:
    - id: nbqa-ruff

7. Shell Script Validation

# shellcheck for shell script quality
- repo: https://github.com/shellcheck-py/shellcheck-py
  hooks:
    - id: shellcheck

Running Pre-commit Hooks

Manual Execution

# Run all hooks on all files
uv run pre-commit run --all-files

# Run specific hook
uv run pre-commit run ruff --all-files
uv run pre-commit run pyright --all-files

# Run hooks on staged files only
uv run pre-commit run

Automatic Execution

Pre-commit hooks run automatically on every commit:

# Install pre-commit hooks (done during setup)
uv run pre-commit install

# Hooks will run automatically on git commit
git commit -m "Your commit message"

🔍 Static Analysis

Pyright Type Checking

Configuration

The framework uses a full pyproject.toml configuration for pyright:

[tool.pyright]
include = ["src", "tests"]
exclude = [
  "**/__pycache__", "**/.venv", "scripts", "memory-bank",
  "documents", "examples",
]
pythonVersion = "3.11"
pythonPlatform = "Linux"
typeCheckingMode = "basic"
reportMissingImports = true
reportMissingTypeStubs = false
reportUnusedImport = true
reportDuplicateImport = true
reportOptionalSubscript = "warning"
reportOptionalMemberAccess = "warning"
reportOptionalCall = "warning"

Key Features

  • Basic Type Checking: Type safety tuned for JAX/scientific computing compatibility
  • JAX Integration: Native support for JAX arrays and transformations
  • FLAX NNX Compatibility: Full type coverage for neural network components
  • SQLAlchemy Integration: Type-safe database operations
  • Scientific Computing Types: Specialized type annotations for scientific computing

Type Safety Achievements

# Example of full type annotations
from jax import Array
from jaxtyping import Float, Complex
from flax import nnx

def neural_operator_forward(
    model: nnx.Module,
    x: Float[Array, "batch spatial_dims channels"],
    training: bool = False
) -> Float[Array, "batch spatial_dims output_channels"]:
    """Type-safe neural operator forward pass."""
    return model(x, training=training)

Ruff Code Quality

Configuration

[tool.ruff]
target-version = "py311"
line-length = 100
src = ["src/opifex"]
extend-include = ["*.ipynb"]

[tool.ruff.lint]
select = [
    "E", "W", "F", "UP", "I001", "N", "S", "B", "A",
    "COM", "C4", "ICN", "PIE", "PT", "Q", "RSE", "RET",
    "SIM", "PL", "TRY", "NPY", "RUF",
    # ... and more (see pyproject.toml for full list)
]
ignore = [
    "E741",    # Ambiguous variable name (mathematical variables)
    "PLR0913", # Too many arguments (scientific functions)
    # ... (see pyproject.toml for full list)
]

Key Features

  • Code Formatting: Consistent code style across the entire codebase
  • Import Sorting: Automatic import organization with isort
  • Complexity Analysis: Detection of overly complex functions and classes
  • Bug Detection: Identification of common Python bugs and anti-patterns
  • Performance Optimization: Suggestions for performance improvements

Recent Fixes Applied



# Before: Line length violation
very_long_variable_name = some_function_with_many_parameters(param1, param2, param3, param4, param5)

# After: Proper line breaking
very_long_variable_name = some_function_with_many_parameters(
    param1, param2, param3, param4, param5
)

# Before: Complex function with 21 branches
def _check_boundary_conditions(self, ...):
    # 51 statements with 21 branches

# After: Refactored into helper methods
def _check_boundary_conditions(self, ...):
    # 10 statements with 2 branches
    return self._check_dirichlet_boundary_condition(...)

def _check_dirichlet_boundary_condition(self, ...):
    # Extracted helper method

🔒 Security Analysis

Bandit Security Scanning

Configuration

[tool.bandit]
exclude_dirs = ["tests", "scripts", "memory-bank", "documents", "custom_modes", ".cursor"]
skips = ["B404", "B603", "B607", "B602", "B110", "B403", "B301"]

Security Features

  • SQL Injection Prevention: Detection of potential SQL injection vulnerabilities
  • Command Injection Prevention: Identification of unsafe shell command usage
  • Cryptography Best Practices: Validation of cryptographic implementations
  • File System Security: Detection of unsafe file operations
  • Network Security: Identification of insecure network operations

📚 Documentation Standards

pydocstyle Compliance

Configuration

[tool.pydocstyle]
convention = "google"
add-ignore = ["D100", "D104", "D105", "D107"]

Documentation Requirements

  • Google Style Docstrings: Consistent documentation format
  • Type Annotations: Complete type information in function signatures
  • Examples: Code examples in docstrings where appropriate
  • Mathematical Notation: LaTeX formatting for mathematical expressions

Example Documentation

def fourier_neural_operator(
    x: Float[Array, "batch spatial_dims channels"],
    modes: int = 12,
    width: int = 64
) -> Float[Array, "batch spatial_dims output_channels"]:
    """Fourier Neural Operator for learning solution operators.

    This function implements the Fourier Neural Operator (FNO) architecture
    for learning mappings between function spaces. The FNO uses spectral
    convolutions in the Fourier domain to capture global dependencies.

    Args:
        x: Input tensor with spatial dimensions and channels.
        modes: Number of Fourier modes to retain in spectral convolution.
        width: Hidden dimension width for the neural network layers.

    Returns:
        Output tensor with the same spatial dimensions but potentially
        different number of channels.

    Example:
        >>> import jax.numpy as jnp
        >>> x = jnp.ones((32, 64, 64, 3))  # batch=32, spatial=64x64, channels=3
        >>> y = fourier_neural_operator(x, modes=12, width=64)
        >>> print(y.shape)  # (32, 64, 64, 1)

    References:
        Li, Z., et al. "Fourier Neural Operator for Parametric Partial
        Differential Equations." ICLR 2021.
    """
    # Implementation here...

🧪 Testing Infrastructure

Test Coverage

Test Categories

  1. Unit Tests: Individual component testing
  2. Integration Tests: Component interaction testing
  3. End-to-End Tests: Complete workflow testing
  4. Performance Tests: Benchmarking and optimization validation
  5. Regression Tests: Prevention of functionality regressions

Test Configuration

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "-vv",
    "--tb=short",
    "--strict-markers",
    "--disable-warnings",
    "--cov=src/opifex",
    "--cov-report=html",
    "--cov-report=term-missing",
    "--cov-fail-under=50",
    "--timeout=300",
]
markers = [
    "slow: marks tests as slow",
    "gpu: marks tests as requiring GPU",
    "integration: marks tests as integration tests",
    "benchmark: marks tests as performance benchmarks",
]

Running Tests

# Run all tests
uv run pytest tests/ -v

# Run with coverage
uv run pytest tests/ --cov=src/opifex --cov-report=html

# Run specific test categories
uv run pytest tests/core/ -v          # Core tests
uv run pytest tests/neural/ -v        # Neural network tests
uv run pytest tests/integration/ -v   # Integration tests
uv run pytest -m "not slow" -v       # Skip slow tests
uv run pytest -m gpu -v              # GPU tests only

🔄 Continuous Integration

GitHub Actions Integration

The pre-commit configuration is designed to match GitHub Actions CI/CD pipeline:

# .github/workflows/ci.yml (example)
name: CI
on: [push, pull_request]
jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          uv sync
      - name: Run pre-commit
        run: uv run pre-commit run --all-files
      - name: Run tests
        run: uv run pytest tests/ -v

Local Development Workflow

  1. Setup: Install pre-commit hooks during environment setup
  2. Development: Write code with type annotations and documentation
  3. Commit: Pre-commit hooks run automatically, fixing issues
  4. Push: Code passes all quality checks before reaching CI/CD
  5. Review: Code review focuses on logic and design, not style

📊 Quality Metrics

🛠️ Maintenance

Regular Quality Checks

# Daily quality check
uv run pre-commit run --all-files

# Weekly full check
uv run pytest tests/ --cov=src/opifex
uv run pre-commit autoupdate

# Monthly dependency update
uv sync --upgrade
uv run pre-commit run --all-files

Quality Standards Enforcement

  1. No Commits Without Passing Hooks: Pre-commit hooks prevent commits with quality issues
  2. Type Safety Required: All new code must include proper type annotations
  3. Documentation Required: All public functions must have docstrings
  4. Security Validation: All code must pass security scanning
  5. Test Coverage: New features must include full tests

🎯 Best Practices

Development Guidelines

  1. Type Annotations: Use full type annotations with jaxtyping
  2. Documentation: Write clear docstrings with examples
  3. Testing: Include unit and integration tests for new features
  4. Error Handling: Implement robust error handling and validation
  5. Performance: Consider performance implications and optimize when necessary

Code Style Guidelines

  1. Line Length: Maximum 100 characters (enforced by ruff)
  2. Import Organization: Automatic sorting with isort
  3. Function Complexity: Keep functions simple and focused
  4. Variable Naming: Use descriptive names following PEP 8
  5. Comments: Use comments sparingly, prefer self-documenting code

Security Guidelines

  1. Input Validation: Validate all external inputs
  2. SQL Safety: Use parameterized queries for database operations
  3. File Operations: Validate file paths and permissions
  4. Network Security: Use secure protocols and validate connections
  5. Dependency Management: Keep dependencies updated and secure

📚 Additional Resources