Opifex Development Setup Guide¶

This guide covers the development setup for the Opifex framework, including environment configuration, code quality standards, and best practices.

🚀 Quick Setup¶

# 1. Clone and setup environment
git clone https://github.com/avitai/opifex.git
cd opifex
./setup.sh

# 2. Activate environment
source ./activate.sh

# 3. Install pre-commit hooks
uv run pre-commit install

Environment Configuration¶

Activation Requirement¶

Always run source ./activate.sh before uv run pytest or any uv run command. The activation script layers environment files in order:

.venv/bin/activate -- Python virtual environment
.opifex.env -- Managed backend config (generated by setup.sh, do not edit)
.env -- User customizations (CUDA paths, JAX flags, compilation cache)
.env.local -- Local machine overrides (gitignored, optional)

Each layer can override variables from the previous one. The .opifex.env file tracks which variables it manages via OPIFEX_MANAGED_ENV_VARS so they can be reset on reactivation.

Running Tests¶

source ./activate.sh
uv run pytest tests/ -v              # Full suite
uv run pytest tests/neural/ -v       # Specific module
uv run pre-commit run --all-files    # Quality checks

Setup Workflow¶

setup.sh is the single entry point for environment creation:

./setup.sh              # Auto-detect backend (GPU if available)
./setup.sh --backend cpu # Force CPU-only
./setup.sh --force-clean # Clean rebuild

It calls scripts/setup_env.py internally to generate .opifex.env. Do not call setup_env.py directly unless debugging the setup process.

Code Quality & Pre-commit Standards¶

Overview¶

The Opifex framework maintains production-grade code quality through full pre-commit hooks and automated tooling. This ensures consistency, security, and maintainability across the codebase.

Pre-commit Hook Configuration¶

Our pre-commit setup includes:

✅ TOML Formatting: sort_pyproject - Maintains organized dependencies
✅ Python Linting: ruff - 100-character line limit, full rule set
✅ Type Checking: pyright - JAX-compatible type validation
✅ Security Scanning: bandit - Production security standards
✅ Documentation: pydocstyle - Google-style docstring enforcement
✅ Shell Scripts: shellcheck - Bash script quality validation

Common Pre-commit Issues & Solutions¶

1. Line Length Violations (E501)¶

Issue: Lines exceeding 100 characters Solution: Break long comments and function calls across multiple lines

# ❌ Bad - Long comment line
# Shape: (batch, in_channels, spectral_size) @ (in_channels, out_channels, spectral_size) -> (batch, out_channels, spectral_size)

# ✅ Good - Split across multiple lines
# Shape: (batch, in_channels, spectral_size) @ (in_channels, out_channels,
# spectral_size) -> (batch, out_channels, spectral_size)

2. Unnecessary Assignments (RET504)¶

Issue: Variable assignment immediately before return Solution: Return expression directly

# ❌ Bad - Unnecessary assignment
result = computation()
return result

# ✅ Good - Direct return
return computation()

3. Complex Functions¶

Issue: Functions exceeding complexity limits Solution: Extract helper methods

# ❌ Bad - Complex function
def complex_function(self, inputs):
    # 50+ lines of logic
    pass

# ✅ Good - Extracted helpers
def complex_function(self, inputs):
    prepared = self._prepare_inputs(inputs)
    processed = self._process_data(prepared)
    return self._finalize_output(processed)

Long-term Code Quality Strategy¶

1. Preventive Measures¶

IDE Integration: Configure your IDE with ruff and pyright extensions
Editor Settings: Set line length to 100 characters with visual guides
Auto-formatting: Enable format-on-save for consistent style
Pre-commit Installation: Always run uv run pre-commit install in new clones

2. Development Workflow¶

# Daily development workflow
git checkout -b feature/my-feature
source activate.sh
# ... make changes ...
uv run pre-commit run --all-files  # Check before commit
git add src/ tests/  # Stage specific files, not everything
git commit -m "feat: descriptive commit message"

3. Handling Complex Scientific Code¶

Scientific computing often requires flexibility in certain rules:

Mathematical Variables: Single-letter variables (x, y, k) are acceptable in mathematical contexts
Long Parameter Lists: Neural network constructors may have many parameters
Complex Algorithms: Numerical algorithms may have higher complexity
Constants: Magic numbers are acceptable for physical/mathematical constants

Our configuration already accounts for these patterns through targeted rule exclusions.

4. Dependency Management¶

TOML Sorting: The sort_pyproject hook automatically organizes dependencies:

Alphabetical ordering within groups
Consistent table key sorting
Inline table formatting

This ensures:

✅ Easier dependency management
✅ Reduced merge conflicts
✅ Professional configuration appearance

5. Documentation Standards¶

Google-style docstrings are enforced for:

Public modules, classes, and functions
Complex algorithms requiring explanation
API interfaces

Relaxed requirements for:

Test files
Internal helper methods
Magic methods (__init__, __call__)

Emergency Fixes¶

Quick Fix Commands¶

# Fix most formatting issues automatically
uv run pre-commit run --all-files

# Fix specific file
uv run ruff check --fix path/to/file.py

# Check types only
uv run pyright

# Skip pre-commit for emergency commits (use sparingly!)
git commit -m "emergency fix" --no-verify

Configuration Updates¶

If patterns emerge that need rule adjustments:

Analyze the pattern: Is it legitimate scientific computing usage?
Update pyproject.toml: Add targeted rule exclusions
Document the decision: Update this guide with rationale
Test comprehensively: Ensure no quality degradation

Best Practices for Scientific Computing¶

Code Organization¶

# ✅ Good - Clear separation of concerns
class NeuralOperator(nnx.Module):
    def __init__(self, ...):
        # Initialization logic
        pass

    def _validate_inputs(self, x):
        # Input validation helper
        pass

    def _compute_spectral_transform(self, x):
        # Core mathematical computation
        pass

    def __call__(self, x):
        # Main interface - compose helpers
        x = self._validate_inputs(x)
        return self._compute_spectral_transform(x)

Mathematical Comments¶

# ✅ Good - Clear mathematical explanation
# Fourier transform: F[f](k) = ∫ f(x) e^(-2πikx) dx
# Discretized: F[f]_k = Σ_j f_j e^(-2πijk/N)
fourier_coeffs = jnp.fft.fft(input_signal)

Performance-Critical Code¶

# ✅ Good - Document performance considerations
@jax.jit  # JIT compilation for GPU acceleration
def spectral_convolution(x, weights):
    """Spectral convolution with GPU optimization.

    Note: Uses static shapes for XLA compatibility.
    """
    # Implementation with XLA-friendly operations
    pass

Quality Metrics & Monitoring¶

Automated Tracking¶

Pre-commit success rate: Monitor hook pass rates
Code coverage: Maintain >80% for core algorithms
Type coverage: Track type annotation completeness
Documentation coverage: Ensure public APIs are documented

Manual Review Points¶

Algorithm correctness: Peer review for mathematical implementations
Performance impact: Benchmark critical paths
API consistency: Maintain interface standards
Error handling: Full error paths with informative messages

IDE Configuration Recommendations¶

VS Code Settings¶

{
    "python.linting.enabled": true,
    "python.linting.ruffEnabled": true,
    "python.formatting.provider": "none",
    "editor.formatOnSave": true,
    "editor.rulers": [100],
    "python.analysis.typeCheckingMode": "basic"
}

PyCharm Settings¶

Code Style: Set line length to 100
Inspections: Enable Ruff and Pyright
Format on Save: Enable with Ruff formatter
Type Hints: Enable type checking

This full approach ensures the Opifex framework maintains world-class code quality while accommodating the unique requirements of scientific computing and machine learning development.

UV Configuration & Performance Optimization¶

Hardlinking Performance Warning¶

You may see this warning during uv operations:

warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.

Root Cause: This occurs when UV's cache directory and the project directory are on different filesystems (e.g., cache on SSD, project on HDD).

Long-term Solution: The Opifex framework now includes full UV configuration to optimize performance:

Manual Configuration¶

If you experience performance issues with UV hardlinking, manually set:

# Temporary (current session)
export UV_LINK_MODE=copy

# Permanent (add to ~/.bashrc or ~/.zshrc)
echo 'export UV_LINK_MODE=copy' >> ~/.bashrc

Performance Impact Analysis¶

Configuration	Installation Time	Impact
Default (hardlink)	~50-100ms	✅ Optimal when same filesystem
Copy mode	~200-500ms	✅ Consistent across all setups
Cross-filesystem	~800ms-2s	❌ Degraded without copy mode

Recommendation: Use copy mode for consistent, predictable performance.

🔧 Troubleshooting & Verification¶

Manual Troubleshooting Commands¶

If you prefer manual verification or need to debug specific issues:

# Check environment configuration
echo "UV_LINK_MODE: $UV_LINK_MODE"
echo "JAX_SKIP_CUDA_CONSTRAINTS_CHECK: $JAX_SKIP_CUDA_CONSTRAINTS_CHECK"

# Test UV performance
time uv sync --quiet  # Should complete in 200-500ms

# Verify pre-commit without warnings
uv run pre-commit run --all-files 2>&1 | grep -i "warning\|failed\|error"

Common Issues & Solutions¶

Issue: "Failed to hardlink files" Warning¶

warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.

Solution: Set the UV_LINK_MODE environment variable:

Set export UV_LINK_MODE=copy in your shell
Or add it permanently to your ~/.bashrc or ~/.zshrc

Issue: Slow Pre-commit Execution (>10 seconds)¶

Symptoms: Pre-commit takes significantly longer than expected

Diagnosis:

# Time the execution
time uv run pre-commit run --all-files

# Check for package reinstallation
uv run pre-commit run --all-files --verbose | grep "Installing"

Solutions:

Clear and rebuild pre-commit cache: uv run pre-commit clean && uv run pre-commit install
Check for package reinstallation in verbose output

Issue: Pre-commit Hooks Failing¶

Symptoms: Individual hooks return non-zero exit codes