Internal Packages Guide¶

This directory contains the internal implementation of the drun execution engine. The codebase is organized into focused, maintainable packages.

Package Overview¶

Package Statistics¶

Total: ~102 files, ~15,800 lines (avg. 155 lines/file)

Package Details¶

ast/ - Abstract Syntax Tree¶

Purpose: Defines all AST node types representing drun language constructs.

Key Files:

• ast.go - Core types: Program, Node, Statement, Expression
• ast_project.go - Project declarations
• ast_task.go - Task definitions
• ast_parameter.go - Parameter types
• ast_control.go - Control flow: if, for, try/catch
• ast_action.go - Base action statements
• ast_expressions.go - Expressions and operators

Domain-Specific:

• ast_shell.go - Shell commands
• ast_file.go - File operations
• ast_docker.go - Docker actions
• ast_git.go - Git actions
• ast_http.go - HTTP requests
• ast_network.go - Network operations
• ast_variable.go - Variable operations
• ast_detection.go - Tool detection

Usage:

import "github.com/phillarmonic/drun/v2/internal/ast"

stmt := &ast.ShellStatement{
    Command: "echo hello",
}

parser/ - Syntax Parser¶

Purpose: Parses drun source code into AST.

Key Files:

• parser.go - Core parser with entry point
• parser_project.go - Parses project declarations
• parser_task.go - Parses task definitions
• parser_parameter.go - Parses parameters
• parser_action.go - Dispatches to action parsers
• parser_control.go - Parses control flow
• parser_error.go - Parses error handling
• parser_helper.go - Helper methods

Domain-Specific Parsers:

• parser_shell.go - Shell command parsing
• parser_file.go - File operation parsing
• parser_docker.go - Docker action parsing
• parser_git.go - Git action parsing
• parser_http.go - HTTP request parsing
• parser_network.go - Network operation parsing
• parser_variable.go - Variable operation parsing
• parser_detection.go - Detection statement parsing

Usage:

import "github.com/phillarmonic/drun/v2/internal/parser"

p := parser.New(lexer)
program, err := p.ParseProgram()

Architecture:

• Each parser file handles one domain
• All parsers share the core Parser struct
• Parsers build AST nodes from tokens

domain/ - Domain Layer¶

Purpose: Business logic layer separating domain concepts from execution concerns.

Status: Fully integrated with engine

Key Files:

• task/task.go - Task entity with validation
• task/registry.go - Task management and lookup (thread-safe)
• task/dependencies.go - Dependency resolution with circular detection
• parameter/parameter.go - Parameter entity
• parameter/validation.go - Parameter validation rules
• project/project.go - Project configuration

Architecture:

CLI Layer → Engine Layer → Domain Layer
                 ↓              ↓
           Orchestration   Business Logic

Key Services:

1. Task Registry (task/registry.go)
2. Registers and manages tasks
3. Preserves insertion order
4. Thread-safe operations

5. Namespace support

6. Dependency Resolver (task/dependencies.go)

7. Resolves task execution order
8. Detects circular dependencies
9. Topological sorting

10. Parallel/sequential grouping

11. Parameter Validator (parameter/validation.go)

12. Validates data types (string, number, boolean, list)
13. Checks constraints (from list)
14. Validates ranges (min/max)
15. Pattern matching (regex, email, semver, etc.)

Usage in Engine:

// Engine struct holds domain services
type Engine struct {
    taskRegistry   *task.Registry
    paramValidator *parameter.Validator
    depResolver    *task.DependencyResolver
    // ...
}

// Register tasks from AST
func (e *Engine) registerTasks(tasks []*ast.TaskStatement, file string) error {
    for _, astTask := range tasks {
        domainTask := task.NewTask(astTask, "", file)
        if err := e.taskRegistry.Register(domainTask); err != nil {
            return err
        }
    }
    return nil
}

// Resolve dependencies
domainTasks, err := e.depResolver.Resolve(taskName)

// Validate parameters
domainParam := &parameter.Parameter{
    Name:        param.Name,
    DataType:    param.DataType,
    Constraints: param.Constraints,
    // ...
}
err := e.paramValidator.Validate(domainParam, typedValue)

Design Principles:

• Domain entities are independent of AST
• Business rules stay in domain layer
• Engine orchestrates, domain validates
• Easily testable in isolation

Test Coverage:

• task/task_test.go - 32 tests
• task/registry_test.go - 16 tests
• task/dependencies_test.go - 21 tests
• parameter/validation_test.go - 17 tests

When to Use Domain Layer:

• Adding new validation rules
• Extending task/parameter properties
• Adding business logic operations
• AST changes (use ast/ instead)
• Execution logic (use engine/ instead)

engine/ - Execution Engine¶

Purpose: Executes AST by orchestrating executors and subsystems.

Core Files¶

• engine.go (911 lines) - Main engine, orchestration
• context.go - Execution context and project context

Subsystems¶

interpolation/ - Variable interpolation

• interpolator.go - Main interpolation logic
• resolvers.go - Variable resolution
• conditional.go - Conditional interpolation
• utilities.go - Helper functions

hooks/ - Lifecycle hooks

• manager.go - Hook registration and execution

includes/ - Include resolution

• resolver.go - Remote/local include handling

Executors¶

Each executor handles one type of statement:

• executor_error.go - try/catch/finally, throw
• executor_control.go - if, for, break, continue
• executor_variables.go - let, set, capture
• executor_shell.go - Shell command execution
• executor_file.go - File operations
• executor_network.go - Network operations, health checks
• executor_docker.go - Docker actions
• executor_git.go - Git actions
• executor_http.go - HTTP requests
• executor_detection.go - Tool detection

Helpers¶

Domain-specific helper functions:

• helpers_builders.go - Command builders
• helpers_conditions.go - Condition evaluation
• helpers_detection.go - Detection helpers
• helpers_download.go - File download with progress
• helpers_expressions.go - Expression evaluation
• helpers_filesystem.go - Filesystem utilities
• helpers_utilities.go - General utilities

Usage:

import "github.com/phillarmonic/drun/v2/internal/engine"

eng := engine.NewEngine(os.Stdout)
eng.LoadProject(program)
err := eng.RunTask("build", params)

Architecture:

Engine (orchestrator)
├── Interpolation System (variable resolution)
├── Hooks System (lifecycle hooks)
├── Includes System (file inclusion)
└── Executors (statement execution)
    ├── Error Executor
    ├── Control Executor
    ├── Variable Executor
    ├── Shell Executor
    ├── File Executor
    ├── Network Executor
    ├── Docker Executor
    ├── Git Executor
    ├── HTTP Executor
    └── Detection Executor

lexer/ - Lexical Analysis¶

Purpose: Tokenizes drun source code.

Key Files:

• lexer.go - Main lexer implementation
• tokens.go - Token type definitions
• keywords.go - Keyword mappings
• semantic_tokens.go - Semantic token support
• position.go - Position tracking
• errors.go - Lexer error types

Usage:

import "github.com/phillarmonic/drun/v2/internal/lexer"

l := lexer.New(source)
token := l.NextToken()

Support Packages¶

builtins/ - Built-in Functions¶

Functions like now(), env(), string operations.

import "github.com/phillarmonic/drun/v2/internal/builtins"

result, err := builtins.Apply("uppercase", "hello")

shell/ - Shell Execution¶

Cross-platform shell command execution.

import "github.com/phillarmonic/drun/v2/internal/shell"

result, err := shell.Execute("echo hello", opts)

detection/ - Tool Detection¶

Detects installed tools and versions.

import "github.com/phillarmonic/drun/v2/internal/detection"

available := detection.IsToolAvailable("docker")
version, _ := detection.GetToolVersion("node")

remote/ - Remote Fetching¶

Fetches files from remote sources (HTTP, GitHub).

import "github.com/phillarmonic/drun/v2/internal/remote"

content, err := remote.FetchContent("https://...")

cache/ - Caching System¶

Caches downloaded files and parsed content.

import "github.com/phillarmonic/drun/v2/internal/cache"

manager := cache.NewManager()
manager.Set("key", data)

errors/ - Error Types¶

Defines custom error types with context.

import "github.com/phillarmonic/drun/v2/internal/errors"

err := errors.NewExecutionError("failed", ctx)

types/ - Type Definitions¶

Common type definitions and utilities.

import "github.com/phillarmonic/drun/v2/internal/types"

value := types.StringValue("hello")

Architecture Principles¶

1. Single Responsibility¶

Each package/file has ONE clear purpose:

• parser_docker.go - Only Docker parsing
• executor_git.go - Only Git execution
• helpers_download.go - Only download helpers

2. Dependency Direction¶

CLI → Engine → Parser → Lexer → AST
          ↓
    Support Packages

Higher-level packages depend on lower-level ones, never the reverse.

3. Domain Organization¶

Files grouped by domain, not by technical layer:

• All Docker-related code: ast_docker.go, parser_docker.go, executor_docker.go
• All Git-related code: ast_git.go, parser_git.go, executor_git.go

4. Testability¶

Every package can be tested independently:

// Test parser without engine
parser := parser.New(lexer.New("task build"))
program, err := parser.ParseProgram()

// Test executor without full engine
ctx := &ExecutionContext{...}
err := executeDocker(stmt, ctx)

Common Patterns¶

Adding a New Action Type¶

1. Define AST in internal/ast/ast_yourfeature.go
2. Add Parser in internal/parser/parser_yourfeature.go
3. Add Executor in internal/engine/executor_yourfeature.go
4. Wire it up in engine.go and parser.go

Accessing Variables¶

// In executor
value, exists := ctx.GetVariable("myvar")
if !exists {
    return fmt.Errorf("variable not found")
}

Interpolating Strings¶

// In executor
interpolated, err := e.interpolator.InterpolateString(input, ctx)
if err != nil {
    return err
}

Executing Shell Commands¶

// In executor
result, err := shell.Execute(command, e.getShellOpts(ctx))
if err != nil {
    return fmt.Errorf("command failed: %w", err)
}

File Size Guidelines¶

After refactoring, we maintain these size limits:

Current Status: All files within guidelines

Navigation Tips¶

Finding Code by Feature¶

Looking for Docker support?

1. AST: internal/ast/ast_docker.go
2. Parser: internal/parser/parser_docker.go
3. Executor: internal/engine/executor_docker.go

Understanding Execution Flow¶

1. Start at cmd/xdrun/main.go - entry point
2. Follow to engine.go - orchestration
3. Look at specific executor - action execution
4. Check helpers - supporting functions

Debugging¶

1. Lexer issues: Check internal/lexer/
2. Parsing errors: Check internal/parser/parser_*.go
3. Runtime errors: Check internal/engine/executor_*.go
4. Variable issues: Check internal/engine/interpolation/

Testing Strategy¶

Unit Tests¶

Each package has its own tests:

internal/parser/parser_docker_test.go
internal/engine/executor_docker_test.go
internal/ast/ast_test.go

Integration Tests¶

Engine tests in internal/engine/*_test.go:

• strict_variables_test.go
• loop_scoping_test.go
• matrix_execution_test.go
• when_otherwise_test.go

Regression Tests¶

All 62 example files in examples/ must pass.

Performance Considerations¶

Hot Paths¶

Most frequently executed code:

1. Variable interpolation (interpolation/)
2. Statement execution (executor_*.go)
3. Context lookups (context.go)

Optimization Points¶

• Variable lookups cached in context
• Includes cached by resolver
• Regex patterns compiled once
• Shell opts reused across commands

Refactoring History¶

Before Refactoring¶

internal/
├── ast.go (1,133 lines)
├── parser.go (4,874 lines)
├── engine.go (5,179 lines)
└── ... (support packages)

After (Current)¶

internal/
├── ast/ (15 files, ~100-200 lines each)
├── parser/ (26 files, ~200-300 lines each)
├── engine/ (36 files, ~150-300 lines each)
└── ... (support packages)

Improvement:

• 10x better file sizes
• Clear responsibilities
• Easier to test
• Easier to understand
• Faster to modify

Contributing¶

When adding new code:

1. Follow existing patterns
2. Keep files under 500 lines
3. Group by domain, not layer
4. Add tests for new code
5. Update documentation

Related Documentation¶

• ARCHITECTURE.md - System architecture overview
• spec/ - Refactoring specifications
• Language specification - Normative language reference

Last Updated: October 5, 2025 Current Version