diff --git a/CLAUDE.md b/CLAUDE.md index f763e7890c..b3a446fbf4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,16 +40,11 @@ make lint # Basic evolution run python openevolve-run.py path/to/initial_program.py path/to/evaluator.py --config path/to/config.yaml --iterations 1000 -# Resume from checkpoint -python openevolve-run.py path/to/initial_program.py path/to/evaluator.py \ - --config path/to/config.yaml \ - --checkpoint path/to/checkpoint_directory \ - --iterations 50 ``` ### Visualization ```bash -# View evolution tree +# View an archived evolution tree from an older checkpoint-based run python scripts/visualizer.py --path examples/function_minimization/openevolve_output/checkpoints/checkpoint_100/ ``` @@ -59,7 +54,7 @@ python scripts/visualizer.py --path examples/function_minimization/openevolve_ou 1. **Controller (`openevolve/controller.py`)**: Main orchestrator that manages the evolution process using ProcessPoolExecutor for parallel iteration execution. -2. **Database (`openevolve/database.py`)**: Implements MAP-Elites algorithm with island-based evolution: +2. **Database (`openevolve/database.py`)**: Query-oriented `ProgramDatabase` interface. `database_memory.py` implements MAP-Elites with island-based evolution: - Programs mapped to multi-dimensional feature grid - Multiple isolated populations (islands) evolve independently - Periodic migration between islands prevents convergence @@ -73,14 +68,14 @@ python scripts/visualizer.py --path examples/function_minimization/openevolve_ou 4. **LLM Integration (`openevolve/llm/`)**: Ensemble approach with multiple models, configurable weights, and async generation with retry logic. -5. **Iteration (`openevolve/iteration.py`)**: Worker process that samples from islands, generates mutations via LLM, evaluates programs, and stores artifacts. +5. **Iteration (`openevolve/process_parallel.py`)**: Queries the database for bounded iteration context, then workers generate mutations and evaluate programs. The controller writes results through the interface. ### Key Architectural Patterns - **Island-Based Evolution**: Multiple populations evolve separately with periodic migration - **MAP-Elites**: Maintains diversity by mapping programs to feature grid cells - **Artifact System**: Side-channel for programs to return debugging data, stored as JSON or files -- **Process Worker Pattern**: Each iteration runs in fresh process with database snapshot +- **Process Worker Pattern**: Each worker receives selected programs and parent artifacts, with no population snapshot - **Double-Selection**: Programs for inspiration differ from those shown to LLM - **Lazy Migration**: Islands migrate based on generation counts, not iterations @@ -103,8 +98,8 @@ YAML-based configuration with hierarchical structure: ### Important Patterns -1. **Checkpoint/Resume**: Automatic saving of entire system state with seamless resume capability -2. **Parallel Evaluation**: Multiple programs evaluated concurrently via TaskPool +1. **Database Sessions**: Inject a `ProgramDatabase` via `database=`. Continue using the same database instance; the in-memory implementation has no save/load operations. PostgreSQL is a future implementation. +2. **Parallel Evaluation**: Multiple programs evaluated concurrently via ProcessPoolExecutor 3. **Error Resilience**: Individual failures don't crash system - extensive retry logic and timeout protection 4. **Prompt Engineering**: Template-based system with context-aware building and evolution history @@ -115,4 +110,4 @@ YAML-based configuration with hierarchical structure: - Tests use unittest framework - Black for code formatting - Artifacts threshold: Small (<10KB) stored in DB, large saved to disk -- Process workers load database snapshots for true parallelism \ No newline at end of file +- Controllers use database queries for each candidate; worker context stays bounded by prompt limits \ No newline at end of file diff --git a/README.md b/README.md index 785a1d5804..0b1ab36f88 100644 --- a/README.md +++ b/README.md @@ -755,9 +755,41 @@ return EvaluationResult( This creates a **feedback loop** where each generation learns from previous mistakes! +## Program database + +`ProgramDatabase` in `openevolve/database.py` is the storage interface. +`InMemoryProgramDatabase` in `openevolve/database_memory.py` implements the existing +MAP-Elites and island policies. The controller queries this interface for each +candidate and sends workers only the selected programs and parent artifacts. +Programs returned by queries are detached values; editing them does not change +the database. Writes go through database operations. + +```python +from openevolve import OpenEvolve +from openevolve.config import load_config +from openevolve.database_memory import InMemoryProgramDatabase + +config = load_config("config.yaml") +database = InMemoryProgramDatabase(config.database) +evolve = OpenEvolve("program.py", "evaluator.py", config, database=database) +# In an async entry point: +best = await evolve.run(iterations=50) +best = await evolve.run(iterations=50) # Continues the same database session +``` + +The library's `run_evolution` function also accepts `database=`. A future PostgreSQL +implementation will use the same interface. The in-memory implementation lasts for +the lifetime of its instance; it does not save or restore population checkpoints. +The `--checkpoint` CLI option and `checkpoint_path` API argument have been removed. +Best-program exports, logs, optional evolution traces, and evaluation artifacts remain +available. Legacy YAML keys `checkpoint_interval` and `database.max_snapshot_artifacts` +are ignored. + ## Visualization -**Real-time evolution tracking** with interactive web interface: +The existing visualizer reads archived checkpoint files from older runs. It has not +yet been adapted to the program database interface, so new runs cannot use it for +live population tracking. ```bash # Install visualization dependencies diff --git a/configs/README.md b/configs/README.md index 6ce24383c1..77fed4b815 100644 --- a/configs/README.md +++ b/configs/README.md @@ -2,6 +2,13 @@ This directory contains configuration files for OpenEvolve with examples for different use cases. +The default program database is `InMemoryProgramDatabase`. Supply a database implementation +through `OpenEvolve(..., database=...)` or `run_evolution(..., database=...)`. +Population checkpoint saving/loading has been removed. Older YAML files may still contain +`checkpoint_interval` and `database.max_snapshot_artifacts`; these keys are ignored. +`database.db_path` is now only a fallback directory for artifacts. `database.in_memory` +is retained for configuration compatibility and does not enable disk persistence. + ## Configuration Files ### `default_config.yaml` @@ -65,9 +72,10 @@ Then use with OpenEvolve: ```python from openevolve import OpenEvolve +from openevolve.config import load_config evolve = OpenEvolve( initial_program_path="program.py", evaluation_file="evaluator.py", - config_path="my_config.yaml" + config=load_config("my_config.yaml") ) ``` diff --git a/configs/default_config.yaml b/configs/default_config.yaml index 14ae54556b..fb6a1ac990 100644 --- a/configs/default_config.yaml +++ b/configs/default_config.yaml @@ -4,7 +4,6 @@ # General settings max_iterations: 100 # Maximum number of evolution iterations -checkpoint_interval: 10 # Save checkpoints every N iterations log_level: "INFO" # Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) log_dir: null # Custom directory for logs (default: output_dir/logs) random_seed: 42 # Random seed for reproducibility (null = random, 42 = default) @@ -92,8 +91,8 @@ prompt: # Database configuration database: # General settings - db_path: null # Path to persist database (null = in-memory only) - in_memory: true # Keep database in memory for faster access + db_path: null # Legacy fallback base directory for artifacts + in_memory: true # Legacy option; use database= to supply an implementation log_prompts: true # If true, log all prompts and responses into the database # Evolutionary parameters diff --git a/examples/tsp_tour_minimization/start_evolution.py b/examples/tsp_tour_minimization/start_evolution.py index 46cc143aa8..c7afe7431f 100644 --- a/examples/tsp_tour_minimization/start_evolution.py +++ b/examples/tsp_tour_minimization/start_evolution.py @@ -1,4 +1,3 @@ -import re import sys import pathlib import asyncio @@ -17,28 +16,8 @@ from utils.code_to_query import * -def latest_checkpoint(dir_path: pathlib.Path) -> str | None: - if not dir_path.exists(): - return None - - candidates: list[tuple[int, pathlib.Path]] = [] - for path in dir_path.glob("checkpoint_*"): - if not path.is_dir(): - continue - - match = re.fullmatch(r"checkpoint_(\d+)", path.name) - if match: - candidates.append((int(match.group(1)), path)) - - if not candidates: - return None - - _, newest_path = max(candidates, key=lambda t: (t[0], t[1].stat().st_mtime)) - return str(newest_path) - - -async def run_evolution(evolve: OpenEvolve, checkpoint_path: str | None) -> None: - best_program = await evolve.run(checkpoint_path=checkpoint_path) +async def run_evolution(evolve: OpenEvolve) -> None: + best_program = await evolve.run() print("Best program metrics:") for name, value in best_program.metrics.items(): @@ -59,7 +38,7 @@ async def run_evolution(evolve: OpenEvolve, checkpoint_path: str | None) -> None "--openevolve_output_dir", type=click.Path(file_okay=False, dir_okay=True, path_type=pathlib.Path), default=BASE_DIR / "openevolve_output", - help="Output directory for OpenEvolve runs (checkpoints, logs, etc.).", + help="Output directory for OpenEvolve results and logs.", ) def cli(initial_program_dir: pathlib.Path, openevolve_output_dir: pathlib.Path) -> None: initial_program_dir = initial_program_dir.resolve() @@ -80,10 +59,7 @@ def cli(initial_program_dir: pathlib.Path, openevolve_output_dir: pathlib.Path) output_dir=str(openevolve_output_dir), ) - checkpoint_path = latest_checkpoint(openevolve_output_dir / "checkpoints") - print(f"Using checkpoint: '{checkpoint_path}'") - - asyncio.run(run_evolution(evolve, checkpoint_path)) + asyncio.run(run_evolution(evolve)) if __name__ == "__main__": diff --git a/openevolve/api.py b/openevolve/api.py index 9452391763..92efbfd2d0 100644 --- a/openevolve/api.py +++ b/openevolve/api.py @@ -13,7 +13,7 @@ from openevolve.controller import OpenEvolve from openevolve.config import Config, load_config, LLMModelConfig -from openevolve.database import Program +from openevolve.database import Program, ProgramDatabase @dataclass @@ -38,7 +38,7 @@ def run_evolution( output_dir: Optional[str] = None, cleanup: bool = True, target_score: Optional[float] = None, - checkpoint_path: Optional[str] = None, + database: Optional[ProgramDatabase] = None, ) -> EvolutionResult: """ Run evolution with flexible inputs - the main library API @@ -58,6 +58,8 @@ def run_evolution( iterations: Number of iterations (overrides config) output_dir: Output directory (None for temp directory) cleanup: If True, clean up temp files after evolution + target_score: Stop once this combined score is reached + database: Session database (defaults to a new in-memory implementation) Returns: EvolutionResult with best program and metrics @@ -92,7 +94,16 @@ def my_evaluator(program_path): ) """ return asyncio.run( - _run_evolution_async(initial_program, evaluator, config, iterations, output_dir, cleanup, target_score, checkpoint_path) + _run_evolution_async( + initial_program, + evaluator, + config, + iterations, + output_dir, + cleanup, + target_score, + database, + ) ) @@ -104,7 +115,7 @@ async def _run_evolution_async( output_dir: Optional[str], cleanup: bool, target_score: Optional[float] = None, - checkpoint_path: Optional[str] = None, + database: Optional[ProgramDatabase] = None, ) -> EvolutionResult: """Async implementation of run_evolution""" @@ -158,9 +169,10 @@ async def _run_evolution_async( evaluation_file=evaluator_path, config=config_obj, output_dir=actual_output_dir, + database=database, ) - best_program = await controller.run(iterations=iterations,target_score=target_score,checkpoint_path=checkpoint_path) + best_program = await controller.run(iterations=iterations, target_score=target_score) # Prepare result best_score = 0.0 diff --git a/openevolve/cli.py b/openevolve/cli.py index d1da28137c..7f491ecc14 100644 --- a/openevolve/cli.py +++ b/openevolve/cli.py @@ -45,12 +45,6 @@ def parse_args() -> argparse.Namespace: default=None, ) - parser.add_argument( - "--checkpoint", - help="Path to checkpoint directory to resume from (e.g., openevolve_output/checkpoints/checkpoint_50)", - default=None, - ) - parser.add_argument("--api-base", help="Base URL for the LLM API", default=None) parser.add_argument("--primary-model", help="Primary LLM model name", default=None) @@ -112,17 +106,6 @@ async def main_async() -> int: output_dir=args.output, ) - # Load from checkpoint if specified - if args.checkpoint: - if not os.path.exists(args.checkpoint): - print(f"Error: Checkpoint directory '{args.checkpoint}' not found") - return 1 - print(f"Loading checkpoint from {args.checkpoint}") - openevolve.database.load(args.checkpoint) - print( - f"Checkpoint loaded successfully (iteration {openevolve.database.last_iteration})" - ) - # Override log level if specified if args.log_level: logging.getLogger().setLevel(getattr(logging, args.log_level)) @@ -131,23 +114,8 @@ async def main_async() -> int: best_program = await openevolve.run( iterations=args.iterations, target_score=args.target_score, - checkpoint_path=args.checkpoint, ) - # Get the checkpoint path - checkpoint_dir = os.path.join(openevolve.output_dir, "checkpoints") - latest_checkpoint = None - if os.path.exists(checkpoint_dir): - checkpoints = [ - os.path.join(checkpoint_dir, d) - for d in os.listdir(checkpoint_dir) - if os.path.isdir(os.path.join(checkpoint_dir, d)) - ] - if checkpoints: - latest_checkpoint = sorted( - checkpoints, key=lambda x: int(x.split("_")[-1]) if "_" in x else 0 - )[-1] - print(f"\nEvolution complete!") print(f"Best program metrics:") for name, value in best_program.metrics.items(): @@ -157,10 +125,6 @@ async def main_async() -> int: else: print(f" {name}: {value}") - if latest_checkpoint: - print(f"\nLatest checkpoint saved at: {latest_checkpoint}") - print(f"To resume, use: --checkpoint {latest_checkpoint}") - return 0 except Exception as e: diff --git a/openevolve/config.py b/openevolve/config.py index c19ab4ca1d..efdc763e8b 100644 --- a/openevolve/config.py +++ b/openevolve/config.py @@ -311,10 +311,10 @@ class DatabaseConfig: """Configuration for the program database""" # General settings - db_path: Optional[str] = None # Path to store database on disk - in_memory: bool = True + db_path: Optional[str] = None # Legacy fallback base directory for artifacts + in_memory: bool = True # Legacy option; implementations are supplied via database= - # Prompt and response logging to programs/.json + # Prompt and response history in the session database log_prompts: bool = True # Evolutionary parameters @@ -359,9 +359,6 @@ class DatabaseConfig: artifact_size_threshold: int = 32 * 1024 # 32KB threshold cleanup_old_artifacts: bool = True artifact_retention_days: int = 30 - max_snapshot_artifacts: Optional[int] = ( - 100 # Max artifacts in worker snapshots (None=unlimited) - ) novelty_llm: Optional["LLMInterface"] = None embedding_model: Optional[str] = None @@ -418,7 +415,6 @@ class Config: # General settings max_iterations: int = 10000 - checkpoint_interval: int = 100 log_level: str = "INFO" log_dir: Optional[str] = None random_seed: Optional[int] = 42 diff --git a/openevolve/controller.py b/openevolve/controller.py index a3f096bf8b..1a829f37bf 100644 --- a/openevolve/controller.py +++ b/openevolve/controller.py @@ -14,6 +14,7 @@ from openevolve.config import Config, load_config from openevolve.database import Program, ProgramDatabase +from openevolve.database_memory import InMemoryProgramDatabase from openevolve.evaluator import Evaluator from openevolve.evolution_trace import EvolutionTracer from openevolve.llm.ensemble import LLMEnsemble @@ -45,6 +46,7 @@ def __init__( evaluation_file: str, config: Config, output_dir: Optional[str] = None, + database: Optional[ProgramDatabase] = None, ): # Load configuration (loaded in main_async) self.config = config @@ -125,7 +127,9 @@ def __init__( self.config.database.random_seed = self.config.random_seed self.config.database.novelty_llm = self.llm_ensemble - self.database = ProgramDatabase(self.config.database) + self.database: ProgramDatabase = ( + database if database is not None else InMemoryProgramDatabase(self.config.database) + ) self.evaluator = Evaluator( self.config.evaluator, @@ -226,7 +230,6 @@ async def run( self, iterations: Optional[int] = None, target_score: Optional[float] = None, - checkpoint_path: Optional[str] = None, ) -> Optional[Program]: """ Run the evolution process with improved parallel processing @@ -234,29 +237,16 @@ async def run( Args: iterations: Maximum number of iterations (uses config if None) target_score: Target score to reach (continues until reached if specified) - checkpoint_path: Path to resume from checkpoint Returns: Best program found """ - max_iterations = iterations or self.config.max_iterations - # Determine starting iteration - start_iteration = 0 - if checkpoint_path and os.path.exists(checkpoint_path): - self._load_checkpoint(checkpoint_path) - start_iteration = self.database.last_iteration + 1 - logger.info(f"Resuming from checkpoint at iteration {start_iteration}") - else: - start_iteration = self.database.last_iteration - - # Only add initial program if starting fresh (not resuming from checkpoint) - should_add_initial = ( - start_iteration == 0 - and len(self.database.programs) == 0 - and not any( - p.code == self.initial_program_code for p in self.database.programs.values() - ) - ) + max_iterations = iterations if iterations is not None else self.config.max_iterations + if max_iterations < 0: + raise ValueError("iterations must be non-negative") + state = self.database.get_state() + should_add_initial = state.program_count == 0 + start_iteration = 0 if should_add_initial else state.last_iteration + 1 if should_add_initial: logger.info("Adding initial program to database") @@ -303,7 +293,7 @@ async def run( else: logger.info( f"Skipping initial program addition (resuming from iteration {start_iteration} " - f"with {len(self.database.programs)} existing programs)" + f"with {state.program_count} existing programs)" ) # Initialize improved parallel processing @@ -346,8 +336,8 @@ def force_exit_handler(signum, frame): # User expects max_iterations evolutionary iterations AFTER the initial program # So we don't need to reduce evolution_iterations - # Run evolution with improved parallel processing and checkpoint callback - await self._run_evolution_with_checkpoints( + # Every iteration selects its context through the database interface. + await self.parallel_controller.run_evolution( evolution_start, evolution_iterations, target_score ) @@ -362,15 +352,7 @@ def force_exit_handler(signum, frame): self.evolution_tracer.close() logger.info("Evolution tracer closed") - # Get the best program - best_program = None - if self.database.best_program_id: - best_program = self.database.get(self.database.best_program_id) - logger.info(f"Using tracked best program: {self.database.best_program_id}") - - if best_program is None: - best_program = self.database.get_best_program() - logger.info("Using calculated best program (tracked program not found)") + best_program = self.database.get_best_program() if best_program: if ( @@ -419,99 +401,6 @@ def _log_iteration( f"(Δ: {improvement_str})" ) - def _save_checkpoint(self, iteration: int) -> None: - """ - Save a checkpoint - - Args: - iteration: Current iteration number - """ - checkpoint_dir = os.path.join(self.output_dir, "checkpoints") - os.makedirs(checkpoint_dir, exist_ok=True) - - # Create specific checkpoint directory - checkpoint_path = os.path.join(checkpoint_dir, f"checkpoint_{iteration}") - os.makedirs(checkpoint_path, exist_ok=True) - - # Save the database - self.database.save(checkpoint_path, iteration) - - # Save the best program found so far - best_program = None - if self.database.best_program_id: - best_program = self.database.get(self.database.best_program_id) - else: - best_program = self.database.get_best_program() - - if best_program: - # Save the best program at this checkpoint - best_program_path = os.path.join(checkpoint_path, f"best_program{self.file_extension}") - with open(best_program_path, "w") as f: - f.write(best_program.code) - - # Save metrics - best_program_info_path = os.path.join(checkpoint_path, "best_program_info.json") - with open(best_program_info_path, "w") as f: - import json - - json.dump( - { - "id": best_program.id, - "generation": best_program.generation, - "iteration": best_program.iteration_found, - "current_iteration": iteration, - "metrics": best_program.metrics, - "language": best_program.language, - "timestamp": best_program.timestamp, - "saved_at": time.time(), - }, - f, - indent=2, - ) - - logger.info( - f"Saved best program at checkpoint {iteration} with metrics: " - f"{format_metrics_safe(best_program.metrics)}" - ) - - logger.info(f"Saved checkpoint at iteration {iteration} to {checkpoint_path}") - - def _load_checkpoint(self, checkpoint_path: str) -> None: - """Load state from a checkpoint directory""" - if not os.path.exists(checkpoint_path): - raise FileNotFoundError(f"Checkpoint directory {checkpoint_path} not found") - - logger.info(f"Loading checkpoint from {checkpoint_path}") - self.database.load(checkpoint_path) - logger.info(f"Checkpoint loaded successfully (iteration {self.database.last_iteration})") - - async def _run_evolution_with_checkpoints( - self, start_iteration: int, max_iterations: int, target_score: Optional[float] - ) -> None: - """Run evolution with checkpoint saving support""" - logger.info(f"Using island-based evolution with {self.config.database.num_islands} islands") - self.database.log_island_status() - - # Run the evolution process with checkpoint callback - await self.parallel_controller.run_evolution( - start_iteration, max_iterations, target_score, checkpoint_callback=self._save_checkpoint - ) - - # Check if shutdown or early stopping was triggered - if self.parallel_controller.shutdown_event.is_set(): - logger.info("Evolution stopped due to shutdown request") - return - elif self.parallel_controller.early_stopping_triggered: - logger.info("Evolution stopped due to early stopping - saving final checkpoint") - # Continue to save final checkpoint for early stopping - - # Save final checkpoint if needed - # Note: start_iteration here is the evolution start (1 for fresh start, not 0) - # max_iterations is the number of evolution iterations to run - final_iteration = start_iteration + max_iterations - 1 - if final_iteration > 0 and final_iteration % self.config.checkpoint_interval == 0: - self._save_checkpoint(final_iteration) - def _save_best_program(self, program: Optional[Program] = None) -> None: """ Save the best program @@ -519,13 +408,8 @@ def _save_best_program(self, program: Optional[Program] = None) -> None: Args: program: Best program (if None, uses the tracked best program) """ - # If no program is provided, use the tracked best program from the database if program is None: - if self.database.best_program_id: - program = self.database.get(self.database.best_program_id) - else: - # Fallback to calculating best program if no tracked best program - program = self.database.get_best_program() + program = self.database.get_best_program() if not program: logger.warning("No best program found to save") diff --git a/openevolve/database.py b/openevolve/database.py index 8abe2bdc0a..bc6d72bd6c 100644 --- a/openevolve/database.py +++ b/openevolve/database.py @@ -1,2587 +1,94 @@ -""" -Program database for OpenEvolve -""" - -import base64 -import json -import logging -import os -import random -import shutil -import time -import uuid -from dataclasses import asdict, dataclass, field, fields - -# FileLock removed - no longer needed with threaded parallel processing -from typing import Any, Dict, List, Optional, Set, Tuple, Union - -import numpy as np - -from openevolve.config import DatabaseConfig -from openevolve.utils.code_utils import calculate_edit_distance -from openevolve.utils.metrics_utils import safe_numeric_average, get_fitness_score - -logger = logging.getLogger(__name__) - - -def _safe_sum_metrics(metrics: Dict[str, Any]) -> float: - """Safely sum only numeric metric values, ignoring strings and other types""" - numeric_values = [ - v for v in metrics.values() if isinstance(v, (int, float)) and not isinstance(v, bool) - ] - return sum(numeric_values) if numeric_values else 0.0 - - -def _safe_avg_metrics(metrics: Dict[str, Any]) -> float: - """Safely calculate average of only numeric metric values""" - numeric_values = [ - v for v in metrics.values() if isinstance(v, (int, float)) and not isinstance(v, bool) - ] - return sum(numeric_values) / max(1, len(numeric_values)) if numeric_values else 0.0 - - -@dataclass -class Program: - """Represents a program in the database""" - - # Program identification - id: str - code: str - changes_description: str = ( - "" # compact program changes description (via LLM) stored per program - ) - language: str = "python" +"""Query-oriented interface for an evolution session's program database. - # Evolution information - parent_id: Optional[str] = None - generation: int = 0 - timestamp: float = field(default_factory=time.time) - iteration_found: int = 0 # Track which iteration this program was found - - # Performance metrics - metrics: Dict[str, float] = field(default_factory=dict) - - # Derived features - complexity: float = 0.0 - diversity: float = 0.0 - - # Metadata - metadata: Dict[str, Any] = field(default_factory=dict) - - # Prompts - prompts: Optional[Dict[str, Any]] = None - - # Artifact storage - artifacts_json: Optional[str] = None # JSON-serialized small artifacts - artifact_dir: Optional[str] = None # Path to large artifact files - - # Embedding vector for novelty rejection sampling - embedding: Optional[List[float]] = None - - def to_dict(self) -> Dict[str, Any]: - """Convert to dictionary representation""" - return asdict(self) +Implementations own selection and population state. Consumers receive individual +programs, bounded selections, or aggregate state; they never access the backing +collections. The interface has no save/load or checkpoint operations. +""" - @classmethod - def from_dict(cls, data: Dict[str, Any]) -> "Program": - """Create from dictionary representation""" - # old DBs don't have changes_description (backward-compatibility) - if "changes_description" not in data: - metadata = data.get("metadata") or {} - if isinstance(metadata, dict): - data = { - **data, - "changes_description": metadata.get("changes_description") - or metadata.get("changes") - or "empty", - } - else: - data = {**data, "changes_description": "empty"} +from dataclasses import dataclass +from typing import Any, Dict, List, Optional, Protocol, Tuple, Union, runtime_checkable - # Get the valid field names for the Program dataclass - valid_fields = {f.name for f in fields(cls)} +from openevolve.program import Program - # Filter the data to only include valid fields - filtered_data = {k: v for k, v in data.items() if k in valid_fields} - # Log if we're filtering out any fields - if len(filtered_data) != len(data): - filtered_out = set(data.keys()) - set(filtered_data.keys()) - logger.debug(f"Filtered out unsupported fields when loading Program: {filtered_out}") +@dataclass(frozen=True) +class DatabaseState: + """Aggregate session state, independent of the storage implementation.""" - return cls(**filtered_data) + program_count: int + last_iteration: int + current_island: int + num_islands: int + feature_dimensions: Tuple[str, ...] -class ProgramDatabase: - """ - Database for storing and sampling programs during evolution +@runtime_checkable +class ProgramDatabase(Protocol): + """Search and result operations implemented in memory or by a DBMS. - The database implements a combination of MAP-Elites algorithm and - island-based population model to maintain diversity during evolution. - It also tracks the absolute best program separately to ensure it's never lost. + Each instance addresses one evolution session. Returned programs are detached + values: changing one does not update the database. Writes must be explicit. + Population and migration policy run behind these operations so a DBMS can + perform them without constructing an authoritative Python population. """ - def __init__(self, config: DatabaseConfig): - self.config = config - - # In-memory program storage - self.programs: Dict[str, Program] = {} - - # Per-island feature grids for MAP-Elites - self.island_feature_maps: List[Dict[str, str]] = [{} for _ in range(config.num_islands)] - - # Handle both int and dict types for feature_bins - if isinstance(config.feature_bins, int): - self.feature_bins = max( - config.feature_bins, - int(pow(config.archive_size, 1 / len(config.feature_dimensions)) + 0.99), - ) - else: - # If dict, keep as is (we'll use feature_bins_per_dim instead) - self.feature_bins = 10 # Default fallback for backward compatibility - - # Island populations - self.islands: List[Set[str]] = [set() for _ in range(config.num_islands)] - - # Island management attributes - self.current_island: int = 0 - self.island_generations: List[int] = [0] * config.num_islands - self.last_migration_generation: int = 0 - self.migration_interval: int = getattr(config, "migration_interval", 10) # Default to 10 - self.migration_rate: float = getattr(config, "migration_rate", 0.1) # Default to 0.1 + def get_state(self) -> DatabaseState: + """Read aggregate progress and the session's island/feature definitions.""" + ... - # Archive of elite programs - self.archive: Set[str] = set() - - # Track the absolute best program separately - self.best_program_id: Optional[str] = None - - # Track best program per island for proper island-based evolution - self.island_best_programs: List[Optional[str]] = [None] * config.num_islands - - # Track the last iteration number (for resuming) - self.last_iteration: int = 0 - - # Load database from disk if path is provided - if config.db_path and os.path.exists(config.db_path): - self.load(config.db_path) - - # Prompt log - self.prompts_by_program: Dict[str, Dict[str, Dict[str, str]]] = None - - # Set random seed for reproducible sampling if specified - if config.random_seed is not None: - import random - - random.seed(config.random_seed) - logger.debug(f"Database: Set random seed to {config.random_seed}") - - # Diversity caching infrastructure - self.diversity_cache: Dict[int, Dict[str, Union[float, float]]] = ( - {} - ) # hash -> {"value": float, "timestamp": float} - self.diversity_cache_size: int = 1000 # LRU cache size - self.diversity_reference_set: List[str] = ( - [] - ) # Reference program codes for consistent diversity - self.diversity_reference_size: int = getattr(config, "diversity_reference_size", 20) - - # Feature scaling infrastructure - self.feature_stats: Dict[str, Dict[str, Union[float, float, List[float]]]] = {} - self.feature_scaling_method: str = "minmax" # Options: minmax, zscore, percentile - - # Per-dimension bins support - if hasattr(config, "feature_bins") and isinstance(config.feature_bins, dict): - self.feature_bins_per_dim = config.feature_bins - else: - # Backward compatibility - use same bins for all dimensions - self.feature_bins_per_dim = { - dim: self.feature_bins for dim in config.feature_dimensions - } - - logger.info(f"Initialized program database with {len(self.programs)} programs") - - # Novelty judge setup - from openevolve.embedding import EmbeddingClient - - self.novelty_llm = config.novelty_llm - self.embedding_client = ( - EmbeddingClient(config.embedding_model) if config.embedding_model else None - ) - self.similarity_threshold = config.similarity_threshold + def record_iteration(self, iteration: int) -> None: + """Record a processed iteration, including one that produced no program.""" + ... def add( - self, program: Program, iteration: int = None, target_island: Optional[int] = None + self, program: Program, iteration: Optional[int] = None, target_island: Optional[int] = None ) -> str: - """ - Add a program to the database - - Args: - program: Program to add - iteration: Current iteration (defaults to last_iteration) - target_island: Specific island to add to (auto-detects parent's island if None) - - Returns: - Program ID - """ - # Store the program - # If iteration is provided, update the program's iteration_found - if iteration is not None: - program.iteration_found = iteration - # Update last_iteration if needed - self.last_iteration = max(self.last_iteration, iteration) - - self.programs[program.id] = program - - # Calculate feature coordinates for MAP-Elites - feature_coords = self._calculate_feature_coords(program) - - # Determine target island - # If target_island is not specified and program has a parent, inherit parent's island - if target_island is None and program.parent_id: - parent = self.programs.get(program.parent_id) - if parent and "island" in parent.metadata: - # Child inherits parent's island to maintain island isolation - island_idx = parent.metadata["island"] - logger.debug( - f"Program {program.id} inheriting island {island_idx} from parent {program.parent_id}" - ) - else: - # Parent not found or has no island, use current_island - island_idx = self.current_island - if parent: - logger.warning( - f"Parent {program.parent_id} has no island metadata, using current_island {island_idx}" - ) - else: - logger.warning( - f"Parent {program.parent_id} not found, using current_island {island_idx}" - ) - elif target_island is not None: - # Explicit target island specified (e.g., for migrants) - island_idx = target_island - else: - # No parent and no target specified, use current island - island_idx = self.current_island - - island_idx = island_idx % len(self.islands) # Ensure valid island - - # Novelty check before adding - if not self._is_novel(program.id, island_idx): - logger.debug( - f"Program {program.id} failed in novelty check and won't be added in the island {island_idx}" - ) - return program.id # Do not add non-novel program - - # Add to island-specific feature map (replacing existing if better) - feature_key = self._feature_coords_to_key(feature_coords) - island_feature_map = self.island_feature_maps[island_idx] - should_replace = feature_key not in island_feature_map - - if not should_replace: - # Check if the existing program still exists before comparing - existing_program_id = island_feature_map[feature_key] - if existing_program_id not in self.programs: - # Stale reference, replace it - should_replace = True - logger.debug( - f"Replacing stale program reference {existing_program_id} in island {island_idx} feature map" - ) - else: - # Program exists, compare fitness - should_replace = self._is_better(program, self.programs[existing_program_id]) - - # Track a program that gets displaced from its cell so we can remove it - # from the population if it ends up orphaned (owning no cell, in no island). - replaced_program_id = None - - if should_replace: - # Log significant MAP-Elites events - coords_dict = { - self.config.feature_dimensions[i]: feature_coords[i] - for i in range(len(feature_coords)) - } - - if feature_key not in island_feature_map: - # New cell occupation in this island - logger.info( - "New MAP-Elites cell occupied in island %d: %s", island_idx, coords_dict - ) - # Check coverage milestone for this island - total_possible_cells = self.feature_bins ** len(self.config.feature_dimensions) - island_coverage = (len(island_feature_map) + 1) / total_possible_cells - if island_coverage in [0.1, 0.25, 0.5, 0.75, 0.9]: - logger.info( - "Island %d MAP-Elites coverage reached %.1f%% (%d/%d cells)", - island_idx, - island_coverage * 100, - len(island_feature_map) + 1, - total_possible_cells, - ) - else: - # Cell replacement - existing program being replaced in this island - existing_program_id = island_feature_map[feature_key] - if existing_program_id in self.programs: - existing_program = self.programs[existing_program_id] - new_fitness = get_fitness_score(program.metrics, self.config.feature_dimensions) - existing_fitness = get_fitness_score( - existing_program.metrics, self.config.feature_dimensions - ) - logger.info( - "Island %d MAP-Elites cell improved: %s (fitness: %.3f -> %.3f)", - island_idx, - coords_dict, - existing_fitness, - new_fitness, - ) - - # use MAP-Elites to manage archive - if existing_program_id in self.archive: - self.archive.discard(existing_program_id) - self.archive.add(program.id) - - # Remove replaced program from island set to keep it consistent with feature map - # This prevents accumulation of stale/replaced programs in the island - self.islands[island_idx].discard(existing_program_id) - replaced_program_id = existing_program_id - - island_feature_map[feature_key] = program.id - - # Add to island - self.islands[island_idx].add(program.id) - - # Track which island this program belongs to - program.metadata["island"] = island_idx - - # Update archive - self._update_archive(program) - - # Enforce population size limit BEFORE updating best program tracking - # This ensures newly added programs aren't immediately removed - self._enforce_population_limit(exclude_program_id=program.id) - - # Update the absolute best program tracking (after population enforcement) - self._update_best_program(program) - - # Update island-specific best program tracking - self._update_island_best_program(program, island_idx) - - # If a program was displaced from its cell by this addition, it may now be - # orphaned - owning no cell and belonging to no island. Such a program is a - # "zombie" that consumes a population slot but can never be sampled again, so - # remove it. This runs after best-program tracking is updated so the newly - # added (better) program is already recorded as best, ensuring we never drop - # the current best program here. - if ( - replaced_program_id is not None - and replaced_program_id != program.id - and replaced_program_id != self.best_program_id - ): - self._remove_program_if_orphaned(replaced_program_id) - - # Save to disk if configured - if self.config.db_path: - self._save_program(program) - - logger.debug(f"Added program {program.id} to island {island_idx}") - - return program.id + """Insert a program and apply the session's population/elite policy.""" + ... def get(self, program_id: str) -> Optional[Program]: - """ - Get a program by ID - - Args: - program_id: Program ID - - Returns: - Program or None if not found - """ - return self.programs.get(program_id) - - def sample(self, num_inspirations: Optional[int] = None) -> Tuple[Program, List[Program]]: - """ - Sample a program and inspirations for the next evolution step - - Args: - num_inspirations: Number of inspiration programs to sample (defaults to 5 for backward compatibility) - - Returns: - Tuple of (parent_program, inspiration_programs) - """ - # Select parent program - parent = self._sample_parent() - - # Select inspirations - if num_inspirations is None: - num_inspirations = 5 # Default for backward compatibility - inspirations = self._sample_inspirations(parent, n=num_inspirations) - - logger.debug(f"Sampled parent {parent.id} and {len(inspirations)} inspirations") - return parent, inspirations + """Look up a program by ID.""" + ... def sample_from_island( self, island_id: int, num_inspirations: Optional[int] = None ) -> Tuple[Program, List[Program]]: - """ - Sample a program and inspirations from a specific island without modifying current_island - - This method is thread-safe and doesn't modify shared state, avoiding race conditions - when multiple workers sample from different islands concurrently. - - Uses the same exploration/exploitation/random strategy as sample() to ensure - consistent behavior between single-process and parallel execution modes. - - Args: - island_id: The island to sample from - num_inspirations: Number of inspiration programs to sample (defaults to 5) - - Returns: - Tuple of (parent_program, inspiration_programs) - """ - # Ensure valid island ID - island_id = island_id % len(self.islands) - - # Get programs from the specific island - island_programs = list(self.islands[island_id]) - - if not island_programs: - # Island is empty, fall back to sampling from all programs - logger.debug(f"Island {island_id} is empty, sampling from all programs") - return self.sample(num_inspirations) - - # Use exploration_ratio and exploitation_ratio to decide sampling strategy - # This matches the logic in _sample_parent() for consistent behavior - rand_val = random.random() - - if rand_val < self.config.exploration_ratio: - # EXPLORATION: Sample randomly from island (diverse sampling) - parent = self._sample_from_island_random(island_id) - sampling_mode = "exploration" - elif rand_val < self.config.exploration_ratio + self.config.exploitation_ratio: - # EXPLOITATION: Sample from archive (elite programs) - parent = self._sample_from_archive_for_island(island_id) - sampling_mode = "exploitation" - else: - # WEIGHTED: Use fitness-weighted sampling (remaining probability) - parent = self._sample_from_island_weighted(island_id) - sampling_mode = "weighted" - - # Select inspirations using the same elite/diversity-aware strategy as sample(). - # Pass the requested island explicitly so parallel workers remain isolated even - # if an archive fallback returns a parent whose metadata points elsewhere. - if num_inspirations is None: - num_inspirations = 5 # Default for backward compatibility - - inspirations = self._sample_inspirations(parent, n=num_inspirations, island_id=island_id) - - logger.debug( - f"Sampled parent {parent.id} and {len(inspirations)} inspirations from island {island_id} " - f"(mode: {sampling_mode}, rand_val: {rand_val:.3f})" - ) - return parent, inspirations + """Select a parent and bounded inspirations using the search policy.""" + ... def get_best_program(self, metric: Optional[str] = None) -> Optional[Program]: - """ - Get the best program based on a metric - - Args: - metric: Metric to use for ranking (uses combined_score or average if None) - - Returns: - Best program or None if database is empty - """ - if not self.programs: - return None - - # If no specific metric and we have a tracked best program, return it - if metric is None and self.best_program_id: - if self.best_program_id in self.programs: - logger.debug(f"Using tracked best program: {self.best_program_id}") - return self.programs[self.best_program_id] - else: - logger.warning( - f"Tracked best program {self.best_program_id} no longer exists, will recalculate" - ) - self.best_program_id = None - - if metric: - # Sort by specific metric - sorted_programs = sorted( - [p for p in self.programs.values() if metric in p.metrics], - key=lambda p: p.metrics[metric], - reverse=True, - ) - if sorted_programs: - logger.debug(f"Found best program by metric '{metric}': {sorted_programs[0].id}") - else: - # Sort by fitness (excluding feature dimensions) - sorted_programs = sorted( - self.programs.values(), - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - reverse=True, - ) - if sorted_programs: - logger.debug(f"Found best program by fitness score: {sorted_programs[0].id}") - - # Update the best program tracking if we found a better program - if sorted_programs and ( - self.best_program_id is None or sorted_programs[0].id != self.best_program_id - ): - old_id = self.best_program_id - self.best_program_id = sorted_programs[0].id - logger.info(f"Updated best program tracking from {old_id} to {self.best_program_id}") - - # Also log the scores to help understand the update - if ( - old_id - and old_id in self.programs - and "combined_score" in self.programs[old_id].metrics - and "combined_score" in self.programs[self.best_program_id].metrics - ): - old_score = self.programs[old_id].metrics["combined_score"] - new_score = self.programs[self.best_program_id].metrics["combined_score"] - logger.info( - f"Score change: {old_score:.4f} → {new_score:.4f} ({new_score-old_score:+.4f})" - ) - - return sorted_programs[0] if sorted_programs else None + """Select the best program by fitness or a specified metric.""" + ... def get_top_programs( self, n: int = 10, metric: Optional[str] = None, island_idx: Optional[int] = None ) -> List[Program]: - """ - Get the top N programs based on a metric - - Args: - n: Number of programs to return - metric: Metric to use for ranking (uses average if None) - island_idx: If specified, only return programs from this island - - Returns: - List of top programs - """ - # Validate island_idx parameter - if island_idx is not None and (island_idx < 0 or island_idx >= len(self.islands)): - raise IndexError(f"Island index {island_idx} is out of range (0-{len(self.islands)-1})") - - if not self.programs: - return [] - - # Get candidate programs - if island_idx is not None: - # Island-specific query - island_programs = [ - self.programs[pid] for pid in self.islands[island_idx] if pid in self.programs - ] - candidates = island_programs - else: - # Global query - candidates = list(self.programs.values()) - - if not candidates: - return [] - - if metric: - # Sort by specific metric - sorted_programs = sorted( - [p for p in candidates if metric in p.metrics], - key=lambda p: p.metrics[metric], - reverse=True, - ) - else: - # Sort by combined_score if available, otherwise by average of all numeric metrics - sorted_programs = sorted( - candidates, - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - reverse=True, - ) - - return sorted_programs[:n] - - def save(self, path: Optional[str] = None, iteration: int = 0) -> None: - """ - Save the database to disk - - Args: - path: Path to save to (uses config.db_path if None) - iteration: Current iteration number - """ - save_path = path or self.config.db_path - if not save_path: - logger.warning("No database path specified, skipping save") - return - - # Perform artifact cleanup before saving - self._cleanup_old_artifacts(save_path) - - # create directory if it doesn't exist - os.makedirs(save_path, exist_ok=True) - - # Save each program - for program in self.programs.values(): - prompts = None - if ( - self.config.log_prompts - and self.prompts_by_program - and program.id in self.prompts_by_program - ): - prompts = self.prompts_by_program[program.id] - self._save_program(program, save_path, prompts=prompts) - - # Save metadata - metadata = { - "island_feature_maps": self.island_feature_maps, - "islands": [list(island) for island in self.islands], - "archive": list(self.archive), - "best_program_id": self.best_program_id, - "island_best_programs": self.island_best_programs, - "last_iteration": iteration or self.last_iteration, - "current_island": self.current_island, - "island_generations": self.island_generations, - "last_migration_generation": self.last_migration_generation, - "feature_stats": self._serialize_feature_stats(), - } - - with open(os.path.join(save_path, "metadata.json"), "w") as f: - json.dump(metadata, f) - - logger.info(f"Saved database with {len(self.programs)} programs to {save_path}") - - def load(self, path: str) -> None: - """ - Load the database from disk - - Args: - path: Path to load from - """ - if not os.path.exists(path): - logger.warning(f"Database path {path} does not exist, skipping load") - return - - # Load metadata first - metadata_path = os.path.join(path, "metadata.json") - saved_islands = [] - if os.path.exists(metadata_path): - with open(metadata_path, "r") as f: - metadata = json.load(f) - - self.island_feature_maps = metadata.get( - "island_feature_maps", [{} for _ in range(self.config.num_islands)] - ) - saved_islands = metadata.get("islands", []) - self.archive = set(metadata.get("archive", [])) - self.best_program_id = metadata.get("best_program_id") - self.island_best_programs = metadata.get( - "island_best_programs", [None] * len(saved_islands) - ) - self.last_iteration = metadata.get("last_iteration", 0) - self.current_island = metadata.get("current_island", 0) - self.island_generations = metadata.get("island_generations", [0] * len(saved_islands)) - self.last_migration_generation = metadata.get("last_migration_generation", 0) - - # Load feature_stats for MAP-Elites grid stability - self.feature_stats = self._deserialize_feature_stats(metadata.get("feature_stats", {})) - - logger.info(f"Loaded database metadata with last_iteration={self.last_iteration}") - if self.feature_stats: - logger.info(f"Loaded feature_stats for {len(self.feature_stats)} dimensions") - - # Load programs - programs_dir = os.path.join(path, "programs") - if os.path.exists(programs_dir): - for program_file in os.listdir(programs_dir): - if program_file.endswith(".json"): - program_path = os.path.join(programs_dir, program_file) - try: - with open(program_path, "r") as f: - program_data = json.load(f) - - program = Program.from_dict(program_data) - self.programs[program.id] = program - except Exception as e: - logger.warning(f"Error loading program {program_file}: {str(e)}") - - # Reconstruct island assignments from metadata - self._reconstruct_islands(saved_islands) - - # Ensure island_generations list has correct length - if len(self.island_generations) != len(self.islands): - self.island_generations = [0] * len(self.islands) - - # Ensure island_best_programs list has correct length - if len(self.island_best_programs) != len(self.islands): - self.island_best_programs = [None] * len(self.islands) - - logger.info(f"Loaded database with {len(self.programs)} programs from {path}") - - # Log the reconstructed island status - self.log_island_status() - - def _reconstruct_islands(self, saved_islands: List[List[str]]) -> None: - """ - Reconstruct island assignments from saved metadata - - Args: - saved_islands: List of island program ID lists from metadata - """ - # Initialize empty islands - num_islands = max(len(saved_islands), self.config.num_islands) - self.islands = [set() for _ in range(num_islands)] - - missing_programs = [] - restored_programs = 0 - - # Restore island assignments - for island_idx, program_ids in enumerate(saved_islands): - if island_idx >= len(self.islands): - continue - - for program_id in program_ids: - if program_id in self.programs: - # Program exists, add to island - self.islands[island_idx].add(program_id) - # Set island metadata on the program - self.programs[program_id].metadata["island"] = island_idx - restored_programs += 1 - else: - # Program missing, track it - missing_programs.append((island_idx, program_id)) - - # Clean up archive - remove missing programs - original_archive_size = len(self.archive) - self.archive = {pid for pid in self.archive if pid in self.programs} - - # Clean up island_feature_maps - remove missing programs - feature_keys_to_remove = [] - for island_idx, island_map in enumerate(self.island_feature_maps): - island_keys_to_remove = [] - for key, program_id in island_map.items(): - if program_id not in self.programs: - island_keys_to_remove.append(key) - feature_keys_to_remove.append((island_idx, key)) - for key in island_keys_to_remove: - del island_map[key] - - # Clean up island best programs - remove stale references - self._cleanup_stale_island_bests() - - # Check best program - if self.best_program_id and self.best_program_id not in self.programs: - logger.warning(f"Best program {self.best_program_id} not found, will recalculate") - self.best_program_id = None - - # Log reconstruction results - if missing_programs: - logger.warning( - f"Found {len(missing_programs)} missing programs during island reconstruction:" - ) - for island_idx, program_id in missing_programs[:5]: # Show first 5 - logger.warning(f" Island {island_idx}: {program_id}") - if len(missing_programs) > 5: - logger.warning(f" ... and {len(missing_programs) - 5} more") - - if original_archive_size > len(self.archive): - logger.info( - f"Removed {original_archive_size - len(self.archive)} missing programs from archive" - ) - - if feature_keys_to_remove: - logger.info( - f"Removed {len(feature_keys_to_remove)} missing programs from island feature maps" - ) - - logger.info(f"Reconstructed islands: restored {restored_programs} programs to islands") - - # If we have programs but no island assignments, distribute them - if self.programs and sum(len(island) for island in self.islands) == 0: - logger.info("No island assignments found, distributing programs across islands") - self._distribute_programs_to_islands() - - def _distribute_programs_to_islands(self) -> None: - """ - Distribute loaded programs across islands when no island metadata exists - """ - program_ids = list(self.programs.keys()) - - # Distribute programs round-robin across islands - for i, program_id in enumerate(program_ids): - island_idx = i % len(self.islands) - self.islands[island_idx].add(program_id) - self.programs[program_id].metadata["island"] = island_idx - - logger.info(f"Distributed {len(program_ids)} programs across {len(self.islands)} islands") - - def _save_program( - self, - program: Program, - base_path: Optional[str] = None, - prompts: Optional[Dict[str, Dict[str, str]]] = None, - ) -> None: - """ - Save a program to disk - - Args: - program: Program to save - base_path: Base path to save to (uses config.db_path if None) - prompts: Optional prompts to save with the program, in the format {template_key: { 'system': str, 'user': str }} - """ - save_path = base_path or self.config.db_path - if not save_path: - return - - # Create programs directory if it doesn't exist - programs_dir = os.path.join(save_path, "programs") - os.makedirs(programs_dir, exist_ok=True) - - # Save program - program_dict = program.to_dict() - if prompts: - program_dict["prompts"] = prompts - program_path = os.path.join(programs_dir, f"{program.id}.json") - - with open(program_path, "w") as f: - json.dump(program_dict, f) - - def _calculate_feature_coords(self, program: Program) -> List[int]: - """ - Calculate feature coordinates for the MAP-Elites grid - - Args: - program: Program to calculate features for - - Returns: - List of feature coordinates - """ - coords = [] - - for dim in self.config.feature_dimensions: - # PRIORITY 1: Check if this is a custom metric from the evaluator - # This allows users to override built-in features with their own implementations - if dim in program.metrics: - # Use custom metric from evaluator - score = program.metrics[dim] - # Update stats and scale - self._update_feature_stats(dim, score) - scaled_value = self._scale_feature_value(dim, score) - num_bins = self.feature_bins_per_dim.get(dim, self.feature_bins) - bin_idx = int(scaled_value * num_bins) - bin_idx = max(0, min(num_bins - 1, bin_idx)) - coords.append(bin_idx) - # PRIORITY 2: Fall back to built-in features if not in metrics - elif dim == "complexity": - # Use code length as complexity measure - complexity = len(program.code) - bin_idx = self._calculate_complexity_bin(complexity) - coords.append(bin_idx) - elif dim == "diversity": - # Use cached diversity calculation with reference set - if len(self.programs) < 2: - bin_idx = 0 - else: - diversity = self._get_cached_diversity(program) - bin_idx = self._calculate_diversity_bin(diversity) - coords.append(bin_idx) - elif dim == "score": - # Use average of numeric metrics - if not program.metrics: - bin_idx = 0 - else: - # Use fitness score for "score" dimension (consistent with rest of system) - avg_score = get_fitness_score(program.metrics, self.config.feature_dimensions) - # Update stats and scale - self._update_feature_stats("score", avg_score) - scaled_value = self._scale_feature_value("score", avg_score) - num_bins = self.feature_bins_per_dim.get("score", self.feature_bins) - bin_idx = int(scaled_value * num_bins) - bin_idx = max(0, min(num_bins - 1, bin_idx)) - coords.append(bin_idx) - else: - # Feature not found - this is an error - raise ValueError( - f"Feature dimension '{dim}' specified in config but not found in program metrics. " - f"Available metrics: {list(program.metrics.keys())}. " - f"Built-in features: 'complexity', 'diversity', 'score'. " - f"Either remove '{dim}' from feature_dimensions or ensure your evaluator returns it." - ) - # Only log coordinates at debug level for troubleshooting - logger.debug( - "MAP-Elites coords: %s", - str({self.config.feature_dimensions[i]: coords[i] for i in range(len(coords))}), - ) - return coords - - def _calculate_complexity_bin(self, complexity: int) -> int: - """ - Calculate the bin index for a given complexity value using feature scaling. - - Args: - complexity: The complexity value (code length) - - Returns: - Bin index in range [0, self.feature_bins - 1] - """ - # Update feature statistics - self._update_feature_stats("complexity", float(complexity)) - - # Scale the value using configured method - scaled_value = self._scale_feature_value("complexity", float(complexity)) - - # Get number of bins for this dimension - num_bins = self.feature_bins_per_dim.get("complexity", self.feature_bins) - - # Convert to bin index - bin_idx = int(scaled_value * num_bins) - - # Ensure bin index is within valid range - bin_idx = max(0, min(num_bins - 1, bin_idx)) - - return bin_idx - - def _calculate_diversity_bin(self, diversity: float) -> int: - """ - Calculate the bin index for a given diversity value using feature scaling. - - Args: - diversity: The average fast code diversity to other programs - - Returns: - Bin index in range [0, self.feature_bins - 1] - """ - # Update feature statistics - self._update_feature_stats("diversity", diversity) - - # Scale the value using configured method - scaled_value = self._scale_feature_value("diversity", diversity) - - # Get number of bins for this dimension - num_bins = self.feature_bins_per_dim.get("diversity", self.feature_bins) - - # Convert to bin index - bin_idx = int(scaled_value * num_bins) - - # Ensure bin index is within valid range - bin_idx = max(0, min(num_bins - 1, bin_idx)) - - return bin_idx - - def _feature_coords_to_key(self, coords: List[int]) -> str: - """ - Convert feature coordinates to a string key - - Args: - coords: Feature coordinates - - Returns: - String key - """ - return "-".join(str(c) for c in coords) - - def _cosine_similarity(self, vec1: List[float], vec2: List[float]) -> float: - """ - Adapted from SakanaAI/ShinkaEvolve (Apache-2.0 License) - Original source: https://github.com/SakanaAI/ShinkaEvolve/blob/main/shinka/database/dbase.py#L1452 - - Compute cosine similarity between two vectors. - """ - if not vec1 or not vec2 or len(vec1) != len(vec2): - return 0.0 - - arr1 = np.array(vec1, dtype=np.float32) - arr2 = np.array(vec2, dtype=np.float32) - - norm_a = np.linalg.norm(arr1) - norm_b = np.linalg.norm(arr2) - - if norm_a == 0 or norm_b == 0: - return 0.0 - - similarity = np.dot(arr1, arr2) / (norm_a * norm_b) - - return float(similarity) - - def _llm_judge_novelty(self, program: Program, similar_program: Program) -> bool: - """ - Use LLM to judge if a program is novel compared to a similar existing program - """ - import asyncio - from openevolve.novelty_judge import NOVELTY_SYSTEM_MSG, NOVELTY_USER_MSG - - user_msg = NOVELTY_USER_MSG.format( - language=program.language, - existing_code=similar_program.code, - proposed_code=program.code, - ) - - try: - # Check if we're already in an event loop - try: - loop = asyncio.get_running_loop() - # We're in an async context, need to run in a new thread - import concurrent.futures - - with concurrent.futures.ThreadPoolExecutor() as executor: - future = executor.submit( - asyncio.run, - self.novelty_llm.generate_with_context( - system_message=NOVELTY_SYSTEM_MSG, - messages=[{"role": "user", "content": user_msg}], - ), - ) - content: str = future.result() - except RuntimeError: - # No event loop running, safe to use asyncio.run() - content: str = asyncio.run( - self.novelty_llm.generate_with_context( - system_message=NOVELTY_SYSTEM_MSG, - messages=[{"role": "user", "content": user_msg}], - ) - ) - - if content is None or content is None: - logger.warning("Novelty LLM returned empty response") - return True - - content = content.strip() - - # Parse the response - NOVEL_i = content.upper().find("NOVEL") - NOT_NOVEL_i = content.upper().find("NOT NOVEL") - - if NOVEL_i == -1 and NOT_NOVEL_i == -1: - logger.warning(f"Unexpected novelty LLM response: {content}") - return True # Assume novel if we can't parse - - if NOVEL_i != -1 and NOT_NOVEL_i != -1: - # Both found, take the one that appears first - is_novel = NOVEL_i < NOT_NOVEL_i - elif NOVEL_i != -1: - is_novel = True - else: - is_novel = False - - return is_novel - - except Exception as e: - logger.error(f"Error in novelty LLM check: {e}") - - return True - - def _is_novel(self, program_id: int, island_idx: int) -> bool: - """ - Determine if a program is novel based on diversity to existing programs - - Args: - program: Program to check - island_idx: Island index - - Returns: - True if novel, False otherwise - """ - if self.embedding_client is None or self.similarity_threshold <= 0.0: - # Novelty checking disabled - return True - - program = self.programs[program_id] - embd = self.embedding_client.get_embedding(program.code) - self.programs[program_id].embedding = embd - - max_smlty = float("-inf") - max_smlty_pid = None - - for pid in self.islands[island_idx]: - other = self.programs[pid] - - if other.embedding is None: - logger.warning(f"Program {other.id} has no embedding, skipping similarity check") - continue - - similarity = self._cosine_similarity(embd, other.embedding) - - if similarity >= max(max_smlty, self.similarity_threshold): - max_smlty = similarity - max_smlty_pid = pid - - if max_smlty_pid is None: - # No similar programs found, consider it novel - return True - - return self._llm_judge_novelty(program, self.programs[max_smlty_pid]) - - def _is_better(self, program1: Program, program2: Program) -> bool: - """ - Determine if program1 has better FITNESS than program2 - - Uses fitness calculation that excludes MAP-Elites feature dimensions - to prevent pollution of fitness comparisons. - - Args: - program1: First program - program2: Second program - - Returns: - True if program1 is better than program2 - """ - # If no metrics, use newest - if not program1.metrics and not program2.metrics: - return program1.timestamp > program2.timestamp - - # If only one has metrics, it's better - if program1.metrics and not program2.metrics: - return True - if not program1.metrics and program2.metrics: - return False - - # Compare fitness (excluding feature dimensions) - fitness1 = get_fitness_score(program1.metrics, self.config.feature_dimensions) - fitness2 = get_fitness_score(program2.metrics, self.config.feature_dimensions) - - return fitness1 > fitness2 - - def _update_archive(self, program: Program) -> None: - """ - Update the archive of elite programs - - Args: - program: Program to consider for archive - """ - # If archive not full, add program - if len(self.archive) < self.config.archive_size: - self.archive.add(program.id) - return - - # Clean up stale references and get valid archive programs - valid_archive_programs = [] - stale_ids = [] - - for pid in self.archive: - if pid in self.programs: - valid_archive_programs.append(self.programs[pid]) - else: - stale_ids.append(pid) - - # Remove stale references from archive - for stale_id in stale_ids: - self.archive.discard(stale_id) - logger.debug(f"Removing stale program {stale_id} from archive") - - # If archive is now not full after cleanup, just add the new program - if len(self.archive) < self.config.archive_size: - self.archive.add(program.id) - return - - # Find worst program among valid programs - if valid_archive_programs: - worst_program = min( - valid_archive_programs, - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - ) - - # Replace if new program is better - if self._is_better(program, worst_program): - self.archive.remove(worst_program.id) - self.archive.add(program.id) - else: - # No valid programs in archive, just add the new one - self.archive.add(program.id) - - def _update_best_program(self, program: Program) -> None: - """ - Update the absolute best program tracking - - Args: - program: Program to consider as the new best - """ - # If we don't have a best program yet, this becomes the best - if self.best_program_id is None: - self.best_program_id = program.id - logger.debug(f"Set initial best program to {program.id}") - return - - # Compare with current best program (if it still exists) - if self.best_program_id not in self.programs: - logger.warning( - f"Best program {self.best_program_id} no longer exists, clearing reference" - ) - self.best_program_id = program.id - logger.info(f"Set new best program to {program.id}") - return - - current_best = self.programs[self.best_program_id] - - # Update if the new program is better - if self._is_better(program, current_best): - old_id = self.best_program_id - self.best_program_id = program.id - - # Log the change - if "combined_score" in program.metrics and "combined_score" in current_best.metrics: - old_score = current_best.metrics["combined_score"] - new_score = program.metrics["combined_score"] - score_diff = new_score - old_score - logger.info( - f"New best program {program.id} replaces {old_id} (combined_score: {old_score:.4f} → {new_score:.4f}, +{score_diff:.4f})" - ) - else: - logger.info(f"New best program {program.id} replaces {old_id}") - - def _update_island_best_program(self, program: Program, island_idx: int) -> None: - """ - Update the best program tracking for a specific island - - Args: - program: Program to consider as the new best for the island - island_idx: Island index - """ - # Ensure island_idx is valid - if island_idx >= len(self.island_best_programs): - logger.warning(f"Invalid island index {island_idx}, skipping island best update") - return - - # If island doesn't have a best program yet, this becomes the best - current_island_best_id = self.island_best_programs[island_idx] - if current_island_best_id is None: - self.island_best_programs[island_idx] = program.id - logger.debug(f"Set initial best program for island {island_idx} to {program.id}") - return - - # Check if current best still exists - if current_island_best_id not in self.programs: - logger.warning( - f"Island {island_idx} best program {current_island_best_id} no longer exists, updating to {program.id}" - ) - self.island_best_programs[island_idx] = program.id - return - - current_island_best = self.programs[current_island_best_id] - - # Update if the new program is better - if self._is_better(program, current_island_best): - old_id = current_island_best_id - self.island_best_programs[island_idx] = program.id - - # Log the change - if ( - "combined_score" in program.metrics - and "combined_score" in current_island_best.metrics - ): - old_score = current_island_best.metrics["combined_score"] - new_score = program.metrics["combined_score"] - score_diff = new_score - old_score - logger.debug( - f"Island {island_idx}: New best program {program.id} replaces {old_id} " - f"(combined_score: {old_score:.4f} → {new_score:.4f}, +{score_diff:.4f})" - ) - else: - logger.debug( - f"Island {island_idx}: New best program {program.id} replaces {old_id}" - ) - - def _sample_parent(self) -> Program: - """ - Sample a parent program from the current island for the next evolution step - - Returns: - Parent program from current island - """ - # Use exploration_ratio and exploitation_ratio to decide sampling strategy - rand_val = random.random() - - if rand_val < self.config.exploration_ratio: - # EXPLORATION: Sample from current island (diverse sampling) - return self._sample_exploration_parent() - elif rand_val < self.config.exploration_ratio + self.config.exploitation_ratio: - # EXPLOITATION: Sample from archive (elite programs) - return self._sample_exploitation_parent() - else: - # RANDOM: Sample from any program (remaining probability) - return self._sample_random_parent() - - def _sample_exploration_parent(self) -> Program: - """ - Sample a parent for exploration (from current island) - """ - current_island_programs = self.islands[self.current_island] - - if not current_island_programs: - # If current island is empty, initialize with best program or random program - if self.best_program_id and self.best_program_id in self.programs: - # Create a copy of best program for the empty island (don't reuse same ID) - best_program = self.programs[self.best_program_id] - copy_program = Program( - id=str(uuid.uuid4()), - code=best_program.code, - changes_description=best_program.changes_description, - language=best_program.language, - parent_id=best_program.id, - generation=best_program.generation, - timestamp=time.time(), - iteration_found=self.last_iteration, - metrics=best_program.metrics.copy(), - complexity=best_program.complexity, - diversity=best_program.diversity, - metadata={"island": self.current_island}, - artifacts_json=best_program.artifacts_json, - artifact_dir=best_program.artifact_dir, - ) - self.programs[copy_program.id] = copy_program - self.islands[self.current_island].add(copy_program.id) - logger.debug( - f"Initialized empty island {self.current_island} with copy of best program" - ) - return copy_program - else: - # Use any available program - return next(iter(self.programs.values())) - - # Clean up stale references and sample from current island - valid_programs = [pid for pid in current_island_programs if pid in self.programs] - - # Remove stale program IDs from island - if len(valid_programs) < len(current_island_programs): - stale_ids = current_island_programs - set(valid_programs) - logger.debug( - f"Removing {len(stale_ids)} stale program IDs from island {self.current_island}" - ) - for stale_id in stale_ids: - self.islands[self.current_island].discard(stale_id) - - # If no valid programs after cleanup, reinitialize island - if not valid_programs: - logger.warning( - f"Island {self.current_island} has no valid programs after cleanup, reinitializing" - ) - if self.best_program_id and self.best_program_id in self.programs: - # Create a copy of best program for the empty island (don't reuse same ID) - best_program = self.programs[self.best_program_id] - copy_program = Program( - id=str(uuid.uuid4()), - code=best_program.code, - changes_description=best_program.changes_description, - language=best_program.language, - parent_id=best_program.id, - generation=best_program.generation, - timestamp=time.time(), - iteration_found=self.last_iteration, - metrics=best_program.metrics.copy(), - complexity=best_program.complexity, - diversity=best_program.diversity, - metadata={"island": self.current_island}, - artifacts_json=best_program.artifacts_json, - artifact_dir=best_program.artifact_dir, - ) - self.programs[copy_program.id] = copy_program - self.islands[self.current_island].add(copy_program.id) - logger.debug( - f"Reinitialized empty island {self.current_island} with copy of best program" - ) - return copy_program - else: - return next(iter(self.programs.values())) - - # Sample from valid programs - parent_id = random.choice(valid_programs) - return self.programs[parent_id] - - def _sample_exploitation_parent(self) -> Program: - """ - Sample a parent for exploitation (from archive/elite programs) - """ - if not self.archive: - # Fallback to exploration if no archive - return self._sample_exploration_parent() - - # Clean up stale references in archive - valid_archive = [pid for pid in self.archive if pid in self.programs] - - # Remove stale program IDs from archive - if len(valid_archive) < len(self.archive): - stale_ids = self.archive - set(valid_archive) - logger.debug(f"Removing {len(stale_ids)} stale program IDs from archive") - for stale_id in stale_ids: - self.archive.discard(stale_id) - - # If no valid archive programs, fallback to exploration - if not valid_archive: - logger.warning( - "Archive has no valid programs after cleanup, falling back to exploration" - ) - return self._sample_exploration_parent() - - # Prefer programs from current island in archive - archive_programs_in_island = [ - pid - for pid in valid_archive - if self.programs[pid].metadata.get("island") == self.current_island - ] - - if archive_programs_in_island: - parent_id = random.choice(archive_programs_in_island) - return self.programs[parent_id] - else: - # Fall back to any valid archive program if current island has none - parent_id = random.choice(valid_archive) - return self.programs[parent_id] - - def _sample_random_parent(self) -> Program: - """ - Sample a completely random parent from all programs - """ - if not self.programs: - raise ValueError("No programs available for sampling") - - # Sample randomly from all programs - program_id = random.choice(list(self.programs.keys())) - return self.programs[program_id] - - def _sample_from_island_weighted(self, island_id: int) -> Program: - """ - Sample a parent from a specific island using fitness-weighted selection - - Args: - island_id: The island to sample from - - Returns: - Parent program selected using fitness-weighted sampling - """ - island_id = island_id % len(self.islands) - island_programs = list(self.islands[island_id]) - - if not island_programs: - # Island is empty, fall back to any available program - logger.debug(f"Island {island_id} is empty, sampling from all programs") - return self._sample_random_parent() - - # Select parent from island programs - if len(island_programs) == 1: - parent_id = island_programs[0] - else: - # Use weighted sampling based on program scores - island_program_objects = [ - self.programs[pid] for pid in island_programs if pid in self.programs - ] - - if not island_program_objects: - # Fallback if programs not found - parent_id = random.choice(island_programs) - else: - # Calculate weights based on fitness scores - weights = [] - for prog in island_program_objects: - fitness = get_fitness_score(prog.metrics, self.config.feature_dimensions) - # Add small epsilon to avoid zero weights - weights.append(max(fitness, 0.001)) - - # Normalize weights - total_weight = sum(weights) - if total_weight > 0: - weights = [w / total_weight for w in weights] - else: - weights = [1.0 / len(island_program_objects)] * len(island_program_objects) - - # Sample parent based on weights - parent = random.choices(island_program_objects, weights=weights, k=1)[0] - parent_id = parent.id - - parent = self.programs.get(parent_id) - if not parent: - # Should not happen, but handle gracefully - logger.error(f"Parent program {parent_id} not found in database") - return self._sample_random_parent() - - return parent - - def _sample_from_island_random(self, island_id: int) -> Program: - """ - Sample a completely random parent from a specific island (uniform distribution) - - Args: - island_id: The island to sample from - - Returns: - Parent program selected uniformly at random - """ - island_id = island_id % len(self.islands) - island_programs = list(self.islands[island_id]) - - if not island_programs: - # Island is empty, fall back to any available program - logger.debug(f"Island {island_id} is empty, sampling from all programs") - return self._sample_random_parent() - - # Clean up stale references - valid_programs = [pid for pid in island_programs if pid in self.programs] - - if not valid_programs: - logger.warning( - f"Island {island_id} has no valid programs, falling back to random sampling" - ) - return self._sample_random_parent() - - # Uniform random selection - parent_id = random.choice(valid_programs) - return self.programs[parent_id] - - def _sample_from_archive_for_island(self, island_id: int) -> Program: - """ - Sample a parent from the archive, preferring programs from the specified island - - Args: - island_id: The island to prefer programs from - - Returns: - Parent program from archive (preferably from the specified island) - """ - if not self.archive: - # Fallback to weighted sampling from island - logger.debug(f"Archive is empty, falling back to weighted island sampling") - return self._sample_from_island_weighted(island_id) - - # Clean up stale references in archive - valid_archive = [pid for pid in self.archive if pid in self.programs] - - if not valid_archive: - logger.warning( - "Archive has no valid programs, falling back to weighted island sampling" - ) - return self._sample_from_island_weighted(island_id) - - island_id = island_id % len(self.islands) - - # Prefer programs from the specified island in archive - archive_programs_in_island = [ - pid for pid in valid_archive if self.programs[pid].metadata.get("island") == island_id - ] - - if archive_programs_in_island: - parent_id = random.choice(archive_programs_in_island) - return self.programs[parent_id] - else: - # Fall back to any valid archive program if island has none - parent_id = random.choice(valid_archive) - return self.programs[parent_id] - - def _sample_inspirations( - self, parent: Program, n: int = 5, island_id: Optional[int] = None - ) -> List[Program]: - """ - Sample inspiration programs for the next evolution step. - - For proper island-based evolution, inspirations are sampled ONLY from the - current island, maintaining genetic isolation between islands. - - Args: - parent: Parent program - n: Number of inspirations to sample - island_id: Explicit island to sample from. If omitted, use the - parent program's island metadata. - - Returns: - List of inspiration programs from the current island - """ - inspirations = [] - - # Prefer an explicitly requested island. This matters for - # sample_from_island(), where archive fallback may return a parent whose - # metadata belongs to a different island. - if island_id is None: - parent_island = parent.metadata.get("island", self.current_island) - else: - parent_island = island_id - - parent_island %= len(self.islands) - - # Get all programs from the current island - island_program_ids = list(self.islands[parent_island]) - island_programs = [self.programs[pid] for pid in island_program_ids if pid in self.programs] - - if not island_programs: - logger.warning(f"Island {parent_island} has no programs for inspiration sampling") - return [] - - # Include the island's best program if available and different from parent - island_best_id = self.island_best_programs[parent_island] - if ( - island_best_id is not None - and island_best_id != parent.id - and island_best_id in self.programs - ): - island_best = self.programs[island_best_id] - inspirations.append(island_best) - logger.debug( - f"Including island {parent_island} best program {island_best_id} in inspirations" - ) - elif island_best_id is not None and island_best_id not in self.programs: - # Clean up stale island best reference - logger.warning( - f"Island {parent_island} best program {island_best_id} no longer exists, clearing reference" - ) - self.island_best_programs[parent_island] = None - - # Add top programs from the island as inspirations - top_n = max(1, int(n * self.config.elite_selection_ratio)) - top_island_programs = self.get_top_programs(n=top_n, island_idx=parent_island) - for program in top_island_programs: - if program.id not in [p.id for p in inspirations] and program.id != parent.id: - inspirations.append(program) - - # Add diverse programs from within the island - if len(island_programs) > n and len(inspirations) < n: - remaining_slots = n - len(inspirations) - - # Try to sample from different feature cells within the island - feature_coords = self._calculate_feature_coords(parent) - nearby_programs = [] - - # Create a mapping of feature cells to island programs for efficient lookup - island_feature_map = {} - for prog_id in island_program_ids: - if prog_id in self.programs: - prog = self.programs[prog_id] - prog_coords = self._calculate_feature_coords(prog) - cell_key = self._feature_coords_to_key(prog_coords) - island_feature_map[cell_key] = prog_id + """Select at most n programs in descending fitness/metric order.""" + ... - # Try to find programs from nearby feature cells within the island - for _ in range(remaining_slots * 3): # Try more times to find nearby programs - # Perturb coordinates - perturbed_coords = [ - max(0, min(self.feature_bins - 1, c + random.randint(-2, 2))) - for c in feature_coords - ] - - cell_key = self._feature_coords_to_key(perturbed_coords) - if cell_key in island_feature_map: - program_id = island_feature_map[cell_key] - if ( - program_id != parent.id - and program_id not in [p.id for p in inspirations] - and program_id not in [p.id for p in nearby_programs] - and program_id in self.programs - ): - nearby_programs.append(self.programs[program_id]) - if len(nearby_programs) >= remaining_slots: - break - - # If we still need more, add random programs from the island - if len(inspirations) + len(nearby_programs) < n: - remaining = n - len(inspirations) - len(nearby_programs) - - # Get available programs from the island - excluded_ids = ( - {parent.id} - .union(p.id for p in inspirations) - .union(p.id for p in nearby_programs) - ) - available_island_ids = [ - pid - for pid in island_program_ids - if pid not in excluded_ids and pid in self.programs - ] - - if available_island_ids: - random_ids = random.sample( - available_island_ids, min(remaining, len(available_island_ids)) - ) - random_programs = [self.programs[pid] for pid in random_ids] - nearby_programs.extend(random_programs) - - inspirations.extend(nearby_programs) - - # Log island isolation info - logger.debug( - f"Sampled {len(inspirations)} inspirations from island {parent_island} " - f"(island has {len(island_programs)} programs total)" - ) - - return inspirations[:n] - - def _remove_program_if_orphaned(self, program_id: str) -> None: - """ - Remove a program from the population if it is orphaned. - - A program is considered orphaned when it no longer owns a MAP-Elites cell - in any island's feature map and is not a member of any island. Such a - program (e.g. one displaced when its cell was improved) can never be - sampled again but still counts against the population size limit, so it is - removed from ``self.programs``, the archive and any lingering references. - - Args: - program_id: ID of the (possibly) orphaned program to check and remove - """ - if program_id not in self.programs: - return - - # Still owns a cell in some island? Then it is not orphaned. - for island_map in self.island_feature_maps: - if program_id in island_map.values(): - return - - # Still a member of some island? Then it is not orphaned. - for island in self.islands: - if program_id in island: - return - - # Fully orphaned - remove from all remaining structures. - del self.programs[program_id] - self.archive.discard(program_id) - self._cleanup_stale_island_bests() - logger.debug(f"Removed orphaned program {program_id} displaced from its cell") - - def _enforce_population_limit(self, exclude_program_id: Optional[str] = None) -> None: - """ - Enforce the population size limit by removing worst programs if needed - - Args: - exclude_program_id: Program ID to never remove (e.g., newly added program) - """ - if len(self.programs) <= self.config.population_size: - return - - # Calculate how many programs to remove - num_to_remove = len(self.programs) - self.config.population_size - - logger.info( - f"Population size ({len(self.programs)}) exceeds limit ({self.config.population_size}), removing {num_to_remove} programs" - ) - - # Collect all MAP-Elites cell owners across every island. These "elite" - # programs represent occupied niches and must be protected from eviction - # to preserve diversity - a low-scoring cell owner should only be removed - # after every non-owning (homeless) program has already been removed. - elite_ids = set() - for island_map in self.island_feature_maps: - elite_ids.update(island_map.values()) - - # Never remove the best program or the excluded (just-added) program - protected_ids = {self.best_program_id, exclude_program_id} - {None} - - all_programs = list(self.programs.values()) - - # Split into non-elite (homeless) and elite (cell owners), each sorted by - # fitness worst-first. Non-elite programs are removed before elite ones. - non_elite = sorted( - [p for p in all_programs if p.id not in elite_ids and p.id not in protected_ids], - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - ) - elite = sorted( - [p for p in all_programs if p.id in elite_ids and p.id not in protected_ids], - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - ) - - # Remove non-elite programs first; only fall back to evicting elite cell - # owners (worst first) if removing all homeless programs is not enough. - programs_to_remove = non_elite[:num_to_remove] - if len(programs_to_remove) < num_to_remove: - remaining = num_to_remove - len(programs_to_remove) - programs_to_remove.extend(elite[:remaining]) - - # Remove the selected programs - for program in programs_to_remove: - program_id = program.id - - # Remove from main programs dict - if program_id in self.programs: - del self.programs[program_id] - - # Remove from island feature maps - for island_idx, island_map in enumerate(self.island_feature_maps): - keys_to_remove = [] - for key, pid in island_map.items(): - if pid == program_id: - keys_to_remove.append(key) - for key in keys_to_remove: - del island_map[key] - - # Remove from islands - for island in self.islands: - island.discard(program_id) - - # Remove from archive - self.archive.discard(program_id) - - logger.debug(f"Removed program {program_id} due to population limit") - - logger.info(f"Population size after cleanup: {len(self.programs)}") - - # Clean up any stale island best program references after removal - self._cleanup_stale_island_bests() - - # Island management methods - def set_current_island(self, island_idx: int) -> None: - """Set which island is currently being evolved""" - self.current_island = island_idx % len(self.islands) - logger.debug(f"Switched to evolving island {self.current_island}") - - def next_island(self) -> int: - """Move to the next island in round-robin fashion""" - self.current_island = (self.current_island + 1) % len(self.islands) - logger.debug(f"Advanced to island {self.current_island}") - return self.current_island + def get_island_stats(self) -> List[Dict[str, Any]]: + """Return aggregate statistics, not island populations.""" + ... def increment_island_generation(self, island_idx: Optional[int] = None) -> None: - """Increment generation counter for an island""" - idx = island_idx if island_idx is not None else self.current_island - self.island_generations[idx] += 1 - logger.debug(f"Island {idx} generation incremented to {self.island_generations[idx]}") + """Advance the generation counter for an island.""" + ... def should_migrate(self) -> bool: - """Check if migration should occur based on generation counters""" - max_generation = max(self.island_generations) - return (max_generation - self.last_migration_generation) >= self.migration_interval + """Query whether the configured migration interval has elapsed.""" + ... def migrate_programs(self) -> None: - """ - Perform migration between islands - - This should be called periodically to share good solutions between islands - """ - if len(self.islands) < 2: - return - - logger.info("Performing migration between islands") - - for i, island in enumerate(self.islands): - if len(island) == 0: - continue - - # Select top programs from this island for migration - island_programs = [self.programs[pid] for pid in island if pid in self.programs] - if not island_programs: - continue - - # Sort by fitness (using combined_score or average metrics) - island_programs.sort( - key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), - reverse=True, - ) - - # Select top programs for migration - num_to_migrate = max(1, int(len(island_programs) * self.migration_rate)) - migrants = island_programs[:num_to_migrate] - - # Migrate to adjacent islands (ring topology) - target_islands = [(i + 1) % len(self.islands), (i - 1) % len(self.islands)] - - for migrant in migrants: - # Prevent re-migration of already migrated programs to avoid exponential duplication. - # Analysis of actual evolution runs shows this causes severe issues: - # - Program cb5d07f2 had 183 descendant copies by iteration 850 - # - Program 5645fbd2 had 31 descendant copies - # - IDs grow exponentially: program_migrant_2_migrant_3_migrant_4_migrant_0... - # - # This is particularly problematic for OpenEvolve's MAP-Elites + Island hybrid architecture: - # 1. All copies have identical code → same complexity/diversity/performance scores - # 2. They all map to the SAME MAP-Elites cell → only 1 survives, rest discarded - # 3. Wastes computation evaluating hundreds of identical programs - # 4. Reduces actual diversity as islands fill with duplicates - # - # By preventing already-migrated programs from migrating again, we ensure: - # - Each program migrates at most once per lineage - # - True diversity is maintained between islands - # - Computational resources aren't wasted on duplicates - # - Aligns with MAP-Elites' one-program-per-cell principle - if migrant.metadata.get("migrant", False): - continue - - for target_island in target_islands: - # Skip migration if target island already has a program with identical code - # Identical code produces identical metrics, so migration would be wasteful - target_island_programs = [ - self.programs[pid] - for pid in self.islands[target_island] - if pid in self.programs - ] - has_duplicate_code = any(p.code == migrant.code for p in target_island_programs) - - if has_duplicate_code: - logger.debug( - f"Skipping migration of program {migrant.id[:8]} to island {target_island} " - f"(duplicate code already exists)" - ) - continue - # Create a copy for migration with simple new UUID - import uuid - - migrant_copy = Program( - id=str(uuid.uuid4()), - code=migrant.code, - changes_description=migrant.changes_description, - language=migrant.language, - parent_id=migrant.id, - generation=migrant.generation, - metrics=migrant.metrics.copy(), - metadata={**migrant.metadata, "island": target_island, "migrant": True}, - ) - - # Use add() method to properly handle MAP-Elites deduplication, - # feature map updates, and island tracking - self.add(migrant_copy, target_island=target_island) - - # Log migration - logger.info( - "Program %s migrated to island %d", - migrant_copy.id[:8], - target_island, - ) - - # Update last migration generation - self.last_migration_generation = max(self.island_generations) - logger.info(f"Migration completed at generation {self.last_migration_generation}") - - # Validate migration results - self._validate_migration_results() - - def _validate_migration_results(self) -> None: - """ - Validate migration didn't create inconsistencies - - Checks that: - 1. Program island metadata matches actual island assignment - 2. No programs are assigned to multiple islands - 3. All island best programs exist and are in correct islands - """ - seen_program_ids = set() - - for i, island in enumerate(self.islands): - for program_id in island: - # Check for duplicate assignments - if program_id in seen_program_ids: - logger.error(f"Program {program_id} assigned to multiple islands") - continue - seen_program_ids.add(program_id) - - # Check program exists - if program_id not in self.programs: - logger.warning(f"Island {i} contains nonexistent program {program_id}") - continue - - # Check metadata consistency - program = self.programs[program_id] - stored_island = program.metadata.get("island") - if stored_island != i: - logger.warning( - f"Island mismatch for program {program_id}: " - f"in island {i} but metadata says {stored_island}" - ) - - # Validate island best programs - for i, best_id in enumerate(self.island_best_programs): - if best_id is not None: - if best_id not in self.programs: - logger.warning(f"Island {i} best program {best_id} does not exist") - elif best_id not in self.islands[i]: - logger.warning(f"Island {i} best program {best_id} not in island") - - def _cleanup_stale_island_bests(self) -> None: - """ - Remove stale island best program references - - Cleans up references to programs that no longer exist in the database - or are not actually in their assigned islands. - """ - cleaned_count = 0 - - for i, best_id in enumerate(self.island_best_programs): - if best_id is not None: - should_clear = False - - # Check if program still exists - if best_id not in self.programs: - logger.debug( - f"Clearing stale island {i} best program {best_id} (program deleted)" - ) - should_clear = True - # Check if program is still in the island - elif best_id not in self.islands[i]: - logger.debug( - f"Clearing stale island {i} best program {best_id} (not in island)" - ) - should_clear = True - - if should_clear: - self.island_best_programs[i] = None - cleaned_count += 1 - - if cleaned_count > 0: - logger.info(f"Cleaned up {cleaned_count} stale island best program references") - - # Recalculate best programs for islands that were cleared - for i, best_id in enumerate(self.island_best_programs): - if best_id is None and len(self.islands[i]) > 0: - # Find new best program for this island - island_programs = [ - self.programs[pid] for pid in self.islands[i] if pid in self.programs - ] - if island_programs: - # Sort by fitness and update - best_program = max( - island_programs, - key=lambda p: p.metrics.get( - "combined_score", safe_numeric_average(p.metrics) - ), - ) - self.island_best_programs[i] = best_program.id - logger.debug(f"Recalculated island {i} best program: {best_program.id}") - - def get_island_stats(self) -> List[dict]: - """Get statistics for each island""" - stats = [] - - for i, island in enumerate(self.islands): - island_programs = [self.programs[pid] for pid in island if pid in self.programs] - - if island_programs: - scores = [ - get_fitness_score(p.metrics, self.config.feature_dimensions) - for p in island_programs - ] - - best_score = max(scores) if scores else 0.0 - avg_score = sum(scores) / len(scores) if scores else 0.0 - diversity = self._calculate_island_diversity(island_programs) - else: - best_score = avg_score = diversity = 0.0 - - stats.append( - { - "island": i, - "population_size": len(island_programs), - "best_score": best_score, - "average_score": avg_score, - "diversity": diversity, - "generation": self.island_generations[i], - "is_current": i == self.current_island, - } - ) - - return stats - - def _calculate_island_diversity(self, programs: List[Program]) -> float: - """Calculate diversity within an island (deterministic version)""" - if len(programs) < 2: - return 0.0 - - total_diversity = 0 - comparisons = 0 - - # Use deterministic sampling instead of random.sample() to ensure consistent results - sample_size = min(5, len(programs)) # Reduced from 10 to 5 - - # Sort programs by ID for deterministic ordering - sorted_programs = sorted(programs, key=lambda p: p.id) - - # Take first N programs instead of random sampling - sample_programs = sorted_programs[:sample_size] - - # Limit total comparisons for performance - max_comparisons = 6 # Maximum comparisons to prevent long delays - - for i, prog1 in enumerate(sample_programs): - for prog2 in sample_programs[i + 1 :]: - if comparisons >= max_comparisons: - break - - # Use fast approximation instead of expensive edit distance - diversity = self._fast_code_diversity(prog1.code, prog2.code) - total_diversity += diversity - comparisons += 1 - - if comparisons >= max_comparisons: - break - - return total_diversity / max(1, comparisons) - - def _fast_code_diversity(self, code1: str, code2: str) -> float: - """ - Fast approximation of code diversity using simple metrics - - Returns diversity score (higher = more diverse) - """ - if code1 == code2: - return 0.0 - - # Length difference (scaled to reasonable range) - len1, len2 = len(code1), len(code2) - length_diff = abs(len1 - len2) - - # Line count difference - lines1 = code1.count("\n") - lines2 = code2.count("\n") - line_diff = abs(lines1 - lines2) - - # Simple character set difference - chars1 = set(code1) - chars2 = set(code2) - char_diff = len(chars1.symmetric_difference(chars2)) - - # Combine metrics (scaled to match original edit distance range) - diversity = length_diff * 0.1 + line_diff * 10 + char_diff * 0.5 - - return diversity - - def _get_cached_diversity(self, program: Program) -> float: - """ - Get diversity score for a program using cache and reference set - - Args: - program: The program to calculate diversity for - - Returns: - Diversity score (cached or newly computed) - """ - code_hash = hash(program.code) - - # Check cache first - if code_hash in self.diversity_cache: - return self.diversity_cache[code_hash]["value"] - - # Update reference set if needed - if ( - not self.diversity_reference_set - or len(self.diversity_reference_set) < self.diversity_reference_size - ): - self._update_diversity_reference_set() - - # Compute diversity against reference set - diversity_scores = [] - for ref_code in self.diversity_reference_set: - if ref_code != program.code: # Don't compare with itself - diversity_scores.append(self._fast_code_diversity(program.code, ref_code)) - - diversity = ( - sum(diversity_scores) / max(1, len(diversity_scores)) if diversity_scores else 0.0 - ) - - # Cache the result with LRU eviction - self._cache_diversity_value(code_hash, diversity) - - return diversity - - def _update_diversity_reference_set(self) -> None: - """Update the reference set for diversity calculation""" - if len(self.programs) == 0: - return - - # Select diverse programs for reference set - all_programs = list(self.programs.values()) - - if len(all_programs) <= self.diversity_reference_size: - self.diversity_reference_set = [p.code for p in all_programs] - else: - # Select programs with maximum diversity - selected = [] - remaining = all_programs.copy() - - # Start with a random program - first_idx = random.randint(0, len(remaining) - 1) - selected.append(remaining.pop(first_idx)) - - # Greedily add programs that maximize diversity to selected set - while len(selected) < self.diversity_reference_size and remaining: - max_diversity = -1 - best_idx = -1 - - for i, candidate in enumerate(remaining): - # Calculate minimum diversity to selected programs - min_div = float("inf") - for selected_prog in selected: - div = self._fast_code_diversity(candidate.code, selected_prog.code) - min_div = min(min_div, div) - - if min_div > max_diversity: - max_diversity = min_div - best_idx = i - - if best_idx >= 0: - selected.append(remaining.pop(best_idx)) - - self.diversity_reference_set = [p.code for p in selected] - - logger.debug( - f"Updated diversity reference set with {len(self.diversity_reference_set)} programs" - ) - - def _cache_diversity_value(self, code_hash: int, diversity: float) -> None: - """Cache a diversity value with LRU eviction""" - # Check if cache is full - if len(self.diversity_cache) >= self.diversity_cache_size: - # Remove oldest entry - oldest_hash = min(self.diversity_cache.items(), key=lambda x: x[1]["timestamp"])[0] - del self.diversity_cache[oldest_hash] - - # Add new entry - self.diversity_cache[code_hash] = {"value": diversity, "timestamp": time.time()} - - def _invalidate_diversity_cache(self) -> None: - """Invalidate the diversity cache when programs change significantly""" - self.diversity_cache.clear() - self.diversity_reference_set = [] - logger.debug("Diversity cache invalidated") - - def _update_feature_stats(self, feature_name: str, value: float) -> None: - """ - Update statistics for a feature dimension - - Args: - feature_name: Name of the feature dimension - value: New value to incorporate into stats - """ - if feature_name not in self.feature_stats: - self.feature_stats[feature_name] = { - "min": value, - "max": value, - "values": [], # Keep recent values for percentile calculation if needed - } - - stats = self.feature_stats[feature_name] - stats["min"] = min(stats["min"], value) - stats["max"] = max(stats["max"], value) - - # Keep recent values for more sophisticated scaling methods - stats["values"].append(value) - if len(stats["values"]) > 1000: # Limit memory usage - stats["values"] = stats["values"][-1000:] - - def _scale_feature_value(self, feature_name: str, value: float) -> float: - """ - Scale a feature value according to the configured scaling method - - Args: - feature_name: Name of the feature dimension - value: Raw feature value - - Returns: - Scaled value in range [0, 1] - """ - if feature_name not in self.feature_stats: - # No stats yet, return normalized by a reasonable default - return min(1.0, max(0.0, value)) - - stats = self.feature_stats[feature_name] - - if self.feature_scaling_method == "minmax": - # Min-max normalization to [0, 1] - min_val = stats["min"] - max_val = stats["max"] - - if max_val == min_val: - return 0.5 # All values are the same - - scaled = (value - min_val) / (max_val - min_val) - return min(1.0, max(0.0, scaled)) # Ensure in [0, 1] - - elif self.feature_scaling_method == "percentile": - # Use percentile ranking - values = stats["values"] - if not values: - return 0.5 - - # Count how many values are less than or equal to this value - count = sum(1 for v in values if v <= value) - percentile = count / len(values) - return percentile - - else: - # Default to min-max if unknown method - return self._scale_feature_value_minmax(feature_name, value) - - def _scale_feature_value_minmax(self, feature_name: str, value: float) -> float: - """Helper for min-max scaling""" - if feature_name not in self.feature_stats: - return min(1.0, max(0.0, value)) - - stats = self.feature_stats[feature_name] - min_val = stats["min"] - max_val = stats["max"] - - if max_val == min_val: - return 0.5 - - scaled = (value - min_val) / (max_val - min_val) - return min(1.0, max(0.0, scaled)) - - def _serialize_feature_stats(self) -> Dict[str, Any]: - """ - Serialize feature_stats for JSON storage - - Returns: - Dictionary that can be JSON-serialized - """ - serialized = {} - for feature_name, stats in self.feature_stats.items(): - # Convert to JSON-serializable format - serialized_stats = {} - for key, value in stats.items(): - if key == "values": - # Limit size to prevent excessive memory usage - # Keep only the most recent 100 values for percentile calculations - if isinstance(value, list) and len(value) > 100: - serialized_stats[key] = value[-100:] - else: - serialized_stats[key] = value - else: - # Convert numpy types to Python native types - if hasattr(value, "item"): # numpy scalar - serialized_stats[key] = value.item() - else: - serialized_stats[key] = value - serialized[feature_name] = serialized_stats - return serialized - - def _deserialize_feature_stats( - self, stats_dict: Dict[str, Any] - ) -> Dict[str, Dict[str, Union[float, List[float]]]]: - """ - Deserialize feature_stats from loaded JSON - - Args: - stats_dict: Dictionary loaded from JSON - - Returns: - Properly formatted feature_stats dictionary - """ - if not stats_dict: - return {} - - deserialized = {} - for feature_name, stats in stats_dict.items(): - if isinstance(stats, dict): - # Ensure proper structure and types - deserialized_stats = { - "min": float(stats.get("min", 0.0)), - "max": float(stats.get("max", 1.0)), - "values": list(stats.get("values", [])), - } - deserialized[feature_name] = deserialized_stats - else: - logger.warning( - f"Skipping malformed feature_stats entry for '{feature_name}': {stats}" - ) - - return deserialized - - def log_island_status(self) -> None: - """Log current status of all islands""" - stats = self.get_island_stats() - logger.info("Island Status:") - for stat in stats: - current_marker = " *" if stat["is_current"] else " " - island_idx = stat["island"] - island_best_id = ( - self.island_best_programs[island_idx] - if island_idx < len(self.island_best_programs) - else None - ) - best_indicator = f" (best: {island_best_id})" if island_best_id else "" - logger.info( - f"{current_marker} Island {stat['island']}: {stat['population_size']} programs, " - f"best={stat['best_score']:.4f}, avg={stat['average_score']:.4f}, " - f"diversity={stat['diversity']:.2f}, gen={stat['generation']}{best_indicator}" - ) - - # Artifact storage and retrieval methods + """Select and migrate programs, updating population and elite state.""" + ... def store_artifacts(self, program_id: str, artifacts: Dict[str, Union[str, bytes]]) -> None: - """ - Store artifacts for a program - - Args: - program_id: ID of the program - artifacts: Dictionary of artifact name to content - """ - if not artifacts: - return - - program = self.get(program_id) - if not program: - logger.warning(f"Cannot store artifacts: program {program_id} not found") - return - - # Check if artifacts are enabled - artifacts_enabled = os.environ.get("ENABLE_ARTIFACTS", "true").lower() == "true" - if not artifacts_enabled: - logger.debug("Artifacts disabled, skipping storage") - return - - # Split artifacts by size - small_artifacts = {} - large_artifacts = {} - size_threshold = getattr(self.config, "artifact_size_threshold", 32 * 1024) # 32KB default - - for key, value in artifacts.items(): - size = self._get_artifact_size(value) - if size <= size_threshold: - small_artifacts[key] = value - else: - large_artifacts[key] = value - - # Store small artifacts as JSON - if small_artifacts: - program.artifacts_json = json.dumps(small_artifacts, default=self._artifact_serializer) - logger.debug(f"Stored {len(small_artifacts)} small artifacts for program {program_id}") - - # Store large artifacts to disk - if large_artifacts: - artifact_dir = self._create_artifact_dir(program_id) - program.artifact_dir = artifact_dir - for key, value in large_artifacts.items(): - self._write_artifact_file(artifact_dir, key, value) - logger.debug(f"Stored {len(large_artifacts)} large artifacts for program {program_id}") + """Attach evaluation evidence to a program.""" + ... def get_artifacts(self, program_id: str) -> Dict[str, Union[str, bytes]]: - """ - Retrieve all artifacts for a program - - Args: - program_id: ID of the program - - Returns: - Dictionary of artifact name to content - """ - program = self.get(program_id) - if not program: - return {} - - artifacts = {} - - # Load small artifacts from JSON - if program.artifacts_json: - try: - small_artifacts = json.loads(program.artifacts_json) - artifacts.update(small_artifacts) - except json.JSONDecodeError as e: - logger.warning(f"Failed to decode artifacts JSON for program {program_id}: {e}") - - # Load large artifacts from disk - if program.artifact_dir and os.path.exists(program.artifact_dir): - disk_artifacts = self._load_artifact_dir(program.artifact_dir) - artifacts.update(disk_artifacts) - - return artifacts - - def _get_artifact_size(self, value: Union[str, bytes]) -> int: - """Get size of an artifact value in bytes""" - if isinstance(value, str): - return len(value.encode("utf-8")) - elif isinstance(value, bytes): - return len(value) - else: - return len(str(value).encode("utf-8")) - - def _artifact_serializer(self, obj): - """JSON serializer for artifacts that handles bytes""" - if isinstance(obj, bytes): - return {"__bytes__": base64.b64encode(obj).decode("utf-8")} - raise TypeError(f"Object of type {type(obj)} is not JSON serializable") - - def _artifact_deserializer(self, dct): - """JSON deserializer for artifacts that handles bytes""" - if "__bytes__" in dct: - return base64.b64decode(dct["__bytes__"]) - return dct - - def _create_artifact_dir(self, program_id: str) -> str: - """Create artifact directory for a program""" - base_path = getattr(self.config, "artifacts_base_path", None) - if not base_path: - base_path = ( - os.path.join(self.config.db_path or ".", "artifacts") - if self.config.db_path - else "./artifacts" - ) - - artifact_dir = os.path.join(base_path, program_id) - os.makedirs(artifact_dir, exist_ok=True) - return artifact_dir - - def _cleanup_old_artifacts(self, checkpoint_path: str) -> None: - """ - Remove artifact directories older than the configured retention period. - - Args: - checkpoint_path: The path of the current checkpoint being saved, which - contains the artifacts folder to be cleaned. - """ - if not self.config.cleanup_old_artifacts: - return - - artifacts_base_path = os.path.join(checkpoint_path, "artifacts") - - if not os.path.isdir(artifacts_base_path): - return - - now = time.time() - retention_seconds = self.config.artifact_retention_days * 24 * 60 * 60 - deleted_count = 0 - - logger.debug(f"Starting artifact cleanup in {artifacts_base_path}...") - - for dirname in os.listdir(artifacts_base_path): - dirpath = os.path.join(artifacts_base_path, dirname) - if os.path.isdir(dirpath): - try: - dir_mod_time = os.path.getmtime(dirpath) - if (now - dir_mod_time) > retention_seconds: - shutil.rmtree(dirpath) - deleted_count += 1 - logger.debug(f"Removed old artifact directory: {dirpath}") - except FileNotFoundError: - # Can happen in race conditions; ignore. - continue - except Exception as e: - logger.error(f"Error removing artifact directory {dirpath}: {e}") - - if deleted_count > 0: - logger.info(f"Cleaned up {deleted_count} old artifact directories.") - - def _write_artifact_file(self, artifact_dir: str, key: str, value: Union[str, bytes]) -> None: - """Write an artifact to a file""" - # Sanitize filename - safe_key = "".join(c for c in key if c.isalnum() or c in "._-") - if not safe_key: - safe_key = "artifact" - - file_path = os.path.join(artifact_dir, safe_key) - - try: - if isinstance(value, str): - with open(file_path, "w", encoding="utf-8") as f: - f.write(value) - elif isinstance(value, bytes): - with open(file_path, "wb") as f: - f.write(value) - else: - # Convert to string and write - with open(file_path, "w", encoding="utf-8") as f: - f.write(str(value)) - except Exception as e: - logger.warning(f"Failed to write artifact {key} to {file_path}: {e}") - - def _load_artifact_dir(self, artifact_dir: str) -> Dict[str, Union[str, bytes]]: - """Load artifacts from a directory""" - artifacts = {} - - try: - for filename in os.listdir(artifact_dir): - file_path = os.path.join(artifact_dir, filename) - if os.path.isfile(file_path): - try: - # Try to read as text first - with open(file_path, "r", encoding="utf-8") as f: - content = f.read() - artifacts[filename] = content - except UnicodeDecodeError: - # If text fails, read as binary - with open(file_path, "rb") as f: - content = f.read() - artifacts[filename] = content - except Exception as e: - logger.warning(f"Failed to read artifact file {file_path}: {e}") - except Exception as e: - logger.warning(f"Failed to list artifact directory {artifact_dir}: {e}") - - return artifacts + """Fetch evidence for one program.""" + ... def log_prompt( self, @@ -2590,27 +97,9 @@ def log_prompt( prompt: Dict[str, str], responses: Optional[List[str]] = None, ) -> None: - """ - Log a prompt for a program. - Only logs if self.config.log_prompts is True. - - Args: - program_id: ID of the program to log the prompt for - template_key: Key for the prompt template - prompt: Prompts in the format {template_key: { 'system': str, 'user': str }}. - responses: Optional list of responses to the prompt, if available. - """ - - if not self.config.log_prompts: - return - - if responses is None: - responses = [] - prompt["responses"] = responses - - if self.prompts_by_program is None: - self.prompts_by_program = {} + """Record the prompt and responses for a program.""" + ... - if program_id not in self.prompts_by_program: - self.prompts_by_program[program_id] = {} - self.prompts_by_program[program_id][template_key] = prompt + def get_prompt_history(self, program_id: str) -> Dict[str, Any]: + """Fetch the recorded prompts and responses for one program.""" + ... diff --git a/openevolve/database_memory.py b/openevolve/database_memory.py new file mode 100644 index 0000000000..53925b67a5 --- /dev/null +++ b/openevolve/database_memory.py @@ -0,0 +1,2259 @@ +""" +Program database for OpenEvolve +""" + +import base64 +import json +import logging +import os +import random +import shutil +import time +import uuid +from copy import deepcopy + +# FileLock removed - no longer needed with threaded parallel processing +from typing import Any, Dict, List, Optional, Set, Tuple, Union + +import numpy as np + +from openevolve.config import DatabaseConfig +from openevolve.database import DatabaseState, ProgramDatabase +from openevolve.program import Program +from openevolve.utils.code_utils import calculate_edit_distance +from openevolve.utils.metrics_utils import safe_numeric_average, get_fitness_score + +logger = logging.getLogger(__name__) + + +def _safe_sum_metrics(metrics: Dict[str, Any]) -> float: + """Safely sum only numeric metric values, ignoring strings and other types""" + numeric_values = [ + v for v in metrics.values() if isinstance(v, (int, float)) and not isinstance(v, bool) + ] + return sum(numeric_values) if numeric_values else 0.0 + + +def _safe_avg_metrics(metrics: Dict[str, Any]) -> float: + """Safely calculate average of only numeric metric values""" + numeric_values = [ + v for v in metrics.values() if isinstance(v, (int, float)) and not isinstance(v, bool) + ] + return sum(numeric_values) / max(1, len(numeric_values)) if numeric_values else 0.0 + + +class InMemoryProgramDatabase(ProgramDatabase): + """ + Database for storing and sampling programs during evolution + + The database implements a combination of MAP-Elites algorithm and + island-based population model to maintain diversity during evolution. + It also tracks the absolute best program separately to ensure it's never lost. + """ + + def __init__(self, config: DatabaseConfig): + self.config = config + + # In-memory program storage + self.programs: Dict[str, Program] = {} + + # Per-island feature grids for MAP-Elites + self.island_feature_maps: List[Dict[str, str]] = [{} for _ in range(config.num_islands)] + + # Handle both int and dict types for feature_bins + if isinstance(config.feature_bins, int): + self.feature_bins = max( + config.feature_bins, + int(pow(config.archive_size, 1 / len(config.feature_dimensions)) + 0.99), + ) + else: + # If dict, keep as is (we'll use feature_bins_per_dim instead) + self.feature_bins = 10 # Default fallback for backward compatibility + + # Island populations + self.islands: List[Set[str]] = [set() for _ in range(config.num_islands)] + + # Island management attributes + self.current_island: int = 0 + self.island_generations: List[int] = [0] * config.num_islands + self.last_migration_generation: int = 0 + self.migration_interval: int = getattr(config, "migration_interval", 10) # Default to 10 + self.migration_rate: float = getattr(config, "migration_rate", 0.1) # Default to 0.1 + + # Archive of elite programs + self.archive: Set[str] = set() + + # Track the absolute best program separately + self.best_program_id: Optional[str] = None + + # Track best program per island for proper island-based evolution + self.island_best_programs: List[Optional[str]] = [None] * config.num_islands + + # Track the last iteration number (for resuming) + self.last_iteration: int = 0 + + # Prompt log + self.prompts_by_program: Dict[str, Dict[str, Dict[str, str]]] = None + + # Set random seed for reproducible sampling if specified + if config.random_seed is not None: + import random + + random.seed(config.random_seed) + logger.debug(f"Database: Set random seed to {config.random_seed}") + + # Diversity caching infrastructure + self.diversity_cache: Dict[int, Dict[str, Union[float, float]]] = ( + {} + ) # hash -> {"value": float, "timestamp": float} + self.diversity_cache_size: int = 1000 # LRU cache size + self.diversity_reference_set: List[str] = ( + [] + ) # Reference program codes for consistent diversity + self.diversity_reference_size: int = getattr(config, "diversity_reference_size", 20) + + # Feature scaling infrastructure + self.feature_stats: Dict[str, Dict[str, Union[float, float, List[float]]]] = {} + self.feature_scaling_method: str = "minmax" # Options: minmax, zscore, percentile + + # Per-dimension bins support + if hasattr(config, "feature_bins") and isinstance(config.feature_bins, dict): + self.feature_bins_per_dim = config.feature_bins + else: + # Backward compatibility - use same bins for all dimensions + self.feature_bins_per_dim = { + dim: self.feature_bins for dim in config.feature_dimensions + } + + logger.info(f"Initialized program database with {len(self.programs)} programs") + + # Novelty judge setup + from openevolve.embedding import EmbeddingClient + + self.novelty_llm = config.novelty_llm + self.embedding_client = ( + EmbeddingClient(config.embedding_model) if config.embedding_model else None + ) + self.similarity_threshold = config.similarity_threshold + + def get_state(self) -> DatabaseState: + return DatabaseState( + program_count=len(self.programs), + last_iteration=self.last_iteration, + current_island=self.current_island, + num_islands=len(self.islands), + feature_dimensions=tuple(self.config.feature_dimensions), + ) + + def record_iteration(self, iteration: int) -> None: + if iteration < 0: + raise ValueError("iteration must be non-negative") + self.last_iteration = max(self.last_iteration, iteration) + + def get_prompt_history(self, program_id: str) -> Dict[str, Any]: + return deepcopy((self.prompts_by_program or {}).get(program_id, {})) + + def add( + self, program: Program, iteration: int = None, target_island: Optional[int] = None + ) -> str: + """ + Add a program to the database + + Args: + program: Program to add + iteration: Current iteration (defaults to last_iteration) + target_island: Specific island to add to (auto-detects parent's island if None) + + Returns: + Program ID + """ + # Store a detached value, matching DBMS row semantics. + program = deepcopy(program) + + # Store the program + # If iteration is provided, update the program's iteration_found + if iteration is not None: + program.iteration_found = iteration + # Update last_iteration if needed + self.last_iteration = max(self.last_iteration, iteration) + + self.programs[program.id] = program + + # Calculate feature coordinates for MAP-Elites + feature_coords = self._calculate_feature_coords(program) + + # Determine target island + # If target_island is not specified and program has a parent, inherit parent's island + if target_island is None and program.parent_id: + parent = self.programs.get(program.parent_id) + if parent and "island" in parent.metadata: + # Child inherits parent's island to maintain island isolation + island_idx = parent.metadata["island"] + logger.debug( + f"Program {program.id} inheriting island {island_idx} from parent {program.parent_id}" + ) + else: + # Parent not found or has no island, use current_island + island_idx = self.current_island + if parent: + logger.warning( + f"Parent {program.parent_id} has no island metadata, using current_island {island_idx}" + ) + else: + logger.warning( + f"Parent {program.parent_id} not found, using current_island {island_idx}" + ) + elif target_island is not None: + # Explicit target island specified (e.g., for migrants) + island_idx = target_island + else: + # No parent and no target specified, use current island + island_idx = self.current_island + + island_idx = island_idx % len(self.islands) # Ensure valid island + + # Novelty check before adding + if not self._is_novel(program.id, island_idx): + logger.debug( + f"Program {program.id} failed in novelty check and won't be added in the island {island_idx}" + ) + return program.id # Do not add non-novel program + + # Add to island-specific feature map (replacing existing if better) + feature_key = self._feature_coords_to_key(feature_coords) + island_feature_map = self.island_feature_maps[island_idx] + should_replace = feature_key not in island_feature_map + + if not should_replace: + # Check if the existing program still exists before comparing + existing_program_id = island_feature_map[feature_key] + if existing_program_id not in self.programs: + # Stale reference, replace it + should_replace = True + logger.debug( + f"Replacing stale program reference {existing_program_id} in island {island_idx} feature map" + ) + else: + # Program exists, compare fitness + should_replace = self._is_better(program, self.programs[existing_program_id]) + + # Track a program that gets displaced from its cell so we can remove it + # from the population if it ends up orphaned (owning no cell, in no island). + replaced_program_id = None + + if should_replace: + # Log significant MAP-Elites events + coords_dict = { + self.config.feature_dimensions[i]: feature_coords[i] + for i in range(len(feature_coords)) + } + + if feature_key not in island_feature_map: + # New cell occupation in this island + logger.info( + "New MAP-Elites cell occupied in island %d: %s", island_idx, coords_dict + ) + # Check coverage milestone for this island + total_possible_cells = self.feature_bins ** len(self.config.feature_dimensions) + island_coverage = (len(island_feature_map) + 1) / total_possible_cells + if island_coverage in [0.1, 0.25, 0.5, 0.75, 0.9]: + logger.info( + "Island %d MAP-Elites coverage reached %.1f%% (%d/%d cells)", + island_idx, + island_coverage * 100, + len(island_feature_map) + 1, + total_possible_cells, + ) + else: + # Cell replacement - existing program being replaced in this island + existing_program_id = island_feature_map[feature_key] + if existing_program_id in self.programs: + existing_program = self.programs[existing_program_id] + new_fitness = get_fitness_score(program.metrics, self.config.feature_dimensions) + existing_fitness = get_fitness_score( + existing_program.metrics, self.config.feature_dimensions + ) + logger.info( + "Island %d MAP-Elites cell improved: %s (fitness: %.3f -> %.3f)", + island_idx, + coords_dict, + existing_fitness, + new_fitness, + ) + + # use MAP-Elites to manage archive + if existing_program_id in self.archive: + self.archive.discard(existing_program_id) + self.archive.add(program.id) + + # Remove replaced program from island set to keep it consistent with feature map + # This prevents accumulation of stale/replaced programs in the island + self.islands[island_idx].discard(existing_program_id) + replaced_program_id = existing_program_id + + island_feature_map[feature_key] = program.id + + # Add to island + self.islands[island_idx].add(program.id) + + # Track which island this program belongs to + program.metadata["island"] = island_idx + + # Update archive + self._update_archive(program) + + # Enforce population size limit BEFORE updating best program tracking + # This ensures newly added programs aren't immediately removed + self._enforce_population_limit(exclude_program_id=program.id) + + # Update the absolute best program tracking (after population enforcement) + self._update_best_program(program) + + # Update island-specific best program tracking + self._update_island_best_program(program, island_idx) + + # If a program was displaced from its cell by this addition, it may now be + # orphaned - owning no cell and belonging to no island. Such a program is a + # "zombie" that consumes a population slot but can never be sampled again, so + # remove it. This runs after best-program tracking is updated so the newly + # added (better) program is already recorded as best, ensuring we never drop + # the current best program here. + if ( + replaced_program_id is not None + and replaced_program_id != program.id + and replaced_program_id != self.best_program_id + ): + self._remove_program_if_orphaned(replaced_program_id) + + logger.debug(f"Added program {program.id} to island {island_idx}") + + return program.id + + def get(self, program_id: str) -> Optional[Program]: + """ + Get a program by ID + + Args: + program_id: Program ID + + Returns: + Program or None if not found + """ + return deepcopy(self.programs.get(program_id)) + + def sample(self, num_inspirations: Optional[int] = None) -> Tuple[Program, List[Program]]: + """ + Sample a program and inspirations for the next evolution step + + Args: + num_inspirations: Number of inspiration programs to sample (defaults to 5 for backward compatibility) + + Returns: + Tuple of (parent_program, inspiration_programs) + """ + if not self.programs: + raise ValueError("Cannot sample from an empty program database") + + # Select parent program + parent = self._sample_parent() + + # Select inspirations + if num_inspirations is None: + num_inspirations = 5 # Default for backward compatibility + inspirations = self._sample_inspirations(parent, n=num_inspirations) + + logger.debug(f"Sampled parent {parent.id} and {len(inspirations)} inspirations") + return deepcopy((parent, inspirations)) + + def sample_from_island( + self, island_id: int, num_inspirations: Optional[int] = None + ) -> Tuple[Program, List[Program]]: + """ + Sample a program and inspirations from a specific island without modifying current_island + + This method is thread-safe and doesn't modify shared state, avoiding race conditions + when multiple workers sample from different islands concurrently. + + Uses the same exploration/exploitation/random strategy as sample() to ensure + consistent behavior between single-process and parallel execution modes. + + Args: + island_id: The island to sample from + num_inspirations: Number of inspiration programs to sample (defaults to 5) + + Returns: + Tuple of (parent_program, inspiration_programs) + """ + if num_inspirations is not None and num_inspirations < 0: + raise ValueError("num_inspirations must be non-negative") + + # Ensure valid island ID + island_id = island_id % len(self.islands) + + # Get programs from the specific island + island_programs = list(self.islands[island_id]) + + if not island_programs: + # Island is empty, fall back to sampling from all programs + logger.debug(f"Island {island_id} is empty, sampling from all programs") + return self.sample(num_inspirations) + + # Use exploration_ratio and exploitation_ratio to decide sampling strategy + # This matches the logic in _sample_parent() for consistent behavior + rand_val = random.random() + + if rand_val < self.config.exploration_ratio: + # EXPLORATION: Sample randomly from island (diverse sampling) + parent = self._sample_from_island_random(island_id) + sampling_mode = "exploration" + elif rand_val < self.config.exploration_ratio + self.config.exploitation_ratio: + # EXPLOITATION: Sample from archive (elite programs) + parent = self._sample_from_archive_for_island(island_id) + sampling_mode = "exploitation" + else: + # WEIGHTED: Use fitness-weighted sampling (remaining probability) + parent = self._sample_from_island_weighted(island_id) + sampling_mode = "weighted" + + # Select inspirations using the same elite/diversity-aware strategy as sample(). + # Pass the requested island explicitly so parallel workers remain isolated even + # if an archive fallback returns a parent whose metadata points elsewhere. + if num_inspirations is None: + num_inspirations = 5 # Default for backward compatibility + + inspirations = self._sample_inspirations(parent, n=num_inspirations, island_id=island_id) + + logger.debug( + f"Sampled parent {parent.id} and {len(inspirations)} inspirations from island {island_id} " + f"(mode: {sampling_mode}, rand_val: {rand_val:.3f})" + ) + return deepcopy((parent, inspirations)) + + def get_best_program(self, metric: Optional[str] = None) -> Optional[Program]: + """ + Get the best program based on a metric + + Args: + metric: Metric to use for ranking (uses combined_score or average if None) + + Returns: + Best program or None if database is empty + """ + if not self.programs: + return None + + # If no specific metric and we have a tracked best program, return it + if metric is None and self.best_program_id: + if self.best_program_id in self.programs: + logger.debug(f"Using tracked best program: {self.best_program_id}") + return deepcopy(self.programs[self.best_program_id]) + else: + logger.warning( + f"Tracked best program {self.best_program_id} no longer exists, will recalculate" + ) + self.best_program_id = None + + if metric: + # Sort by specific metric + sorted_programs = sorted( + [p for p in self.programs.values() if metric in p.metrics], + key=lambda p: p.metrics[metric], + reverse=True, + ) + if sorted_programs: + logger.debug(f"Found best program by metric '{metric}': {sorted_programs[0].id}") + else: + # Sort by fitness (excluding feature dimensions) + sorted_programs = sorted( + self.programs.values(), + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + reverse=True, + ) + if sorted_programs: + logger.debug(f"Found best program by fitness score: {sorted_programs[0].id}") + + # Update the best program tracking if we found a better program + if ( + metric is None + and sorted_programs + and (self.best_program_id is None or sorted_programs[0].id != self.best_program_id) + ): + old_id = self.best_program_id + self.best_program_id = sorted_programs[0].id + logger.info(f"Updated best program tracking from {old_id} to {self.best_program_id}") + + # Also log the scores to help understand the update + if ( + old_id + and old_id in self.programs + and "combined_score" in self.programs[old_id].metrics + and "combined_score" in self.programs[self.best_program_id].metrics + ): + old_score = self.programs[old_id].metrics["combined_score"] + new_score = self.programs[self.best_program_id].metrics["combined_score"] + logger.info( + f"Score change: {old_score:.4f} → {new_score:.4f} ({new_score-old_score:+.4f})" + ) + + return deepcopy(sorted_programs[0]) if sorted_programs else None + + def get_top_programs( + self, n: int = 10, metric: Optional[str] = None, island_idx: Optional[int] = None + ) -> List[Program]: + """ + Get the top N programs based on a metric + + Args: + n: Number of programs to return + metric: Metric to use for ranking (uses average if None) + island_idx: If specified, only return programs from this island + + Returns: + List of top programs + """ + if n < 0: + raise ValueError("n must be non-negative") + + # Validate island_idx parameter + if island_idx is not None and (island_idx < 0 or island_idx >= len(self.islands)): + raise IndexError(f"Island index {island_idx} is out of range (0-{len(self.islands)-1})") + + if not self.programs: + return [] + + # Get candidate programs + if island_idx is not None: + # Island-specific query + island_programs = [ + self.programs[pid] for pid in self.islands[island_idx] if pid in self.programs + ] + candidates = island_programs + else: + # Global query + candidates = list(self.programs.values()) + + if not candidates: + return [] + + if metric: + # Sort by specific metric + sorted_programs = sorted( + [p for p in candidates if metric in p.metrics], + key=lambda p: p.metrics[metric], + reverse=True, + ) + else: + # Sort by combined_score if available, otherwise by average of all numeric metrics + sorted_programs = sorted( + candidates, + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + reverse=True, + ) + + return deepcopy(sorted_programs[:n]) + + def _calculate_feature_coords(self, program: Program) -> List[int]: + """ + Calculate feature coordinates for the MAP-Elites grid + + Args: + program: Program to calculate features for + + Returns: + List of feature coordinates + """ + coords = [] + + for dim in self.config.feature_dimensions: + # PRIORITY 1: Check if this is a custom metric from the evaluator + # This allows users to override built-in features with their own implementations + if dim in program.metrics: + # Use custom metric from evaluator + score = program.metrics[dim] + # Update stats and scale + self._update_feature_stats(dim, score) + scaled_value = self._scale_feature_value(dim, score) + num_bins = self.feature_bins_per_dim.get(dim, self.feature_bins) + bin_idx = int(scaled_value * num_bins) + bin_idx = max(0, min(num_bins - 1, bin_idx)) + coords.append(bin_idx) + # PRIORITY 2: Fall back to built-in features if not in metrics + elif dim == "complexity": + # Use code length as complexity measure + complexity = len(program.code) + bin_idx = self._calculate_complexity_bin(complexity) + coords.append(bin_idx) + elif dim == "diversity": + # Use cached diversity calculation with reference set + if len(self.programs) < 2: + bin_idx = 0 + else: + diversity = self._get_cached_diversity(program) + bin_idx = self._calculate_diversity_bin(diversity) + coords.append(bin_idx) + elif dim == "score": + # Use average of numeric metrics + if not program.metrics: + bin_idx = 0 + else: + # Use fitness score for "score" dimension (consistent with rest of system) + avg_score = get_fitness_score(program.metrics, self.config.feature_dimensions) + # Update stats and scale + self._update_feature_stats("score", avg_score) + scaled_value = self._scale_feature_value("score", avg_score) + num_bins = self.feature_bins_per_dim.get("score", self.feature_bins) + bin_idx = int(scaled_value * num_bins) + bin_idx = max(0, min(num_bins - 1, bin_idx)) + coords.append(bin_idx) + else: + # Feature not found - this is an error + raise ValueError( + f"Feature dimension '{dim}' specified in config but not found in program metrics. " + f"Available metrics: {list(program.metrics.keys())}. " + f"Built-in features: 'complexity', 'diversity', 'score'. " + f"Either remove '{dim}' from feature_dimensions or ensure your evaluator returns it." + ) + # Only log coordinates at debug level for troubleshooting + logger.debug( + "MAP-Elites coords: %s", + str({self.config.feature_dimensions[i]: coords[i] for i in range(len(coords))}), + ) + return coords + + def _calculate_complexity_bin(self, complexity: int) -> int: + """ + Calculate the bin index for a given complexity value using feature scaling. + + Args: + complexity: The complexity value (code length) + + Returns: + Bin index in range [0, self.feature_bins - 1] + """ + # Update feature statistics + self._update_feature_stats("complexity", float(complexity)) + + # Scale the value using configured method + scaled_value = self._scale_feature_value("complexity", float(complexity)) + + # Get number of bins for this dimension + num_bins = self.feature_bins_per_dim.get("complexity", self.feature_bins) + + # Convert to bin index + bin_idx = int(scaled_value * num_bins) + + # Ensure bin index is within valid range + bin_idx = max(0, min(num_bins - 1, bin_idx)) + + return bin_idx + + def _calculate_diversity_bin(self, diversity: float) -> int: + """ + Calculate the bin index for a given diversity value using feature scaling. + + Args: + diversity: The average fast code diversity to other programs + + Returns: + Bin index in range [0, self.feature_bins - 1] + """ + # Update feature statistics + self._update_feature_stats("diversity", diversity) + + # Scale the value using configured method + scaled_value = self._scale_feature_value("diversity", diversity) + + # Get number of bins for this dimension + num_bins = self.feature_bins_per_dim.get("diversity", self.feature_bins) + + # Convert to bin index + bin_idx = int(scaled_value * num_bins) + + # Ensure bin index is within valid range + bin_idx = max(0, min(num_bins - 1, bin_idx)) + + return bin_idx + + def _feature_coords_to_key(self, coords: List[int]) -> str: + """ + Convert feature coordinates to a string key + + Args: + coords: Feature coordinates + + Returns: + String key + """ + return "-".join(str(c) for c in coords) + + def _cosine_similarity(self, vec1: List[float], vec2: List[float]) -> float: + """ + Adapted from SakanaAI/ShinkaEvolve (Apache-2.0 License) + Original source: https://github.com/SakanaAI/ShinkaEvolve/blob/main/shinka/database/dbase.py#L1452 + + Compute cosine similarity between two vectors. + """ + if not vec1 or not vec2 or len(vec1) != len(vec2): + return 0.0 + + arr1 = np.array(vec1, dtype=np.float32) + arr2 = np.array(vec2, dtype=np.float32) + + norm_a = np.linalg.norm(arr1) + norm_b = np.linalg.norm(arr2) + + if norm_a == 0 or norm_b == 0: + return 0.0 + + similarity = np.dot(arr1, arr2) / (norm_a * norm_b) + + return float(similarity) + + def _llm_judge_novelty(self, program: Program, similar_program: Program) -> bool: + """ + Use LLM to judge if a program is novel compared to a similar existing program + """ + import asyncio + from openevolve.novelty_judge import NOVELTY_SYSTEM_MSG, NOVELTY_USER_MSG + + user_msg = NOVELTY_USER_MSG.format( + language=program.language, + existing_code=similar_program.code, + proposed_code=program.code, + ) + + try: + # Check if we're already in an event loop + try: + loop = asyncio.get_running_loop() + # We're in an async context, need to run in a new thread + import concurrent.futures + + with concurrent.futures.ThreadPoolExecutor() as executor: + future = executor.submit( + asyncio.run, + self.novelty_llm.generate_with_context( + system_message=NOVELTY_SYSTEM_MSG, + messages=[{"role": "user", "content": user_msg}], + ), + ) + content: str = future.result() + except RuntimeError: + # No event loop running, safe to use asyncio.run() + content: str = asyncio.run( + self.novelty_llm.generate_with_context( + system_message=NOVELTY_SYSTEM_MSG, + messages=[{"role": "user", "content": user_msg}], + ) + ) + + if content is None or content is None: + logger.warning("Novelty LLM returned empty response") + return True + + content = content.strip() + + # Parse the response + NOVEL_i = content.upper().find("NOVEL") + NOT_NOVEL_i = content.upper().find("NOT NOVEL") + + if NOVEL_i == -1 and NOT_NOVEL_i == -1: + logger.warning(f"Unexpected novelty LLM response: {content}") + return True # Assume novel if we can't parse + + if NOVEL_i != -1 and NOT_NOVEL_i != -1: + # Both found, take the one that appears first + is_novel = NOVEL_i < NOT_NOVEL_i + elif NOVEL_i != -1: + is_novel = True + else: + is_novel = False + + return is_novel + + except Exception as e: + logger.error(f"Error in novelty LLM check: {e}") + + return True + + def _is_novel(self, program_id: int, island_idx: int) -> bool: + """ + Determine if a program is novel based on diversity to existing programs + + Args: + program: Program to check + island_idx: Island index + + Returns: + True if novel, False otherwise + """ + if self.embedding_client is None or self.similarity_threshold <= 0.0: + # Novelty checking disabled + return True + + program = self.programs[program_id] + embd = self.embedding_client.get_embedding(program.code) + self.programs[program_id].embedding = embd + + max_smlty = float("-inf") + max_smlty_pid = None + + for pid in self.islands[island_idx]: + other = self.programs[pid] + + if other.embedding is None: + logger.warning(f"Program {other.id} has no embedding, skipping similarity check") + continue + + similarity = self._cosine_similarity(embd, other.embedding) + + if similarity >= max(max_smlty, self.similarity_threshold): + max_smlty = similarity + max_smlty_pid = pid + + if max_smlty_pid is None: + # No similar programs found, consider it novel + return True + + return self._llm_judge_novelty(program, self.programs[max_smlty_pid]) + + def _is_better(self, program1: Program, program2: Program) -> bool: + """ + Determine if program1 has better FITNESS than program2 + + Uses fitness calculation that excludes MAP-Elites feature dimensions + to prevent pollution of fitness comparisons. + + Args: + program1: First program + program2: Second program + + Returns: + True if program1 is better than program2 + """ + # If no metrics, use newest + if not program1.metrics and not program2.metrics: + return program1.timestamp > program2.timestamp + + # If only one has metrics, it's better + if program1.metrics and not program2.metrics: + return True + if not program1.metrics and program2.metrics: + return False + + # Compare fitness (excluding feature dimensions) + fitness1 = get_fitness_score(program1.metrics, self.config.feature_dimensions) + fitness2 = get_fitness_score(program2.metrics, self.config.feature_dimensions) + + return fitness1 > fitness2 + + def _update_archive(self, program: Program) -> None: + """ + Update the archive of elite programs + + Args: + program: Program to consider for archive + """ + # If archive not full, add program + if len(self.archive) < self.config.archive_size: + self.archive.add(program.id) + return + + # Clean up stale references and get valid archive programs + valid_archive_programs = [] + stale_ids = [] + + for pid in self.archive: + if pid in self.programs: + valid_archive_programs.append(self.programs[pid]) + else: + stale_ids.append(pid) + + # Remove stale references from archive + for stale_id in stale_ids: + self.archive.discard(stale_id) + logger.debug(f"Removing stale program {stale_id} from archive") + + # If archive is now not full after cleanup, just add the new program + if len(self.archive) < self.config.archive_size: + self.archive.add(program.id) + return + + # Find worst program among valid programs + if valid_archive_programs: + worst_program = min( + valid_archive_programs, + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + ) + + # Replace if new program is better + if self._is_better(program, worst_program): + self.archive.remove(worst_program.id) + self.archive.add(program.id) + else: + # No valid programs in archive, just add the new one + self.archive.add(program.id) + + def _update_best_program(self, program: Program) -> None: + """ + Update the absolute best program tracking + + Args: + program: Program to consider as the new best + """ + # If we don't have a best program yet, this becomes the best + if self.best_program_id is None: + self.best_program_id = program.id + logger.debug(f"Set initial best program to {program.id}") + return + + # Compare with current best program (if it still exists) + if self.best_program_id not in self.programs: + logger.warning( + f"Best program {self.best_program_id} no longer exists, clearing reference" + ) + self.best_program_id = program.id + logger.info(f"Set new best program to {program.id}") + return + + current_best = self.programs[self.best_program_id] + + # Update if the new program is better + if self._is_better(program, current_best): + old_id = self.best_program_id + self.best_program_id = program.id + + # Log the change + if "combined_score" in program.metrics and "combined_score" in current_best.metrics: + old_score = current_best.metrics["combined_score"] + new_score = program.metrics["combined_score"] + score_diff = new_score - old_score + logger.info( + f"New best program {program.id} replaces {old_id} (combined_score: {old_score:.4f} → {new_score:.4f}, +{score_diff:.4f})" + ) + else: + logger.info(f"New best program {program.id} replaces {old_id}") + + def _update_island_best_program(self, program: Program, island_idx: int) -> None: + """ + Update the best program tracking for a specific island + + Args: + program: Program to consider as the new best for the island + island_idx: Island index + """ + # Ensure island_idx is valid + if island_idx >= len(self.island_best_programs): + logger.warning(f"Invalid island index {island_idx}, skipping island best update") + return + + # If island doesn't have a best program yet, this becomes the best + current_island_best_id = self.island_best_programs[island_idx] + if current_island_best_id is None: + self.island_best_programs[island_idx] = program.id + logger.debug(f"Set initial best program for island {island_idx} to {program.id}") + return + + # Check if current best still exists + if current_island_best_id not in self.programs: + logger.warning( + f"Island {island_idx} best program {current_island_best_id} no longer exists, updating to {program.id}" + ) + self.island_best_programs[island_idx] = program.id + return + + current_island_best = self.programs[current_island_best_id] + + # Update if the new program is better + if self._is_better(program, current_island_best): + old_id = current_island_best_id + self.island_best_programs[island_idx] = program.id + + # Log the change + if ( + "combined_score" in program.metrics + and "combined_score" in current_island_best.metrics + ): + old_score = current_island_best.metrics["combined_score"] + new_score = program.metrics["combined_score"] + score_diff = new_score - old_score + logger.debug( + f"Island {island_idx}: New best program {program.id} replaces {old_id} " + f"(combined_score: {old_score:.4f} → {new_score:.4f}, +{score_diff:.4f})" + ) + else: + logger.debug( + f"Island {island_idx}: New best program {program.id} replaces {old_id}" + ) + + def _sample_parent(self) -> Program: + """ + Sample a parent program from the current island for the next evolution step + + Returns: + Parent program from current island + """ + # Use exploration_ratio and exploitation_ratio to decide sampling strategy + rand_val = random.random() + + if rand_val < self.config.exploration_ratio: + # EXPLORATION: Sample from current island (diverse sampling) + return self._sample_exploration_parent() + elif rand_val < self.config.exploration_ratio + self.config.exploitation_ratio: + # EXPLOITATION: Sample from archive (elite programs) + return self._sample_exploitation_parent() + else: + # RANDOM: Sample from any program (remaining probability) + return self._sample_random_parent() + + def _sample_exploration_parent(self) -> Program: + """ + Sample a parent for exploration (from current island) + """ + current_island_programs = self.islands[self.current_island] + + if not current_island_programs: + # If current island is empty, initialize with best program or random program + if self.best_program_id and self.best_program_id in self.programs: + # Create a copy of best program for the empty island (don't reuse same ID) + best_program = self.programs[self.best_program_id] + copy_program = Program( + id=str(uuid.uuid4()), + code=best_program.code, + changes_description=best_program.changes_description, + language=best_program.language, + parent_id=best_program.id, + generation=best_program.generation, + timestamp=time.time(), + iteration_found=self.last_iteration, + metrics=best_program.metrics.copy(), + complexity=best_program.complexity, + diversity=best_program.diversity, + metadata={"island": self.current_island}, + artifacts_json=best_program.artifacts_json, + artifact_dir=best_program.artifact_dir, + ) + self.programs[copy_program.id] = copy_program + self.islands[self.current_island].add(copy_program.id) + logger.debug( + f"Initialized empty island {self.current_island} with copy of best program" + ) + return copy_program + else: + # Use any available program + return next(iter(self.programs.values())) + + # Clean up stale references and sample from current island + valid_programs = [pid for pid in current_island_programs if pid in self.programs] + + # Remove stale program IDs from island + if len(valid_programs) < len(current_island_programs): + stale_ids = current_island_programs - set(valid_programs) + logger.debug( + f"Removing {len(stale_ids)} stale program IDs from island {self.current_island}" + ) + for stale_id in stale_ids: + self.islands[self.current_island].discard(stale_id) + + # If no valid programs after cleanup, reinitialize island + if not valid_programs: + logger.warning( + f"Island {self.current_island} has no valid programs after cleanup, reinitializing" + ) + if self.best_program_id and self.best_program_id in self.programs: + # Create a copy of best program for the empty island (don't reuse same ID) + best_program = self.programs[self.best_program_id] + copy_program = Program( + id=str(uuid.uuid4()), + code=best_program.code, + changes_description=best_program.changes_description, + language=best_program.language, + parent_id=best_program.id, + generation=best_program.generation, + timestamp=time.time(), + iteration_found=self.last_iteration, + metrics=best_program.metrics.copy(), + complexity=best_program.complexity, + diversity=best_program.diversity, + metadata={"island": self.current_island}, + artifacts_json=best_program.artifacts_json, + artifact_dir=best_program.artifact_dir, + ) + self.programs[copy_program.id] = copy_program + self.islands[self.current_island].add(copy_program.id) + logger.debug( + f"Reinitialized empty island {self.current_island} with copy of best program" + ) + return copy_program + else: + return next(iter(self.programs.values())) + + # Sample from valid programs + parent_id = random.choice(valid_programs) + return self.programs[parent_id] + + def _sample_exploitation_parent(self) -> Program: + """ + Sample a parent for exploitation (from archive/elite programs) + """ + if not self.archive: + # Fallback to exploration if no archive + return self._sample_exploration_parent() + + # Clean up stale references in archive + valid_archive = [pid for pid in self.archive if pid in self.programs] + + # Remove stale program IDs from archive + if len(valid_archive) < len(self.archive): + stale_ids = self.archive - set(valid_archive) + logger.debug(f"Removing {len(stale_ids)} stale program IDs from archive") + for stale_id in stale_ids: + self.archive.discard(stale_id) + + # If no valid archive programs, fallback to exploration + if not valid_archive: + logger.warning( + "Archive has no valid programs after cleanup, falling back to exploration" + ) + return self._sample_exploration_parent() + + # Prefer programs from current island in archive + archive_programs_in_island = [ + pid + for pid in valid_archive + if self.programs[pid].metadata.get("island") == self.current_island + ] + + if archive_programs_in_island: + parent_id = random.choice(archive_programs_in_island) + return self.programs[parent_id] + else: + # Fall back to any valid archive program if current island has none + parent_id = random.choice(valid_archive) + return self.programs[parent_id] + + def _sample_random_parent(self) -> Program: + """ + Sample a completely random parent from all programs + """ + if not self.programs: + raise ValueError("No programs available for sampling") + + # Sample randomly from all programs + program_id = random.choice(list(self.programs.keys())) + return self.programs[program_id] + + def _sample_from_island_weighted(self, island_id: int) -> Program: + """ + Sample a parent from a specific island using fitness-weighted selection + + Args: + island_id: The island to sample from + + Returns: + Parent program selected using fitness-weighted sampling + """ + island_id = island_id % len(self.islands) + island_programs = list(self.islands[island_id]) + + if not island_programs: + # Island is empty, fall back to any available program + logger.debug(f"Island {island_id} is empty, sampling from all programs") + return self._sample_random_parent() + + # Select parent from island programs + if len(island_programs) == 1: + parent_id = island_programs[0] + else: + # Use weighted sampling based on program scores + island_program_objects = [ + self.programs[pid] for pid in island_programs if pid in self.programs + ] + + if not island_program_objects: + # Fallback if programs not found + parent_id = random.choice(island_programs) + else: + # Calculate weights based on fitness scores + weights = [] + for prog in island_program_objects: + fitness = get_fitness_score(prog.metrics, self.config.feature_dimensions) + # Add small epsilon to avoid zero weights + weights.append(max(fitness, 0.001)) + + # Normalize weights + total_weight = sum(weights) + if total_weight > 0: + weights = [w / total_weight for w in weights] + else: + weights = [1.0 / len(island_program_objects)] * len(island_program_objects) + + # Sample parent based on weights + parent = random.choices(island_program_objects, weights=weights, k=1)[0] + parent_id = parent.id + + parent = self.programs.get(parent_id) + if not parent: + # Should not happen, but handle gracefully + logger.error(f"Parent program {parent_id} not found in database") + return self._sample_random_parent() + + return parent + + def _sample_from_island_random(self, island_id: int) -> Program: + """ + Sample a completely random parent from a specific island (uniform distribution) + + Args: + island_id: The island to sample from + + Returns: + Parent program selected uniformly at random + """ + island_id = island_id % len(self.islands) + island_programs = list(self.islands[island_id]) + + if not island_programs: + # Island is empty, fall back to any available program + logger.debug(f"Island {island_id} is empty, sampling from all programs") + return self._sample_random_parent() + + # Clean up stale references + valid_programs = [pid for pid in island_programs if pid in self.programs] + + if not valid_programs: + logger.warning( + f"Island {island_id} has no valid programs, falling back to random sampling" + ) + return self._sample_random_parent() + + # Uniform random selection + parent_id = random.choice(valid_programs) + return self.programs[parent_id] + + def _sample_from_archive_for_island(self, island_id: int) -> Program: + """ + Sample a parent from the archive, preferring programs from the specified island + + Args: + island_id: The island to prefer programs from + + Returns: + Parent program from archive (preferably from the specified island) + """ + if not self.archive: + # Fallback to weighted sampling from island + logger.debug(f"Archive is empty, falling back to weighted island sampling") + return self._sample_from_island_weighted(island_id) + + # Clean up stale references in archive + valid_archive = [pid for pid in self.archive if pid in self.programs] + + if not valid_archive: + logger.warning( + "Archive has no valid programs, falling back to weighted island sampling" + ) + return self._sample_from_island_weighted(island_id) + + island_id = island_id % len(self.islands) + + # Prefer programs from the specified island in archive + archive_programs_in_island = [ + pid for pid in valid_archive if self.programs[pid].metadata.get("island") == island_id + ] + + if archive_programs_in_island: + parent_id = random.choice(archive_programs_in_island) + return self.programs[parent_id] + else: + # Fall back to any valid archive program if island has none + parent_id = random.choice(valid_archive) + return self.programs[parent_id] + + def _sample_inspirations( + self, parent: Program, n: int = 5, island_id: Optional[int] = None + ) -> List[Program]: + """ + Sample inspiration programs for the next evolution step. + + For proper island-based evolution, inspirations are sampled ONLY from the + current island, maintaining genetic isolation between islands. + + Args: + parent: Parent program + n: Number of inspirations to sample + island_id: Explicit island to sample from. If omitted, use the + parent program's island metadata. + + Returns: + List of inspiration programs from the current island + """ + inspirations = [] + + # Prefer an explicitly requested island. This matters for + # sample_from_island(), where archive fallback may return a parent whose + # metadata belongs to a different island. + if island_id is None: + parent_island = parent.metadata.get("island", self.current_island) + else: + parent_island = island_id + + parent_island %= len(self.islands) + + # Get all programs from the current island + island_program_ids = list(self.islands[parent_island]) + island_programs = [self.programs[pid] for pid in island_program_ids if pid in self.programs] + + if not island_programs: + logger.warning(f"Island {parent_island} has no programs for inspiration sampling") + return [] + + # Include the island's best program if available and different from parent + island_best_id = self.island_best_programs[parent_island] + if ( + island_best_id is not None + and island_best_id != parent.id + and island_best_id in self.programs + ): + island_best = self.programs[island_best_id] + inspirations.append(island_best) + logger.debug( + f"Including island {parent_island} best program {island_best_id} in inspirations" + ) + elif island_best_id is not None and island_best_id not in self.programs: + # Clean up stale island best reference + logger.warning( + f"Island {parent_island} best program {island_best_id} no longer exists, clearing reference" + ) + self.island_best_programs[parent_island] = None + + # Add top programs from the island as inspirations + top_n = max(1, int(n * self.config.elite_selection_ratio)) + top_island_programs = self.get_top_programs(n=top_n, island_idx=parent_island) + for program in top_island_programs: + if program.id not in [p.id for p in inspirations] and program.id != parent.id: + inspirations.append(program) + + # Add diverse programs from within the island + if len(island_programs) > n and len(inspirations) < n: + remaining_slots = n - len(inspirations) + + # Try to sample from different feature cells within the island + feature_coords = self._calculate_feature_coords(parent) + nearby_programs = [] + + # Create a mapping of feature cells to island programs for efficient lookup + island_feature_map = {} + for prog_id in island_program_ids: + if prog_id in self.programs: + prog = self.programs[prog_id] + prog_coords = self._calculate_feature_coords(prog) + cell_key = self._feature_coords_to_key(prog_coords) + island_feature_map[cell_key] = prog_id + + # Try to find programs from nearby feature cells within the island + for _ in range(remaining_slots * 3): # Try more times to find nearby programs + # Perturb coordinates + perturbed_coords = [ + max(0, min(self.feature_bins - 1, c + random.randint(-2, 2))) + for c in feature_coords + ] + + cell_key = self._feature_coords_to_key(perturbed_coords) + if cell_key in island_feature_map: + program_id = island_feature_map[cell_key] + if ( + program_id != parent.id + and program_id not in [p.id for p in inspirations] + and program_id not in [p.id for p in nearby_programs] + and program_id in self.programs + ): + nearby_programs.append(self.programs[program_id]) + if len(nearby_programs) >= remaining_slots: + break + + # If we still need more, add random programs from the island + if len(inspirations) + len(nearby_programs) < n: + remaining = n - len(inspirations) - len(nearby_programs) + + # Get available programs from the island + excluded_ids = ( + {parent.id} + .union(p.id for p in inspirations) + .union(p.id for p in nearby_programs) + ) + available_island_ids = [ + pid + for pid in island_program_ids + if pid not in excluded_ids and pid in self.programs + ] + + if available_island_ids: + random_ids = random.sample( + available_island_ids, min(remaining, len(available_island_ids)) + ) + random_programs = [self.programs[pid] for pid in random_ids] + nearby_programs.extend(random_programs) + + inspirations.extend(nearby_programs) + + # Log island isolation info + logger.debug( + f"Sampled {len(inspirations)} inspirations from island {parent_island} " + f"(island has {len(island_programs)} programs total)" + ) + + return inspirations[:n] + + def _remove_program_if_orphaned(self, program_id: str) -> None: + """ + Remove a program from the population if it is orphaned. + + A program is considered orphaned when it no longer owns a MAP-Elites cell + in any island's feature map and is not a member of any island. Such a + program (e.g. one displaced when its cell was improved) can never be + sampled again but still counts against the population size limit, so it is + removed from ``self.programs``, the archive and any lingering references. + + Args: + program_id: ID of the (possibly) orphaned program to check and remove + """ + if program_id not in self.programs: + return + + # Still owns a cell in some island? Then it is not orphaned. + for island_map in self.island_feature_maps: + if program_id in island_map.values(): + return + + # Still a member of some island? Then it is not orphaned. + for island in self.islands: + if program_id in island: + return + + # Fully orphaned - remove from all remaining structures. + del self.programs[program_id] + self.archive.discard(program_id) + self._cleanup_stale_island_bests() + logger.debug(f"Removed orphaned program {program_id} displaced from its cell") + + def _enforce_population_limit(self, exclude_program_id: Optional[str] = None) -> None: + """ + Enforce the population size limit by removing worst programs if needed + + Args: + exclude_program_id: Program ID to never remove (e.g., newly added program) + """ + if len(self.programs) <= self.config.population_size: + return + + # Calculate how many programs to remove + num_to_remove = len(self.programs) - self.config.population_size + + logger.info( + f"Population size ({len(self.programs)}) exceeds limit ({self.config.population_size}), removing {num_to_remove} programs" + ) + + # Collect all MAP-Elites cell owners across every island. These "elite" + # programs represent occupied niches and must be protected from eviction + # to preserve diversity - a low-scoring cell owner should only be removed + # after every non-owning (homeless) program has already been removed. + elite_ids = set() + for island_map in self.island_feature_maps: + elite_ids.update(island_map.values()) + + # Never remove the best program or the excluded (just-added) program + protected_ids = {self.best_program_id, exclude_program_id} - {None} + + all_programs = list(self.programs.values()) + + # Split into non-elite (homeless) and elite (cell owners), each sorted by + # fitness worst-first. Non-elite programs are removed before elite ones. + non_elite = sorted( + [p for p in all_programs if p.id not in elite_ids and p.id not in protected_ids], + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + ) + elite = sorted( + [p for p in all_programs if p.id in elite_ids and p.id not in protected_ids], + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + ) + + # Remove non-elite programs first; only fall back to evicting elite cell + # owners (worst first) if removing all homeless programs is not enough. + programs_to_remove = non_elite[:num_to_remove] + if len(programs_to_remove) < num_to_remove: + remaining = num_to_remove - len(programs_to_remove) + programs_to_remove.extend(elite[:remaining]) + + # Remove the selected programs + for program in programs_to_remove: + program_id = program.id + + # Remove from main programs dict + if program_id in self.programs: + del self.programs[program_id] + + # Remove from island feature maps + for island_idx, island_map in enumerate(self.island_feature_maps): + keys_to_remove = [] + for key, pid in island_map.items(): + if pid == program_id: + keys_to_remove.append(key) + for key in keys_to_remove: + del island_map[key] + + # Remove from islands + for island in self.islands: + island.discard(program_id) + + # Remove from archive + self.archive.discard(program_id) + + logger.debug(f"Removed program {program_id} due to population limit") + + logger.info(f"Population size after cleanup: {len(self.programs)}") + + # Clean up any stale island best program references after removal + self._cleanup_stale_island_bests() + + # Island management methods + def set_current_island(self, island_idx: int) -> None: + """Set which island is currently being evolved""" + self.current_island = island_idx % len(self.islands) + logger.debug(f"Switched to evolving island {self.current_island}") + + def next_island(self) -> int: + """Move to the next island in round-robin fashion""" + self.current_island = (self.current_island + 1) % len(self.islands) + logger.debug(f"Advanced to island {self.current_island}") + return self.current_island + + def increment_island_generation(self, island_idx: Optional[int] = None) -> None: + """Increment generation counter for an island""" + idx = island_idx if island_idx is not None else self.current_island + self.island_generations[idx] += 1 + logger.debug(f"Island {idx} generation incremented to {self.island_generations[idx]}") + + def should_migrate(self) -> bool: + """Check if migration should occur based on generation counters""" + max_generation = max(self.island_generations) + return (max_generation - self.last_migration_generation) >= self.migration_interval + + def migrate_programs(self) -> None: + """ + Perform migration between islands + + This should be called periodically to share good solutions between islands + """ + if len(self.islands) < 2: + return + + logger.info("Performing migration between islands") + + for i, island in enumerate(self.islands): + if len(island) == 0: + continue + + # Select top programs from this island for migration + island_programs = [self.programs[pid] for pid in island if pid in self.programs] + if not island_programs: + continue + + # Sort by fitness (using combined_score or average metrics) + island_programs.sort( + key=lambda p: get_fitness_score(p.metrics, self.config.feature_dimensions), + reverse=True, + ) + + # Select top programs for migration + num_to_migrate = max(1, int(len(island_programs) * self.migration_rate)) + migrants = island_programs[:num_to_migrate] + + # Migrate to adjacent islands (ring topology) + target_islands = [(i + 1) % len(self.islands), (i - 1) % len(self.islands)] + + for migrant in migrants: + # Prevent re-migration of already migrated programs to avoid exponential duplication. + # Analysis of actual evolution runs shows this causes severe issues: + # - Program cb5d07f2 had 183 descendant copies by iteration 850 + # - Program 5645fbd2 had 31 descendant copies + # - IDs grow exponentially: program_migrant_2_migrant_3_migrant_4_migrant_0... + # + # This is particularly problematic for OpenEvolve's MAP-Elites + Island hybrid architecture: + # 1. All copies have identical code → same complexity/diversity/performance scores + # 2. They all map to the SAME MAP-Elites cell → only 1 survives, rest discarded + # 3. Wastes computation evaluating hundreds of identical programs + # 4. Reduces actual diversity as islands fill with duplicates + # + # By preventing already-migrated programs from migrating again, we ensure: + # - Each program migrates at most once per lineage + # - True diversity is maintained between islands + # - Computational resources aren't wasted on duplicates + # - Aligns with MAP-Elites' one-program-per-cell principle + if migrant.metadata.get("migrant", False): + continue + + for target_island in target_islands: + # Skip migration if target island already has a program with identical code + # Identical code produces identical metrics, so migration would be wasteful + target_island_programs = [ + self.programs[pid] + for pid in self.islands[target_island] + if pid in self.programs + ] + has_duplicate_code = any(p.code == migrant.code for p in target_island_programs) + + if has_duplicate_code: + logger.debug( + f"Skipping migration of program {migrant.id[:8]} to island {target_island} " + f"(duplicate code already exists)" + ) + continue + # Create a copy for migration with simple new UUID + import uuid + + migrant_copy = Program( + id=str(uuid.uuid4()), + code=migrant.code, + changes_description=migrant.changes_description, + language=migrant.language, + parent_id=migrant.id, + generation=migrant.generation, + metrics=migrant.metrics.copy(), + metadata={**migrant.metadata, "island": target_island, "migrant": True}, + ) + + # Use add() method to properly handle MAP-Elites deduplication, + # feature map updates, and island tracking + self.add(migrant_copy, target_island=target_island) + + # Log migration + logger.info( + "Program %s migrated to island %d", + migrant_copy.id[:8], + target_island, + ) + + # Update last migration generation + self.last_migration_generation = max(self.island_generations) + logger.info(f"Migration completed at generation {self.last_migration_generation}") + + # Validate migration results + self._validate_migration_results() + + def _validate_migration_results(self) -> None: + """ + Validate migration didn't create inconsistencies + + Checks that: + 1. Program island metadata matches actual island assignment + 2. No programs are assigned to multiple islands + 3. All island best programs exist and are in correct islands + """ + seen_program_ids = set() + + for i, island in enumerate(self.islands): + for program_id in island: + # Check for duplicate assignments + if program_id in seen_program_ids: + logger.error(f"Program {program_id} assigned to multiple islands") + continue + seen_program_ids.add(program_id) + + # Check program exists + if program_id not in self.programs: + logger.warning(f"Island {i} contains nonexistent program {program_id}") + continue + + # Check metadata consistency + program = self.programs[program_id] + stored_island = program.metadata.get("island") + if stored_island != i: + logger.warning( + f"Island mismatch for program {program_id}: " + f"in island {i} but metadata says {stored_island}" + ) + + # Validate island best programs + for i, best_id in enumerate(self.island_best_programs): + if best_id is not None: + if best_id not in self.programs: + logger.warning(f"Island {i} best program {best_id} does not exist") + elif best_id not in self.islands[i]: + logger.warning(f"Island {i} best program {best_id} not in island") + + def _cleanup_stale_island_bests(self) -> None: + """ + Remove stale island best program references + + Cleans up references to programs that no longer exist in the database + or are not actually in their assigned islands. + """ + cleaned_count = 0 + + for i, best_id in enumerate(self.island_best_programs): + if best_id is not None: + should_clear = False + + # Check if program still exists + if best_id not in self.programs: + logger.debug( + f"Clearing stale island {i} best program {best_id} (program deleted)" + ) + should_clear = True + # Check if program is still in the island + elif best_id not in self.islands[i]: + logger.debug( + f"Clearing stale island {i} best program {best_id} (not in island)" + ) + should_clear = True + + if should_clear: + self.island_best_programs[i] = None + cleaned_count += 1 + + if cleaned_count > 0: + logger.info(f"Cleaned up {cleaned_count} stale island best program references") + + # Recalculate best programs for islands that were cleared + for i, best_id in enumerate(self.island_best_programs): + if best_id is None and len(self.islands[i]) > 0: + # Find new best program for this island + island_programs = [ + self.programs[pid] for pid in self.islands[i] if pid in self.programs + ] + if island_programs: + # Sort by fitness and update + best_program = max( + island_programs, + key=lambda p: p.metrics.get( + "combined_score", safe_numeric_average(p.metrics) + ), + ) + self.island_best_programs[i] = best_program.id + logger.debug(f"Recalculated island {i} best program: {best_program.id}") + + def get_island_stats(self) -> List[dict]: + """Get statistics for each island""" + stats = [] + + for i, island in enumerate(self.islands): + island_programs = [self.programs[pid] for pid in island if pid in self.programs] + + if island_programs: + scores = [ + get_fitness_score(p.metrics, self.config.feature_dimensions) + for p in island_programs + ] + + best_score = max(scores) if scores else 0.0 + avg_score = sum(scores) / len(scores) if scores else 0.0 + diversity = self._calculate_island_diversity(island_programs) + else: + best_score = avg_score = diversity = 0.0 + + stats.append( + { + "island": i, + "population_size": len(island_programs), + "best_score": best_score, + "average_score": avg_score, + "diversity": diversity, + "generation": self.island_generations[i], + "is_current": i == self.current_island, + } + ) + + return stats + + def _calculate_island_diversity(self, programs: List[Program]) -> float: + """Calculate diversity within an island (deterministic version)""" + if len(programs) < 2: + return 0.0 + + total_diversity = 0 + comparisons = 0 + + # Use deterministic sampling instead of random.sample() to ensure consistent results + sample_size = min(5, len(programs)) # Reduced from 10 to 5 + + # Sort programs by ID for deterministic ordering + sorted_programs = sorted(programs, key=lambda p: p.id) + + # Take first N programs instead of random sampling + sample_programs = sorted_programs[:sample_size] + + # Limit total comparisons for performance + max_comparisons = 6 # Maximum comparisons to prevent long delays + + for i, prog1 in enumerate(sample_programs): + for prog2 in sample_programs[i + 1 :]: + if comparisons >= max_comparisons: + break + + # Use fast approximation instead of expensive edit distance + diversity = self._fast_code_diversity(prog1.code, prog2.code) + total_diversity += diversity + comparisons += 1 + + if comparisons >= max_comparisons: + break + + return total_diversity / max(1, comparisons) + + def _fast_code_diversity(self, code1: str, code2: str) -> float: + """ + Fast approximation of code diversity using simple metrics + + Returns diversity score (higher = more diverse) + """ + if code1 == code2: + return 0.0 + + # Length difference (scaled to reasonable range) + len1, len2 = len(code1), len(code2) + length_diff = abs(len1 - len2) + + # Line count difference + lines1 = code1.count("\n") + lines2 = code2.count("\n") + line_diff = abs(lines1 - lines2) + + # Simple character set difference + chars1 = set(code1) + chars2 = set(code2) + char_diff = len(chars1.symmetric_difference(chars2)) + + # Combine metrics (scaled to match original edit distance range) + diversity = length_diff * 0.1 + line_diff * 10 + char_diff * 0.5 + + return diversity + + def _get_cached_diversity(self, program: Program) -> float: + """ + Get diversity score for a program using cache and reference set + + Args: + program: The program to calculate diversity for + + Returns: + Diversity score (cached or newly computed) + """ + code_hash = hash(program.code) + + # Check cache first + if code_hash in self.diversity_cache: + return self.diversity_cache[code_hash]["value"] + + # Update reference set if needed + if ( + not self.diversity_reference_set + or len(self.diversity_reference_set) < self.diversity_reference_size + ): + self._update_diversity_reference_set() + + # Compute diversity against reference set + diversity_scores = [] + for ref_code in self.diversity_reference_set: + if ref_code != program.code: # Don't compare with itself + diversity_scores.append(self._fast_code_diversity(program.code, ref_code)) + + diversity = ( + sum(diversity_scores) / max(1, len(diversity_scores)) if diversity_scores else 0.0 + ) + + # Cache the result with LRU eviction + self._cache_diversity_value(code_hash, diversity) + + return diversity + + def _update_diversity_reference_set(self) -> None: + """Update the reference set for diversity calculation""" + if len(self.programs) == 0: + return + + # Select diverse programs for reference set + all_programs = list(self.programs.values()) + + if len(all_programs) <= self.diversity_reference_size: + self.diversity_reference_set = [p.code for p in all_programs] + else: + # Select programs with maximum diversity + selected = [] + remaining = all_programs.copy() + + # Start with a random program + first_idx = random.randint(0, len(remaining) - 1) + selected.append(remaining.pop(first_idx)) + + # Greedily add programs that maximize diversity to selected set + while len(selected) < self.diversity_reference_size and remaining: + max_diversity = -1 + best_idx = -1 + + for i, candidate in enumerate(remaining): + # Calculate minimum diversity to selected programs + min_div = float("inf") + for selected_prog in selected: + div = self._fast_code_diversity(candidate.code, selected_prog.code) + min_div = min(min_div, div) + + if min_div > max_diversity: + max_diversity = min_div + best_idx = i + + if best_idx >= 0: + selected.append(remaining.pop(best_idx)) + + self.diversity_reference_set = [p.code for p in selected] + + logger.debug( + f"Updated diversity reference set with {len(self.diversity_reference_set)} programs" + ) + + def _cache_diversity_value(self, code_hash: int, diversity: float) -> None: + """Cache a diversity value with LRU eviction""" + # Check if cache is full + if len(self.diversity_cache) >= self.diversity_cache_size: + # Remove oldest entry + oldest_hash = min(self.diversity_cache.items(), key=lambda x: x[1]["timestamp"])[0] + del self.diversity_cache[oldest_hash] + + # Add new entry + self.diversity_cache[code_hash] = {"value": diversity, "timestamp": time.time()} + + def _invalidate_diversity_cache(self) -> None: + """Invalidate the diversity cache when programs change significantly""" + self.diversity_cache.clear() + self.diversity_reference_set = [] + logger.debug("Diversity cache invalidated") + + def _update_feature_stats(self, feature_name: str, value: float) -> None: + """ + Update statistics for a feature dimension + + Args: + feature_name: Name of the feature dimension + value: New value to incorporate into stats + """ + if feature_name not in self.feature_stats: + self.feature_stats[feature_name] = { + "min": value, + "max": value, + "values": [], # Keep recent values for percentile calculation if needed + } + + stats = self.feature_stats[feature_name] + stats["min"] = min(stats["min"], value) + stats["max"] = max(stats["max"], value) + + # Keep recent values for more sophisticated scaling methods + stats["values"].append(value) + if len(stats["values"]) > 1000: # Limit memory usage + stats["values"] = stats["values"][-1000:] + + def _scale_feature_value(self, feature_name: str, value: float) -> float: + """ + Scale a feature value according to the configured scaling method + + Args: + feature_name: Name of the feature dimension + value: Raw feature value + + Returns: + Scaled value in range [0, 1] + """ + if feature_name not in self.feature_stats: + # No stats yet, return normalized by a reasonable default + return min(1.0, max(0.0, value)) + + stats = self.feature_stats[feature_name] + + if self.feature_scaling_method == "minmax": + # Min-max normalization to [0, 1] + min_val = stats["min"] + max_val = stats["max"] + + if max_val == min_val: + return 0.5 # All values are the same + + scaled = (value - min_val) / (max_val - min_val) + return min(1.0, max(0.0, scaled)) # Ensure in [0, 1] + + elif self.feature_scaling_method == "percentile": + # Use percentile ranking + values = stats["values"] + if not values: + return 0.5 + + # Count how many values are less than or equal to this value + count = sum(1 for v in values if v <= value) + percentile = count / len(values) + return percentile + + else: + # Default to min-max if unknown method + return self._scale_feature_value_minmax(feature_name, value) + + def _scale_feature_value_minmax(self, feature_name: str, value: float) -> float: + """Helper for min-max scaling""" + if feature_name not in self.feature_stats: + return min(1.0, max(0.0, value)) + + stats = self.feature_stats[feature_name] + min_val = stats["min"] + max_val = stats["max"] + + if max_val == min_val: + return 0.5 + + scaled = (value - min_val) / (max_val - min_val) + return min(1.0, max(0.0, scaled)) + + def log_island_status(self) -> None: + """Log current status of all islands""" + stats = self.get_island_stats() + logger.info("Island Status:") + for stat in stats: + current_marker = " *" if stat["is_current"] else " " + island_idx = stat["island"] + island_best_id = ( + self.island_best_programs[island_idx] + if island_idx < len(self.island_best_programs) + else None + ) + best_indicator = f" (best: {island_best_id})" if island_best_id else "" + logger.info( + f"{current_marker} Island {stat['island']}: {stat['population_size']} programs, " + f"best={stat['best_score']:.4f}, avg={stat['average_score']:.4f}, " + f"diversity={stat['diversity']:.2f}, gen={stat['generation']}{best_indicator}" + ) + + # Artifact storage and retrieval methods + + def store_artifacts(self, program_id: str, artifacts: Dict[str, Union[str, bytes]]) -> None: + """ + Store artifacts for a program + + Args: + program_id: ID of the program + artifacts: Dictionary of artifact name to content + """ + if not artifacts: + return + + program = self.programs.get(program_id) + if not program: + logger.warning(f"Cannot store artifacts: program {program_id} not found") + return + + # Check if artifacts are enabled + artifacts_enabled = os.environ.get("ENABLE_ARTIFACTS", "true").lower() == "true" + if not artifacts_enabled: + logger.debug("Artifacts disabled, skipping storage") + return + + # Split artifacts by size + small_artifacts = {} + large_artifacts = {} + size_threshold = getattr(self.config, "artifact_size_threshold", 32 * 1024) # 32KB default + + for key, value in artifacts.items(): + size = self._get_artifact_size(value) + if size <= size_threshold: + small_artifacts[key] = value + else: + large_artifacts[key] = value + + # Store small artifacts as JSON + if small_artifacts: + program.artifacts_json = json.dumps(small_artifacts, default=self._artifact_serializer) + logger.debug(f"Stored {len(small_artifacts)} small artifacts for program {program_id}") + + # Artifact retention is independent of population persistence. + self._cleanup_old_artifacts() + + # Store large artifacts to disk + if large_artifacts: + artifact_dir = self._create_artifact_dir(program_id) + program.artifact_dir = artifact_dir + for key, value in large_artifacts.items(): + self._write_artifact_file(artifact_dir, key, value) + logger.debug(f"Stored {len(large_artifacts)} large artifacts for program {program_id}") + + def get_artifacts(self, program_id: str) -> Dict[str, Union[str, bytes]]: + """ + Retrieve all artifacts for a program + + Args: + program_id: ID of the program + + Returns: + Dictionary of artifact name to content + """ + program = self.programs.get(program_id) + if not program: + return {} + + artifacts = {} + + # Load small artifacts from JSON + if program.artifacts_json: + try: + small_artifacts = json.loads(program.artifacts_json) + artifacts.update(small_artifacts) + except json.JSONDecodeError as e: + logger.warning(f"Failed to decode artifacts JSON for program {program_id}: {e}") + + # Load large artifacts from disk + if program.artifact_dir and os.path.exists(program.artifact_dir): + disk_artifacts = self._load_artifact_dir(program.artifact_dir) + artifacts.update(disk_artifacts) + + return artifacts + + def _get_artifact_size(self, value: Union[str, bytes]) -> int: + """Get size of an artifact value in bytes""" + if isinstance(value, str): + return len(value.encode("utf-8")) + elif isinstance(value, bytes): + return len(value) + else: + return len(str(value).encode("utf-8")) + + def _artifact_serializer(self, obj): + """JSON serializer for artifacts that handles bytes""" + if isinstance(obj, bytes): + return {"__bytes__": base64.b64encode(obj).decode("utf-8")} + raise TypeError(f"Object of type {type(obj)} is not JSON serializable") + + def _artifact_deserializer(self, dct): + """JSON deserializer for artifacts that handles bytes""" + if "__bytes__" in dct: + return base64.b64decode(dct["__bytes__"]) + return dct + + def _artifacts_base_path(self) -> str: + return self.config.artifacts_base_path or os.path.join( + self.config.db_path or ".", "artifacts" + ) + + def _create_artifact_dir(self, program_id: str) -> str: + """Create artifact directory for a program.""" + artifact_dir = os.path.join(self._artifacts_base_path(), program_id) + os.makedirs(artifact_dir, exist_ok=True) + return artifact_dir + + def _cleanup_old_artifacts(self) -> None: + """Remove artifacts older than the configured retention period.""" + if not self.config.cleanup_old_artifacts: + return + artifacts_base_path = self._artifacts_base_path() + + if not os.path.isdir(artifacts_base_path): + return + + now = time.time() + retention_seconds = self.config.artifact_retention_days * 24 * 60 * 60 + deleted_count = 0 + + logger.debug(f"Starting artifact cleanup in {artifacts_base_path}...") + + for dirname in os.listdir(artifacts_base_path): + dirpath = os.path.join(artifacts_base_path, dirname) + if os.path.isdir(dirpath): + try: + dir_mod_time = os.path.getmtime(dirpath) + if (now - dir_mod_time) > retention_seconds: + shutil.rmtree(dirpath) + deleted_count += 1 + logger.debug(f"Removed old artifact directory: {dirpath}") + except FileNotFoundError: + # Can happen in race conditions; ignore. + continue + except Exception as e: + logger.error(f"Error removing artifact directory {dirpath}: {e}") + + if deleted_count > 0: + logger.info(f"Cleaned up {deleted_count} old artifact directories.") + + def _write_artifact_file(self, artifact_dir: str, key: str, value: Union[str, bytes]) -> None: + """Write an artifact to a file""" + # Sanitize filename + safe_key = "".join(c for c in key if c.isalnum() or c in "._-") + if not safe_key: + safe_key = "artifact" + + file_path = os.path.join(artifact_dir, safe_key) + + try: + if isinstance(value, str): + with open(file_path, "w", encoding="utf-8") as f: + f.write(value) + elif isinstance(value, bytes): + with open(file_path, "wb") as f: + f.write(value) + else: + # Convert to string and write + with open(file_path, "w", encoding="utf-8") as f: + f.write(str(value)) + except Exception as e: + logger.warning(f"Failed to write artifact {key} to {file_path}: {e}") + + def _load_artifact_dir(self, artifact_dir: str) -> Dict[str, Union[str, bytes]]: + """Load artifacts from a directory""" + artifacts = {} + + try: + for filename in os.listdir(artifact_dir): + file_path = os.path.join(artifact_dir, filename) + if os.path.isfile(file_path): + try: + # Try to read as text first + with open(file_path, "r", encoding="utf-8") as f: + content = f.read() + artifacts[filename] = content + except UnicodeDecodeError: + # If text fails, read as binary + with open(file_path, "rb") as f: + content = f.read() + artifacts[filename] = content + except Exception as e: + logger.warning(f"Failed to read artifact file {file_path}: {e}") + except Exception as e: + logger.warning(f"Failed to list artifact directory {artifact_dir}: {e}") + + return artifacts + + def log_prompt( + self, + program_id: str, + template_key: str, + prompt: Dict[str, str], + responses: Optional[List[str]] = None, + ) -> None: + """ + Log a prompt for a program. + Only logs if self.config.log_prompts is True. + + Args: + program_id: ID of the program to log the prompt for + template_key: Key for the prompt template + prompt: Prompts in the format {template_key: { 'system': str, 'user': str }}. + responses: Optional list of responses to the prompt, if available. + """ + + if not self.config.log_prompts: + return + + if responses is None: + responses = [] + prompt = deepcopy(prompt) + prompt["responses"] = list(responses) + + if self.prompts_by_program is None: + self.prompts_by_program = {} + + if program_id not in self.prompts_by_program: + self.prompts_by_program[program_id] = {} + self.prompts_by_program[program_id][template_key] = prompt diff --git a/openevolve/evaluator.py b/openevolve/evaluator.py index b1142ece50..ba6b35ad8a 100644 --- a/openevolve/evaluator.py +++ b/openevolve/evaluator.py @@ -18,7 +18,6 @@ import traceback from openevolve.config import EvaluatorConfig -from openevolve.database import ProgramDatabase from openevolve.evaluation_result import EvaluationResult from openevolve.database import ProgramDatabase from openevolve.llm.ensemble import LLMEnsemble @@ -563,7 +562,9 @@ async def _llm_evaluate(self, program_code: str, program_id: str = "") -> Dict[s try: # Create prompt for LLM - feature_dimensions = self.database.config.feature_dimensions if self.database else [] + feature_dimensions = ( + list(self.database.get_state().feature_dimensions) if self.database else [] + ) prompt = self.prompt_sampler.build_prompt( current_program=program_code, template_key="evaluation", diff --git a/openevolve/process_parallel.py b/openevolve/process_parallel.py index b2cfeab788..c324f73530 100644 --- a/openevolve/process_parallel.py +++ b/openevolve/process_parallel.py @@ -5,12 +5,11 @@ import asyncio import logging import multiprocessing as mp -import pickle import signal import time from concurrent.futures import Future, ProcessPoolExecutor from concurrent.futures import TimeoutError as FutureTimeoutError -from dataclasses import asdict, dataclass +from dataclasses import asdict, dataclass, replace from pathlib import Path from typing import Any, Dict, List, Optional, Tuple @@ -21,6 +20,18 @@ logger = logging.getLogger(__name__) +@dataclass +class IterationContext: + """Selected values for one worker; its size does not grow with the population.""" + + parent: Program + inspirations: List[Program] + top_programs: List[Program] + parent_artifacts: Dict[str, Any] + target_island: int + feature_dimensions: Tuple[str, ...] + + @dataclass class SerializableResult: """Result that can be pickled and sent between processes""" @@ -131,42 +142,16 @@ def _lazy_init_worker_components(): ) -def _run_iteration_worker( - iteration: int, db_snapshot: Dict[str, Any], parent_id: str, inspiration_ids: List[str] -) -> SerializableResult: +def _run_iteration_worker(iteration: int, context: IterationContext) -> SerializableResult: """Run a single iteration in a worker process""" try: # Lazy initialization _lazy_init_worker_components() - # Reconstruct programs from snapshot - programs = {pid: Program(**prog_dict) for pid, prog_dict in db_snapshot["programs"].items()} - - parent = programs[parent_id] - inspirations = [programs[pid] for pid in inspiration_ids if pid in programs] - - # Get parent artifacts if available - parent_artifacts = db_snapshot["artifacts"].get(parent_id) - - # Get island-specific programs for context - parent_island = parent.metadata.get("island", db_snapshot["current_island"]) - island_programs = [ - programs[pid] for pid in db_snapshot["islands"][parent_island] if pid in programs - ] - - # Sort by metrics for top programs - island_programs.sort( - key=lambda p: p.metrics.get("combined_score", safe_numeric_average(p.metrics)), - reverse=True, - ) - - # Use config values for limits instead of hardcoding - # Programs for LLM display (includes both top and diverse for inspiration) - programs_for_prompt = island_programs[ - : _worker_config.prompt.num_top_programs + _worker_config.prompt.num_diverse_programs - ] - # Best programs only (for previous attempts section, focused on top performers) - best_programs_only = island_programs[: _worker_config.prompt.num_top_programs] + parent = context.parent + inspirations = context.inspirations + programs_for_prompt = context.top_programs + best_programs_only = programs_for_prompt[: _worker_config.prompt.num_top_programs] # Build prompt if _worker_config.prompt.programs_as_changes_description: @@ -188,8 +173,8 @@ def _run_iteration_worker( language=_worker_config.language, evolution_round=iteration, diff_based_evolution=_worker_config.diff_based_evolution, - program_artifacts=parent_artifacts, - feature_dimensions=db_snapshot.get("feature_dimensions", []), + program_artifacts=context.parent_artifacts, + feature_dimensions=list(context.feature_dimensions), current_changes_description=parent_changes_desc, ) @@ -307,15 +292,12 @@ def _run_iteration_worker( metadata={ "changes": changes_summary, "parent_metrics": parent.metrics, - "island": parent_island, + "island": context.target_island, }, ) iteration_time = time.time() - iteration_start - # Get target island from snapshot (where child should be placed) - target_island = db_snapshot.get("sampling_island") - return SerializableResult( child_program_dict=child_program.to_dict(), parent_id=parent.id, @@ -324,7 +306,7 @@ def _run_iteration_worker( llm_response=llm_response, artifacts=artifacts, iteration=iteration, - target_island=target_island, + target_island=context.target_island, ) except Exception as e: @@ -410,7 +392,7 @@ def __init__( # Number of worker processes self.num_workers = config.evaluator.parallel_evaluations - self.num_islands = config.database.num_islands + self.num_islands = database.get_state().num_islands logger.info(f"Initialized process parallel controller with {self.num_workers} workers") @@ -418,8 +400,8 @@ def _serialize_config(self, config: Config) -> dict: """Serialize config object to a dictionary that can be pickled""" # Manual serialization to handle nested objects properly - # The asdict() call itself triggers the deepcopy which tries to serialize novelty_llm. Remove it first. - config.database.novelty_llm = None + # Keep runtime clients out of worker configuration without changing the store. + database_config = replace(config.database, novelty_llm=None) return { "llm": { @@ -435,10 +417,9 @@ def _serialize_config(self, config: Config) -> dict: "retry_delay": config.llm.retry_delay, }, "prompt": asdict(config.prompt), - "database": asdict(config.database), + "database": asdict(database_config), "evaluator": asdict(config.evaluator), "max_iterations": config.max_iterations, - "checkpoint_interval": config.checkpoint_interval, "log_level": config.log_level, "log_dir": config.log_dir, "random_seed": config.random_seed, @@ -495,42 +476,11 @@ def request_shutdown(self) -> None: logger.info("Graceful shutdown requested...") self.shutdown_event.set() - def _create_database_snapshot(self) -> Dict[str, Any]: - """Create a serializable snapshot of the database state""" - # Only include necessary data for workers - snapshot = { - "programs": {pid: prog.to_dict() for pid, prog in self.database.programs.items()}, - "islands": [list(island) for island in self.database.islands], - "current_island": self.database.current_island, - "feature_dimensions": self.database.config.feature_dimensions, - "artifacts": {}, # Will be populated selectively - } - - # Include artifacts for programs that might be selected - # This limits artifacts (execution outputs/errors) to avoid large snapshot sizes. - # This does NOT affect program code - all programs are fully serialized above. - # With max_artifact_bytes=20KB and population_size=1000, artifacts could be 20MB total, - # which would significantly slow worker process initialization. The default limit of 100 - # keeps artifact data under 2MB while still providing execution context for recent programs. - # Workers can still evolve properly as they have access to ALL program code. - # Configure via database.max_snapshot_artifacts (None for unlimited). - max_artifacts = self.database.config.max_snapshot_artifacts - program_ids = list(self.database.programs.keys()) - if max_artifacts is not None: - program_ids = program_ids[:max_artifacts] - for pid in program_ids: - artifacts = self.database.get_artifacts(pid) - if artifacts: - snapshot["artifacts"][pid] = artifacts - - return snapshot - async def run_evolution( self, start_iteration: int, max_iterations: int, target_score: Optional[float] = None, - checkpoint_callback=None, ): """Run evolution with process-based parallelism""" if not self.executor: @@ -636,8 +586,12 @@ async def run_evolution( ) if parent_program: # Determine island ID - island_id = child_program.metadata.get( - "island", self.database.current_island + island_id = ( + result.target_island + if result.target_island is not None + else child_program.metadata.get( + "island", self.database.get_state().current_island + ) ) self.evolution_tracer.log_trace( @@ -669,7 +623,13 @@ async def run_evolution( # Island management # get current program island id - island_id = child_program.metadata.get("island", self.database.current_island) + island_id = ( + result.target_island + if result.target_island is not None + else child_program.metadata.get( + "island", self.database.get_state().current_island + ) + ) # use this to increment island generation self.database.increment_island_generation(island_idx=island_id) @@ -677,7 +637,7 @@ async def run_evolution( if self.database.should_migrate(): logger.info(f"Performing migration at iteration {completed_iteration}") self.database.migrate_programs() - self.database.log_island_status() + logger.info("Island statistics: %s", self.database.get_island_stats()) # Log progress logger.info( @@ -714,25 +674,13 @@ async def run_evolution( self._warned_about_combined_score = True # Check for new best - if self.database.best_program_id == child_program.id: + best_program = self.database.get_best_program() + if best_program and best_program.id == child_program.id: logger.info( f"🌟 New best solution found at iteration {completed_iteration}: " f"{child_program.id}" ) - # Checkpoint callback - # Don't checkpoint at iteration 0 (that's just the initial program) - if ( - completed_iteration > 0 - and completed_iteration % self.config.checkpoint_interval == 0 - ): - logger.info( - f"Checkpoint interval reached at iteration {completed_iteration}" - ) - self.database.log_island_status() - if checkpoint_callback: - checkpoint_callback(completed_iteration) - # Check target score if target_score is not None and child_program.metrics: if ( @@ -811,6 +759,9 @@ async def run_evolution( except Exception as e: logger.error(f"Error processing result from iteration {completed_iteration}: {e}") + finally: + self.database.record_iteration(completed_iteration) + completed_iterations += 1 # Remove completed iteration from island tracking @@ -849,36 +800,36 @@ async def run_evolution( return self.database.get_best_program() + def _select_iteration_context(self, island_id: Optional[int] = None) -> IterationContext: + """Query only the programs and evidence needed for this candidate.""" + state = self.database.get_state() + target_island = island_id if island_id is not None else state.current_island + parent, inspirations = self.database.sample_from_island( + island_id=target_island, + num_inspirations=self.config.prompt.num_diverse_programs, + ) + # Preserve the existing prompt context when sampling falls back to another island. + context_island = parent.metadata.get("island", target_island) + top_programs = self.database.get_top_programs( + n=self.config.prompt.num_top_programs + self.config.prompt.num_diverse_programs, + island_idx=context_island, + ) + return IterationContext( + parent=parent, + inspirations=inspirations, + top_programs=top_programs, + parent_artifacts=self.database.get_artifacts(parent.id), + target_island=target_island, + feature_dimensions=state.feature_dimensions, + ) + def _submit_iteration( self, iteration: int, island_id: Optional[int] = None ) -> Optional[Future]: """Submit an iteration to the process pool, optionally pinned to a specific island""" try: - # Use specified island or current island - target_island = island_id if island_id is not None else self.database.current_island - - # Use thread-safe sampling that doesn't modify shared state - # This fixes the race condition from GitHub issue #246 - # Inspirations are the diverse/creative examples; size them by - # num_diverse_programs (not num_top_programs) so the config parameter - # actually controls the inspiration count (GitHub issue #452). - parent, inspirations = self.database.sample_from_island( - island_id=target_island, - num_inspirations=self.config.prompt.num_diverse_programs, - ) - - # Create database snapshot - db_snapshot = self._create_database_snapshot() - db_snapshot["sampling_island"] = target_island # Mark which island this is for - - # Submit to process pool - future = self.executor.submit( - _run_iteration_worker, - iteration, - db_snapshot, - parent.id, - [insp.id for insp in inspirations], - ) + context = self._select_iteration_context(island_id) + future = self.executor.submit(_run_iteration_worker, iteration, context) return future diff --git a/openevolve/program.py b/openevolve/program.py new file mode 100644 index 0000000000..d54c761766 --- /dev/null +++ b/openevolve/program.py @@ -0,0 +1,80 @@ +"""Program value shared by all program database implementations.""" + +import logging +import time +from dataclasses import asdict, dataclass, field, fields +from typing import Any, Dict, List, Optional + +logger = logging.getLogger(__name__) + + +@dataclass +class Program: + """Represents a program in the database""" + + # Program identification + id: str + code: str + changes_description: str = ( + "" # compact program changes description (via LLM) stored per program + ) + language: str = "python" + + # Evolution information + parent_id: Optional[str] = None + generation: int = 0 + timestamp: float = field(default_factory=time.time) + iteration_found: int = 0 # Track which iteration this program was found + + # Performance metrics + metrics: Dict[str, float] = field(default_factory=dict) + + # Derived features + complexity: float = 0.0 + diversity: float = 0.0 + + # Metadata + metadata: Dict[str, Any] = field(default_factory=dict) + + # Prompts + prompts: Optional[Dict[str, Any]] = None + + # Artifact storage + artifacts_json: Optional[str] = None # JSON-serialized small artifacts + artifact_dir: Optional[str] = None # Path to large artifact files + + # Embedding vector for novelty rejection sampling + embedding: Optional[List[float]] = None + + def to_dict(self) -> Dict[str, Any]: + """Convert to dictionary representation""" + return asdict(self) + + @classmethod + def from_dict(cls, data: Dict[str, Any]) -> "Program": + """Create from dictionary representation""" + # old DBs don't have changes_description (backward-compatibility) + if "changes_description" not in data: + metadata = data.get("metadata") or {} + if isinstance(metadata, dict): + data = { + **data, + "changes_description": metadata.get("changes_description") + or metadata.get("changes") + or "empty", + } + else: + data = {**data, "changes_description": "empty"} + + # Get the valid field names for the Program dataclass + valid_fields = {f.name for f in fields(cls)} + + # Filter the data to only include valid fields + filtered_data = {k: v for k, v in data.items() if k in valid_fields} + + # Log if we're filtering out any fields + if len(filtered_data) != len(data): + filtered_out = set(data.keys()) - set(filtered_data.keys()) + logger.debug(f"Filtered out unsupported fields when loading Program: {filtered_out}") + + return cls(**filtered_data) diff --git a/tests/integration/test_checkpoint_with_llm.py b/tests/integration/test_checkpoint_with_llm.py deleted file mode 100644 index e801f9cfbe..0000000000 --- a/tests/integration/test_checkpoint_with_llm.py +++ /dev/null @@ -1,170 +0,0 @@ -""" -Integration tests for checkpoint functionality with real LLM inference -""" - -import pytest -import asyncio -from openevolve.controller import OpenEvolve - - -class TestCheckpointWithLLM: - """Test checkpoints with real LLM generation""" - - @pytest.mark.slow - @pytest.mark.asyncio - async def test_checkpoint_intervals_with_real_llm( - self, - optillm_server, - evolution_config, - test_program_file, - test_evaluator_file, - evolution_output_dir - ): - """Test checkpoints occur at correct intervals with real evolution""" - evolution_config.checkpoint_interval = 2 - evolution_config.max_iterations = 4 # Much smaller for CI speed - evolution_config.evaluator.timeout = 15 # Shorter timeout for CI - - checkpoint_calls = [] - - controller = OpenEvolve( - initial_program_path=str(test_program_file), - evaluation_file=str(test_evaluator_file), - config=evolution_config, - output_dir=str(evolution_output_dir) - ) - - # Track checkpoint calls - original_save = controller._save_checkpoint - controller._save_checkpoint = lambda i: checkpoint_calls.append(i) or original_save(i) - - await controller.run(iterations=4) - - # Check that some checkpoints were called - # Note: Checkpoints only occur on successful iterations - print(f"Checkpoint calls: {checkpoint_calls}") - - # We expect checkpoints at multiples of 2, but only for successful iterations - # So we might see some subset of [2, 4] depending on how many iterations succeeded - expected_checkpoints = [2, 4] - successful_checkpoints = [cp for cp in expected_checkpoints if cp in checkpoint_calls] - - # At least one checkpoint should have occurred if any iterations succeeded - if len(controller.database.programs) > 1: # More than just initial program - assert len(checkpoint_calls) > 0, "Should have at least one checkpoint call if evolution succeeded" - - @pytest.mark.slow - @pytest.mark.asyncio - async def test_checkpoint_resume_functionality( - self, - optillm_server, - evolution_config, - test_program_file, - test_evaluator_file, - evolution_output_dir - ): - """Test checkpoint save and resume with real LLM""" - evolution_config.checkpoint_interval = 4 - evolution_config.max_iterations = 8 - - # Run first phase - controller1 = OpenEvolve( - initial_program_path=str(test_program_file), - evaluation_file=str(test_evaluator_file), - config=evolution_config, - output_dir=str(evolution_output_dir) - ) - - await controller1.run(iterations=6) - - # Check if checkpoint was created - checkpoints_dir = evolution_output_dir / "checkpoints" - if checkpoints_dir.exists(): - checkpoint_dirs = [d for d in checkpoints_dir.iterdir() if d.is_dir() and d.name.startswith("checkpoint_")] - print(f"Found checkpoint directories: {[d.name for d in checkpoint_dirs]}") - - if checkpoint_dirs: - # Find the latest checkpoint - latest_checkpoint = max(checkpoint_dirs, key=lambda d: int(d.name.split("_")[1])) - checkpoint_iter = int(latest_checkpoint.name.split("_")[1]) - - # Test resume (simplified - just verify the checkpoint directory structure) - assert (latest_checkpoint / "database.json").exists(), "Database checkpoint should exist" - print(f"Successfully created checkpoint at iteration {checkpoint_iter}") - else: - print("No checkpoints created (likely due to all iterations failing)") - else: - print("No checkpoints directory created") - - @pytest.mark.slow - @pytest.mark.asyncio - async def test_final_checkpoint_creation( - self, - optillm_server, - evolution_config, - test_program_file, - test_evaluator_file, - evolution_output_dir - ): - """Test that final checkpoint is created regardless of interval""" - evolution_config.checkpoint_interval = 100 # Large interval - evolution_config.max_iterations = 5 - - checkpoint_calls = [] - - controller = OpenEvolve( - initial_program_path=str(test_program_file), - evaluation_file=str(test_evaluator_file), - config=evolution_config, - output_dir=str(evolution_output_dir) - ) - - original_save = controller._save_checkpoint - controller._save_checkpoint = lambda i: checkpoint_calls.append(i) or original_save(i) - - await controller.run(iterations=5) - - print(f"Final checkpoint calls: {checkpoint_calls}") - - # Final checkpoint may be created at the end even if no interval checkpoints occurred - # This depends on the controller logic, so we just verify the system didn't crash - assert len(controller.database.programs) >= 1, "Should have at least the initial program" - - @pytest.mark.slow - @pytest.mark.asyncio - async def test_checkpoint_with_best_program_save( - self, - optillm_server, - evolution_config, - test_program_file, - test_evaluator_file, - evolution_output_dir - ): - """Test that checkpoints include best program information""" - evolution_config.checkpoint_interval = 3 - evolution_config.max_iterations = 6 - - controller = OpenEvolve( - initial_program_path=str(test_program_file), - evaluation_file=str(test_evaluator_file), - config=evolution_config, - output_dir=str(evolution_output_dir) - ) - - await controller.run(iterations=6) - - # Check best program directory - best_dir = evolution_output_dir / "best" - if best_dir.exists(): - best_files = list(best_dir.glob("*")) - print(f"Best program files: {[f.name for f in best_files]}") - - # Should have best program file and info - program_files = [f for f in best_files if f.suffix == ".py"] - info_files = [f for f in best_files if f.name.endswith("_info.json")] - - if program_files: - assert len(program_files) >= 1, "Should have best program file" - - if info_files: - assert len(info_files) >= 1, "Should have best program info file" \ No newline at end of file diff --git a/tests/integration/test_evolution_pipeline.py b/tests/integration/test_evolution_pipeline.py index 489d815e35..f87e599ee9 100644 --- a/tests/integration/test_evolution_pipeline.py +++ b/tests/integration/test_evolution_pipeline.py @@ -23,7 +23,6 @@ async def test_full_evolution_loop( """Test complete evolution with real LLM""" # Configure smaller iteration count for testing evolution_config.max_iterations = 8 - evolution_config.checkpoint_interval = 4 # Run evolution controller = OpenEvolve( diff --git a/tests/integration/test_examples_validation.py b/tests/integration/test_examples_validation.py index 56002badab..852efa775d 100644 --- a/tests/integration/test_examples_validation.py +++ b/tests/integration/test_examples_validation.py @@ -232,7 +232,9 @@ def tearDown(self): def test_database_stores_and_retrieves_programs(self): """Test that the database can store and retrieve programs""" - from openevolve.database import ProgramDatabase, Program, DatabaseConfig + from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase + from openevolve.database import Program + from openevolve.config import DatabaseConfig config = DatabaseConfig(population_size=100) db = ProgramDatabase(config) @@ -253,7 +255,9 @@ def test_database_stores_and_retrieves_programs(self): def test_program_evolution_tracking(self): """Test that program generations are tracked correctly""" - from openevolve.database import ProgramDatabase, Program, DatabaseConfig + from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase + from openevolve.database import Program + from openevolve.config import DatabaseConfig config = DatabaseConfig(population_size=100) db = ProgramDatabase(config) diff --git a/tests/integration/test_iteration_counting_with_llm.py b/tests/integration/test_iteration_counting_with_llm.py index 3faf95db2b..5e25009693 100644 --- a/tests/integration/test_iteration_counting_with_llm.py +++ b/tests/integration/test_iteration_counting_with_llm.py @@ -1,5 +1,5 @@ """ -Integration test for controller iteration/checkpoint behavior with real LLM inference. +Integration test for controller iteration behavior with real LLM inference. Ported from tests/test_iteration_counting.py, which previously guarded this test with a runtime `skipTest` when no optillm server was reachable. It now lives here so @@ -13,7 +13,7 @@ class TestIterationCountingWithLLM: - """Real-LLM checks for iteration counting and checkpoint alignment.""" + """Real-LLM checks for iteration counting.""" @pytest.mark.slow @pytest.mark.asyncio @@ -25,9 +25,8 @@ async def test_controller_iteration_behavior( test_evaluator_file, evolution_output_dir, ): - """Run a short real evolution and verify checkpoints align with the interval.""" + """Run a short real evolution and verify processed iteration progress.""" evolution_config.max_iterations = 8 - evolution_config.checkpoint_interval = 4 evolution_config.evaluator.parallel_evaluations = 1 evolution_config.evaluator.timeout = 30 # Longer timeout for small model @@ -38,25 +37,8 @@ async def test_controller_iteration_behavior( output_dir=str(evolution_output_dir), ) - # Track checkpoint calls - checkpoint_calls = [] - original_save = controller._save_checkpoint - controller._save_checkpoint = lambda i: checkpoint_calls.append(i) or original_save(i) - await controller.run(iterations=8) - print(f"Checkpoint calls: {checkpoint_calls}") - print(f"Total programs: {len(controller.database.programs)}") - - # Should always have at least the initial program. - assert len(controller.database.programs) >= 1, "Should have at least the initial program" - - # If any evolution succeeded, checkpoints must be a subset of the expected - # interval multiples (4, 8) - never at an unexpected iteration. - if len(controller.database.programs) > 1: - assert all( - cp % evolution_config.checkpoint_interval == 0 for cp in checkpoint_calls - ), f"Checkpoints must align with interval; got {checkpoint_calls}" - assert set(checkpoint_calls).issubset( - {4, 8} - ), f"Unexpected checkpoint iterations: {checkpoint_calls}" + state = controller.database.get_state() + assert state.program_count >= 1 + assert state.last_iteration == 8 diff --git a/tests/integration/test_library_api.py b/tests/integration/test_library_api.py index d050d07bb6..ee13431eb9 100644 --- a/tests/integration/test_library_api.py +++ b/tests/integration/test_library_api.py @@ -22,7 +22,6 @@ def _get_library_test_config(port: int = 8000) -> Config: """Get config for library API tests with optillm server""" config = Config() config.max_iterations = 100 - config.checkpoint_interval = 1 config.database.in_memory = True config.evaluator.cascade_evaluation = False config.evaluator.parallel_evaluations = 1 @@ -272,7 +271,7 @@ def evaluate(program_path): output_path = Path(result.output_dir) assert output_path.exists() assert (output_path / "best").exists() - assert (output_path / "checkpoints").exists() + assert not (output_path / "checkpoints").exists() print(f"✅ run_evolution completed successfully!") print(f" Best score: {result.best_score}") diff --git a/tests/integration/test_session_with_llm.py b/tests/integration/test_session_with_llm.py new file mode 100644 index 0000000000..868bae6470 --- /dev/null +++ b/tests/integration/test_session_with_llm.py @@ -0,0 +1,63 @@ +"""Integration checks for database sessions with real model generation.""" + +import json + +import pytest + +from openevolve.controller import OpenEvolve +from openevolve.database_memory import InMemoryProgramDatabase + + +class TestSessionWithLLM: + @pytest.mark.slow + @pytest.mark.asyncio + async def test_continue_with_same_database( + self, + optillm_server, + evolution_config, + test_program_file, + test_evaluator_file, + evolution_output_dir, + ): + database = InMemoryProgramDatabase(evolution_config.database) + first = OpenEvolve( + str(test_program_file), + str(test_evaluator_file), + evolution_config, + output_dir=str(evolution_output_dir), + database=database, + ) + await first.run(iterations=2) + assert database.get_state().last_iteration == 2 + second = OpenEvolve( + str(test_program_file), + str(test_evaluator_file), + evolution_config, + output_dir=str(evolution_output_dir), + database=database, + ) + await second.run(iterations=2) + assert database.get_state().last_iteration == 4 + assert not (evolution_output_dir / "checkpoints").exists() + + @pytest.mark.slow + @pytest.mark.asyncio + async def test_best_result_export( + self, + optillm_server, + evolution_config, + test_program_file, + test_evaluator_file, + evolution_output_dir, + ): + controller = OpenEvolve( + str(test_program_file), + str(test_evaluator_file), + evolution_config, + output_dir=str(evolution_output_dir), + ) + best = await controller.run(iterations=2) + assert best is not None + best_dir = evolution_output_dir / "best" + assert (best_dir / "best_program.py").read_text() == best.code + assert json.loads((best_dir / "best_program_info.json").read_text())["id"] == best.id diff --git a/tests/test_artifacts.py b/tests/test_artifacts.py index 5ccbce55f9..0f9c97ec57 100644 --- a/tests/test_artifacts.py +++ b/tests/test_artifacts.py @@ -9,7 +9,8 @@ from unittest.mock import Mock, patch from openevolve.config import DatabaseConfig, EvaluatorConfig, PromptConfig -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase from openevolve.evaluation_result import EvaluationResult from openevolve.evaluator import Evaluator from openevolve.prompt.sampler import PromptSampler diff --git a/tests/test_artifacts_integration.py b/tests/test_artifacts_integration.py index 852b32c660..28b3c330f2 100644 --- a/tests/test_artifacts_integration.py +++ b/tests/test_artifacts_integration.py @@ -9,7 +9,8 @@ from unittest.mock import Mock, patch from openevolve.config import Config, DatabaseConfig, EvaluatorConfig, PromptConfig -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase from openevolve.evaluation_result import EvaluationResult from openevolve.evaluator import Evaluator from openevolve.prompt.sampler import PromptSampler @@ -257,7 +258,7 @@ async def run_test(): self.assertIn("successful", artifacts["stdout"].lower()) -class TestArtifactsPersistence(unittest.TestCase): +class TestArtifactsRoundTrip(unittest.TestCase): """Test that artifacts persist correctly across save/load cycles""" def setUp(self): @@ -283,8 +284,8 @@ def tearDown(self): if pending: self.loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True)) - def test_save_load_artifacts(self): - """Test that artifacts survive database save/load cycle""" + def test_small_and_large_artifacts_round_trip(self): + """Test that stored artifacts can be fetched without saving the database""" # Create program with artifacts program = Program(id="persist_test_1", code="print('test')", metrics={"score": 0.8}) @@ -298,15 +299,7 @@ def test_save_load_artifacts(self): self.database.add(program) self.database.store_artifacts(program.id, artifacts) - # Save database - self.database.save() - - # Create new database instance and load - new_database = ProgramDatabase(DatabaseConfig(db_path=self.temp_dir)) - new_database.load(self.temp_dir) - - # Check that artifacts are preserved - loaded_artifacts = new_database.get_artifacts(program.id) + loaded_artifacts = self.database.get_artifacts(program.id) self.assertEqual(loaded_artifacts["stderr"], artifacts["stderr"]) self.assertEqual(loaded_artifacts["stdout"], artifacts["stdout"]) diff --git a/tests/test_concurrent_island_access.py b/tests/test_concurrent_island_access.py index 3f42bbea72..91566fe2dd 100644 --- a/tests/test_concurrent_island_access.py +++ b/tests/test_concurrent_island_access.py @@ -9,7 +9,7 @@ from concurrent.futures import ThreadPoolExecutor from unittest.mock import MagicMock, patch -from openevolve.database import ProgramDatabase +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase from openevolve.config import Config from openevolve.database import Program diff --git a/tests/test_database.py b/tests/test_database.py index d9677dcb47..9565d91533 100644 --- a/tests/test_database.py +++ b/tests/test_database.py @@ -5,7 +5,8 @@ import unittest import uuid from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestProgramDatabase(unittest.TestCase): @@ -97,7 +98,7 @@ def test_island_operations_basic(self): # Should be in island 0 self.assertIn("island_test", self.db.islands[0]) - self.assertEqual(program.metadata.get("island"), 0) + self.assertEqual(self.db.get(program.id).metadata.get("island"), 0) def test_multi_island_setup(self): """Test database with multiple islands""" @@ -122,7 +123,7 @@ def test_multi_island_setup(self): # Verify assignment self.assertIn(f"test_island_{i}", multi_db.islands[i]) - self.assertEqual(program.metadata.get("island"), i) + self.assertEqual(multi_db.get(program.id).metadata.get("island"), i) def test_feature_coordinates_calculation(self): """Test MAP-Elites feature coordinate calculation""" diff --git a/tests/test_database_cleanup.py b/tests/test_database_cleanup.py index 0ec0125656..335bfb298b 100644 --- a/tests/test_database_cleanup.py +++ b/tests/test_database_cleanup.py @@ -7,7 +7,8 @@ import unittest from openevolve.config import DatabaseConfig -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestArtifactCleanup(unittest.TestCase): @@ -44,8 +45,9 @@ def test_artifact_cleanup(self): self.assertTrue(os.path.exists(dir_to_keep)) self.assertTrue(os.path.exists(dir_to_delete)) - # 4. Call the save method, which should trigger the cleanup - db.save() + # 4. Storing evaluation evidence triggers retention cleanup + db.add(Program(id="new", code="pass")) + db.store_artifacts("new", {"stdout": "new evidence"}) # 5. Assert that the old directory was deleted and the new one was kept self.assertTrue( diff --git a/tests/test_database_contract.py b/tests/test_database_contract.py new file mode 100644 index 0000000000..e9354a0f98 --- /dev/null +++ b/tests/test_database_contract.py @@ -0,0 +1,173 @@ +"""Backend contract tests; future database implementations can reuse this mixin.""" + +import tempfile +import unittest +from pathlib import Path + +from openevolve.config import DatabaseConfig +from openevolve.database import Program, ProgramDatabase +from openevolve.database_memory import InMemoryProgramDatabase + + +class ProgramDatabaseContract: + """Use only the public interface when checking backend behavior.""" + + def make_database(self, config: DatabaseConfig) -> ProgramDatabase: + raise NotImplementedError + + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.addCleanup(self.temp_dir.cleanup) + self.config = DatabaseConfig( + num_islands=2, + feature_dimensions=["slot"], + exploration_ratio=1.0, + exploitation_ratio=0.0, + artifacts_base_path=self.temp_dir.name, + ) + self.db = self.make_database(self.config) + + def seed(self): + for island, score in enumerate([0.4, 0.8]): + self.db.add( + Program( + id=f"p{island}", + code=f"return {island}", + metrics={"combined_score": score, "slot": 0.5, "other": 1 - score}, + ), + target_island=island, + ) + + def test_empty_database(self): + self.assertIsInstance(self.db, ProgramDatabase) + self.assertEqual(self.db.get_state().program_count, 0) + self.assertIsNone(self.db.get("missing")) + self.assertIsNone(self.db.get_best_program()) + self.assertEqual(self.db.get_top_programs(3), []) + self.assertEqual(self.db.get_artifacts("missing"), {}) + self.assertEqual(self.db.get_prompt_history("missing"), {}) + with self.assertRaises(ValueError): + self.db.sample_from_island(0) + + def test_writes_and_reads_are_detached(self): + program = Program( + id="value", + code="pass", + metrics={"combined_score": 1.0, "slot": 0.5}, + metadata={"nested": ["original"]}, + ) + self.db.add(program, iteration=7, target_island=1) + program.code = "caller changed this" + program.metrics["combined_score"] = -100 + program.metadata["nested"].append("changed") + stored = self.db.get("value") + self.assertEqual(stored.code, "pass") + self.assertEqual(stored.metadata, {"nested": ["original"], "island": 1}) + self.assertEqual(stored.iteration_found, 7) + stored.metrics["combined_score"] = -100 + self.assertEqual(self.db.get("value").metrics["combined_score"], 1.0) + + def test_bounded_ranked_queries_and_detached_results(self): + self.seed() + self.assertEqual([p.id for p in self.db.get_top_programs(1)], ["p1"]) + self.assertEqual(self.db.get_top_programs(0), []) + self.assertEqual([p.id for p in self.db.get_top_programs(4, island_idx=0)], ["p0"]) + with self.assertRaises(ValueError): + self.db.get_top_programs(-1) + with self.assertRaises(IndexError): + self.db.get_top_programs(1, island_idx=2) + best = self.db.get_best_program() + top = self.db.get_top_programs(1)[0] + best.metrics["combined_score"] = -100 + top.code = "changed" + self.assertEqual(self.db.get("p1").metrics["combined_score"], 0.8) + self.assertEqual(self.db.get("p1").code, "return 1") + + def test_custom_metric_query_does_not_change_default_best(self): + self.seed() + self.assertEqual(self.db.get_best_program(metric="other").id, "p0") + self.assertEqual(self.db.get_best_program().id, "p1") + self.assertEqual(self.db.get_top_programs(10, metric="missing"), []) + + def test_sampling_is_bounded_and_does_not_expose_stored_records(self): + self.seed() + for limit in [0, 1, 5]: + parent, inspirations = self.db.sample_from_island(1, num_inspirations=limit) + self.assertEqual(parent.id, "p1") + self.assertLessEqual(len(inspirations), limit) + parent.code = "changed" + for program in inspirations: + program.code = "changed" + self.assertEqual(self.db.get("p1").code, "return 1") + self.assertEqual(self.db.get_state().current_island, 0) + + def test_explicit_target_island_overrides_parent(self): + self.seed() + self.db.add( + Program( + id="child", + code="return 3", + parent_id="p0", + metrics={"combined_score": 0.9, "slot": 0.5}, + ), + iteration=2, + target_island=1, + ) + self.assertEqual(self.db.get("child").metadata["island"], 1) + self.assertEqual(self.db.get_top_programs(1, island_idx=1)[0].id, "child") + + def test_progress_is_monotonic_even_without_new_programs(self): + self.seed() + for iteration in [4, 2, 7, 7]: + self.db.record_iteration(iteration) + state = self.db.get_state() + self.assertEqual(state.last_iteration, 7) + self.assertEqual(state.program_count, 2) + self.assertEqual(state.num_islands, 2) + self.assertEqual(state.feature_dimensions, ("slot",)) + with self.assertRaises(ValueError): + self.db.record_iteration(-1) + + def test_artifacts_and_prompt_history_are_explicit_writes(self): + self.seed() + artifacts = {"stderr": "evidence", "large": "x" * (40 * 1024)} + self.db.store_artifacts("p0", artifacts) + self.assertEqual(self.db.get_artifacts("p0"), artifacts) + retrieved = self.db.get_artifacts("p0") + retrieved["stderr"] = "changed" + self.assertEqual(self.db.get_artifacts("p0")["stderr"], "evidence") + prompt = {"system": "system", "user": "user"} + responses = ["answer"] + self.db.log_prompt("p0", "rewrite", prompt, responses) + prompt["user"] = "changed" + responses.append("changed") + history = self.db.get_prompt_history("p0") + self.assertEqual( + history["rewrite"], {"system": "system", "user": "user", "responses": ["answer"]} + ) + history["rewrite"]["responses"].append("changed") + self.assertEqual(self.db.get_prompt_history("p0")["rewrite"]["responses"], ["answer"]) + + def test_generation_and_migration_queries(self): + self.seed() + self.assertFalse(self.db.should_migrate()) + self.db.increment_island_generation(1) + stats = self.db.get_island_stats() + self.assertEqual(stats[1]["generation"], 1) + stats[1]["generation"] = 100 + self.assertEqual(self.db.get_island_stats()[1]["generation"], 1) + + +class TestInMemoryDatabase(ProgramDatabaseContract, unittest.TestCase): + def make_database(self, config): + return InMemoryProgramDatabase(config) + + def test_no_population_files_or_restore(self): + config = DatabaseConfig(db_path=self.temp_dir.name, in_memory=False) + db = InMemoryProgramDatabase(config) + db.add(Program(id="p", code="pass")) + self.assertEqual(list(Path(self.temp_dir.name).iterdir()), []) + # A new instance is a new empty session, even with the same legacy path. + self.assertEqual(InMemoryProgramDatabase(config).get_state().program_count, 0) + self.assertFalse(hasattr(db, "save")) + self.assertFalse(hasattr(db, "load")) diff --git a/tests/test_feature_stats_persistence.py b/tests/test_feature_stats_persistence.py deleted file mode 100644 index 95a64602c6..0000000000 --- a/tests/test_feature_stats_persistence.py +++ /dev/null @@ -1,284 +0,0 @@ -""" -Unit tests for feature_stats persistence in ProgramDatabase checkpoints -""" - -import json -import os -import tempfile -import shutil -import unittest -from unittest.mock import patch - -from openevolve.database import ProgramDatabase, Program -from openevolve.config import DatabaseConfig - - -class TestFeatureStatsPersistence(unittest.TestCase): - """Test feature_stats are correctly saved and loaded in checkpoints""" - - def setUp(self): - """Set up test environment""" - self.test_dir = tempfile.mkdtemp() - self.config = DatabaseConfig( - db_path=self.test_dir, - feature_dimensions=["score", "custom_metric1", "custom_metric2"], - feature_bins=10, - ) - - def tearDown(self): - """Clean up test environment""" - shutil.rmtree(self.test_dir) - - def test_feature_stats_saved_and_loaded(self): - """Test that feature_stats are correctly saved and loaded from checkpoints""" - # Create database and add programs to build feature_stats - db1 = ProgramDatabase(self.config) - - programs = [] - for i in range(5): - program = Program( - id=f"test_prog_{i}", - code=f"# Test program {i}", - metrics={ - "combined_score": 0.1 + i * 0.2, - "custom_metric1": 10 + i * 20, - "custom_metric2": 100 + i * 50, - }, - ) - programs.append(program) - db1.add(program) - - # Verify feature_stats were built - self.assertIn("score", db1.feature_stats) - self.assertIn("custom_metric1", db1.feature_stats) - self.assertIn("custom_metric2", db1.feature_stats) - - # Store original feature_stats for comparison - original_stats = { - dim: {"min": stats["min"], "max": stats["max"], "values": stats["values"].copy()} - for dim, stats in db1.feature_stats.items() - } - - # Save checkpoint - db1.save(self.test_dir, iteration=42) - - # Load into new database - db2 = ProgramDatabase(self.config) - db2.load(self.test_dir) - - # Verify feature_stats were loaded correctly - self.assertEqual(len(db2.feature_stats), len(original_stats)) - - for dim, original in original_stats.items(): - self.assertIn(dim, db2.feature_stats) - loaded = db2.feature_stats[dim] - - self.assertAlmostEqual(loaded["min"], original["min"], places=5) - self.assertAlmostEqual(loaded["max"], original["max"], places=5) - self.assertEqual(loaded["values"], original["values"]) - - def test_empty_feature_stats_handling(self): - """Test handling of empty feature_stats""" - db1 = ProgramDatabase(self.config) - - # Save without any programs (empty feature_stats) - db1.save(self.test_dir, iteration=1) - - # Load and verify - db2 = ProgramDatabase(self.config) - db2.load(self.test_dir) - - self.assertEqual(db2.feature_stats, {}) - - def test_backward_compatibility_missing_feature_stats(self): - """Test loading checkpoints that don't have feature_stats (backward compatibility)""" - # Create a checkpoint manually without feature_stats - os.makedirs(self.test_dir, exist_ok=True) - - # Create metadata without feature_stats (simulating old checkpoint) - metadata = { - "island_feature_maps": [{}], # Updated to new format - "islands": [[]], - "archive": [], - "best_program_id": None, - "island_best_programs": [None], - "last_iteration": 10, - "current_island": 0, - "island_generations": [0], - "last_migration_generation": 0, - # Note: no "feature_stats" key - } - - with open(os.path.join(self.test_dir, "metadata.json"), "w") as f: - json.dump(metadata, f) - - # Load should work without errors - db = ProgramDatabase(self.config) - db.load(self.test_dir) - - # feature_stats should be empty but not None - self.assertEqual(db.feature_stats, {}) - - def test_feature_stats_serialization_edge_cases(self): - """Test feature_stats serialization handles edge cases correctly""" - db = ProgramDatabase(self.config) - - # Test with various edge cases - db.feature_stats = { - "normal_case": {"min": 1.0, "max": 10.0, "values": [1.0, 5.0, 10.0]}, - "single_value": {"min": 5.0, "max": 5.0, "values": [5.0]}, - "large_values_list": { - "min": 0.0, - "max": 200.0, - "values": list(range(200)), # Should be truncated to 100 - }, - "empty_values": {"min": 0.0, "max": 1.0, "values": []}, - } - - # Test serialization - serialized = db._serialize_feature_stats() - - # Check that large values list was truncated - self.assertLessEqual(len(serialized["large_values_list"]["values"]), 100) - - # Test deserialization - deserialized = db._deserialize_feature_stats(serialized) - - # Verify structure is maintained - self.assertIn("normal_case", deserialized) - self.assertIn("single_value", deserialized) - self.assertIn("large_values_list", deserialized) - self.assertIn("empty_values", deserialized) - - # Verify types are correct - for dim, stats in deserialized.items(): - self.assertIsInstance(stats["min"], float) - self.assertIsInstance(stats["max"], float) - self.assertIsInstance(stats["values"], list) - - def test_feature_stats_preservation_during_load(self): - """Test that feature_stats ranges are preserved when loading from checkpoint""" - # Create database with programs - db1 = ProgramDatabase(self.config) - - test_programs = [] - - for i in range(3): - program = Program( - id=f"stats_test_{i}", - code=f"# Stats test {i}", - metrics={ - "combined_score": 0.2 + i * 0.3, - "custom_metric1": 20 + i * 30, - "custom_metric2": 200 + i * 100, - }, - ) - test_programs.append(program) - db1.add(program) - - # Record original feature ranges - original_ranges = {} - for dim, stats in db1.feature_stats.items(): - original_ranges[dim] = {"min": stats["min"], "max": stats["max"]} - - # Save checkpoint - db1.save(self.test_dir, iteration=50) - - # Load into new database - db2 = ProgramDatabase(self.config) - db2.load(self.test_dir) - - # Verify feature ranges are preserved - for dim, original_range in original_ranges.items(): - self.assertIn(dim, db2.feature_stats) - loaded_stats = db2.feature_stats[dim] - - self.assertAlmostEqual( - loaded_stats["min"], - original_range["min"], - places=5, - msg=f"Min value changed for {dim}: {original_range['min']} -> {loaded_stats['min']}", - ) - self.assertAlmostEqual( - loaded_stats["max"], - original_range["max"], - places=5, - msg=f"Max value changed for {dim}: {original_range['max']} -> {loaded_stats['max']}", - ) - - # Test that adding a new program within existing ranges doesn't break anything - new_program = Program( - id="range_test", - code="# Program to test range stability", - metrics={ - "combined_score": 0.35, # Within existing range - "custom_metric1": 35, # Within existing range - "custom_metric2": 250, # Within existing range - }, - ) - - # Adding this program should not cause issues - db2.add(new_program) - new_coords = db2._calculate_feature_coords(new_program) - - # Should get valid coordinates - self.assertEqual(len(new_coords), len(self.config.feature_dimensions)) - for coord in new_coords: - self.assertIsInstance(coord, int) - self.assertGreaterEqual(coord, 0) - - def test_feature_stats_with_numpy_types(self): - """Test that numpy types are correctly handled in serialization""" - import numpy as np - - db = ProgramDatabase(self.config) - - # Simulate feature_stats with numpy types - db.feature_stats = { - "numpy_test": { - "min": np.float64(1.5), - "max": np.float64(9.5), - "values": [np.float64(x) for x in [1.5, 5.0, 9.5]], - } - } - - # Test serialization doesn't fail - serialized = db._serialize_feature_stats() - - # Verify numpy types were converted to Python types - self.assertIsInstance(serialized["numpy_test"]["min"], float) - self.assertIsInstance(serialized["numpy_test"]["max"], float) - - # Test deserialization - deserialized = db._deserialize_feature_stats(serialized) - self.assertIsInstance(deserialized["numpy_test"]["min"], float) - self.assertIsInstance(deserialized["numpy_test"]["max"], float) - - def test_malformed_feature_stats_handling(self): - """Test handling of malformed feature_stats during deserialization""" - db = ProgramDatabase(self.config) - - # Test with malformed data - malformed_data = { - "valid_entry": {"min": 1.0, "max": 10.0, "values": [1.0, 5.0, 10.0]}, - "invalid_entry": "this is not a dict", - "missing_keys": { - "min": 1.0 - # missing "max" and "values" - }, - } - - with patch("openevolve.database.logger") as mock_logger: - deserialized = db._deserialize_feature_stats(malformed_data) - - # Should have valid entry and skip invalid ones - self.assertIn("valid_entry", deserialized) - self.assertNotIn("invalid_entry", deserialized) - self.assertIn("missing_keys", deserialized) # Should be created with defaults - - # Should have logged warning for invalid entry - mock_logger.warning.assert_called() - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_grid_stability.py b/tests/test_grid_stability.py index 86a3875113..8d2e262266 100644 --- a/tests/test_grid_stability.py +++ b/tests/test_grid_stability.py @@ -1,5 +1,5 @@ """ -Integration tests for MAP-Elites grid stability across checkpoints +Integration tests for MAP-Elites grid stability during evolution """ import os @@ -7,12 +7,13 @@ import shutil import unittest -from openevolve.database import ProgramDatabase, Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase +from openevolve.database import Program from openevolve.config import DatabaseConfig class TestGridStability(unittest.TestCase): - """Integration tests for MAP-Elites grid stability when resuming from checkpoints""" + """Integration tests for MAP-Elites grid stability as programs are added""" def setUp(self): """Set up test environment""" @@ -22,8 +23,8 @@ def tearDown(self): """Clean up test environment""" shutil.rmtree(self.test_dir) - def test_feature_ranges_preserved_across_checkpoints(self): - """Test that feature ranges are preserved across checkpoint save/load cycles""" + def test_feature_ranges_do_not_contract(self): + """Test that feature ranges are preserved as more programs are added""" config = DatabaseConfig( db_path=self.test_dir, feature_dimensions=["score", "prompt_length", "reasoning_sophistication"], @@ -55,15 +56,7 @@ def test_feature_ranges_preserved_across_checkpoints(self): "value_count": len(stats["values"]), } - # Save checkpoint - db1.save(self.test_dir, iteration=25) - - # Phase 2: Resume from checkpoint - db2 = ProgramDatabase(config) - db2.load(self.test_dir) - - # Verify all programs were loaded - self.assertEqual(len(db2.programs), len(test_cases)) + db2 = db1 # Verify feature ranges are preserved for dim, original_range in original_ranges.items(): @@ -136,12 +129,8 @@ def test_grid_expansion_behavior(self): original_time_min = db1.feature_stats["execution_time"]["min"] original_time_max = db1.feature_stats["execution_time"]["max"] - # Save checkpoint - db1.save(self.test_dir, iteration=30) - - # Phase 2: Resume and add program outside range - db2 = ProgramDatabase(config) - db2.load(self.test_dir) + # Phase 2: Add a program outside the established range + db2 = db1 # Verify ranges were preserved self.assertAlmostEqual(db2.feature_stats["score"]["min"], original_score_min) @@ -167,76 +156,8 @@ def test_grid_expansion_behavior(self): self.assertLessEqual(db2.feature_stats["execution_time"]["min"], original_time_min) self.assertGreaterEqual(db2.feature_stats["execution_time"]["max"], 50) - def test_feature_stats_consistency_across_cycles(self): - """Test that feature_stats remain consistent across multiple save/load cycles""" - config = DatabaseConfig( - db_path=self.test_dir, feature_dimensions=["score", "memory_usage"], feature_bins=4 - ) - - # Initial program to establish baseline - reference_program = Program( - id="reference", - code="# Reference program for consistency testing", - metrics={"combined_score": 0.5, "memory_usage": 1024}, - ) - - # Cycle 1: Establish initial feature stats - db1 = ProgramDatabase(config) - db1.add(reference_program) - - # Record initial feature stats - cycle1_stats = {} - for dim, stats in db1.feature_stats.items(): - cycle1_stats[dim] = {"min": stats["min"], "max": stats["max"]} - - db1.save(self.test_dir, iteration=10) - - # Cycle 2: Load and verify stats preservation - db2 = ProgramDatabase(config) - db2.load(self.test_dir) - - # Verify feature stats were preserved - for dim, original_stats in cycle1_stats.items(): - self.assertIn(dim, db2.feature_stats) - self.assertAlmostEqual(db2.feature_stats[dim]["min"], original_stats["min"]) - self.assertAlmostEqual(db2.feature_stats[dim]["max"], original_stats["max"]) - - # Add another program and save again - db2.add( - Program( - id="cycle2_program", - code="# Cycle 2 program", - metrics={"combined_score": 0.3, "memory_usage": 512}, - ) - ) - - # Record expanded stats after adding new program - cycle2_stats = {} - for dim, stats in db2.feature_stats.items(): - cycle2_stats[dim] = {"min": stats["min"], "max": stats["max"]} - - db2.save(self.test_dir, iteration=20) - - # Cycle 3: Verify stats are still preserved - db3 = ProgramDatabase(config) - db3.load(self.test_dir) - - # Verify expanded feature stats were preserved - for dim, cycle2_stats_dim in cycle2_stats.items(): - self.assertIn(dim, db3.feature_stats) - self.assertAlmostEqual( - db3.feature_stats[dim]["min"], - cycle2_stats_dim["min"], - msg=f"Min value changed for {dim} in cycle 3", - ) - self.assertAlmostEqual( - db3.feature_stats[dim]["max"], - cycle2_stats_dim["max"], - msg=f"Max value changed for {dim} in cycle 3", - ) - def test_feature_stats_accumulation(self): - """Test that feature_stats accumulate correctly across checkpoint cycles""" + """Test that feature_stats accumulate correctly as programs are added""" config = DatabaseConfig( db_path=self.test_dir, feature_dimensions=["score", "complexity"], feature_bins=10 ) @@ -256,11 +177,8 @@ def test_feature_stats_accumulation(self): phase1_score_values = set(db1.feature_stats["score"]["values"]) phase1_complexity_values = set(db1.feature_stats["complexity"]["values"]) - db1.save(self.test_dir, iteration=15) - - # Cycle 2: Load and add more programs - db2 = ProgramDatabase(config) - db2.load(self.test_dir) + # Phase 2: Add more programs + db2 = db1 for i in range(2): program = Program( @@ -277,11 +195,11 @@ def test_feature_stats_accumulation(self): # Phase 1 values should be preserved (subset relationship) self.assertTrue( phase1_score_values.issubset(phase2_score_values), - "Phase 1 score values were lost after loading checkpoint", + "Phase 1 score values were lost while adding programs", ) self.assertTrue( phase1_complexity_values.issubset(phase2_complexity_values), - "Phase 1 complexity values were lost after loading checkpoint", + "Phase 1 complexity values were lost while adding programs", ) diff --git a/tests/test_initial_program_artifacts.py b/tests/test_initial_program_artifacts.py index 079ff9eea1..f6223efcf4 100644 --- a/tests/test_initial_program_artifacts.py +++ b/tests/test_initial_program_artifacts.py @@ -75,13 +75,11 @@ def test_initial_program_artifacts_are_stored(self): # Neutralize the parallel evolution loop - we only want the initial-program # handling in run() to execute. with ( - patch.object( - controller, "_run_evolution_with_checkpoints", new=AsyncMock(return_value=None) - ), patch("openevolve.controller.ProcessParallelController") as mock_ppc, ): # start()/stop()/request_shutdown() are called on the instance instance = mock_ppc.return_value + instance.run_evolution = AsyncMock(return_value=None) instance.start.return_value = None instance.stop.return_value = None @@ -111,11 +109,9 @@ def test_no_artifacts_when_evaluator_returns_none(self): ) with ( - patch.object( - controller, "_run_evolution_with_checkpoints", new=AsyncMock(return_value=None) - ), patch("openevolve.controller.ProcessParallelController") as mock_ppc, ): + mock_ppc.return_value.run_evolution = AsyncMock(return_value=None) mock_ppc.return_value.start.return_value = None mock_ppc.return_value.stop.return_value = None asyncio.run(controller.run(iterations=1)) diff --git a/tests/test_island_child_placement.py b/tests/test_island_child_placement.py index f911b4ec7f..c4fb58dea5 100644 --- a/tests/test_island_child_placement.py +++ b/tests/test_island_child_placement.py @@ -8,7 +8,8 @@ import unittest from openevolve.config import Config, DatabaseConfig -from openevolve.database import ProgramDatabase, Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase +from openevolve.database import Program class TestIslandChildPlacement(unittest.TestCase): @@ -43,7 +44,7 @@ def test_child_inherits_parent_island_when_no_target_specified(self): self.db.add(child) # No target_island specified # Child should inherit parent's island (island 0) - self.assertEqual(child.metadata.get("island"), 0) + self.assertEqual(self.db.get(child.id).metadata.get("island"), 0) self.assertIn("child_0", self.db.islands[0]) def test_child_placed_in_target_island_when_specified(self): @@ -68,7 +69,7 @@ def test_child_placed_in_target_island_when_specified(self): self.db.add(child, target_island=2) # Child should be in island 2, NOT island 0 - self.assertEqual(child.metadata.get("island"), 2) + self.assertEqual(self.db.get(child.id).metadata.get("island"), 2) self.assertIn("child_1", self.db.islands[2]) self.assertNotIn("child_1", self.db.islands[0]) @@ -143,7 +144,7 @@ def test_child_should_go_to_target_island_not_parent_island(self): # Child should be in island 1 (target), not island 0 (parent's island) self.assertEqual( - child.metadata.get("island"), 1, + self.db.get(child.id).metadata.get("island"), 1, "Child should be in target island 1, not parent's island 0." ) self.assertIn("child_for_island_1", self.db.islands[1]) @@ -169,7 +170,7 @@ def test_explicit_target_island_overrides_parent_inheritance(self): self.db.add(child, target_island=2) # This should work - explicit target_island is respected - self.assertEqual(child.metadata.get("island"), 2) + self.assertEqual(self.db.get(child.id).metadata.get("island"), 2) self.assertIn("child_for_island_2", self.db.islands[2]) @@ -286,7 +287,7 @@ def test_without_target_island_child_inherits_parent(self): # Without target_island, child inherits parent's island (0), not target (2) # This is the BUG - child should be in island 2 but ends up in island 0 self.assertEqual( - child.metadata.get("island"), 0, + self.db.get(child.id).metadata.get("island"), 0, "Without target_island, child incorrectly inherits parent's island" ) @@ -320,7 +321,7 @@ def test_with_target_island_child_goes_to_target(self): # With target_island, child goes to island 2 (correct) self.assertEqual( - child.metadata.get("island"), 2, + self.db.get(child.id).metadata.get("island"), 2, "With target_island, child should go to target island" ) diff --git a/tests/test_island_isolation.py b/tests/test_island_isolation.py index 5de584878c..7286eda8fa 100644 --- a/tests/test_island_isolation.py +++ b/tests/test_island_isolation.py @@ -7,7 +7,8 @@ import asyncio from openevolve.config import Config, DatabaseConfig, EvaluatorConfig -from openevolve.database import ProgramDatabase, Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase +from openevolve.database import Program from openevolve.process_parallel import ProcessParallelController @@ -56,10 +57,10 @@ def test_submit_iteration_uses_correct_island(self): # Get the snapshot that was passed to worker call_args = mock_executor.submit.call_args[0] - db_snapshot = call_args[2] # Third argument is db_snapshot + context = call_args[2] # Third argument is the selected iteration context # Verify snapshot has island marking - self.assertEqual(db_snapshot["sampling_island"], 1) + self.assertEqual(context.target_island, 1) def test_island_isolation_during_evolution(self): """Test that parallel workers maintain island isolation""" diff --git a/tests/test_island_map_elites.py b/tests/test_island_map_elites.py index 750cfdeb1a..2693a2c1a2 100644 --- a/tests/test_island_map_elites.py +++ b/tests/test_island_map_elites.py @@ -9,7 +9,8 @@ import unittest import uuid from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestIslandMapElites(unittest.TestCase): @@ -170,41 +171,6 @@ def test_no_migrant_suffix_generation(self): self.assertEqual(len(migrant_programs), 0, f"Found programs with _migrant suffix: {migrant_programs}") - def test_checkpoint_serialization_preserves_island_maps(self): - """Test that saving/loading preserves island feature maps correctly""" - import tempfile - import shutil - - # Add programs to different islands - prog1 = self._create_test_program("prog1", 0.8, [0.1, 0.2], island=0) - prog2 = self._create_test_program("prog2", 0.7, [0.3, 0.4], island=1) - - self.db.add(prog1, target_island=0) - self.db.add(prog2, target_island=1) - - # Get the current state - original_maps = [dict(island_map) for island_map in self.db.island_feature_maps] - - # Save to temporary directory - temp_dir = tempfile.mkdtemp() - try: - self.db.save(temp_dir) - - # Create new database and load from checkpoint - config = Config() - config.database.in_memory = True - config.database.num_islands = 3 - new_db = ProgramDatabase(config.database) - new_db.load(temp_dir) - - # Verify island feature maps are preserved - self.assertEqual(len(new_db.island_feature_maps), 3) - for i, (original_map, loaded_map) in enumerate(zip(original_maps, new_db.island_feature_maps)): - self.assertEqual(original_map, loaded_map, - f"Island {i} feature map not preserved correctly") - - finally: - shutil.rmtree(temp_dir) if __name__ == '__main__': diff --git a/tests/test_island_migration.py b/tests/test_island_migration.py index 5765bd33d4..6c235cb0ae 100644 --- a/tests/test_island_migration.py +++ b/tests/test_island_migration.py @@ -4,7 +4,8 @@ import unittest from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestIslandMigration(unittest.TestCase): diff --git a/tests/test_island_parent_consistency.py b/tests/test_island_parent_consistency.py index 6764fca18c..1a4825b1d0 100644 --- a/tests/test_island_parent_consistency.py +++ b/tests/test_island_parent_consistency.py @@ -4,7 +4,8 @@ import unittest from openevolve.config import Config -from openevolve.database import ProgramDatabase, Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase +from openevolve.database import Program class TestIslandParentConsistency(unittest.TestCase): @@ -24,7 +25,7 @@ def test_parent_child_island_consistency(self): # Verify initial program is on island 0 self.assertIn("initial", database.islands[0]) - self.assertEqual(initial_program.metadata.get("island"), 0) + self.assertEqual(database.get(initial_program.id).metadata.get("island"), 0) # Now switch to island 1 database.next_island() @@ -94,7 +95,7 @@ def test_multiple_generations_island_drift(self): ) database.add(prog) - programs.append(prog) + programs.append(database.get(prog.id)) # Switch islands periodically (simulating what happens in evolution) if i % 3 == 0: @@ -155,7 +156,7 @@ def test_explicit_migration_override(self): # Verify migrant went to island 2, not parent's island 0 self.assertIn("migrant", database.islands[2]) self.assertNotIn("migrant", database.islands[0]) - self.assertEqual(migrant_child.metadata.get("island"), 2) + self.assertEqual(database.get(migrant_child.id).metadata.get("island"), 2) # Parent should still be on island 0 self.assertEqual(database.programs["parent"].metadata.get("island"), 0) diff --git a/tests/test_island_tracking.py b/tests/test_island_tracking.py index 160f9161c5..c2921729d7 100644 --- a/tests/test_island_tracking.py +++ b/tests/test_island_tracking.py @@ -4,7 +4,8 @@ import unittest from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestIslandTracking(unittest.TestCase): diff --git a/tests/test_iteration_context.py b/tests/test_iteration_context.py new file mode 100644 index 0000000000..42544d07af --- /dev/null +++ b/tests/test_iteration_context.py @@ -0,0 +1,99 @@ +"""Workers receive bounded query results, with evidence for the actual parent.""" + +import pickle +import unittest +from unittest.mock import AsyncMock, Mock, patch + +from openevolve import process_parallel as workers +from openevolve.config import Config +from openevolve.database import Program, ProgramDatabase +from openevolve.database_memory import InMemoryProgramDatabase +from openevolve.process_parallel import IterationContext, ProcessParallelController + + +class TestIterationContext(unittest.TestCase): + def test_large_population_does_not_expand_worker_context(self): + config = Config() + config.database.num_islands = 110 + config.prompt.num_top_programs = 1 + config.prompt.num_diverse_programs = 0 + store = InMemoryProgramDatabase(config.database) + for i in range(110): + store.add( + Program(id=f"p{i}", code=f"return {i}", metrics={"combined_score": i}), + target_island=i, + ) + store.store_artifacts(f"p{i}", {"stdout": f"evidence {i}"}) + database = Mock(spec_set=ProgramDatabase, wraps=store) + controller = ProcessParallelController(config, "unused.py", database) + # This parent fell outside the previous first-100-program artifact snapshot. + context = controller._select_iteration_context(109) + self.assertEqual(context.parent.id, "p109") + self.assertEqual(context.parent_artifacts, {"stdout": "evidence 109"}) + self.assertEqual([p.id for p in context.top_programs], ["p109"]) + self.assertEqual(context.inspirations, []) + database.get_artifacts.assert_called_once_with("p109") + database.get_top_programs.assert_called_once_with(n=1, island_idx=109) + wire_data = pickle.dumps(context) + self.assertNotIn(b"evidence 108", wire_data) + self.assertNotIn(b"return 108", wire_data) + self.assertEqual(pickle.loads(wire_data), context) + + def test_each_candidate_queries_the_current_database(self): + config = Config() + config.database.num_islands = 1 + config.database.feature_dimensions = ["score"] + config.prompt.num_top_programs = 1 + store = InMemoryProgramDatabase(config.database) + database = Mock(spec_set=ProgramDatabase, wraps=store) + database.add(Program(id="first", code="return 1", metrics={"combined_score": 0.5})) + controller = ProcessParallelController(config, "unused.py", database) + first_context = controller._select_iteration_context(0) + database.add(Program(id="better", code="return 2", metrics={"combined_score": 0.9})) + second_context = controller._select_iteration_context(0) + self.assertEqual(first_context.top_programs[0].id, "first") + self.assertEqual(second_context.top_programs[0].id, "better") + self.assertEqual(database.sample_from_island.call_count, 2) + self.assertEqual(database.get_top_programs.call_count, 2) + + def test_worker_uses_selected_context_and_explicit_target_island(self): + config = Config() + config.language = "python" + config.diff_based_evolution = False + config.prompt.num_top_programs = 1 + parent = Program( + id="parent", code="return 1", metrics={"combined_score": 0.5}, metadata={"island": 0} + ) + context = IterationContext( + parent=parent, + inspirations=[], + top_programs=[parent], + parent_artifacts={"stderr": "parent evidence"}, + target_island=1, + feature_dimensions=("complexity",), + ) + sampler = Mock() + sampler.build_prompt.return_value = {"system": "system", "user": "user"} + llm = Mock() + llm.generate_with_context = AsyncMock(return_value="```python\ndef solve(): return 2\n```") + evaluator = Mock() + evaluator.evaluate_program = AsyncMock(return_value={"combined_score": 0.9}) + evaluator.get_pending_artifacts.return_value = {"stdout": "child evidence"} + with ( + patch.object(workers, "_lazy_init_worker_components"), + patch.object(workers, "_worker_config", config, create=True), + patch.object(workers, "_worker_prompt_sampler", sampler, create=True), + patch.object(workers, "_worker_llm_ensemble", llm, create=True), + patch.object(workers, "_worker_evaluator", evaluator, create=True), + ): + result = workers._run_iteration_worker(7, pickle.loads(pickle.dumps(context))) + self.assertIsNone(result.error) + self.assertEqual(result.target_island, 1) + self.assertEqual(result.child_program_dict["metadata"]["island"], 1) + self.assertEqual(result.child_program_dict["iteration_found"], 7) + self.assertEqual(result.parent_id, "parent") + self.assertEqual(result.artifacts, {"stdout": "child evidence"}) + prompt_args = sampler.build_prompt.call_args.kwargs + self.assertEqual(prompt_args["program_artifacts"], {"stderr": "parent evidence"}) + self.assertEqual(prompt_args["previous_programs"], [parent.to_dict()]) + self.assertEqual(prompt_args["feature_dimensions"], ["complexity"]) diff --git a/tests/test_iteration_counting.py b/tests/test_iteration_counting.py index f7bc1aba53..78598d9838 100644 --- a/tests/test_iteration_counting.py +++ b/tests/test_iteration_counting.py @@ -1,150 +1,178 @@ -""" -Tests for iteration counting and checkpoint behavior -""" +"""Exercise progress and continuation against the database interface.""" -import os +import asyncio import tempfile import unittest +from concurrent.futures import Future +from pathlib import Path +from unittest.mock import AsyncMock, Mock, patch + +from openevolve.config import Config, LLMModelConfig +from openevolve.api import run_evolution +from openevolve.controller import OpenEvolve +from openevolve.database import Program, ProgramDatabase +from openevolve.database_memory import InMemoryProgramDatabase +from openevolve.process_parallel import ProcessParallelController, SerializableResult + + +class ImmediateExecutor: + """Complete worker results locally without making model requests.""" + + def __init__(self, succeed=False): + self.iterations = [] + self.contexts = [] + self.succeed = succeed + + def submit(self, fn, iteration, context): + self.iterations.append(iteration) + self.contexts.append(context) + future = Future() + if self.succeed: + child = Program( + id=f"child-{iteration}", + code=f"return {iteration}", + parent_id=context.parent.id, + metrics={"combined_score": 1.0}, + ) + future.set_result( + SerializableResult( + iteration=iteration, + child_program_dict=child.to_dict(), + parent_id=context.parent.id, + target_island=context.target_island, + artifacts={"stderr": "child evidence"}, + prompt={"system": "system", "user": "user"}, + llm_response="response", + ) + ) + else: + future.set_result(SerializableResult(iteration=iteration, error="No valid code")) + return future -# Set dummy API key for testing -os.environ["OPENAI_API_KEY"] = "test" - -from openevolve.config import Config + def shutdown(self, **kwargs): + pass class TestIterationCounting(unittest.TestCase): - """Tests for correct iteration counting behavior""" - def setUp(self): - """Set up test environment""" - self.test_dir = tempfile.mkdtemp() - - # Create test program - self.program_content = """# EVOLVE-BLOCK-START -def compute(x): - return x * 2 -# EVOLVE-BLOCK-END -""" - self.program_file = os.path.join(self.test_dir, "test_program.py") - with open(self.program_file, "w") as f: - f.write(self.program_content) - - # Create test evaluator - self.eval_content = """ -def evaluate(program_path): - return {"score": 0.5, "performance": 0.6} -""" - self.eval_file = os.path.join(self.test_dir, "evaluator.py") - with open(self.eval_file, "w") as f: - f.write(self.eval_content) - - def tearDown(self): - """Clean up test environment""" - import shutil - - shutil.rmtree(self.test_dir, ignore_errors=True) - - def test_fresh_start_iteration_counting(self): - """Test that fresh start correctly handles iteration 0 as special""" - # Test the logic without actually running evolution - config = Config() - config.max_iterations = 20 - config.checkpoint_interval = 10 - - # Simulate fresh start - start_iteration = 0 - should_add_initial = True - - # Apply the logic from controller.py - evolution_start = start_iteration - evolution_iterations = config.max_iterations - - if should_add_initial and start_iteration == 0: - evolution_start = 1 - - # Verify - self.assertEqual(evolution_start, 1, "Evolution should start at iteration 1") - self.assertEqual(evolution_iterations, 20, "Should run 20 evolution iterations") - - # Simulate what process_parallel would do - total_iterations = evolution_start + evolution_iterations - self.assertEqual(total_iterations, 21, "Total range should be 21 (1 through 20)") - - # Check checkpoint alignment - expected_checkpoints = [] - for i in range(evolution_start, total_iterations): - if i > 0 and i % config.checkpoint_interval == 0: - expected_checkpoints.append(i) - - self.assertEqual(expected_checkpoints, [10, 20], "Checkpoints should be at 10 and 20") - - def test_resume_iteration_counting(self): - """Test that resume correctly continues from checkpoint""" - config = Config() - config.max_iterations = 10 - config.checkpoint_interval = 10 - - # Simulate resume from checkpoint 10 - start_iteration = 11 # Last iteration was 10, so start at 11 - should_add_initial = False - - # Apply the logic - evolution_start = start_iteration - evolution_iterations = config.max_iterations - - if should_add_initial and start_iteration == 0: - evolution_start = 1 - - # Verify - self.assertEqual(evolution_start, 11, "Evolution should continue from iteration 11") - self.assertEqual(evolution_iterations, 10, "Should run 10 more iterations") - - # Total iterations - total_iterations = evolution_start + evolution_iterations - self.assertEqual(total_iterations, 21, "Should run through iteration 20") - - # Check checkpoint at 20 - expected_checkpoints = [] - for i in range(evolution_start, total_iterations): - if i > 0 and i % config.checkpoint_interval == 0: - expected_checkpoints.append(i) - - self.assertEqual(expected_checkpoints, [20], "Should checkpoint at 20") - - def test_checkpoint_boundary_conditions(self): - """Test checkpoint behavior at various boundaries""" - test_cases = [ - # (start_iter, max_iter, checkpoint_interval, expected_checkpoints) - (1, 100, 10, list(range(10, 101, 10))), # Standard case - (1, 99, 10, list(range(10, 100, 10))), # Just short of last checkpoint - (1, 101, 10, list(range(10, 101, 10))), # Just past checkpoint - (0, 20, 5, [5, 10, 15, 20]), # Special case with iteration 0 - ] - - for start, max_iter, interval, expected in test_cases: - # Apply fresh start logic - evolution_start = start - if start == 0: - evolution_start = 1 - - total = evolution_start + max_iter - - checkpoints = [] - for i in range(evolution_start, total): - if i > 0 and i % interval == 0: - checkpoints.append(i) - - self.assertEqual( - checkpoints, - expected, - f"Failed for start={start}, max={max_iter}, interval={interval}", + self.temp_dir = tempfile.TemporaryDirectory() + self.addCleanup(self.temp_dir.cleanup) + self.program_path = Path(self.temp_dir.name) / "program.py" + self.program_path.write_text("def solve(): return 1\n") + self.eval_path = Path(self.temp_dir.name) / "evaluator.py" + self.eval_path.write_text("def evaluate(path): return {'combined_score': 0.5}\n") + self.config = Config() + self.config.database.num_islands = 2 + self.config.evaluator.parallel_evaluations = 1 + self.config.evaluator.cascade_evaluation = False + self.store = InMemoryProgramDatabase(self.config.database) + # This facade raises AttributeError for backing maps, config, and progress fields. + self.database = Mock(spec_set=ProgramDatabase, wraps=self.store) + + def controller(self): + with patch("openevolve.controller.LLMEnsemble"), patch.object(OpenEvolve, "_setup_logging"): + controller = OpenEvolve( + str(self.program_path), + str(self.eval_path), + self.config, + output_dir=self.temp_dir.name, + database=self.database, ) - - # NOTE: The real-LLM test that exercised controller.run() against a live optillm - # server was moved to tests/integration/test_iteration_counting_with_llm.py so it - # runs against the shared model fixtures instead of being skipped when no server - # is reachable. This module now contains only server-free logic tests. - - -if __name__ == "__main__": - unittest.main() + controller.evaluator.evaluate_program = AsyncMock(return_value={"combined_score": 0.5}) + controller.evaluator.get_pending_artifacts = Mock(return_value={"stdout": "initial"}) + return controller + + def test_fresh_run_and_continuation_count_failed_iterations(self): + controller = self.controller() + executor = ImmediateExecutor() + + def start(parallel): + parallel.executor = executor + + with patch.object(ProcessParallelController, "start", start), patch("signal.signal"): + asyncio.run(controller.run(iterations=3)) + self.assertEqual(executor.iterations, [1, 2, 3]) + self.assertEqual(self.database.get_state().last_iteration, 3) + # A second controller can continue against the same database instance. + next_controller = self.controller() + asyncio.run(next_controller.run(iterations=2)) + + self.assertEqual(executor.iterations, [1, 2, 3, 4, 5]) + self.assertEqual(self.database.get_state().last_iteration, 5) + self.assertEqual(self.database.get_state().program_count, 1) + controller.evaluator.evaluate_program.assert_awaited_once() + next_controller.evaluator.evaluate_program.assert_not_awaited() + self.assertEqual(self.database.sample_from_island.call_count, 5) + self.assertEqual(self.database.get_top_programs.call_count, 5) + self.assertFalse((Path(self.temp_dir.name) / "checkpoints").exists()) + self.assertTrue((Path(self.temp_dir.name) / "best" / "best_program.py").exists()) + + def test_zero_iterations_only_evaluates_initial_program(self): + controller = self.controller() + executor = ImmediateExecutor() + with ( + patch.object( + ProcessParallelController, "start", lambda p: setattr(p, "executor", executor) + ), + patch("signal.signal"), + ): + asyncio.run(controller.run(iterations=0)) + self.assertEqual(executor.iterations, []) + self.assertEqual(self.database.get_state().last_iteration, 0) + self.assertEqual(self.database.get_state().program_count, 1) + controller.evaluator.evaluate_program.assert_awaited_once() + + def test_library_api_accepts_an_existing_database(self): + self.config.llm.models = [LLMModelConfig(name="test")] + self.database.add(Program(id="existing", code="return 1", metrics={"combined_score": 0.5})) + self.database.record_iteration(10) + executor = ImmediateExecutor() + with ( + patch.object( + ProcessParallelController, "start", lambda p: setattr(p, "executor", executor) + ), + patch.object(OpenEvolve, "_setup_logging"), + patch("openevolve.controller.LLMEnsemble"), + patch("signal.signal"), + ): + result = run_evolution( + self.program_path, + self.eval_path, + config=self.config, + iterations=1, + output_dir=self.temp_dir.name, + cleanup=False, + database=self.database, + ) + self.assertEqual(executor.iterations, [11]) + self.assertEqual(self.database.get_state().last_iteration, 11) + self.assertEqual(result.best_program.id, "existing") + + def test_successful_result_uses_only_interface_and_records_target_stop(self): + self.database.add( + Program(id="initial", code="return 0", metrics={"combined_score": 0.5}), target_island=0 + ) + tracer = Mock() + parallel = ProcessParallelController( + self.config, str(self.eval_path), self.database, tracer + ) + parallel.executor = ImmediateExecutor(succeed=True) + result = asyncio.run(parallel.run_evolution(1, 1, target_score=1.0)) + self.assertEqual(result.id, "child-1") + self.assertEqual(self.database.get_state().last_iteration, 1) + self.assertEqual(self.database.get_artifacts(result.id), {"stderr": "child evidence"}) + self.assertEqual( + self.database.get_prompt_history(result.id)["diff_user"]["responses"], ["response"] + ) + self.assertEqual(self.database.get_island_stats()[0]["generation"], 1) + tracer.log_trace.assert_called_once() + + def test_worker_exception_also_advances_progress(self): + self.database.add(Program(id="initial", code="pass")) + parallel = ProcessParallelController(self.config, str(self.eval_path), self.database) + parallel.executor = Mock() + future = Future() + future.set_exception(RuntimeError("worker exited")) + parallel.executor.submit.return_value = future + asyncio.run(parallel.run_evolution(1, 1)) + self.assertEqual(self.database.get_state().last_iteration, 1) diff --git a/tests/test_map_elites_features.py b/tests/test_map_elites_features.py index 223cff9f30..49bdeb0cdc 100644 --- a/tests/test_map_elites_features.py +++ b/tests/test_map_elites_features.py @@ -5,7 +5,8 @@ import unittest from unittest.mock import MagicMock, patch from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestMapElitesFeatures(unittest.TestCase): diff --git a/tests/test_migration_no_duplicates.py b/tests/test_migration_no_duplicates.py index 4337b2dd9d..e472bb3fcf 100644 --- a/tests/test_migration_no_duplicates.py +++ b/tests/test_migration_no_duplicates.py @@ -10,7 +10,8 @@ import uuid import re from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestMigrationNoDuplicates(unittest.TestCase): diff --git a/tests/test_novelty_asyncio_issue.py b/tests/test_novelty_asyncio_issue.py index 46fb03475f..b0e6c6f16e 100644 --- a/tests/test_novelty_asyncio_issue.py +++ b/tests/test_novelty_asyncio_issue.py @@ -11,7 +11,8 @@ import asyncio from unittest.mock import AsyncMock, MagicMock, patch, Mock from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class MockLLM: diff --git a/tests/test_orphan_program_removal.py b/tests/test_orphan_program_removal.py index f7831344a6..520ee6da80 100644 --- a/tests/test_orphan_program_removal.py +++ b/tests/test_orphan_program_removal.py @@ -12,7 +12,8 @@ import unittest from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase def _make_program(pid, fitness, island=0): diff --git a/tests/test_population_elite_protection.py b/tests/test_population_elite_protection.py index 6992011857..4edc2d48d5 100644 --- a/tests/test_population_elite_protection.py +++ b/tests/test_population_elite_protection.py @@ -13,7 +13,8 @@ import unittest from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestPopulationEliteProtection(unittest.TestCase): diff --git a/tests/test_process_parallel.py b/tests/test_process_parallel.py index 23c6f92197..6b1bb65c25 100644 --- a/tests/test_process_parallel.py +++ b/tests/test_process_parallel.py @@ -22,7 +22,8 @@ def _slow_test_worker(marker_path: str) -> str: os.environ["OPENAI_API_KEY"] = "test" from openevolve.config import Config, DatabaseConfig, EvaluatorConfig, LLMConfig, PromptConfig -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase from openevolve import process_parallel as process_parallel_module from openevolve.process_parallel import ProcessParallelController, SerializableResult @@ -43,7 +44,6 @@ def setUp(self): # displaced (MAP-Elites removes programs displaced from their cell). self.config.database.num_islands = 3 self.config.database.in_memory = True - self.config.checkpoint_interval = 5 # Create test evaluation file self.eval_content = """ @@ -144,24 +144,18 @@ def test_process_pool_shutdown_escalates_to_kill(self): process.terminate.assert_called_once_with() process.kill.assert_called_once_with() - def test_database_snapshot_creation(self): - """Test creating database snapshot for workers""" + def test_select_iteration_context(self): + """Workers receive selected programs and the parent's artifacts.""" controller = ProcessParallelController(self.config, self.eval_file, self.database) + self.database.store_artifacts("test_2", {"stderr": "parent evidence"}) - snapshot = controller._create_database_snapshot() + context = controller._select_iteration_context(2) - # Verify snapshot structure - self.assertIn("programs", snapshot) - self.assertIn("islands", snapshot) - self.assertIn("current_island", snapshot) - self.assertIn("artifacts", snapshot) - - # Verify programs are serialized - self.assertEqual(len(snapshot["programs"]), 3) - for pid, prog_dict in snapshot["programs"].items(): - self.assertIsInstance(prog_dict, dict) - self.assertIn("id", prog_dict) - self.assertIn("code", prog_dict) + self.assertEqual(context.target_island, 2) + self.assertEqual(context.parent.id, "test_2") + self.assertEqual(context.parent_artifacts, {"stderr": "parent evidence"}) + self.assertEqual([p.id for p in context.top_programs], ["test_2"]) + self.assertLessEqual(len(context.inspirations), self.config.prompt.num_diverse_programs) def test_run_evolution_basic(self): """Test basic evolution run""" diff --git a/tests/test_process_parallel_fix.py b/tests/test_process_parallel_fix.py index e106895afc..f39de8d4ae 100644 --- a/tests/test_process_parallel_fix.py +++ b/tests/test_process_parallel_fix.py @@ -9,7 +9,8 @@ from openevolve.process_parallel import ProcessParallelController from openevolve.config import Config -from openevolve.database import ProgramDatabase, Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase +from openevolve.database import Program class TestProcessParallelFix(unittest.TestCase): diff --git a/tests/test_sample_from_island_inspirations.py b/tests/test_sample_from_island_inspirations.py index 884853e333..252c7b3967 100644 --- a/tests/test_sample_from_island_inspirations.py +++ b/tests/test_sample_from_island_inspirations.py @@ -4,7 +4,8 @@ from unittest.mock import patch from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestSampleFromIslandInspirations(unittest.TestCase): @@ -54,7 +55,8 @@ def test_sample_from_island_reuses_strategy_aware_inspiration_sampler(self): num_inspirations=2, ) - self.assertIs(parent, self.parent) + self.assertEqual(parent, self.parent) + self.assertIsNot(parent, self.parent) self.assertEqual(inspirations, [self.elite]) sample_inspirations.assert_called_once_with( self.parent, diff --git a/tests/test_sample_from_island_ratios.py b/tests/test_sample_from_island_ratios.py index 54eacb53c4..7063b81004 100644 --- a/tests/test_sample_from_island_ratios.py +++ b/tests/test_sample_from_island_ratios.py @@ -8,7 +8,8 @@ import random import unittest from openevolve.config import Config -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class TestSampleFromIslandRatios(unittest.TestCase): diff --git a/tests/test_checkpoint_resume.py b/tests/test_session_continuation.py similarity index 96% rename from tests/test_checkpoint_resume.py rename to tests/test_session_continuation.py index 791d5d5d84..f2e554cd4e 100644 --- a/tests/test_checkpoint_resume.py +++ b/tests/test_session_continuation.py @@ -1,5 +1,5 @@ """ -Tests for checkpoint resume functionality and initial program deduplication +Tests for continuation within a database session and initial program deduplication """ import asyncio @@ -15,7 +15,8 @@ from openevolve.config import Config from openevolve.controller import OpenEvolve -from openevolve.database import Program, ProgramDatabase +from openevolve.database import Program +from openevolve.database_memory import InMemoryProgramDatabase as ProgramDatabase class MockEvaluator: @@ -42,8 +43,8 @@ def get_pending_artifacts(self, program_id): return None -class TestCheckpointResume(unittest.TestCase): - """Tests for checkpoint resume functionality""" +class TestSessionContinuation(unittest.TestCase): + """Tests for continuation within a database session""" def setUp(self): """Set up test environment""" @@ -76,7 +77,6 @@ def evaluate(program_path): # Create test config self.config = Config() self.config.max_iterations = 2 # Keep tests fast - self.config.checkpoint_interval = 1 self.config.database.in_memory = True def tearDown(self): @@ -184,8 +184,8 @@ async def run_test(): # Verify evaluator was not called for initial program self.assertEqual(mock_evaluator.call_count, 0) - def test_checkpoint_resume_skips_initial_program(self): - """Test that initial program is not re-added when resuming from checkpoint""" + def test_existing_session_skips_initial_program(self): + """Test that initial program is not re-added when continuing a session""" async def run_test(): with patch("openevolve.controller.Evaluator") as mock_evaluator_class: @@ -199,7 +199,7 @@ async def run_test(): output_dir=self.test_dir, ) - # Simulate existing database state (as if loaded from checkpoint) + # Simulate existing database state (as if already processed) existing_program = Program( id="existing_program_id", code=self.test_program_content, # Same content as initial program diff --git a/tests/test_snapshot_artifacts_limit.py b/tests/test_snapshot_artifacts_limit.py deleted file mode 100644 index 776f439f27..0000000000 --- a/tests/test_snapshot_artifacts_limit.py +++ /dev/null @@ -1,145 +0,0 @@ -""" -Tests for configurable max_snapshot_artifacts limit. -Controls how many artifacts are included in worker process snapshots. -""" - -import unittest - -from openevolve.config import Config, DatabaseConfig -from openevolve.database import ProgramDatabase, Program - - -class TestMaxSnapshotArtifactsConfig(unittest.TestCase): - """Tests for max_snapshot_artifacts configuration""" - - def test_default_value_is_100(self): - """Test that max_snapshot_artifacts defaults to 100""" - config = Config() - self.assertEqual(config.database.max_snapshot_artifacts, 100) - - def test_database_config_default(self): - """Test DatabaseConfig default for max_snapshot_artifacts""" - db_config = DatabaseConfig() - self.assertEqual(db_config.max_snapshot_artifacts, 100) - - def test_custom_value_from_dict(self): - """Test loading custom max_snapshot_artifacts from config dict""" - config_dict = { - "llm": {"primary_model": "gpt-4"}, - "database": { - "max_snapshot_artifacts": 500, - }, - } - config = Config.from_dict(config_dict) - self.assertEqual(config.database.max_snapshot_artifacts, 500) - - def test_unlimited_artifacts_with_none(self): - """Test setting unlimited artifacts with None""" - config_dict = { - "llm": {"primary_model": "gpt-4"}, - "database": { - "max_snapshot_artifacts": None, - }, - } - config = Config.from_dict(config_dict) - self.assertIsNone(config.database.max_snapshot_artifacts) - - def test_zero_artifacts(self): - """Test setting max_snapshot_artifacts to 0""" - config_dict = { - "llm": {"primary_model": "gpt-4"}, - "database": { - "max_snapshot_artifacts": 0, - }, - } - config = Config.from_dict(config_dict) - self.assertEqual(config.database.max_snapshot_artifacts, 0) - - -class TestArtifactStorageWithLimit(unittest.TestCase): - """Tests for artifact storage respecting the limit""" - - def test_store_artifacts_within_limit(self): - """Test storing artifacts when within the limit""" - # One island per program so each program owns its cell and none is - # displaced (MAP-Elites removes programs displaced from their cell). - db_config = DatabaseConfig(max_snapshot_artifacts=5, num_islands=3) - db = ProgramDatabase(db_config) - - # Add programs with artifacts - for i in range(3): - program = Program( - id=f"prog_{i}", - code=f"def func_{i}(): pass", - generation=0, - metrics={"score": i * 0.1}, - ) - db.add(program, target_island=i) - db.store_artifacts(f"prog_{i}", {"output": f"result_{i}"}) - - # All artifacts should be retrievable - for i in range(3): - artifacts = db.get_artifacts(f"prog_{i}") - self.assertEqual(artifacts.get("output"), f"result_{i}") - - def test_store_many_artifacts(self): - """Test storing more artifacts than the limit""" - # One island per program so each program owns its cell and none is - # displaced (MAP-Elites removes programs displaced from their cell). - db_config = DatabaseConfig(max_snapshot_artifacts=5, num_islands=10) - db = ProgramDatabase(db_config) - - # Add 10 programs with artifacts - for i in range(10): - program = Program( - id=f"prog_{i}", - code=f"def func_{i}(): pass", - generation=0, - metrics={"score": i * 0.1}, - ) - db.add(program, target_island=i) - db.store_artifacts(f"prog_{i}", {"output": f"result_{i}"}) - - # All artifacts should still be stored in the database - # (the limit only affects snapshots, not storage) - for i in range(10): - artifacts = db.get_artifacts(f"prog_{i}") - self.assertEqual(artifacts.get("output"), f"result_{i}") - - def test_artifacts_for_nonexistent_program_returns_empty(self): - """Test retrieving artifacts for non-existent program""" - db_config = DatabaseConfig() - db = ProgramDatabase(db_config) - - artifacts = db.get_artifacts("nonexistent_id") - self.assertEqual(artifacts, {}) - - def test_store_artifacts_for_nonexistent_program_logs_warning(self): - """Test that storing artifacts for non-existent program doesn't crash""" - db_config = DatabaseConfig() - db = ProgramDatabase(db_config) - - # Should not raise an error - db.store_artifacts("nonexistent", {"output": "test"}) - - -class TestSnapshotCreation(unittest.TestCase): - """Tests for snapshot creation with artifact limits""" - - def test_config_accessible_from_database(self): - """Test that max_snapshot_artifacts is accessible from database config""" - db_config = DatabaseConfig(max_snapshot_artifacts=50) - db = ProgramDatabase(db_config) - - self.assertEqual(db.config.max_snapshot_artifacts, 50) - - def test_unlimited_config_is_none(self): - """Test that unlimited artifacts config is None""" - db_config = DatabaseConfig(max_snapshot_artifacts=None) - db = ProgramDatabase(db_config) - - self.assertIsNone(db.config.max_snapshot_artifacts) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_utils.py b/tests/test_utils.py index 5482b24e3e..69d084c73d 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -124,7 +124,6 @@ def get_integration_config(port: int = DEFAULT_PORT) -> Config: """Get config for integration tests with optillm""" config = Config() config.max_iterations = 5 # Very small for CI speed - config.checkpoint_interval = 2 config.database.in_memory = True config.evaluator.parallel_evaluations = 2 config.evaluator.timeout = 10 # Short timeout for CI diff --git a/trellis-research-design.md b/trellis-research-design.md new file mode 100644 index 0000000000..7a7dd84ceb --- /dev/null +++ b/trellis-research-design.md @@ -0,0 +1,204 @@ +**Trellis in OpenEvolve: research and implementation design** + +Prepared September 8, 2026 against checkout `411fb59`; revised September 9 to extract a program database interface first, retain the existing algorithm in an in-memory implementation, and then implement PostgreSQL. This is a new proposal derived from the Trellis paper, the current code, and the research questions in this conversation. It does not use the existing project plan as its specification. The working assumption is that the first deliverable is a research prototype with controlled experiments. This document describes the research roadmap. The interface extraction is the first development step; the PostgreSQL and Trellis research capabilities below remain proposed work. + +**Objective.** Turn OpenEvolve into a reproducible testbed for agents that improve through accumulated, shared experience. Establish whether experience helps, whether graph structure contributes beyond flat retrieval, and whether discoveries transfer between independent searchers. Follow with training and systems experiments using the same operational data. + +The first research milestone is a controlled comparison of no historical memory, prior winners, flat retrieval, and graph retrieval on held-out tasks. A database-backed evolution run is an intermediate implementation milestone. + +**Architecture.** Separate four responsibilities: search policy chooses what to explore; retrieval policy chooses historical evidence; a worker generates and evaluates an attempt; an experience store owns durable records and current search state. An experiment runner configures these components and controls which historical data each run can see. + +PostgreSQL is the program database from the outset. Each request for a parent, inspirations, an elite, or pending work executes a query over committed database state. The policy chooses the query and its parameters; reusable parameterized statements are sufficient. The DBMS performs selection, filtering, ordering, sampling, aggregation, and updates. Python retains the current candidate and the bounded context needed to execute it. + +There are no application checkpoints in this execution model: no periodic population save, no deserialization on restart, and no authoritative Python map mirrored into SQL. Restart means connecting to the same database and querying the session and eligible work. Historical event reconstruction and immutable experiment corpora support analysis; they are not a prerequisite for continuing execution. + +```mermaid +flowchart TD + E[Experiment runner and corpus manifests] --> C[Evolution controller] + C --> S[Search policy] + S --> D[PostgreSQL program database] + C --> W[Worker: claim work and fetch context] + W --> R[Retrieval policy and context builder] + R --> D + R --> L[LLM or agent adapter] + L --> V[Evaluator] + V --> D + L --> D + D --> A[Analysis, visualization, and training exports] +``` + +Use PostgreSQL for the first implementation. Add pgvector when building historical similarity retrieval and an artifact store when large content needs it. Start vector retrieval with exact search; measure approximate retrieval separately when scale requires it. pgvector supports both modes, with a recall/speed tradeoff for approximate indexes. [pgvector documentation](https://github.com/pgvector/pgvector#querying) + +Implement lineage queries using bounded recursive SQL and typed links. PostgreSQL supports recursive queries over hierarchical data. This gives a concrete reference implementation before evaluating a specialized graph or hybrid-query backend. [PostgreSQL recursive queries](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-RECURSIVE) + +Keep the existing CLI and library task inputs, and replace their state-management path with PostgreSQL sessions. Preserve ordinary OpenEvolve at a pinned checkout as a separate experimental baseline. Worker requests carry work IDs and bounded context. The existing process pool can remain the local worker launcher. Remove checkpoint callbacks, checkpoint configuration, and population snapshot construction from the PostgreSQL execution path. + +The existing in-memory operations become database operations: + +| Current operation | PostgreSQL operation | +| --- | --- | +| `programs[id]` | Point query for a node and its evaluation/artifact. | +| Choose a parent | Query the eligible population with the policy's island, fitness, and sampling conditions. | +| Choose inspirations | Query ranked or diverse eligible nodes with explicit limits. | +| Read a MAP-Elites cell | Query the elite for a session, island, and feature-cell key. | +| Add a candidate | Insert the attempt/evaluation and transactionally update its search eligibility and cell ownership. | +| Enforce population size | Rank eligible members and update membership; retain historical attempts. | +| Migrate between islands | Query eligible migrants and update or insert the appropriate membership and provenance records. | +| Find the best result | Query valid evaluations using the task's fitness ordering. | +| Continue after restart | Query session progress, pending updates, and eligible work. | + +A dictionary-shaped adapter that loads rows into a Python population does not satisfy this design. A query result can be materialized for the current prompt without becoming an authoritative population cache. + +**Data model.** An attempt, a code artifact, an evaluation, and membership in a search population are different records. The same code may occur in several attempts, receive several evaluations, or participate in multiple islands. Code hashes deduplicate artifact bytes; they do not erase attempts. + +| Record | Required content and purpose | +| --- | --- | +| Task | Specification, family and instance parameters, initial artifact, evaluator version, metric definitions and direction, correctness contract, environment manifest. | +| Session | Task, searcher/model configuration, search-policy version, seed, budgets, status, progress, experiment assignment, permitted corpus. Store configuration without credentials. | +| Node | Attempt ID allocated before generation, originating session, primary parent, lifecycle status, generated artifact reference when available, outcome references, generation, timestamps. Failures can have no code. | +| Additional links | Sources used as inspirations or seeds, alternate derivation links, and explicitly supported repair relationships. Preserve source node IDs across sessions. A retrieved reference records exposure, not proof that it caused an improvement. | +| LLM calls | Ordered messages and responses for each invocation and retry, requested and reported model, generation parameters, usage, latency, provider request ID when available, and terminal status. | +| Evaluations | Node and artifact, evaluator/environment versions, raw metrics, validity, fitness within that task's metric contract, diagnostics, duration, and retry/replicate identity. Re-evaluation appends evidence. | +| Artifacts | Content hash, media type, size, and durable location for code, diagnostics, traces, and other outputs. | +| Search state | Active membership, island membership, archive eligibility, cell elites, feature coordinates and mapping version, plus policy-specific counters. Eviction updates this state without deleting historical nodes. | +| Work and events | Work status, lease and claim generation, reserved budget, and ordered lifecycle/search-state changes. | +| Context selections | Retrieval query/version, corpus version, candidate IDs and scores, selected evidence, rendered excerpts, truncation, token counts, and the search-state values used for the decision. | +| Corpus and experiment manifests | Immutable permitted source IDs/revisions, task splits, experimental conditions, configuration hashes, and replicate identity. | + +Use relational columns for frequently queried state and JSON for heterogeneous evidence. Every query that assembles context must respect the permitted corpus, including each node reached through graph expansion. + +Score comparisons require a shared evaluation contract. A score from one task or hardware configuration cannot automatically become fitness in another task. Retrieved programs can supply reference context; programs inserted as starting candidates must be evaluated on the target task. Unknown correctness stays unknown. + +Record mutable search-state changes with a per-session revision assigned under a short session write lock. Apply the state change and its event in the same transaction. Record the actual values used by each decision as well: submitted iteration numbers, completion order, and wall-clock timestamps are not interchangeable, and a sequence number alone does not establish concurrent commit order. This supports reconstructing observed decisions and later historical queries without using future statistics. + +**Interfaces and code placement.** First extract `ProgramDatabase` in `openevolve/database.py`, shared `Program` values in `program.py`, and the existing policy in `database_memory.py`. The controller accepts an implementation via `database=` and workers receive bounded `IterationContext` values. The interface includes insertion, selection, ranked lookup, artifacts, prompts, and aggregate progress; it has no checkpoint operations. Add PostgreSQL behind that interface next. As experience capture grows, introduce the richer interfaces below with bounded reads and explicit transactions. Avoid a store API that returns the entire graph for callers to filter in Python. + +| Proposed module | Responsibility | +| --- | --- | +| `openevolve/experience/records.py` | Task, node, call, evaluation, context, work, and event records. | +| `openevolve/experience/store.py` | ExperienceStore protocol: lifecycle, claims, graph reads, bounded selection, context, and result commits. | +| `openevolve/experience/postgres.py` and `sql/` | Schema, migrations, queries, and transaction implementations. | +| `openevolve/experience/artifacts.py` | Content-addressed local/shared artifact storage behind an interface that can later support object storage. | +| `openevolve/search/base.py`, `map_elites.py`, `greedy.py` | Versioned search policies. Start with MAP-Elites; add greedy for transfer experiments. | +| `openevolve/memory/retrieval.py` and `context.py` | Historical retrieval policies, evidence ranking, graph expansion, and prompt-budget allocation. | +| `openevolve/experiments/` | Task/corpus manifests, experiment execution, usage accounting, and analysis. | +| `openevolve/experience/export.py` | Versioned analysis and training exports with provenance. | + +The store should expose operations such as `begin_attempt`, `claim_work`, `record_call`, `record_evaluation`, `commit_search_update`, `select_parent`, `select_local_inspirations`, `get_ancestors`, `get_siblings`, `retrieve_evidence`, and `get_session_progress`. Concrete signatures should use typed query objects and pagination where needed. + +The search policy specifies selection strategy, fitness semantics, diversity rules, and proposed transitions. The store executes the corresponding bounded queries and commits transitions against current state. This keeps SQL out of the controller while allowing different search policies over the same records. + +Derive independent random streams from the session seed and stable work identity, and define deterministic tie ordering. Record chosen parents and inspirations. Concurrency still changes which committed outcomes are available at selection time, so a reproducible experiment records the schedule and observed decision inputs rather than promising identical live trajectories from a seed alone. + +The main existing integration points are: + +- `controller.py`: connect to/create sessions, configure policies, enforce session budgets, query progress and the final result, and remove checkpoint orchestration. +- `process_parallel.py`: allocate attempts before work, replace population snapshots with work IDs, and persist intermediate boundaries instead of returning the only copy of evidence to the parent process. +- `database.py`: define the query contract shared by `database_memory.py` and a future PostgreSQL implementation. PostgreSQL owns its population state and selection; extract MAP-Elites policy parameters and separate active membership from history as the research model grows. +- `llm/base.py`, `llm/ensemble.py`, and provider adapters: expose structured call results and per-retry records. Preserve a text-returning compatibility wrapper for existing callers. +- `evaluator.py` and `evaluation_result.py`: retain stage outcomes, validity, diagnostics, and repeated measurements. Adapt existing evaluator return formats. +- `prompt/sampler.py`: accept an explicit historical-evidence bundle alongside current-session context, then retain the exact rendered messages. +- `evolution_trace.py` and `scripts/visualizer.py`: consume store queries/exports and display source evidence, failures, and cross-session links. + +**Attempt lifecycle and recovery.** Commit a node/work reservation before invoking a model. A worker claims it with a lease, obtains bounded context, and persists the selected evidence and messages. Persist the raw model response before parsing it; persist the candidate artifact before evaluating it; persist evaluation evidence before applying population changes. + +Represent lifecycle outcomes explicitly: queued, claimed, generating, generated, evaluating, evaluated, and failed/interrupted/cancelled. Invalid generation, failed correctness, timeout, and infrastructure failure remain distinct. Evaluation completion can be followed by a pending search update, so a restart can apply that update without re-running evaluation. + +Claim work with a short transaction and row locking. PostgreSQL's `SKIP LOCKED` is suitable for queue consumers; use it for work claims rather than assuming it provides a consistent view for ranking the search frontier. [PostgreSQL locking documentation](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) + +Never hold a database transaction open during LLM generation or evaluation. Renew leases during long work. Fence commits with a claim-generation token, make result commits idempotent, and recover expired work from the latest persisted boundary. Permit multiple distinct expansion jobs for the same parent when the search policy requests them. + +A crash after a provider finishes but before its response is stored can still lose that response. Record the interrupted invocation and reconcile it only when the provider supports doing so; otherwise a retry is separate work with separate cost. The system guarantees retention of committed evidence, not exactly-once external computation. + +Commit population changes, progress, consumed work, and associated events atomically. Reserve model/evaluation budget when scheduling concurrent jobs so parallelism cannot silently multiply the experimental allowance. Track permitted overshoot from already-running calls explicitly. + +Recheck elite replacement and population limits against the state locked for commit; a worker's earlier view cannot unconditionally overwrite a newer winner. Begin with serialized, short session-state commits for straightforward correctness, then measure whether finer-grained concurrency is needed. Selection reads should be side-effect free: update feature statistics through explicit writes, and version any resulting behavior change relative to the legacy policy. + +Publish artifact content before committing its database reference, verify its hash, and retain it for the lifetime of any experiment that references it. A shared worker demonstration requires shared artifact access as well as database access. + +**Historical retrieval.** Keep current-session search context consistent across experimental conditions. Initially vary only access to prior experience: + +| Policy | Historical context | +| --- | --- | +| `none` | No prior-session evidence. Still record the new run. | +| `winners` | Relevant successful artifacts with comparable evaluation metadata. | +| `flat` | Relevant individual attempts, including failures and diagnostics, without graph expansion. | +| `graph` | Relevant attempts expanded into bounded ancestors, siblings, and failure/repair evidence. | + +Treat initialization from a prior winner as a separate experimental factor. Combining a better starting program with graph retrieval in only one condition would confound the result. + +Apply corpus and compatibility filters, retrieve seeds from task descriptions and structured node diagnostics, expand allowed graph neighborhoods, then rank and deduplicate evidence under a prompt-token budget. Store embedding model/version and input hashes. OpenEvolve's existing code-embedding novelty check serves a different purpose and should remain a separately controlled feature. + +Start with deterministic ranking, bounded depth, and explicit evidence quotas. Keep the same seed retrieval and token budget in flat-versus-graph comparisons. Add component ablations that remove links, failed attempts, or rewards while retaining other inputs. Include matched-node-content comparisons where feasible to distinguish benefits of relational presentation from benefits of finding different nodes. + +Preserve parent/current code priority in prompt assembly. Record all truncation and actual message tokens. Empty retrieval is an observable outcome; do not fill the budget with arbitrary material. Add injection frequency, diversity-aware reranking, and stale-evidence handling as later versioned policies. An inferred repair relation must carry its inference method and evidence. + +**Implementation sequence and completion checks.** + +0. **Extract the program database interface.** Move the existing population and selection logic into `InMemoryProgramDatabase`. Route controller and evaluator access through bounded query methods and explicit writes. Pass only the selected context to workers. Remove checkpoint orchestration and population save/load from the in-memory implementation. Test the shared contract and run the controller through a facade exposing only interface methods. Completion: ordinary evolution works without direct map access outside the implementation; the in-memory version remains the development reference. + +1. **Build the PostgreSQL program database.** Pin the current checkout as an external baseline. Define the minimal relational task/session/node/evaluation and active-search tables, including parent links, island membership, feature cells, fitness, and attempt status. Implement actual SQL operations for insertion, parent/inspiration selection, elite lookup/update, population enforcement, and best-result lookup. Use a real temporary PostgreSQL database and deterministic fixtures from the first tests. Completion: committed rows and query results alone determine the next selection; the PostgreSQL implementation does not reconstruct or mirror an authoritative Python population. Reuse the interface contract tests against a real database. + +2. **Run evolution directly against PostgreSQL.** Inject the PostgreSQL implementation behind the extracted interface and exercise the SQL operations from step 1. Allocate attempts before generation and persist messages, responses, evaluations, and terminal statuses. Start with a single worker and scripted generation through the actual pipeline. Remove application checkpoint saving/loading from this path. Completion: every iteration obtains its parent and context through DBMS queries and commits its result; a newly constructed controller continues the session by querying PostgreSQL. Failed attempts and displaced ancestors remain queryable. A capture mirror under the existing Python map is not an intermediate deliverable. + +3. **Add concurrent execution and research controls.** Introduce work leases, fenced/idempotent commits, budget reservations, and interruption tests against PostgreSQL. Version feature mappings and persist assigned cells; any change to existing scaling/sampling behavior becomes a separately named policy variant applied consistently across experiment arms. Add structured provider-usage reporting, task specifications, metric contracts, experiment manifests, and a parameterized sorting family with immutable discovery/development/test splits. Completion: competing workers do not duplicate committed results or corrupt elites, all state needed to continue is queryable, and rerunning a manifest produces auditable measurements without corpus leakage. + +4. **Implement retrieval and run the first controlled study.** Add the four policies, corpus boundaries, token budgeting, retrieval provenance, and source-evidence visualization. Generate a small independent discovery corpus, freeze it, tune retrieval only on development tasks, and run held-out comparisons. Completion: a reproducible report compares conditions on validated quality, success, total tokens, elapsed time, repeated failures, and diversity. A measured lack of graph advantage is a valid research result. + +5. **Test accumulation and collective reuse.** Build nested immutable corpus versions at several historical-compute budgets, then compare performance on fixed held-out tasks. Add greedy search and a second measurable model backend. Compare isolated, pooled-frozen-history, and live-shared-history runs at equal total compute. Add a second task family. Completion: experience-growth curves and transfer matrices identify where prior work helps, plateaus, or harms exploration. + +6. **Close a training loop.** Export successful trajectories, matched-context preference pairs, and decision-time graph features with dataset manifests. Train a small generation model or frontier-value model through a separate training toolchain, evaluate on untouched tasks, and append another round of exploration. Completion: compare equal training budgets and evaluate with retrieval both enabled and disabled. Preference pairs require the same effective prompt and evaluation contract; sharing a parent alone is insufficient. Keep group-relative online training as a later extension requiring compatible rollout and policy metadata. + +7. **Evaluate the data foundation as a system.** Replay the measured operational workload at larger scale: frontier selection, mixed graph/vector retrieval, concurrent commits, artifact access, and training extraction. Compare query composition strategies on equivalent outputs or measured retrieval recall. Add provenance-based invalidation of exports and scoped access across raw and derived data before making governance claims. Completion: report latency distributions, throughput, recovery waste, storage growth, and search impact under contention. A specialized query-engine adapter should be justified by these results. + +Steps 1-4 form the first research release. Steps 5-7 establish broader claims without making them prerequisites for the initial result. Model training can begin after complete capture, decision provenance, and corpus isolation are reliable; it need not wait for specialized query execution. + +**Benchmark design.** Parameterize existing examples rather than treating repeated runs of one fixed problem as evidence of generalization. Start with adaptive sorting over distinct input-distribution families and sizes. Extend to packing instances with different counts/constraints, or numerical optimization families with independently verified objectives. The current circle-packing evaluator is fixed to 26 circles, and the minimization example uses fixed reference values, so both require explicit task-family adapters before supporting these studies. + +Separate discovery tasks that populate memory, development tasks that tune retrieval, and final evaluation tasks. Within each task, separate feedback tests visible during search from final validation tests. Verify outputs independently, including fitness values claimed by generated code. Run final validation under a fixed environment and preserve evaluator/test hashes. Validate artifact identity across generation and execution. + +Use frozen corpus manifests for causal comparisons. Final evaluation outcomes cannot enter a corpus used by another nominally independent test run. For online accumulation experiments, define chronological cohorts and sharing groups explicitly; each decision records exactly which revisions were visible. Split at the problem-family or generator level when simple instance splits would leak near-duplicates. + +Match model settings, current-session policy, initial program, task, evaluation allowance, and completion-token limits across memory conditions. Cap added context uniformly, and report total input/output tokens instead of assuming identical context length. Include retrieval, embedding, summarization, failed calls, retries, compilation, and evaluation costs. Distinguish provider-reported usage from estimates and unavailable values; unavailable cost must never be silently treated as zero. + +The current text-only provider adapters do not expose enough data for these measurements. Begin published cost comparisons with backends that expose usage reliably. A CLI or agent backend needs an adapter for the structured events it actually exposes; record missing internal tool/model spans explicitly rather than claiming a complete transcript. + +Use a small pilot to estimate variance and choose replication counts. Analyze independent tasks, sessions, and corpus replicates as experimental units; nodes within a session are correlated. Report uncertainty, every failed run, and the fraction that never reaches the target. Do not average only successful runs. Normalize comparisons within each task before aggregating across domains. + +The first release should generate these artifacts from its manifest: + +- Fitness versus total tokens and elapsed time, with independent final validation. +- Target-reaching success and cost, including runs that exhaust their budget. +- A comparison of historical-memory policies and component ablations. +- Corpus growth versus marginal new-task benefit and amortized cost. +- A lineage/evidence view that explains selected concrete successes and failures without replacing aggregate measurements. + +Collect a database workload trace from the outset: query class, latency, rows/bytes returned, history size, active population size, and concurrent workers. This makes later systems experiments representative of the agent workload. + +**Configuration and user workflow.** Add explicit database/session, retrieval, search-policy, and experiment configuration. Continue a run by session ID. Original OpenEvolve runs remain available in the pinned baseline checkout. The exact spelling below is illustrative: + +```yaml +experience: + backend: postgres + dsn_env: OPENEVOLVE_EXPERIENCE_DSN + artifact_root: ./experience_artifacts +search: + policy: map_elites_v1 +retrieval: + policy: graph + corpus: sorting_discovery_v1 + max_context_tokens: 3000 + max_graph_depth: 3 +experiment: + manifest: sorting_graph_ablation_v1 +``` + +Provide commands or library functions to register tasks, run/resume a session, freeze a corpus, execute an experiment manifest, inspect node evidence, and export results. Experimental conditions should differ through these policies and manifests rather than separate controller forks. Keep PostgreSQL dependencies in an optional package extra and provide a reproducible local database setup for contributors. + +**Verification.** Test the PostgreSQL query contract, graph traversal and corpus boundaries, prompt reconstruction, retry accounting, population invariants, and concurrent claims against a real database. Inject failures at transaction boundaries. Check that failures retain a node even without generated code, inactive ancestors remain accessible, target-task fitness is re-evaluated, and graph expansion cannot import held-out records. Construct a fresh controller after committed changes and verify its next selection reflects those rows without loading a checkpoint. Inspect query results and memory use to ensure no authoritative Python population is reconstructed. Controlled single-worker tests can compare normalized deterministic decisions; randomized and concurrent search needs invariants and distributional checks rather than an identical trajectory requirement. + +Adapt tests that directly manipulate `database.programs`, `islands`, or cell maps to verify public behavior on the new path. Retain the pinned legacy baseline independently. Do not use the existing smoke assertion that at least one program survives as proof of meaningful search improvement. + +**Initial scope choices.** Extract the interface and an in-memory implementation first, then add PostgreSQL queries using one local worker launcher and MAP-Elites. Neither implementation needs application checkpoints. The PostgreSQL backend owns its rows and selection policy directly; the in-memory implementation serves as a reference and test backend. Add another search policy to answer a transfer question. Preserve event and evidence provenance early because missing history cannot be recovered later. Expand into distributed services, additional storage engines, learned retrieval, training infrastructure, and fine-grained governance when the associated experiment requires them. + +The architecture and scientific motivation follow the experience-graph, reuse, training, and research-agenda discussions in [trellis.pdf](trellis.pdf), particularly sections 4-7. PostgreSQL results would establish a reproducible implementation of the logical architecture; claims about a particular Trellis optimizer or physical design require a corresponding implementation and controlled systems comparison.