Extract program database interface and remove population checkpoints

CLAUDE.md

@@ -40,16 +40,11 @@ make lint
 # Basic evolution run
 python openevolve-run.py path/to/initial_program.py path/to/evaluator.py --config path/to/config.yaml --iterations 1000
 
-# Resume from checkpoint
-python openevolve-run.py path/to/initial_program.py path/to/evaluator.py \
-  --config path/to/config.yaml \
-  --checkpoint path/to/checkpoint_directory \
-  --iterations 50
 ```
 
 ### Visualization
 ```bash
-# View evolution tree
+# View an archived evolution tree from an older checkpoint-based run
 python scripts/visualizer.py --path examples/function_minimization/openevolve_output/checkpoints/checkpoint_100/
 ```
 
@@ -59,7 +54,7 @@ python scripts/visualizer.py --path examples/function_minimization/openevolve_ou
 
 1. **Controller (`openevolve/controller.py`)**: Main orchestrator that manages the evolution process using ProcessPoolExecutor for parallel iteration execution.
 
-2. **Database (`openevolve/database.py`)**: Implements MAP-Elites algorithm with island-based evolution:
+2. **Database (`openevolve/database.py`)**: Query-oriented `ProgramDatabase` interface. `database_memory.py` implements MAP-Elites with island-based evolution:
    - Programs mapped to multi-dimensional feature grid
    - Multiple isolated populations (islands) evolve independently
    - Periodic migration between islands prevents convergence
@@ -73,14 +68,14 @@ python scripts/visualizer.py --path examples/function_minimization/openevolve_ou
 
 4. **LLM Integration (`openevolve/llm/`)**: Ensemble approach with multiple models, configurable weights, and async generation with retry logic.
 
-5. **Iteration (`openevolve/iteration.py`)**: Worker process that samples from islands, generates mutations via LLM, evaluates programs, and stores artifacts.
+5. **Iteration (`openevolve/process_parallel.py`)**: Queries the database for bounded iteration context, then workers generate mutations and evaluate programs. The controller writes results through the interface.
 
 ### Key Architectural Patterns
 
 - **Island-Based Evolution**: Multiple populations evolve separately with periodic migration
 - **MAP-Elites**: Maintains diversity by mapping programs to feature grid cells
 - **Artifact System**: Side-channel for programs to return debugging data, stored as JSON or files
-- **Process Worker Pattern**: Each iteration runs in fresh process with database snapshot
+- **Process Worker Pattern**: Each worker receives selected programs and parent artifacts, with no population snapshot
 - **Double-Selection**: Programs for inspiration differ from those shown to LLM
 - **Lazy Migration**: Islands migrate based on generation counts, not iterations
 
@@ -103,8 +98,8 @@ YAML-based configuration with hierarchical structure:
 
 ### Important Patterns
 
-1. **Checkpoint/Resume**: Automatic saving of entire system state with seamless resume capability
-2. **Parallel Evaluation**: Multiple programs evaluated concurrently via TaskPool
+1. **Database Sessions**: Inject a `ProgramDatabase` via `database=`. Continue using the same database instance; the in-memory implementation has no save/load operations. PostgreSQL is a future implementation.
+2. **Parallel Evaluation**: Multiple programs evaluated concurrently via ProcessPoolExecutor
 3. **Error Resilience**: Individual failures don't crash system - extensive retry logic and timeout protection
 4. **Prompt Engineering**: Template-based system with context-aware building and evolution history
 
@@ -115,4 +110,4 @@ YAML-based configuration with hierarchical structure:
 - Tests use unittest framework
 - Black for code formatting
 - Artifacts threshold: Small (<10KB) stored in DB, large saved to disk
-- Process workers load database snapshots for true parallelism
+- Controllers use database queries for each candidate; worker context stays bounded by prompt limits

README.md

@@ -755,9 +755,41 @@ return EvaluationResult(
 
 This creates a **feedback loop** where each generation learns from previous mistakes!
 
+## Program database
+
+`ProgramDatabase` in `openevolve/database.py` is the storage interface.
+`InMemoryProgramDatabase` in `openevolve/database_memory.py` implements the existing
+MAP-Elites and island policies. The controller queries this interface for each
+candidate and sends workers only the selected programs and parent artifacts.
+Programs returned by queries are detached values; editing them does not change
+the database. Writes go through database operations.
+
+```python
+from openevolve import OpenEvolve
+from openevolve.config import load_config
+from openevolve.database_memory import InMemoryProgramDatabase
+
+config = load_config("config.yaml")
+database = InMemoryProgramDatabase(config.database)
+evolve = OpenEvolve("program.py", "evaluator.py", config, database=database)
+# In an async entry point:
+best = await evolve.run(iterations=50)
+best = await evolve.run(iterations=50)  # Continues the same database session
+```
+
+The library's `run_evolution` function also accepts `database=`. A future PostgreSQL
+implementation will use the same interface. The in-memory implementation lasts for
+the lifetime of its instance; it does not save or restore population checkpoints.
+The `--checkpoint` CLI option and `checkpoint_path` API argument have been removed.
+Best-program exports, logs, optional evolution traces, and evaluation artifacts remain
+available. Legacy YAML keys `checkpoint_interval` and `database.max_snapshot_artifacts`
+are ignored.
+
 ## Visualization
 
-**Real-time evolution tracking** with interactive web interface:
+The existing visualizer reads archived checkpoint files from older runs. It has not
+yet been adapted to the program database interface, so new runs cannot use it for
+live population tracking.
 
 ```bash
 # Install visualization dependencies

configs/README.md

@@ -2,6 +2,13 @@
 
 This directory contains configuration files for OpenEvolve with examples for different use cases.
 
+The default program database is `InMemoryProgramDatabase`. Supply a database implementation
+through `OpenEvolve(..., database=...)` or `run_evolution(..., database=...)`.
+Population checkpoint saving/loading has been removed. Older YAML files may still contain
+`checkpoint_interval` and `database.max_snapshot_artifacts`; these keys are ignored.
+`database.db_path` is now only a fallback directory for artifacts. `database.in_memory`
+is retained for configuration compatibility and does not enable disk persistence.
+
 ## Configuration Files
 
 ### `default_config.yaml`
@@ -65,9 +72,10 @@ Then use with OpenEvolve:
 
 ```python
 from openevolve import OpenEvolve
+from openevolve.config import load_config
 evolve = OpenEvolve(
     initial_program_path="program.py",
     evaluation_file="evaluator.py", 
-    config_path="my_config.yaml"
+    config=load_config("my_config.yaml")
 )
 ```

configs/default_config.yaml

@@ -4,7 +4,6 @@
 
 # General settings
 max_iterations: 100                  # Maximum number of evolution iterations
-checkpoint_interval: 10               # Save checkpoints every N iterations
 log_level: "INFO"                     # Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
 log_dir: null                         # Custom directory for logs (default: output_dir/logs)
 random_seed: 42                       # Random seed for reproducibility (null = random, 42 = default)
@@ -92,8 +91,8 @@ prompt:
 # Database configuration
 database:
   # General settings
-  db_path: null                       # Path to persist database (null = in-memory only)
-  in_memory: true                     # Keep database in memory for faster access
+  db_path: null                       # Legacy fallback base directory for artifacts
+  in_memory: true                     # Legacy option; use database= to supply an implementation
   log_prompts: true                  # If true, log all prompts and responses into the database
 
   # Evolutionary parameters

examples/tsp_tour_minimization/start_evolution.py

@@ -1,4 +1,3 @@
-import re
 import sys
 import pathlib
 import asyncio
@@ -17,28 +16,8 @@
 from utils.code_to_query import *
 
 
-def latest_checkpoint(dir_path: pathlib.Path) -> str | None:
-    if not dir_path.exists():
-        return None
-
-    candidates: list[tuple[int, pathlib.Path]] = []
-    for path in dir_path.glob("checkpoint_*"):
-        if not path.is_dir():
-            continue
-
-        match = re.fullmatch(r"checkpoint_(\d+)", path.name)
-        if match:
-            candidates.append((int(match.group(1)), path))
-
-    if not candidates:
-        return None
-
-    _, newest_path = max(candidates, key=lambda t: (t[0], t[1].stat().st_mtime))
-    return str(newest_path)
-
-
-async def run_evolution(evolve: OpenEvolve, checkpoint_path: str | None) -> None:
-    best_program = await evolve.run(checkpoint_path=checkpoint_path)
+async def run_evolution(evolve: OpenEvolve) -> None:
+    best_program = await evolve.run()
 
     print("Best program metrics:")
     for name, value in best_program.metrics.items():
@@ -59,7 +38,7 @@ async def run_evolution(evolve: OpenEvolve, checkpoint_path: str | None) -> None
     "--openevolve_output_dir",
     type=click.Path(file_okay=False, dir_okay=True, path_type=pathlib.Path),
     default=BASE_DIR / "openevolve_output",
-    help="Output directory for OpenEvolve runs (checkpoints, logs, etc.).",
+    help="Output directory for OpenEvolve results and logs.",
 )
 def cli(initial_program_dir: pathlib.Path, openevolve_output_dir: pathlib.Path) -> None:
     initial_program_dir = initial_program_dir.resolve()
@@ -80,10 +59,7 @@ def cli(initial_program_dir: pathlib.Path, openevolve_output_dir: pathlib.Path)
         output_dir=str(openevolve_output_dir),
     )
 
-    checkpoint_path = latest_checkpoint(openevolve_output_dir / "checkpoints")
-    print(f"Using checkpoint: '{checkpoint_path}'")
-
-    asyncio.run(run_evolution(evolve, checkpoint_path))
+    asyncio.run(run_evolution(evolve))
 
 
 if __name__ == "__main__":

openevolve/api.py

@@ -13,7 +13,7 @@
 
 from openevolve.controller import OpenEvolve
 from openevolve.config import Config, load_config, LLMModelConfig
-from openevolve.database import Program
+from openevolve.database import Program, ProgramDatabase
 
 
 @dataclass
@@ -38,7 +38,7 @@ def run_evolution(
     output_dir: Optional[str] = None,
     cleanup: bool = True,
     target_score: Optional[float] = None,
-    checkpoint_path: Optional[str] = None,
+    database: Optional[ProgramDatabase] = None,
 ) -> EvolutionResult:
     """
     Run evolution with flexible inputs - the main library API
@@ -58,6 +58,8 @@ def run_evolution(
         iterations: Number of iterations (overrides config)
         output_dir: Output directory (None for temp directory)
         cleanup: If True, clean up temp files after evolution
+        target_score: Stop once this combined score is reached
+        database: Session database (defaults to a new in-memory implementation)
 
     Returns:
         EvolutionResult with best program and metrics
@@ -92,7 +94,16 @@ def my_evaluator(program_path):
         )
     """
     return asyncio.run(
-        _run_evolution_async(initial_program, evaluator, config, iterations, output_dir, cleanup, target_score, checkpoint_path)
+        _run_evolution_async(
+            initial_program,
+            evaluator,
+            config,
+            iterations,
+            output_dir,
+            cleanup,
+            target_score,
+            database,
+        )
     )
 
 
@@ -104,7 +115,7 @@ async def _run_evolution_async(
     output_dir: Optional[str],
     cleanup: bool,
     target_score: Optional[float] = None,
-    checkpoint_path: Optional[str] = None,
+    database: Optional[ProgramDatabase] = None,
 ) -> EvolutionResult:
     """Async implementation of run_evolution"""
 
@@ -158,9 +169,10 @@ async def _run_evolution_async(
             evaluation_file=evaluator_path,
             config=config_obj,
             output_dir=actual_output_dir,
+            database=database,
         )
 
-        best_program = await controller.run(iterations=iterations,target_score=target_score,checkpoint_path=checkpoint_path)
+        best_program = await controller.run(iterations=iterations, target_score=target_score)
 
         # Prepare result
         best_score = 0.0

openevolve/cli.py

@@ -45,12 +45,6 @@ def parse_args() -> argparse.Namespace:
         default=None,
     )
 
-    parser.add_argument(
-        "--checkpoint",
-        help="Path to checkpoint directory to resume from (e.g., openevolve_output/checkpoints/checkpoint_50)",
-        default=None,
-    )
-
     parser.add_argument("--api-base", help="Base URL for the LLM API", default=None)
 
     parser.add_argument("--primary-model", help="Primary LLM model name", default=None)
@@ -112,17 +106,6 @@ async def main_async() -> int:
             output_dir=args.output,
         )
 
-        # Load from checkpoint if specified
-        if args.checkpoint:
-            if not os.path.exists(args.checkpoint):
-                print(f"Error: Checkpoint directory '{args.checkpoint}' not found")
-                return 1
-            print(f"Loading checkpoint from {args.checkpoint}")
-            openevolve.database.load(args.checkpoint)
-            print(
-                f"Checkpoint loaded successfully (iteration {openevolve.database.last_iteration})"
-            )
-
         # Override log level if specified
         if args.log_level:
             logging.getLogger().setLevel(getattr(logging, args.log_level))
@@ -131,23 +114,8 @@ async def main_async() -> int:
         best_program = await openevolve.run(
             iterations=args.iterations,
             target_score=args.target_score,
-            checkpoint_path=args.checkpoint,
         )
 
-        # Get the checkpoint path
-        checkpoint_dir = os.path.join(openevolve.output_dir, "checkpoints")
-        latest_checkpoint = None
-        if os.path.exists(checkpoint_dir):
-            checkpoints = [
-                os.path.join(checkpoint_dir, d)
-                for d in os.listdir(checkpoint_dir)
-                if os.path.isdir(os.path.join(checkpoint_dir, d))
-            ]
-            if checkpoints:
-                latest_checkpoint = sorted(
-                    checkpoints, key=lambda x: int(x.split("_")[-1]) if "_" in x else 0
-                )[-1]
-
         print(f"\nEvolution complete!")
         print(f"Best program metrics:")
         for name, value in best_program.metrics.items():
@@ -157,10 +125,6 @@ async def main_async() -> int:
             else:
                 print(f"  {name}: {value}")
 
-        if latest_checkpoint:
-            print(f"\nLatest checkpoint saved at: {latest_checkpoint}")
-            print(f"To resume, use: --checkpoint {latest_checkpoint}")
-
         return 0
 
     except Exception as e:

openevolve/config.py

@@ -311,10 +311,10 @@ class DatabaseConfig:
     """Configuration for the program database"""
 
     # General settings
-    db_path: Optional[str] = None  # Path to store database on disk
-    in_memory: bool = True
+    db_path: Optional[str] = None  # Legacy fallback base directory for artifacts
+    in_memory: bool = True  # Legacy option; implementations are supplied via database=
 
-    # Prompt and response logging to programs/<id>.json
+    # Prompt and response history in the session database
     log_prompts: bool = True
 
     # Evolutionary parameters
@@ -359,9 +359,6 @@ class DatabaseConfig:
     artifact_size_threshold: int = 32 * 1024  # 32KB threshold
     cleanup_old_artifacts: bool = True
     artifact_retention_days: int = 30
-    max_snapshot_artifacts: Optional[int] = (
-        100  # Max artifacts in worker snapshots (None=unlimited)
-    )
 
     novelty_llm: Optional["LLMInterface"] = None
     embedding_model: Optional[str] = None
@@ -418,7 +415,6 @@ class Config:
 
     # General settings
     max_iterations: int = 10000
-    checkpoint_interval: int = 100
     log_level: str = "INFO"
     log_dir: Optional[str] = None
     random_seed: Optional[int] = 42