Skip to content

0.16.10

Deployed: 09-10-2025

Latest features and fixes for Data Catering include:

YAML Plan and Task Execution via UI/API

  • Automatic YAML Plan Discovery: UI/API can now execute YAML-defined plans and tasks seamlessly

    • When UI sends a plan with only task summaries, system automatically discovers and loads full YAML plan by name
    • Searches configured plan directory for matching YAML files
    • Supports mixing UI-created plans with YAML-defined tasks
    • Check here for documentation

  • Connection Configuration Auto-Merge: Connection details from application.conf automatically merge with task configurations

    • Connection format (e.g., json, postgres, kafka) derived from top-level configuration key
    • Eliminates duplication of connection details across task files
    • Step-specific options override connection defaults (proper precedence)
    • Check here for documentation

Core Engine Improvements

  • PlanProcessor:

    • Add intelligent YAML plan detection when task definitions are missing from UI requests
    • Implement findYamlPlanFile() to locate YAML plans by name in configured directory
    • Create YamlPlanRun class to properly merge connection configs into task step options

  • ConnectionRepository:

    • Add support for loading connections from both saved files and application.conf
    • Implement getConnectionFromConfig() and getConnectionsFromConfig() methods
    • Connection details automatically available to both UI and YAML plans

  • PlanRepository:

    • Add support for listing and loading YAML plans alongside UI-created plans
    • YAML plan discovery from configured plan directory
    • Seamless integration of YAML and UI plan workflows

Bug Fixes

  • Connection Configuration Loading: Fixed "No format specified for data source" error
    • Root cause: Connection configs from application.conf were not merged into step options
    • Solution: YamlPlanRun now explicitly merges connection configuration into each task's step options
    • Ensures format field and other connection properties are available during execution

Documentation

  • API Documentation:

    • Add comprehensive "YAML Plans and Connection Configuration" section
    • Include example workflow showing connection definition, YAML tasks, and plan execution
    • Document benefits: reduced duplication, centralized config, seamless integration

  • Connection Documentation:

    • Add "Automatic Connection Merging with YAML Plans" section
    • Explain connection format detection, merge precedence, and override behavior
    • Include complete example with application.conf, YAML task, and YAML plan

Tests

  • YamlPlanIntegrationTest: New comprehensive integration test for YAML plan execution

    • Test YAML plan discovery and loading
    • Test connection configuration merging
    • Verify UI can execute YAML plans seamlessly

  • Test Fixes:

    • Update DataGeneratorProcessorTest to use correct connection config keys
    • Make YamlPlanIntegrationTest more robust to handle test file variations

Benefits

  • Reduced Duplication: Define connection details once in application.conf, reuse across all tasks
  • Centralized Configuration: Manage all connection credentials and settings in one place
  • Seamless Integration: Mix and match UI-created plans with YAML-defined tasks and configurations
  • Environment Flexibility: Override connection settings using environment variables without changing task files
  • Better Separation: Keep connection credentials separate from task definitions for improved security