Skip to content

0.17.0

Deployed: 23-10-2025

Latest features and fixes for Data Catering include major enhancements to YAML support, Java API improvements, and expanded testing infrastructure.

YAML Support Enhancements

  • YAML Metadata Source: Reference YAML field definitions across data sources with metadataSource.yamlTask() for reusable step libraries and DRY configurations. Docs

  • YamlBuilder API: Load and extend YAML plans/tasks programmatically with methods like yaml.plan(), yaml.taskByFile(), yaml.stepByName(), etc. Enables hybrid YAML + code approach. Docs

  • Enhanced Processing: Task/step filtering by name, improved type inference, better error handling, comprehensive test infrastructure

Java API Improvements

  • WeightedValue Class: WeightedValue.of(value, weight) for type-safe weighted field generation in Java
  • Enhanced Integration: Better Java-Scala interoperability, improved type safety, enhanced method signatures

UI and API Server Enhancements

  • Sample Generation Endpoints: New REST endpoints for generating sample data from plans/tasks/steps with fastMode (45-58% faster) and raw data responses
  • Plan Execution Endpoints: Execute plans/tasks/steps by name with filtering (POST /run/plans/{planName}/tasks/{taskName}/steps/{stepName})
  • FastSampleGenerator: Multi-schema support, improved error handling, content-type negotiation for JSON/CSV/Parquet
  • PlanRepository: YAML plan loading with task/step filtering

Core Engine Improvements

  • YamlDataSourceMetadata: Refined type inference, nested structures support, validation extraction
  • PlanParser: Better error handling and validation for YAML
  • Data Generators: Optimized pipeline, enhanced sample generation, improved memory management
  • SinkFactory: Single file consolidation, better error handling, cleanup improvements

Testing and Quality Assurance

  • YAML Test Suite: YamlBuilderTest, YamlMetadataSourceTest, YamlMetadataTest, YamlMetadataReferenceTest, YamlMetadataReuseTest
  • End-to-End Tests: PlanApiEndToEndTest (950+ lines) covering API endpoints, sample generation, plan execution with filtering
  • Java API Tests: JavaApiImprovementsTest for WeightedValue and Java-Scala compatibility

Examples and Configuration

  • Example Plans: YamlEnvironmentSpecificPlan, YamlPlanWithExtensionsPlan, YamlReferenceExamplePlan, YamlSchemaMultipleOutputsPlan, YamlTaskWithOverridesPlan, YamlWithForeignKeysPlan, YamlWithValidationsPlan, YamlMetadataReuseMultipleOutputs (Java/Scala), DynamicCompanyPaymentsPlan
  • Constants: New YAML constants (YAML_PLAN_FILE, YAML_TASK_FILE, YAML_STEP_NAME, YAML_TASK_NAME)
  • Configuration: Enhanced YAML integration with application settings, connection merging

Documentation

  • YAML Guide: Comprehensive YAML configurations guide with examples, best practices, migration patterns
  • REST API: Enhanced endpoint documentation with curl/wget examples, fast mode benchmarks, content-type handling
  • Version Updates: Docker images and quick-start guides updated to 0.17.0