Provisioning Config for Component Types

Below document outlines default and override-able params that can be used in the provisioning config for various component types.

Exposed configs for component types

Component type with special parameter are mentioned separately.

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_ec2","gcp_container"], "type": "string"}

params.instance_pool

array

No

Sample provisioning config will look like

[
    {
        "component_name": "my-aerospike",
        "deployment_type": "container"
    }
]

or

[
    {
        "component_name": "my-aerospike",
        "deployment_type": "aws_ec2",
        "params": {
            "instance_pool": ["c5.xlarge"]
        }
    }
]

application

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_ec2","gcp_container"], "type": "string"}

params.lb_type

string

No

{"enum": ["elb", "alb"], "type": "string"}

params.instance_pool

array

No

Sample provisioning config will look like

[
    {
        "component_name": "my-application",
        "deployment_type": "container"
    }
]

or

[
    {
        "component_name": "my-application",
        "deployment_type": "aws_ec2",
        "params": {
            "instance_pool": ["c5.xlarge"],
            "lb_type": "elb"
        }
    }
]

aerospike

Note - Only supports Aerospike version 6.3.0.7

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["optimus_aws_ec2"], "type": "string"}

params.aerospike.num_instances

integer

No

params.kafka_outbound.num_instances

integer

No

params.namespace_memory

string

No

params.file

string

No

params.filesize

string

No

params.write_block_size

string

No

params.data_in_memory

string

No

params.remove_ebs_vol

string

No

{ "enum": [ "yes", "no" ], "type": "string" }

params.replication_factor

integer

No

{"type":"integer","minimum":1,"exclusiveMinimum":false}

Spark

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{ "enum": [ "aws_ec2", "aws_ec2_ha" ], "type": "string" }

params.worker_cleanup_ttl

string

No

params.s3_cron

string

No

params.max_size

string

No

params.on_demand_percentage

string

No

params.spot_allocation_strategy

string

No

params.ebs_root_volume_size

string

No

params.disable_api_termination

string

No

params.num_on_demand_instances

string

No

params.master_instance_pool

array

No

{ "type": "array", "items": { "type": "string" }, "minItems": 1, "maxItems": 1, "uniqueItems": true }

params.master_num_instances

string

No

params.worker_instance_pool

array

No

{ "type": "array", "items": { "type": "string" }, "minItems": 1, "maxItems": 1, "uniqueItems": true }

params.worker_num_instances

string

No

params.history_num_instances

string

No

params.history_instance_pool

string

No

{ "type": "array", "items": { "type": "string" }, "minItems": 1, "maxItems": 1, "uniqueItems": true }

params.spark_ha_route

string

No

To deploy Spark HA, providedeployment_type as aws_ec2_hain the provisioning file.

  • you can access the Host using url (spark-$COMPONENT_NAME-multimaster-alb$TEAM_SUFFIX.dream11$VPC_SUFFIX.local). you can also override the Host url using the param spark_ha_route.

  • to access the spark UI, use the route from spark exposed configs.

neptune

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_neptune","gcp_container"], "type": "string"}

params.cluster_parameters

array

No

{"type": "array", "items": {"type": "object", "required": ["name", "value"], "properties": {"name": {"type": "string"}, "value": {}, "apply_immediately": {"enum": ["pending-reboot", "immediate"], "type": "string"}}}}

params.instance_parameters

array

No

{"type": "array", "items": {"type": "object", "required": ["name", "value"], "properties": {"name": {"type": "string"}, "value": {}, "apply_immediately": {"enum": ["pending-reboot", "immediate"], "type": "string"}}}}

postgresql

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_rds_aurora","gcp_container"], "type": "string"}

params.cluster_parameter_group_name

string

No

params.parameter_group_name

string

No

serverless

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_serverless"], "type": "string"}

params.extra_envs

array

No

{"type": "array", "items": {"minLength": 1}, "uniqueItems": true}

stepfunction

Exposed configs

Config
Data Type
Mandatory
Schema

deployment_type

string

No

{"enum": ["container", "aws_stepfunction"], "type": "string"}

params.extra_envs

array

No

{"type": "array", "items": {"minLength": 1}, "uniqueItems": true}

Config
Description
Data Type
Mandatory
Schema

deployment_type

Platform on which the component should be deployed

String

No

params.flink.jobmanager.instance_type

Type of the EC2 instance for jobmanager nodes

String

No

params.flink.taskmanager.instance_type

Type of the EC2 instance for taskmanager nodes

String

No

Last updated