Configuring a Distributed API Manager Deployment with Gateway and Control Plane

In the distributed setup, the API-M server profiles are deployed as separate API-M nodes. Followings are the APIM nodes you can have in a distributed deployment support for our pattern.

• Gateway Worker Node - API-M nodes running the Gateway profile.
• Control Plane Node - API-M nodes running the Control Plane profile. The Control Plane includes the Traffic Manager, Key Manager, Publisher, and Developer Portal components.

1. Install WSO2 API Manager

• Download the WSO2 API Manager.
• Create copies of the API-M distribution for the individual profiles

2. Install and configure the databases

You can create the required databases for the API-M deployment in a separate server and point to the databases from the respective nodes.

Below are the default databases which will be used within API Manager.

• WSO2AM_DB - WSO2 API Manager has this database keeping its specific API-M related data.
• WSO2SHARED_DB - This database contains the registry and user management data.
• WSO2CARBON_DB - This database has the internal data related to the product.

Databases used by API-M profiles:

	WSO2AM_DB	WSO2SHARED_DB	WSO2CARBON_DB
Control Plane profile	Yes	Yes	Yes
Gateway profile	No	Yes	Yes

It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. WSO2 products are shipped with scripts for creating the required tables in all the required databases: The scripts for creating tables for API-M, user management, and registry data are stored in the <API-M_HOME>/dbscripts directory.

In this example i'm using PostgreSQL.

3. Create and import SSL certificates

Create an SSL certificate for each of the WSO2 API-M nodes and import them to the keystore and the truststore. This ensures that hostname mismatch issues in the certificates will not occur.

Note: The same primary keystore should be used for all API Manager instances to decrypt the registry resources.

For more details:

https://apim.docs.wso2.com/en/latest/install-and-setup/setup/security/configuring-keystores/keystore-basics/creating-new-keystores/

https://apim.docs.wso2.com/en/latest/install-and-setup/setup/security/configuring-keystores/configuring-keystores-in-wso2-api-manager/

4. Configure and start the profiles

4.1 Configure the Gateway Nodes

Follow these steps to configure the Gateway to communicate with the Control Plane.

Open the <API-M_HOME>/repository/conf/deployment.toml file of the Gateway node.

Add the following configurations to the deployment.toml file.

Note: Update the parameters as necessary.

--------------------------------------------------------------------------------------------------------

[server]

hostname = "{{apim gateway lb host}}"

node_ip = "{{node_ip}}"

server_role = "gateway-worker"

offset= 0

[user_store]

type = "database_unique_id"

[super_admin]

username = "$secret{admin_username}"

password = "$secret{admin_password}"

create_admin_account = true

#Databases

[database.apim_db]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/apim_db"

username = "$secret{wso2am_db_username}"

password = "$secret{wso2am_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

pool_options.maxActive = 50

pool_options.maxWait = 30000

[database.shared_db]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/shared_db"

username = "$secret{wso2shared_db_username}"

password = "$secret{wso2shared_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

pool_options.maxActive = 50

pool_options.maxWait = 10000

pool_options.validationInterval = 10000

[database.local]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/carbon_db"

username = "$secret{wso2carbon_db_username}"

password = "$secret{wso2carbon_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

 # JWT Generation

[apim.jwt]

enable = "true"

encoding = "base64"

generator_impl = "org.wso2.carbon.apimgt.keymgt.token.JWTGenerator"

claim_dialect = "http://wso2.org/claims"

header = "X-JWT-Assertion"

signing_algorithm = "SHA256withRSA"

enable_user_claims = "false"

claims_extractor_impl = "org.wso2.carbon.apimgt.impl.token.ExtendedDefaultClaimsRetriever"

[apim.sync_runtime_artifacts.gateway]

gateway_labels = ["Default"]

artifact_retriever = "DBRetriever"

deployment_retry_duration = 15000

data_retrieval_mode = "sync"

event_waiting_time = 5000

[apim.key_manager]

service_url = "https://{{apim control plane lb host}}/services/"

username = "$ref{super_admin.username}"

password = "$ref{super_admin.password}"

[apim.throttling]

enable_data_publishing = true

enable_policy_deploy = true

enable_blacklist_condition = true

enable_persistence = true

username = "$ref{super_admin.username}"

password = "$ref{super_admin.password}"

service_url = "https://{{apim control plane lb host}}/services/"

throttle_decision_endpoints = ["tcp://{{apim control plane 1 host}}:5672","tcp://{{apim control plane 2 host}}:5672"]

enable_unlimited_tier = true

enable_header_based_throttling = false

enable_jwt_claim_based_throttling = false

enable_query_param_based_throttling = false

[[apim.throttling.url_group]]

traffic_manager_urls = ["tcp://{{apim control plane 1 host}}:9611"]

traffic_manager_auth_urls = ["ssl://{{apim control plane 1 host}}:9711"]

[[apim.throttling.url_group]]

traffic_manager_urls = ["tcp://{{apim control plane 2 host}}:9611"]

traffic_manager_auth_urls = ["ssl://{{apim control plane 2 host}}:9711"]

[transport.http]

properties.port = 9763

properties.proxyPort = 80

[transport.https]

properties.port = 9443

properties.proxyPort = 443

[apim.analytics]

enable = false

auth_token = "auth_token"

# Caches

[apim.cache.gateway_token]

enable = true

expiry_time = 15

[apim.cache.resource]

enable = true

[apim.cache.jwt_claim]

enable = true

expiry_time = 900

[apim.oauth_config]

enable_outbound_auth_header = true

auth_header = "Authorization"

[apim.cors]

allow_origins = "*"

allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]

allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction"]

allow_credentials = false

[oauth.grant_type.token_exchange]

enable = true

allow_refresh_tokens = true

iat_validity_period = "1h"

# Condition check for whether the Certification Import Option is enabled

[keystore.primary]

file_name = "primary.pfx"

password = "$secret{primaryKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{primaryKeyPassword}"

type = "PKCS12"

[keystore.internal]

file_name = "internal.pfx"

password = "$secret{internalKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{internalKeyPassword}"

type = "PKCS12"

[keystore.tls]

file_name = "tls.pfx"

password = "$secret{tlsKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{tlsKeyPassword}"

type = "PKCS12"

[truststore]

file_name = "client-truststore.jks"

password = "$secret{truststrorePassword}"

alias = "symmetric.key.value"

algorithm = "AES"

[secrets]

admin_username = "[{{ admin_username }}]"

admin_password = "[{{ admin_password }}]"

wso2am_db_username = "[{{ wso2am_db_username }}]"

wso2am_db_password = "[{{ wso2am_db_password }}]"

wso2shared_db_username = "[{{ wso2shared_db_username }}]"

wso2shared_db_password = "[{{ wso2shared_db_password }}]"

wso2carbon_db_username = "[{{ wso2carbon_db_username }}]"

wso2carbon_db_password = "[{{ wso2carbon_db_password }}]"

truststrorePassword = "[{{truststore.storePassword}}]"

primaryKeyStorePassword  = "[{{primary.privateKeyPasssword}}]"

primaryKeyPassword  = "[{{primary.privateKeyPasssword}}]"

internalKeyStorePassword  = "[{{internal.privateKeyPasssword}}]"

internalKeyPassword  = "[{{internal.privateKeyPasssword}}]"

tlsKeyStorePassword  = "[{{tls.privateKeyPasssword}}]"

tlsKeyPassword  = "[{{tls.privateKeyPasssword}}]"

---------------------------------------------------------------------------------------------------------

4.2 Configure the Control Plane Nodes

Follow these steps to configure the Control Plane to communicate with the Gateway.

Open the <API-M_HOME>/repository/conf/deployment.toml file of the Gateway node.

Add the following configurations to the deployment.toml file.

Note: Update the parameters as necessary.

------------------------------------------------------------------------------------------------------------

[server]

hostname = "{{apim control plane lb host}}"

node_ip = "{{node_ip}}"

server_role = "control-plane"

base_path = "${carbon.protocol}://${carbon.host}:${carbon.management.port}"

[user_store]

type = "database_unique_id"

[super_admin]

username = "$secret{admin_username}"

password = "$secret{admin_password}"

create_admin_account = true

#Databases

[database.apim_db]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/apim_db"

username = "$secret{wso2am_db_username}"

password = "$secret{wso2am_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

pool_options.maxActive = 50

pool_options.maxWait = 30000

[database.shared_db]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/shared_db"

username = "$secret{wso2shared_db_username}"

password = "$secret{wso2shared_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

pool_options.maxActive = 50

pool_options.maxWait = 10000

pool_options.validationInterval = 10000

[database.local]

type = "postgre"

url = "jdbc:postgresql://{{db host}}:5432/carbon_db"

username = "$secret{wso2carbon_db_username}"

password = "$secret{wso2carbon_db_password}"

driver = "org.postgresql.Driver"

validationQuery = "SELECT 1"

[tenant_mgt]

enable_email_domain = true

[apim.devportal]

url = "https://{{apim control plane lb host}}/devportal"

[transport.http]

properties.port = 9763

properties.proxyPort = 80

[transport.https]

properties.port = 9443

properties.proxyPort = 443

 # JWT Generation

[apim.jwt]

enable = "true"

encoding = "base64"

generator_impl = "org.wso2.carbon.apimgt.keymgt.token.JWTGenerator"

claim_dialect = "http://wso2.org/claims"

header = "X-JWT-Assertion"

signing_algorithm = "SHA256withRSA"

enable_user_claims = "false"

claims_extractor_impl = "org.wso2.carbon.apimgt.impl.token.ExtendedDefaultClaimsRetriever"

[apim.sync_runtime_artifacts.publisher]

artifact_saver = "DBSaver"

publish_directly_to_gateway = "false"

[[apim.gateway.environment]]

name = "Default"

type = "hybrid"

display_in_api_console = true

description = "This is a hybrid gateway that handles both production and sandbox token traffic."

show_as_token_endpoint_url = true

service_url = "https://{{apim gateway lb host}}/services/"

ws_endpoint = "ws://{{apim gateway lb host}}:9099"

wss_endpoint = "wss://{{apim gateway lb host}}:8099"

http_endpoint = "http://{{apim gateway lb host}}"

https_endpoint = "https://{{apim gateway lb host}}"

username= "${admin.username}"

password= "${admin.password}"

websub_event_receiver_http_endpoint = "http://{{apim gateway lb host}}:9021"

websub_event_receiver_https_endpoint = "https://{{apim gateway lb host}}:8021"

# provider = "wso2"

# Event Hub configurations

[apim.event_hub]

enable = true

username= "$ref{super_admin.username}"

password= "$ref{super_admin.password}"

service_url = "https://{{apim control plane lb host}}/services/"

event_listening_endpoints = ["tcp://{{apim control plane 1 host}}:5672"]

event_duplicate_url = ["tcp://{{apim control plane 2 host}}:5672"]

[[apim.event_hub.publish.url_group]]

urls = ["tcp://{{apim control plane 1 host}}:9611"]

auth_urls = ["ssl://{{apim control plane 1 host}}:9711"]

[[apim.event_hub.publish.url_group]]

urls = ["tcp://{{apim control plane 2 host}}:9611"]

auth_urls = ["ssl://{{apim control plane 2 host}}:9711"]

# key manager implementation

[apim.key_manager]

service_url = "https://{{apim control plane lb host}}/services/"

[apim.analytics]

enable = false

auth_token = "auth_token"

[apim.cache_invalidation]

enabled = true

domain = "control-plane-domain"

# Caches

[apim.cache.gateway_token]

enable = true

expiry_time = 15

[apim.cache.resource]

enable = true

[apim.cache.jwt_claim]

enable = true

expiry_time = 900

[apim.oauth_config]

enable_outbound_auth_header = true

auth_header = "Authorization"

[apim.cors]

allow_origins = "*"

allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]

allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction"]

allow_credentials = false

[[event_listener]]

id = "token_revocation"

type = "org.wso2.carbon.identity.core.handler.AbstractIdentityHandler"

name = "org.wso2.is.notification.ApimOauthEventInterceptor"

order = 1

[event_listener.properties]

notification_endpoint = "https://{{apim control plane lb host}}/internal/data/v1/notify"

username = "${super_admin.username}"

password = "${super_admin.password}"

'header.X-WSO2-KEY-MANAGER' = "default"

[oauth.grant_type.token_exchange]

enable = true

allow_refresh_tokens = true

iat_validity_period = "1h"

# Condition check for whether the Certification Import Option is enabled

[keystore.primary]

file_name = "primary.pfx"

password = "$secret{primaryKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{primaryKeyPassword}"

type = "PKCS12"

[keystore.internal]

file_name = "internal.pfx"

password = "$secret{internalKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{internalKeyPassword}"

type = "PKCS12"

[keystore.tls]

file_name = "tls.pfx"

password = "$secret{tlsKeyStorePassword}"

alias = "wso2carbon"

key_password = "$secret{tlsKeyPassword}"

type = "PKCS12"

[truststore]

file_name = "client-truststore.jks"

password = "$secret{truststrorePassword}"

alias = "symmetric.key.value"

algorithm = "AES"

[secrets]

admin_username = "[{{ admin_username }}]"

admin_password = "[{{ admin_password }}]"

wso2am_db_username = "[{{ wso2am_db_username }}]"

wso2am_db_password = "[{{ wso2am_db_password }}]"

wso2shared_db_username = "[{{ wso2shared_db_username }}]"

wso2shared_db_password = "[{{ wso2shared_db_password }}]"

wso2carbon_db_username = "[{{ wso2carbon_db_username }}]"

wso2carbon_db_password = "[{{ wso2carbon_db_password }}]"

truststrorePassword = "[{{truststore.storePassword}}]"

primaryKeyStorePassword  = "[{{primary.privateKeyPasssword}}]"

primaryKeyPassword  = "[{{primary.privateKeyPasssword}}]"

internalKeyStorePassword  = "[{{internal.privateKeyPasssword}}]"

internalKeyPassword  = "[{{internal.privateKeyPasssword}}]"

tlsKeyStorePassword  = "[{{tls.privateKeyPasssword}}]"

tlsKeyPassword  = "[{{tls.privateKeyPasssword}}]"

-----------------------------------------------------------------------------------------------------

Open the server's /etc/hosts file and map the hostnames to IPs.

Format: 

<GATEWAY-LB-IP> gw.wso2.com

<GATEWAY-1-IP> gw-1.wso2.com

<GATEWAY-2-IP> gw-2.wso2.com

<CONTROL-PLANE-LB-IP> cp.wso2.com

<CONTROL-PLANE-1-IP> cp-1.wso2.com

<CONTROL-PLANE-2-IP> cp-2.wso2.com

4.3 Configure carbon.xml

Configure carbon.xml file if you using custom keystores as below.

<Security>

        <!--

            KeyStore which will be used for encrypting/decrypting passwords

            and other sensitive information.

        -->

        <KeyStore>

            <!-- Keystore file location-->

            <Location>${carbon.home}/repository/resources/security/primary.pfx</Location>

            <!-- Keystore type (JKS/PKCS12 etc.)-->

            <Type>PKCS12</Type>

            <!-- Keystore password-->

            <Password>$secret{primaryKeyStorePassword}</Password>

            <!-- Private Key alias-->

            <KeyAlias>wso2carbon</KeyAlias>

            <!-- Private Key password-->

            <KeyPassword>$secret{primaryKeyPassword}</KeyPassword>

        </KeyStore>

        <!--

            The KeyStore which is used for encrypting/decrypting internal data.

            This block is read by Carbon Crypto Service.

        -->

        <InternalKeyStore>

            <!-- Keystore file location-->

            <Location>${carbon.home}/repository/resources/security/internal.pfx</Location>

            <!-- Keystore type (JKS/PKCS12 etc.)-->

            <Type>PKCS12</Type>

            <!-- Keystore password-->

            <Password>$secret{internalKeyStorePassword}</Password>

            <!-- Private Key alias-->

            <KeyAlias>wso2carbon</KeyAlias>

            <!-- Private Key password-->

            <KeyPassword>$secret{internalKeyPassword}</KeyPassword>

        </InternalKeyStore>

        <UserStorePasswordEncryption>InternalKeyStore</UserStorePasswordEncryption>

        <!--

            System wide trust-store which is used to maintain the certificates of all

            the trusted parties.

        -->

        <TrustStore>

            <!-- trust-store file location -->

            <Location>${carbon.home}/repository/resources/security/client-truststore.jks</Location>

            <!-- trust-store type (JKS/PKCS12 etc.) -->

            <Type>JKS</Type>

            <!-- trust-store password -->

            <Password>$secret{truststrorePassword}</Password>

        </TrustStore>

<Security>

4.4 Configure load balancers

You need to properly configure load balancers fronting the two Control Plane nodes and two Gateway nodes.

4.5 Start the API-M nodes

Before starts the servers,

Execute the control plane optimization:

cd <API-M_HOME>/bin/

sh profileSetup.sh -Dprofile=control-plane

Execute the gateway optimization:

cd <API-M_HOME>/bin/

sh profileSetup.sh -Dprofile=gateway-worker

Once you have successfully configured all the API-M nodes in the deployment, you can start the servers.

Starting the Gateway nodes

Open a terminal, navigate to the <API-M-GATEWAY-HOME>/bin folder, and execute the following command

cd <API-M_HOME>/bin/

sh api-manager.sh -Dprofile=gateway-worker

Start the Control Plane nodes

Open a terminal, navigate to the <API-M-CONTROL-PLANE-HOME>/bin folder, and execute the following command:

cd <API-M_HOME>/bin/

sh api-manager.sh -Dprofile=control-plane

5. Sample APIM deployment architecture in Azure cloud.

This is the sample architecture of the APIM deployment in Azure using Azure VMs, Azure scaleset, Azure Load-balancer and Azure Active Directory.