By clicking the Manage Connectors button at the bottom left of each Connect Cluster card, you can view the list and overall status of the connectors currently running in the Connect Cluster.
Connector Details
Status Overview
You can easily check the key metrics for the connector at a glance.
| Item | Description |
|---|---|
| Connector status | The current status of the connector instance |
| Record count (POLL,READ) | (Source connector) The number of records the Source connector has polled into tasks [poll] (Sink connector) The number of records the Sink connector has read from Kafka [read] |
| Record count (PUT,WRITE) | (Source connector) The number of records the Source connector has written to Kafka [write] (Sink connector) The number of records the sync connector has sent to external systems [send] |
| Record batch average time | (Source connector) The average time taken for the Source connector to [poll] record batches from external systems (Sink connector) The average time taken for the Sink connector to [put] record batches to external systems |
| Error count | The number of records that encountered errors during processing |
Task Status
In the Tasks tab, you can check the status and details of the tasks run by each connector, restart tasks, or view logs for tasks that have encountered errors.
- Restarting
Click Restart button to restart each task. If the connector is stopped or has encountered an error, restarting the task may not be the solution for it. In this case, restart the connector itself or adjust the connector settings to resolve the issue.
- Viewing Error Logs
If a task is in an error state, the View Error Logs button is activated. Click on this button to see the detailed error logs for the task. You can fix the error by restarting the task or adjusting the connector settings after reviewing the error logs.
Metrics
In the Metrics tab, you can check the metrics information for each connector.
Total Record Count
You can see the number of records that the connector is currently reading from or sending to external systems. The Total Record Count indicates the number of records handled by the connector. For Source connectors, you will see the POLL and WRITE metrics, and for Sink connectors, you will see the READ and SEND metrics.
Total Error/Failed Record Count
Total Error Record Count
This is the total number of records that encountered errors, when the transformer or connector attempted to transform data. This can occur in the following situations.
Occurrence Situation
- Data format mismatch: When the incoming data format does not match the expected format during data transformation process
- Transformation: When the logic defined in the transformer fails or an exception occurs
- Schema compatibility issue: When the input data schema does not match the expected output schema
Failed Record Count
This is the total number of records that failed while the connector transmits data to external systems. This can occur in the following situations.
Occurrence Situation
- System error: Errors in the target system such as databases or file systems
- Network issue: Problems with network connection during data transmission
- Validation failure: Records that fail the validation checks before transmission
- Retry failure: When all retry attempts for data transmission fail
Dead Letter Topic (DLT) Failed Attempt Count
You can see the number of records that failed to be written to the DLT (Dead Letter Topic), and the number of attempts made. If there are write attempts to the DLT, it means the connector had encountered records it could not process. Therefore, check the DLT message content to identify which records failed to be processed.
This metric is only available for connectors with DLT settings configured. If a connector does not have DLT settings, the DLT metrics will not be displayed.
Configs
In the Configs tab, you can view and modify the configuration information for each connector. You might fail to modify if you enter incorrect configuration information.
Connector Action
You can hover over each connector in the list or click on the [Connector Details > Actions] button to perform various actions.
Changing Connector Status
You can pause a running connector or resume a paused connector.
Restarting Connector
You can restart the connector instance, with options to include task instances in the restarting as well. If no option is selected, only the connector instance will be restarted.
- Restart Including All Tasks
Restart all currently running tasks along with the connector. - Restart Including Failed Tasks Only
Restart only with the tasks that are currently in a failed state.
Deleting Connector
You can delete a connector instance. Deleting a connector will also delete all its tasks, so please proceed with caution.
Adding Connector
Click Add Connector to see the list of available connector plugins you can add.
Use various types of Connector Plugins to easily connect to different data sources like databases, cloud services, and file systems. Plugin helps automate data transfer, synchronization, and transformation tasks, simplifying and increasing the efficiency of your data management process.
Step 1. Viewing Plugin List & Selecting Plugin
You can check the list of connector plugins that are available to be added to the current Connect Cluster. Click Add of the connector plugin you want to add, to proceed to the next step.
Step 2. Setting Connector Details
Enter in the required values for each connector. Under the Recommended Settings List on the left, you can view the values by Importance Levels and Categories, and add more values by clicking on the check button. For detailed explanations of each setting, click the Expand Description toggle. Once the values are added, they will appear in the JSON Editor on the right, where you can also edit or delete them.
Required/recommended values for each connector might slightly differ from those provided in the dashboard, so please refer to the official document for each connector when you have any issues.
Step 3. Validating & Adding
Verify that the settings you have entered for the connector are valid, and then click Add to add the connector.