When you create a source connector in Kafka Connect, by default it will use the broker’s default batch.size which is 16384 bytes (16Kb). This can have a massive impact on the ingestion time of larger files. Let’s use a file with 20,000,000+ lines as an example and retrieve some metrics:

Learn how to launch an Apache Kafka with the Apache Kafka Raft (KRaft) consensus protocol and SASL/PLAIN authentication. This article is a continuation of my previous article Running Kafka in Kubernetes with KRaft mode. PLAIN versus PLAINTEXT: Do not confuse the SASL mechanism PLAIN with the no TLS/SSL encryption option,which…

Learn how to launch an Apache Kafka with the Apache Kafka Raft (KRaft) consensus protocol and SSL encryption. This article is a continuation of my previous article Running Kafka in Kubernetes with KRaft mode. Prerequisites An understanding of Apache Kafka, Kubernetes, and Minikube. The following steps were initially taken on a…

Learn how to launch an Apache Kafka with the Apache Kafka Raft (KRaft) consensus protocol, removing Kafka’s dependency on Apache Zookeeper for metadata management. Before jumping into the technical part, let me give yousome context as to why and how KRaft came into existence. Why Kafka Without ZooKeeper? Apache ZooKeeper has been an integral…

Learn about Kubernetes Auditing and how to use it with Kubernetes Secrets. As stated in K8s documentation, “auditing provides a chronological set of records documenting the sequence of actions in a cluster.” It intends to answer questions like: What happened? When did it happen? Who initiated it? Kubernetes Secrets contains…

After configuring Istio with STRICT mTLS and using a ClusterIP service, pod-to-pod communication breaks. Problem Pods configured to use ClusterIP as Service stop communicating with each other after configuring Istio PeerAuthentication to use STRICT mTLS. In the following sections, let’s simulate the error and how to reestablish inter-pod communication. Test Environment The test…

A K8s Pod can return an Error when trying to reach the network before the Istio Sidecar is Running. Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the istio-proxy sidecar container is not ready. This article…

Use a Gateway to manage inbound and outbound traffic for your mesh, letting you specify which traffic you want to enter or leave the Istio mesh. In a previous article, I explained the concept of a Service Mesh. This article demonstrates how to expose Kubernetes services deployed in the Service…

How Trivy helps us scan Docker images, Kubernetes, and Terraform code to detect potential configuration issues and minimise the risk of attack. Vulnerabilities are everywhere! I’d like to describe how I’m using Trivy to scan Docker images, Kubernetes configuration files, and Terraform code to find vulnerabilities before deploying them in…