Apache Kafka

Apache Kafka Consumer not receiving messages in Order

I am trying a POC for Kafka in my project and created two console apps in .net core 2.1 using Confluent.kafka library. I have installed Kafka on …

JavaScript

Writing the output of Batch Queries to Kafka for Spark version 2.1.1

Can somebody give me pointers on how can I load the output of Batch Queries to kafka.I researched a lot in stackoverflow and other articles but I was …

Stack Overflow

java.lang.NoSuchMethodError: kafka.api.TopicMetadata.errorCode()S while witing into kafka from Spark job

I am attempting to write json message into kafka from my spark job.I have added the following into my dependencies : spark-sql-kafka-0-10_2.11, …

Ubiquitous Computing

How we build a robust analytics platform using Spark, Kafka and Cassandra Lambda architecture

In today’s online world, supply chain is one of the most important pillars of any online shop. Not just quality products, but customers also want …

Big Data

Instaclustr: 9 tips to improve Apache Kafka management - Open Source Insider

<i>This is a guest post for the Computer Weekly Open Source Inside blog written by by</i> <i>Ben Slater</i> <i>in his role as chief product officer at</i></i> …

Ubiquitous Computing

Confluent Streaming Data: Product Overview and Analysis

By using a fully-managed Confluent streaming data service on Google Cloud, users can offload the operating burden of Apache Kafka and stream data at …

Ubiquitous Computing

How to process files using Spark Structured Streaming chunk by chunk?

I am treating a large amount of files, and I want to treat these files chunk by chunk, let's say that during each batch, I want to treat each 50 …

Python Programming

Architecture help - alternative to ETL data flow & processing

I am looking for some guidance on building an architecture for a simple ETL job. I have already built a solution but I am looking for ways to improve …

Big Data

Decoupling Systems with Apache Kafka, Schema Registry and Avro

As your Apache Kafka® deployment starts to grow, the benefits of using a schema registry quickly become compelling. Confluent Schema Registry, which …

Microservices

Self Driven Data Science — Issue #57

Weekly rundown of interesting news and insights focused on data science, machine learning, and artificial intelligence<p>Here’s this weeks lineup of …

Data Science

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used …

Machine Learning

SD Times Open-Source Project of the Week: Faust

The financial services company Robinhood has announced it is open-sourcing the distributed stream processing library Faust. According to the company, …

Microservices

Partner Webcast – Building event driven microservices with Oracle Event Hub CS

Digital Marketing Specialist<p>Join this webinar during which you will learn how to configure and manage the OEHCS Kafka cluster and how to use it to …

Microservices

Confluent release adds enterprise, developer, IoT savvy to Apache Kafka

Confluent, the company founded by the creators of streaming data platform Apache Kafka, is announcing a new release today. Confluent Platform 5.0, …

Ubiquitous Computing

Apache Kafka book using Scala

I am looking for Apache Kafka book using Scala. I found the book "Kafka Definitive Guide", but the code is in Java.Could anyone let me if there is a …

Ubiquitous Computing

Why does executing Structured Streaming application fail with "Failed to find data source: kafka"?

I am trying to connect Spark Structured Streaming with kafka and it throws the below error:Exception in thread "main" …

Schadenfreude

3 big data platforms look beyond Hadoop

A distributed file system, a MapReduce programming framework, and an extended family of tools for processing huge data sets on large clusters of …

Big Data

Kafka is establishing its toehold

Data pipelines were the headline from the third annual survey of Apache Kafka use. Behind anecdotal evidence of a growing user base, Kafka is still …

Software Development

What’s New in KNIME Analytics Platform 3.6 and KNIME Server 4.7 | KNIME

This year's summer release, on July 11, 2018, is a major KNIME® Software update. Here we highlight some of the major changes, new features, and …

Databases

Create Apache Flink Table From Kafka DataStream

I have a DataStream of typewhich I created using a (Flink) Kafka consumer. I wish to create a Flink table from this DataStream. I have looked through …

Ubiquitous Computing

Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow

Machine Learning / Deep Learning models can be used in different ways to do predictions. My preferred way is to deploy an analytic model directly …

Ubiquitous Computing

Scala: Splitting the data coming from kafka vi a DStream

I am receiving the data from kafka in the form ofI want to access the email id and first name and want to compare it with data coming from cassandra …

Ubiquitous Computing

Apache Kafka Producer.send randomly hangs

I was trying to get a basic kafka stream working, and I created a producer and usedto send a ProducerRecord to the kafka stream. It was hanging, so I …

Hadoop

JSON send as kafka producer message and consuming by spark structured streaming -parquet

I would like to know how to send a JSON string as message to kafka topic using scala function and Consumed by the using readstream() in spark …

Big Data

This Week in Numbers: Apache Kafka's Metamorphosis

The Apache Kafka distributed streaming platform is not changing but its typical use cases are. Commercial Apache distributor Confluent issued its …

Big Data

Survey Reveals Apache Kafka® Will Be Mission-Critical to 90 Percent of Data and Application Infrastructures in 2018

Confluent, provider of the leading streaming platform based on Apache Kafka®, today announced the results of the third annual survey of the Kafka …

Ubiquitous Computing

Supercharging Kafka — Enable Realtime Web Streaming by Adding Pushpin

Exposing Kafka messages via a public HTTP streaming API<p>Apache Kafka is the new hotness when it comes to adding realtime messaging capabilities to …

Microservices

How to name outputs of Kafka-HDFS-Ingestion job containing Apache Kafka topic names in Apache Gobblin?

I have tested Gobblin with Hadoop and Apache Kafka using Kafka-HDFS-Ingestion Job. The example is available here. In Kafka, I have 2 topic and I can …

Linux

online time series anomalies detection with apache spark

we have a data pipeline systemapache kafka---->spark steaming----->spark mlibthe data consumed is time series data (e.g. each record is in the form …

Apache Spark

Kafkaesque: Instaclustr creates Kafka-as-a-Service - Open Source Insider

Instaclustr has announced Kafka-as-a-Service in bid to provide an easier route to the real-time data streaming platform<p>An open source player from the …

Google Cloud Platform