Apache Kafka

Does Apache Kafka Store the messages internally in HDFS or Some other File system

We have a project requirement of testing the data at Kafka Layer. So JSON files are moving into hadoop area and kafka is reading the live data in …

Big Data

Which directory does apache kafka store the data in broker nodes

I can see a property in config/server.properties called log.dir? Is it the same directory for storing logs and data both?<p>apache-kafka<p>Midhun Mathew …

Big Data

Event-Driven Microservices with RabbitMQ and Ruby

In my last post we discussed how event sourcing with Apache Kafka can be used to create robust reactive microservices. While event sourcing as an …

JavaScript

Where does Big Data go from here?

The term Big Data has been bandied about since the 1990s. It was meant to reflect the explosion of data – structured and unstructured — with which …

Computer Science

Kafka optimal retention and deletion policy

I am fairly new to kafka so forgive me if this question is trivial. I have a very simple setup for purposes of timing tests as follows:<p>Machine A -> …

Big Data

CISOs Should Examine Commercial SOAPA Offerings in 2018

For over a year now, I’ve written about a burgeoning security technology initiative that ESG calls a security operations and analytics platform …

Cybersecurity

Ingesting Golden Gate Records From Apache Kafka and Automagically Populating Any JDBC Tables

Sometimes, you need to process any number of table changes sent from tools via Apache Kafka. As long as they have proper header data and records in …

Big Data

Last week in Azure: Azure HDInsight, Azure Cosmos DB, and more

Welcome to 2018! This edition of Last Week in Azure covers the final two weeks of 2017. Just before everyone disappeared for the holiday break here …

Cloud Computing

Transparently Securing Kafka, Istio-style, with up to 300% Higher Performance than Native TLS in Microservice Environments

A couple of months back, I wrote an article about some unexpected results we obtained while using an Istio-style approach to secure Kafka. After my …

Cloud Computing

Intel aims to reduce HPC memory bottleneck with FPGA technology | Scientific Computing World

Intel is aiming to tackle the challenges of increasing memory bandwidth for HPC applications with an FPGA that includes High Bandwidth Memory DRAM …

Computer Science

Apache Kafka Introduction

development<p>No Comments<p>Integrating systems that every day grow larger is a complex task. Apache Kafka is a software that tries to solve this by using …

Big Data

Streamline Software Integration: An Apache Camel Tutorial

Software rarely, if at all, exists in an informational vacuum. At least, that is the assumption we software engineers can make for most of the …

DevOps

Apache Kafka

Download this new Refcard to get started with Apache Kafka. Learn more about its background, steps to run the quickstart script, components, and how …

Big Data

A Blockchain Experiment With Apache Kafka

Blockchain technology and Apache Kafka share characteristics which suggest a natural affinity. For instance, both share the concept of an ‘immutable …

Bitcoin

Building a Distributed Log from Scratch, Part 1: Storage Mechanics

The log is a totally-ordered, append-only data structure. It’s a powerful yet simple abstraction—a sequence of immutable events. It’s something that …

Hadoop

Events, Flows and Long-Running Services: A Modern Approach to Workflow Automation

Key Takeaways<p>Recent discussions around the microservice architectural style has promoted the idea of event-driven-architectures to effectively …

Software Development

How to Generate Your Avro Schema Automatically

In a previous post, I explained how StreamSets Data Collector (SDC) can work with Apache Kafka and Confluent Schema Registry to handle data drift via …

Hadoop

Machine Learning on OpenShift and Kubernetes

Learn how we've teamed up with Google and other members of the Kubernetes community to create a strong open source community for artificial …

Open Source Movement

Apache Kafka Lands on Microsoft's Cloud-Based Big Data Platform

After a yearlong beta period, Apache Kafka on Azure HDInsight is now ready to take on production workloads with Microsoft's rollout of the …

Big Data

Announcing Apache Kafka for Azure HDInsight general availability

Big Data

Apache Kafka on HDInsight

Raghav Mohan joins Scott Hanselman to talk about Apache Kafka on HDInsight, which added the open-source distributed streaming platform last year to …

Big Data

Kafka & Redis Streams

Let’s talk about queue design.We have a very long book which we would like many people to read. Some can read during their lunch hour, some read on …

Hadoop

Thorough Introduction to Apache Kafka™

Introduction<p>Kafka is a word that gets heard a lot nowadays… A lot of leading digital companies seem to use it as well. But what is it actually?<p>Kafka …

Big Data

YugaByte: 7 core IoT developer skills - CW Developer Network

YugaByte is a newly established company that sets out to deliver what it describes as turnkey distributed consistent and highly available database …

Big Data

A Few Cybersecurity Predictions for 2018

Over the past few weeks, dozens of people have reached out to me with their cybersecurity predictions for 2018. Some prophecies are fairly obvious …

Cybersecurity

Confluent Platform: Schema Registry Subjects

Working with Confluent Platform, the platform offered by the creators of Apache Kafka, and I have a question:<p>In the documentation of the Schema …

Big Data

Run spark job in parallel and using single spark context running in local mode

I need to run some HQLs via spark. I have a jar having class that creates dataset from JSON, perform HQL and create JSON. Finally, save that JSON to …

Big Data

Problems With Kafka Streams

Before diving straight into the main topic, let me introduce you to Kafka Streams first.We have all heard about Apache Kafka, as it has been used …

Big Data

Pivotal Unveils Expansion of Pivotal Cloud Foundry and Announces Serverless Computing Product

Most Innovative Release Since Launch of Pivotal Cloud Foundry<p>On December 5, 2017, Pivotal is announcing the next era of Pivotal Cloud Foundry® (PCF), …

Cloud Foundry

Kafka and firewall rules

We have a fairly strict network segmentation policy. I am using a cloud foundry instance to deploy an app to. The firewall rules have been set up to …

Cloud Foundry