2018-10-22         Stanford

KafkaConsumer : erro on consumer.subscribe(Arrays.asList(topic)) error

I'm trying to config my KafkaConsumer to read data from KafkaProducer by the 'kafkatopic'.My scala code is : package com.sundogsoftware.sparkstreaming import java.util import java.util.Properties import org.apache.kafka.clients.consumer.KafkaConsumer import scala.collection.JavaConverters._ import org.apache.kafka.clients.consumer.{ ConsumerRecords, KafkaConsumer } import org.apache.kafka.clients.producer.{ KafkaProducer, Producer, ProducerConfig, ProducerRecord } import java.util.Arrays;object ConsumerExample extends App { val props = new Propertie...

 scala                     1 answers                     8 view
 2018-10-22         Nina

Alpakka XML content between tags

Alpakka XML processing flow allows to read xml file element by element. But how to extract data between particular StartElement and EndElement including StartElement data? subslice is not an option because there is no constant prefix for needed elements. There is no operator like that out of the box, but you can write your own logic that filters out unnecessary elements that are emitted from XmlParsing.parser.Drawing inspiration from subslice implementation can be a good start. [XXX]

 scala                     1 answers                     11 view
 2018-10-22         Natalie

Spark Scala FPGrowth without any results?

I'm trying to get some frequent item sets and assocation rules out of Spark MLLib using Scala. But actually I don't get anything, not even an error.The code (a spark/databricks notebook) and the data input file could be found here.Either the algorithm does not find any frequent item sets and/or association rules, but there is some evidence that this is wrong. I mostly did the same with KNIME (a non programming analytics platform) but using the Borgelt algorithm for association rule learning. There I get the antecedent, consequent mapping with lift and all the other require...

 scala                     1 answers                     16 view
 2018-10-22         Wright

Serializing generic avro records as an Array[Byte] keeps the schema in the object

SituationI'm currently writing a consumer/producer using AVRO and a schema repository.From what I gather My options for serializing this data is either use the Confluent's avro serializer, or go with Twitter's Bijection.It seemed Bijection looked the most straightforward.So I want to produce date in the following format ProducerRecord[String,Array[Byte]], this comes down to [some string ID, serialized GenericRecord] (note: I'm going for Generic records as this codebase has to handle thousands of schema's that get parsed from Json/csv/...)Question:The whole reason I serializ...

 scala                     1 answers                     17 view
 2018-10-22         Dale

How can you create a partition on a Kafka topic using Samza?

I have a few Samza jobs running all reading messages off of a Kafka topic and writing a new message to a new topic. To send the new messages, I am using Samza's built in OutgoingMessageEnvelope. Also using a MessageCollector to send out the new message. It looks something like this:collector.send(new OutgoingMessageEnvelope(SystemStream, newMessage))Is there a way I can use this to add partitions to the Kafka topic? Such as partitioning on a user ID or something like that.Or if there is a better way I would love to hear it! You should be able to send messages using a par...

 scala                     1 answers                     18 view
 2018-10-22         Rodney

Using message bus as replacement for regular message passing between actors (e.g., in scala)

I have a Java web-service that I am going to reimplement from scratch in Scala. I have an actor-based design for the new code, with around 10-20 actors. One of the use-cases has a flow like this: Actor A gets a message a, creates tens of b messages to be handled by Actor B (possibly multiple instances, for load balancing), producing multiple c messages for Actor C, and so on.In the scenario above, one message a could lead to a few thousand messages being sent back and forth, but I don't expect more than a handful of a messages a day (yes, it is not a busy service at the mo...

 scala                     1 answers                     21 view
 2018-10-22         Tyrone

Deserialising Avro formatted data from Kafka in Spark Streaming gives empty String and 0 for long

I'm struggling to deserialise Avro serialised data coming off Kafka in Spark Streaming. This is the file I am running through spark-submit:package com.example.mymessageimport org.apache.avro.Schemaimport org.apache.avro.generic.{GenericDatumReader, GenericRecord}import org.apache.avro.io.DecoderFactoryimport org.apache.log4j.{Level, Logger}import org.apache.spark.{Logging, SparkConf}import org.apache.spark.streaming._import org.apache.spark.streaming.kafka._object MyMessageCount extends Logging { def main(args: Array[String]) { if (args.length < 4) { System.err....

 scala                     1 answers                     63 view
 2018-10-22         Curitis

No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig)

I am trying to connect Apache Kafka to Apache Flink by creating a Kafka Consumer in Scala. This is my Consumer code:val properties = new Properties()properties.setProperty("bootstrap.servers", "localhost:9092")properties.setProperty("zookeeper.connect", "localhost:2181")properties.setProperty("group.id", "test")val env = StreamExecutionEnvironment.getExecutionEnvironmentval stream = env .addSource(new FlinkKafkaConsumer09[String]("test", new SimpleStringSchema(), properties)) .printenv.enableCheckpointing(5000)When I run this program I get some warnin...

 scala                     1 answers                     69 view
 2018-10-22         Dylan

Scala infix notation for generics

Can generics be used in Scala infix notations?For example:// Example from play-jsonJson.obj("name" -> "Joe") \ "name" as[String] // Generics in infix notation // error: type application is not allowed for postfix operatorsIdeally, I would like to achieve API like:Json.obj("name" -> "Joe") \ "name" as String // But I think its implossibleOf course, using standard dot notation with parentheses it works fine:(Json.obj("name" -> "Joe") \ "name").as[String]The only worthy discussion that I found: https://groups.google.com/for...

 scala                     1 answers                     14 view
 2018-10-22         Crystal

Play framework custom headers ignored

I have this very simple Play controller:@Singletonclass Application @Inject()(cc: ControllerComponents) extends AbstractController(cc) { def index = Action { Ok(views.html.index(SharedMessages.itWorks)) .withHeaders("Content-Security-Policy" -> "script-src 'unsafe-eval'") }}But the added header is ignored. The Content Security Policy in the rendered page is the default one:Content-Security-Policy: default-src 'self'Why is that? Do you have Play!'s security filter enabled? In that case you have to set the CSP header in the application.conf configuration file ...

 scala                     1 answers                     38 view
 2018-10-22         Leila

Unable to Send Spark Data Frame to Kafka (java.lang.ClassNotFoundException: Failed to find data source: kafka.)

I am facing issue while pushing data to Kafka with Spark data frame.Let me explain my scenario in detail with sample example. I want to load the data to spark and send the spark output to kafka. I am using Gradle 3.5 and Spark 2.3.1 & Kafka 1.0.1Here is build.gradlebuildscript {ext { springBootVersion = '1.5.15.RELEASE'}repositories { mavenCentral()}dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}") } }apply plugin: 'scala'apply plugin: 'java'apply plugin: 'eclipse'apply plugin: 'org.springframework.boot'group =...

 scala                     1 answers                     37 view
 2018-10-22         Ethel

Install scala source without change pom

I can install java source from command line: mvn source:jar installSo how can I install scala source only with command? I think this answer applies to your question.Scala is a dependency of maven project, so you can fetch it bymvn dependency:sourcesTo install jar with sources without updating pom.xmlmvn source:jar install [XXX]

 scala                     1 answers                     26 view
 2018-10-22         Mike

Unable to authenticate cassandra cluster through spark scala program

Please suggest me to solve the below issue, or suggest me any different approach to achieve my problem statement.I am getting data from somewhere and inserting it into cassandra daily basis then I need to retrieve the data from cassandra for whole week and do some processing and insert result back onto cassandra.i have lot of records, each record executing most of the below operations. According to my previous post Repreparing preparedstatement warning suggestion, to avoid repreparing the prepared statement,tried to keep a map of query string vs prepared statements.I tried ...

 scala                     3 answers                     14 view
 2018-10-22         Les

Spark Processing file with different structure

My file contains multiple rows that have different structure. Each column is recognized by position depending on the type of row.For example, we could have a file like this:row_type1 first_name1 last_name1 info1 info2row_type2 last_name1 first_name1 info3 info2row_type3info4info1last_name1first_name1 We know the position of every column for every row type, we can use substring to get them.The target dataframe will be "first_name1,last_name1,info1,info2,info3,info4) with no duplicated (first_name1,last_name1)The info1 for example is duplicated in the first and 3rd row....

 scala                     1 answers                     14 view
 2018-10-22         Ted

Spark-rdd manupulating data

I have a sample data like below :UserId,ProductId,Category,Action1,111,Electronics,Browse2,112,Fashion,Click3,113,Kids,AddtoCart4,114,Food,Purchase5,115,Books,Logout6,114,Food,Click7,113,Kids,AddtoCart8,115,Books,Purchase9,111,Electronics,Click10,112,Fashion,Purchase3,112,Fashion,ClickI need to generate list of users who are interested in either “Fashion” category or “Electronics” category but not in both categories. User is interested if he/she has performed any of these actions (Click / AddToCart / Purchase) using spark/scala code I have done up till below :val rrd1 = sc....

 scala                     1 answers                     20 view
 2018-10-22         Lena

How to update dataframe column in Spark Scala after join?

Join of two dataframes results into almost 60 columns. Most of them suppose to stay as is, but some require update based on values in other columns. Is there a way to update those columns w/o calculating new, removing the originals and renaming the calculated back?Simplified example: the revenue in $"Sales column from the left dataframe is supposed to be weighted by the $"Weight in the join results. Is there an efficient way to make the calculation w/o generating the $"SalesWeighted as a new column, dropping the original $Sales and re-naming $SalesWeighted into $Sales?val l...

 scala                     2 answers                     20 view

Page 1 of 478  |  Show More Pages:  Top Prev Next Last