Pyspark stores RDD into mysql

Simply put: Storing RDD in mysql requires converting RDD to Dataframe, and then storing dataframe in mysql

Here are some examples of how to configure this. Without further elaboration, first you need to have an RDD. My RDD looks like this

Then turn the RDD into a Dataframe

from pyspark.sql import Row
emp = rdd.map(lambda p : Row(url=p))
df = spark.createDataFrame(emp)
Copy the code

Finally, store the dataframe in mysql

Url = "JDBC: mysql: / / 192.168.14.90:3306 / hy88? user=spider&password=111111" df.write.jdbc(url=url,mode="append",table="com_url",properties={"driver":"com.mysql.jdbc.Driver"})Copy the code