pyspark使用pandas_udf时的一个坑

Jsong_

在学习pyspark [size=1.1em]pandas_udf时
使用了pyspark官方文档给的代码

发生了如下报错：
19/11/14 15:59:36 ERROR TaskSetManager: Task 44 in stage 10.0 failed 1 times; aborting job
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/dataframe.py", line 380, in show
    print(self._jdf.showString(n, 20, vertical))
  File "/opt/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/opt/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/opt/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o64.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 44 in stage 10.0 failed 1 times, most recent failure: Lost task 44.0 in stage 10.0 (TID 133, localhost, executor driver): java.lang.IllegalArgumentException
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
        at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:543)
        at org.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:58)
        at org.apache.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:132)
        at org.apache.arrow.vector.ipc.ArrowReader.initialize(ArrowReader.java:181)
..........................

一开始以为是JDK的版本过高，换成了官方推荐的JDK8，结果还是报错
#￥%……&#￥
最后的最后才找到是：pyarrow版本过高
重新安装pyarrow 0.14.1或者更低版本就可以解决这个问题了

jiangpeng59

楼主这么热血的么？  怎么把大数据相关的帖子发在这里？

Jsong_

jiangpeng59 发表于 2019-11-14 19:08
楼主这么热血的么？  怎么把大数据相关的帖子发在这里？

上次注册的号被封了，来52是想混逆向区的，可最近要做这个相关的东西，没空去那边，又不想号被封，就来这儿发发踩过的坑，万一有人来52检索相关资料呢对吧（其实主要是想保住这次的号( &#8226;&#768; ω &#8226;&#769; )&#10023;）

wjkanm

谢谢楼主，刚好也准备学习pyspark