Question

我跟踪了官方文件在本地视窗11的机器上设置了阿帕奇公园

这一设置包括:

Proper installation of Apache Spark, setting up the env variables etc.
Creation of a virtual env specifically for Python 3.9 to ensure compatibility with PySpark.

Despite these steps, I m encountering a ShowString error in VS Code: While I can initiate a Spark session successfully and it starts without errors, I run into problems when trying to use df.show() to display DataFrame contents. The method fails and returns a ShowString error.

不知道目前版本的 java17 space3.5 是否支持 Win11 的表演。

但任何建议都受到高度赞赏:)

[enter image description here](https://i.sstatic.net/2fBDWU3M.png)
[enter image description here](https://i.sstatic.net/3mjFNGlD.png)
[enter image description here](https://i.sstatic.net/gTm3ecIz.png)

我尝试过多个调试步骤- 验证当前信封变量是否被指向, 并确保火花会话启动。

Answer 1

错误消息 :

--------------------------------------------------------------------------- Py4JJavaError                             Traceback (most recent call last)
Cell In[45], line 2
      1 df = spark.createDataFrame([(1,  Alice ), (2,  Bob )], [ id ,  name ])
----> 2 df.show()

File c:UsersDocumentspyspark_venvlibsite-packagespysparksqldataframe.py:945, in DataFrame.show(self, n, truncate, vertical)
    885 def show(self, n: int = 20, truncate: Union[bool, int] = True, vertical: bool = False) -> None:
    886     """Prints the first ``n`` rows to the console.
    887 
    888     .. versionadded:: 1.3.0
   (...)
    943     name | Bob
    944     """
--> 945     print(self._show_string(n, truncate, vertical))

File c:UsersDocumentspyspark_venvlibsite-packagespysparksqldataframe.py:963, in DataFrame._show_string(self, n, truncate, vertical)
    957     raise PySparkTypeError(
    958         error_class="NOT_BOOL",
    959         message_parameters={"arg_name": "vertical", "arg_type": type(vertical).__name__},
    960     )
    962 if isinstance(truncate, bool) and truncate:
--> 963     return self._jdf.showString(n, 20, vertical)
    964 else:
    965     try:
...
    at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:499)
    at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:158)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
    ... 34 more

Answer 2

并意识到SparkEnv基本上是在寻找一个 python3 可执行文件

creating an alias of python3.exe using python.exe and explicitly specifying in the python path helped.

import os
from pyspark.sql import SparkSession

# Define the Python executable path or alias
python_path = r C:Users\AppDataLocalProgramsPythonPython39python3.exe   
os.environ[ PYSPARK_PYTHON ] = python_path
os.environ[ PYSPARK_DRIVER_PYTHON ] = python_path

spark = SparkSession.builder 
    .appName("Your App Name") 
    .config("spark.python.worker.exec", python_path) 
    .getOrCreate()

df = spark.createDataFrame([(1,  Alice ), (2,  Bob )], [ id ,  name ])
df.show()

+---+-----+
| id| name|
+---+-----+
|  1|Alice|
|  2|  Bob|
+---+-----+

友情链接