使用以下代码创建符号链接而不是使用ln命令:
from pyspark.ml.util import *
JavaWrapper = SparkContext._jvm.JavaWrapper
JavaClass = SparkContext._jvm.JavaClass
def createSymlink(target, link):
hadoop = SparkContext._jvm.org.apache.hadoop
fs = hadoop.fs.FileSystem
conf = hadoop.conf.Configuration()
uri = hadoop.fs.Path(target).toUri()
tgtFs = fs.get(uri, conf)
linkPath = JavaWrapper("org.apache.hadoop.fs.Path")(link)
tgtPath = JavaWrapper("org.apache.hadoop.fs.Path")(target)
tgtFs.createSymlink(tgtPath, linkPath, True)
然后使用以下代码调用方便的'createSymlink”函数来创建符号链接:
createSymlink("/mnt/source", "/dbfs/mnt/symlink")