这两天在执行hadoop streaming (用的脚本是python),一直出现subprocess failed with code 这个错误,看了streaming 源码,上面写的是系统抛出的异常,但是怎么也搞不明白为什么系统会抛出异常呢?百思不得其解啊!看了http://taoo.iteye.com/blog/1486650这个博客给我了一点提醒,原来是python问题,集群中环境不一致导致的。
把这个问题记下来吧,以备以后查阅。
hadoop streaming pyhton 时候出现:subprocess failed with code 127,code 后面错误码不管是什么,都是 脚本环境的问题
在python脚本里面声明的:#!/usr/bin/env python,这里声明了位置,所以必须要放到 /usr/bin 下面,才能有效。
python 执行程序必须放到 /usr/bin 下面,在环境变量的其他位置不管用。
/usr/bin/env: python: No such file or directory
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
这个主要是环境存在的问题,code后面的错误不管是几。都是streaming中执行某个脚本导致的。
分享到:
相关推荐
Building Data Streaming Applications with Apache Kafka_Code 源码 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
Building Data Streaming Applications with Apache Kafka 英文azw3 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system...using Apache Kafka and to design efficient streaming data applications with it.
Streaming XML with xmpp.pdf
spark Streaming和structed streaming分析,理解整个 Spark Streaming 的模块划分和代码逻辑。
Hadoop Streaming 官方中文文档
保证完整,目前网上找个完整的不容易(2019-06-...Stream Processing with Apache Flink(Flink布道者)完整书签高清pdf以及epub; 以及评价超高的Streaming Systems(谷歌大脑Tyler Akidau) ; 以及18年10月Flink技术沙
spark2018欧洲峰会中关于StructuredStreaming中stateful stream processing的ppt
With this practical guide, data engineers, data scientists, and developers will learn how to work with streaming data in a conceptual and platform-agnostic way. Expanded from Tyler Akidau’s popular ...
In the same way, streaming data applications can accomplish amazing tasks like reading live location data to recommend nearby services, tracking faults with machinery in real time, and sending ...
Spark核心概念简介: Spark使用maven进行打包(减少jar包大小): Spark中的(弹性分布式数据集)简称RDD: ...SparkStreaming中的正常操作(每读2秒就计算一次): Spark中的local[2]: Spark中的处理流程图像:
Pro Spark Streaming walks you through end-to-end real-time application development using real-world applications, data, and code. Taking an application-first approach, each chapter introduces use ...
With this practical guide, data engineers, data scientists, and developers will learn how to work with streaming data in a conceptual and platform-agnostic way. Expanded from Tyler Akidau’s popular...
ngwsx-0.8.39 with h264 streaming module. features: *) native port of nginx on windows. *) single process mode. *) windows service. *) iocp event method. *) file aio. *) zlib-1.2.3. (statically link...
赠送jar包:flink-streaming-java_2.11-1.13.2.jar; 赠送原API文档:flink-streaming-java_2.11-1.13.2-javadoc.jar; 赠送源代码:flink-streaming-java_2.11-1.13.2-sources.jar; 赠送Maven依赖信息文件:flink-...
Spark streaming tutorial, with advanced analysis.!
Example Playlist Files for use with HTTP Live Streaming.doc HLS_Sample_Encryption.pdf HTTP_Live_Streaming_Metadata_Spec.pdf StreamingMediaGuide.pdf Understanding the EXT-X-VERSION tag.doc Using_...
spark streaming streaming
英文版的、实时的Real-Time Analytics with Spark Streaming
Structured Spark Streaming-as-a-Service with Hopsworks