Question

如果工作流动没有结束,我怎么能用oto子来补充等待亚马孙婴儿死亡率的工作流动?

I ve在亚马孙弹性地图上建立了互动工作流程。当我用Boto s emr_conn.add_jobflow_pans(......)将新步骤转至工作流程时,工作流程在完成或失败后即告终止。

我知道,我可以开始使用<代码>run_jobflow和_keep_alive para amount——的博托来进行工作流程,但我要与已经运行的流动量合作。

Answer 1

如果正确填写,则不应以<代码>keep_alive=True终止。尽管如此,它通常会因失败而退出,因此,请在<条码>上添加<>终止_on_failure=”CONTINUE>。

Answer 2

我也这样做。

生产

import boto.emr

conn = boto.emr.connect_to_region( us-west-2 )
jobid = conn.run_jobflow(name= cluster-name ,
                     ec2_keyname="yourkeyhere",
                     num_instances=3,
                     master_instance_type= m1.medium ,
                     slave_instance_type= m1.medium ,
                     keep_alive=True,
)

在现有组别中增加一项工作(对于组群在等候状态中几乎没有工作)

import boto.emr

conn = boto.emr.connect_to_region( us-west-2 )
# get the list of waiting cluster and take the first one
jobid =  conn.describe_jobflows(states=["WAITING"])[0].jobflowid
print jobid
flow_steps = list()
runThing = boto.emr.step.ScriptRunnerStep(
                        name="job step name",
                        step_args = ["s3://yours3bucket/dosmthg.sh"])
flow_steps.append(runThing)
conn.add_jobflow_steps(jobid, flow_steps)

notes

you need to have ~/.aws/credentials filled up (aws configure)
amazon region us-west-2 currently has the more recent ami version
you can may have to add bootstrap_actions= if you need hive, pig or custom installation steps

Answer 3

你们也可以这样做。

response = emr.run_job_flow(
        Name="start-my-cluster",
        ReleaseLabel="emr-5.3.1",
        LogUri= s3://logs ,
        Instances={
             InstanceGroups : [
                { Name :  EmrMaster ,
                  InstanceRole :  MASTER ,
                  InstanceType :  m3.xlarge ,
                  InstanceCount : 1},
                { Name :  EmrCore ,
                  InstanceRole :  CORE ,
                  InstanceType :  m3.xlarge ,
                  InstanceCount : 2}
            ],
             Ec2KeyName :  my-key-name ,
             KeepJobFlowAliveWhenNoSteps  : True,
        },
        Applications=[{ Name :  Hadoop }, { Name :  Spark }, { Name :  Hive }],
        JobFlowRole= EMR_EC2_DefaultRole ,
        ServiceRole= EMR_DefaultRole ,
        VisibleToAllUsers=True,
        Steps=[
            # steps go here...
        ]
        )

友情链接