-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(aws-stepfunctions-tasks): GlueStartJobRunProps missing support of AllocatedCapacity or MaxCapacity #12757
Comments
The documentation is outdated. There are a few new parameters introduced by This is related to: awsdocs/aws-step-functions-developer-guide#46 |
I understood that we can use |
Hi any update? |
Hi I'm running into this as well; I really need to increase the number of workers for certain jobs. As far as I can tell, there is no way to drop down to the |
@cprice404-aws have you found any workaround for this? |
Unfortunately I think the (awful) solution that I came up with was to set the default number of workers to the highest number I might need, when I defined the job. When invoking the job via other means besides step functions I pass in an override to reduce the number. Not a very good approach but I didn't find any other way to do it. |
@shivlaks any update on this? |
A workaround has been given by the Glue team. Since With that in mind, I am changing this to a Feature Request and am investigating what it would take to add those fields as we speak. 😸 😷 |
the documentation for the service pattern needs to be updated as well. @NGL321 - the main question to answer re: adding support is whether the new properties can be specified as part of the state input path (dynamically rather than static allocation to a type). That is what will determine whether I'm happy to take a look. Going to have to try this out myself since it seems we have some documentation gaps here. @johnhill2424 Another workaround would be to bypass the use of the |
@shivlaks I think you meant @johnhill2424 |
@joehillen thanks for that! corrected. |
update: checking with the Step Functions team whether they're aware of these changes given that it's still not documented Remaining
|
Do we have any updates on this? |
Bumping this issue- there are a bunch of parameters exposed in the JobRun structure that aren't exposed in CDK L2 constructs. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html Here is a custom construct my team uses to customize import { Construct } from 'constructs';
import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
export interface RetryProps {
maxAttempts: number;
backoffRate: number;
}
export interface GlueStartJobRunProps {
name: string;
arguments: { [name: string]: string };
retry: RetryProps;
numWorkers: string;
workerType: string;
}
/**
* Custom construct for a glue job. A custom construct is required,
* because fn.GlueStartJobRun is missing key features, like the
* ability to configure NumberOfWorkers and WorkerType.
*/
export class GlueStartJobRun extends sfn.CustomState {
constructor(scope: Construct, id: string, props: GlueStartJobRunProps) {
let stateJson = {
Type: 'Task',
ResultPath: null,
Retry: [
{
ErrorEquals: ['States.ALL'],
MaxAttempts: props.retry.maxAttempts,
BackoffRate: props.retry.backoffRate,
},
],
Resource: 'arn:aws:states:::glue:startJobRun.sync',
Parameters: {
'JobName': props.name,
'Arguments': props.arguments,
'NumberOfWorkers.$': props.numWorkers,
'WorkerType.$': props.workerType,
},
};
super(scope, id, { stateJson: stateJson });
}
} |
Is there any update for this issue? Here is a workaround we are using now. export interface GlueStartJobRunProps extends tasks.GlueStartJobRunProps {
readonly numberOfWorkers?: number;
readonly workerType?: WorkerType;
}
export class GlueStartJobRun extends tasks.GlueStartJobRun {
constructor(scope: Construct, id: string, private readonly enhancedProps: GlueStartJobRunProps) {
super(scope, id, enhancedProps);
}
protected override _renderTask(): any {
const state = super._renderTask();
if (this.enhancedProps.workerType) {
state.Parameters.WorkerType = this.enhancedProps.workerType.name;
}
if (this.enhancedProps.numberOfWorkers) {
state.Parameters.NumberOfWorkers = this.enhancedProps.numberOfWorkers;
}
return state;
}
} |
Any updates on this ? |
…StartJobRun class (#30319) ### Issue # (if applicable) Closes #12757. ### Reason for this change Missing property ### Description of changes Add workerType and numberOfWorkers to GlueStartJobRun class. The reasons for this change are as follows: * AllocatedCapacity is deprecated. * MaxCapacity can only be used with Glue version 1 and earlier, which have already reached end of support (EOS). * Glue version 2 and later use WorkerType and NumberOfWorkers. For mor information, see also the documents below. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html#aws-glue-api-jobs-runs-StartJobRun https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html ### Description of how you validated changes Add unit tests and integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
…StartJobRun class (aws#30319) ### Issue # (if applicable) Closes aws#12757. ### Reason for this change Missing property ### Description of changes Add workerType and numberOfWorkers to GlueStartJobRun class. The reasons for this change are as follows: * AllocatedCapacity is deprecated. * MaxCapacity can only be used with Glue version 1 and earlier, which have already reached end of support (EOS). * Glue version 2 and later use WorkerType and NumberOfWorkers. For mor information, see also the documents below. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html#aws-glue-api-jobs-runs-StartJobRun https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html ### Description of how you validated changes Add unit tests and integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
…StartJobRun class (aws#30319) ### Issue # (if applicable) Closes aws#12757. ### Reason for this change Missing property ### Description of changes Add workerType and numberOfWorkers to GlueStartJobRun class. The reasons for this change are as follows: * AllocatedCapacity is deprecated. * MaxCapacity can only be used with Glue version 1 and earlier, which have already reached end of support (EOS). * Glue version 2 and later use WorkerType and NumberOfWorkers. For mor information, see also the documents below. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html#aws-glue-api-jobs-runs-StartJobRun https://docs.aws.amazon.com/glue/latest/dg/glue-version-support-policy.html ### Description of how you validated changes Add unit tests and integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
❓ General Issue
The Question
Today in CDK
GlueStartJobRunProps
does not supportAllocatedCapacity
orMaxCapacity
to configure Glue job run DPU. But based on Step Functions service integration, SFn does support parameterAllocatedCapacity
when invoking GlueStartJobRun
API. WillGlueStartJobRunProps
support acceptingAllocatedCapacity
orMaxCapacity
parameters? (see details of these two parameters here)Environment
Other information
The text was updated successfully, but these errors were encountered: