A headline figure, not a decision
The order of magnitude of $1,296 per month for a Glue job launched every 5 minutes circulates as an argument from authority. Yet it is not enough to decide. Cost depends first on the type of job, its version, the number of allocated DPUs and the actually billable duration. Without these four parameters, the figure remains one hypothesis among others.
Presenting a single amount as a conclusion amounts to hiding the variables that produce it. A Spark job does not have the same billing behavior as a job of another nature, and a recent version does not apply the same rules as an earlier version. The first precaution is therefore to treat the figure as a starting point, not as a verdict.
The billing minimum is not universal
AWS announced on August 10, 2020 a billing minimum reduced from 10 to 1 minute for Glue 2.0 Spark jobs. Presenting 10 minutes as a universal minimum is therefore misleading. The applicable floor varies by version and job type. The first check is to record this minimum on the configuration concerned, not to assume it.
This distinction changes the order of magnitude. A billable duration of one minute and a billable duration of ten minutes do not produce the same cost for the same number of executions. The debate about the engine or the alternative loses part of its meaning as long as this parameter is not established.
Illustrative example: 2 DPUs, 1 billable minute
Let us take an illustrative example, with 2 DPUs, an assumed rate of $0.44 per DPU-hour and a billable duration of one minute per execution. Over 30 days, a trigger every 5 minutes represents 8,640 executions. The unit cost is calculated as follows: 2 × 1/60 × $0.44, or about $0.01467. The total reaches $126.72 per month.
If the billable duration is 10 minutes, the same assumptions give $1,267.20. One detail matters: rounding each execution before multiplying distorts the total. Over 8,640 lines, the rounding difference accumulates and can shift the result appreciably. It is better to keep precision until the total, then round once.
What these simulations leave out
These amounts are calculation simulations. They exclude storage, logs, transfers and other associated services. The regional rate and the actually billed duration must be recorded on your configuration. In other words, the full cost of a job is not read from a single generic figure. It is reconstructed from observed billing, service by service.
A Glue job can also entail indirect costs: orchestration, monitoring, intermediate storage, recovery after incidents. These items do not appear in the DPU-hour calculation. Ignoring them amounts to comparing different scopes, which distorts the trade-off between several options.
Four checks before changing engines
Before considering a change of engine, four points deserve examination. First, check the job type, its version, the number of workers and the billable duration. Next, count the triggers that occur without new data to process. Then test grouping processing according to the freshness expected by use cases. Finally, compare the full cost, maintenance and lead time with the available alternatives.
These steps do not designate a single solution: they inform the trade-off. They also assume reliable measurements are available. Without a record of the billed duration, any comparison remains theoretical. A change of engine may be justified, but it must rely on figures specific to the environment concerned, not on a general order of magnitude.
Frequency is often the real lever
The most useful question may not be the price of the engine, but the number of frequent jobs that start without useful data to process. If this number is high, the first lever is not necessarily changing the engine: it may lie in the trigger frequency, the launch conditions or the grouping of processing.
A frequency aligned with the freshness actually expected often costs less than heavy technical optimization. Conversely, reducing frequency without checking the need can degrade the service delivered. The calculation and the use case must be examined together. The right question is not only how much a job every 5 minutes costs, but how many of these executions actually serve a purpose.
The post behind this insight
Expanded from the LinkedIn post. The links below come from the original post; listing them does not imply independent verification.
LinkedIn ↗