2022-12-21

How to measure element processing time by apache beam pipeline?

I have a beam pipeline coded in python and running on Flink, which reads from kafka topic, applies few Do.Fn transformations and writes back to another kafka topic. What would be a reasonable approach to measure end-to-end processing time of one element in apache beam pipeline?

Does it make sense apply MeasureTime(beam.DoFn) on the first and last transformation and count the difference or there's a better way.

https://github.com/apache/beam/blob/1988284a89b10b60eea48325f8a3b370b551c77c/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py#L406

Thanks



No comments:

Post a Comment