如何获取Dataflow Python进程的线程转储？

When a Dataflow worker becomes stuck it would be helpful to be able to get the thread dumps of Python process that the worker is having trouble with.

当Dataflow工作程序陷入困境时，能够获得工作程序遇到问题的Python进程的线程转储将会很有帮助。

How would I identify and obtain the threadz dump of a stuck Python process?

我如何识别并获取卡住的Python进程的threadz转储？

1 个解决方案

#1

SSH into the worker machine.

SSH进入工作机器。

Run docker ps. Example output may look like:

运行docker ps。示例输出可能如下所示：

CONTAINER ID   IMAGE                                ... NAMES
...
0649a3f76a4a   dataflow.gcr.io/v1beta3/python:2.4.0 ... k8s_python.b30867f0_dataflow-wordcount-ma-07022224-6ad4-harness-r7cy_default_a9d4faf05c7e404114002f06ca702ecc_6e2cabdf
...

Note the CONTAINER ID for the container named ...python....

请注意名为... python ....的容器的CONTAINER ID。

Log into the container with docker exec -it $CONTAINER_ID bash.

使用docker exec -it $ CONTAINER_ID bash登录容器。
Find the threadz port for the worker process in the worker logs with:

在工作日志中找到工作进程的threadz端口：

cat /var/log/dataflow/python-dataflow-*-json.log | grep 'Status HTTP server'.

cat /var/log/dataflow/python-dataflow-*-json.log | grep'状态HTTP服务器'。

Example output may look like:

示例输出可能如下所示：
```
{"severity": "INFO",..., "message": "Status HTTP server running at localhost:37735", "logger": "root:batchworker.py:status_server"}
```
Note that there may be multiple worker processes running, each having their own threadz port.

请注意，可能有多个工作进程在运行，每个进程都有自己的threadz端口。
Sample threadz content with curl localhost:$PORT.

使用curl localhost示例threadz内容：$ PORT。

#1

SSH into the worker machine.

SSH进入工作机器。