从MySQL复制到MongoDB时强制关闭连接

I've written a script that transforms my data from MySQL to MongoDB. During the handling of a table with 4,000,000 rows I got (when it was almost done):

我编写了一个脚本,将我的数据从MySQL转换为MongoDB。在处理一个有4,000,000行的表时我得到了(当它差不多完成时):

Traceback (most recent call last):
  File "C:\Python32\lib\site-packages\pymongo\connection.py", line 822, in _send_message
    sock_info.sock.sendall(data)
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kolibri_to_mongo.py", line 94, in <module>
    coll.update(..., upsert=True)
  File "C:\Python32\lib\site-packages\pymongo\collection.py", line 411, in update
    _check_keys, self.__uuid_subtype), safe)
  File "C:\Python32\lib\site-packages\pymongo\connection.py", line 837, in _send_message
    raise AutoReconnect(str(e))
pymongo.errors.AutoReconnect: [Errno 10054] An existing connection was forcibly closed by the remote host
Exception mysql.connector.errors.InternalError: InternalError() in <bound method SqlConn.__del__ of SQLConn(?)> ignored

Is that a PyMongo Error or an SQL Error? Can I check for any limits (size or timeout) on the MySQL or MongoDB side? Or did just someone kill my query?

这是PyMongo错误还是SQL错误?我可以检查MySQL或MongoDB端的任何限制(大小或超时)吗?或者只是有人杀了我的查询?

EDIT: I've notice that now I cannot connect to the MongoDB anymore with a timeout error :( Are there any limits in MongoDB that need to be changed is it more likely to be another IT/Hardware problem?

编辑:我注意到现在我无法再通过超时错误连接到MongoDB :( MongoDB中是否有任何需要更改的限制是否更有可能是另一个IT /硬件问题?

2 个解决方案

#1

That error is coming from MySQL, however, as to why is a bit of an unknown. The error appears to indicate that the remote end, in this instance, MongoDB closed the connection.

然而,这个错误来自MySQL,至于为什么有点未知。该错误似乎表明远程端,在这种情况下,MongoDB关闭了连接。

I would recommend looking in the pymongo logs to see if there's any further information there.

我建议查看pymongo日志,看看那里是否有任何进一步的信息。

I would also run tcpdump on the MySQL side to see if the MongoDB server is rejected the connection attempts with a RST or if the SYNs are leaving the MySQL server and simply being ignored (for the latter, you should see the syns spaced out accordingly based on the retransmission timer, e.g. Attempt 1 @ 0s, 2 @ +3s, 3 @ +9s, 4 @ +21s).

我也会在MySQL端运行tcpdump来查看MongoDB服务器是否被拒绝连接尝试使用RST,或者SYN是否正在离开MySQL服务器而只是被忽略(对于后者,你应该看到syns间隔相应的在重传定时器上,例如尝试1 @ 0s,2 + 3s,3 + 9s,4 + 21s)。

On the MongoDB server, does netstat -an | grep LIST or sudo lsof -c mongod show that MongoDB is still listening on port 27017 (assuming you haven't changed the default)?

在MongoDB服务器上,netstat -an | grep LIST或sudo lsof -c mongod显示MongoDB仍在侦听端口27017(假设你没有更改默认值)?

With regard to MongoDB connections errors, the classic case is where ulimit settings are too low and the server runs out of file descriptors. Here are two good links for you to read:

关于MongoDB连接错误,经典案例是ulimit设置太低而服务器用完了文件描述符。以下是两个很好的链接供您阅读:

#2

This is a disconnect on the MySQL server - I wonder if you are hitting a query timeout or a deadlock.

这是MySQL服务器上的断开连接 - 我想知道你是否遇到了查询超时或死锁。

Do you have any slow queries or long queries or are you getting any errors in your MySQL logs?

您有任何慢查询或长查询,或者您的MySQL日志中有任何错误?

#1