多模态搜索 - milvus 文本相似性搜索 - 《人工智能》

文本相似度推荐参考
图片.png
2021-07-09 12:45:38,842 ｜ ERROR ｜ mysql_helpers.py ｜ delete_table ｜ 69 ｜ MYSQL ERROR: (2013, 'Lost connection to MySQL server during query ([Errno 110] Connection timed out)') with sql: drop table if exists text_search;
2021-07-09 12:45:38,849 ｜ ERROR ｜ h11_impl.py ｜ run_asgi ｜ 372 ｜ Exception in ASGI application
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 66, in delete_table
    self.cursor.execute(sql)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 740, in _read_bytes
    "Lost connection to MySQL server during query (%s)" % (e,),
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 110] Connection timed out)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/applications.py", line 201, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 217, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/main.py", line 52, in drop_tables
    status = do_drop(table_name, MILVUS_CLI, MYSQL_CLI)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/operations/drop.py", line 15, in do_drop
    mysql_cli.delete_table(table_name)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 70, in delete_table
    sys.exit(1)
SystemExit: 1
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 66, in delete_table
    self.cursor.execute(sql)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 740, in _read_bytes
    "Lost connection to MySQL server during query (%s)" % (e,),
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 110] Connection timed out)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/applications.py", line 201, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 217, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/main.py", line 52, in drop_tables
    status = do_drop(table_name, MILVUS_CLI, MYSQL_CLI)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/operations/drop.py", line 15, in do_drop
    mysql_cli.delete_table(table_name)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 70, in delete_table
    sys.exit(1)
SystemExit: 1
INFO:     192.168.2.13:35564 - "POST /text/drop?table_name=text_search HTTP/1.1" 500 Internal Server Error
INFO:     192.168.2.13:37796 - "OPTIONS /text/load HTTP/1.1" 200 OK
2021-07-09 12:45:50,028 ｜ ERROR ｜ mysql_helpers.py ｜ load_data_to_mysql ｜ 43 ｜ MYSQL ERROR: (1054, "Unknown column 'title' in 'field list'") with sql: insert into text_search (milvus_id,title,text) values (%s,%s,%s);
2021-07-09 12:45:50,029 ｜ ERROR ｜ h11_impl.py ｜ run_asgi ｜ 372 ｜ Exception in ASGI application
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 39, in load_data_to_mysql
    self.cursor.executemany(sql, data)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 179, in executemany
    self._get_db().encoding,
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 211, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1054, "Unknown column 'title' in 'field list'")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/applications.py", line 201, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 217, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/main.py", line 75, in load_text
    total_num = import_data(table_name, fname_path ,MILVUS_CLI, MYSQL_CLI)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/operations/load.py", line 55, in import_data
    mysql_cli.load_data_to_mysql(collection_name, format_data(ids, title_data, text_data))
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 44, in load_data_to_mysql
    sys.exit(1)
SystemExit: 1
INFO:     192.168.2.13:37796 - "POST /text/load HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 39, in load_data_to_mysql
    self.cursor.executemany(sql, data)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 179, in executemany
    self._get_db().encoding,
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 211, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1054, "Unknown column 'title' in 'field list'")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/applications.py", line 201, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 217, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/home/milvus/pycharm_workspaces/bootcamp/venv/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/main.py", line 75, in load_text
    total_num = import_data(table_name, fname_path ,MILVUS_CLI, MYSQL_CLI)
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/operations/load.py", line 55, in import_data
    mysql_cli.load_data_to_mysql(collection_name, format_data(ids, title_data, text_data))
  File "/home/milvus/pycharm_workspaces/bootcamp/solutions/text_search_engine/quick_deploy/Milvus-bert-server/src/mysql_helpers.py", line 44, in load_data_to_mysql
    sys.exit(1)
SystemExit: 1
文本相似度推荐参考

个性化新闻推荐系统：https://blog.csdn.net/qq_32690999/article/details/77434381
个性化新闻推荐系统-实现：https://github.com/bluemapleman/NewsRecommendSystem
基于内容推荐算法详解(比较全面的文章)：
https://blog.csdn.net/nicajonh/article/details/79657317
参考：
NLP之文本相似度：https://blog.csdn.net/miner_zhu/article/details/81566456
word2vec利用词向量进行中文新闻标题相似度分析：https://blog.csdn.net/li_huifei/article/details/80265046
TF-IDF算法解析与Python实现：https://github.com/CWSchen/ML-PYTHON3-PROJECTS/blob/cc988cbd136ea04b26c946b08c9c3639fdb6558a/NLP/99-Python-TF-IDF%E7%AE%97%E6%B3%95%E8%A7%A3%E6%9E%90.py
基于内容相似度的推荐与TF-IDF算法：https://blog.csdn.net/qq_40006058/article/details/83477866
今日头条的新闻推荐算法原理：https://blog.csdn.net/u010332284/article/details/80415336
文本分类：bert
优质文章：https://leovan.me/cn/2020/10/text-similarity/