在本实验中，您将了解Web代理服务器的工作原理及其基本功能之一 —— 缓存。

您的任务是开发一个能够缓存网页的小型Web代理服务器。这是一个很简单的代理服务器，它只能理解简单的GET请求，但能够处理各种对象 —— 不仅仅是HTML页面，还包括图片。

通常，当客户端发出一个请求时，请求将被直接发送到Web服务器。然后Web服务器处理该请求并将响应消息发送客户端。为了提高性能，我们在客户端和Web服务器之间建立一个代理服务器。现在，客户端发送的请求消息和Web服务器返回的响应消息都要经过代理服务器。换句话说，客户端通过代理服务器请求对象。代理服务器将客户端的请求转发到Web服务器。然后，Web服务器将生成响应消息并将其传递给代理服务器，代理服务器又将其发送给客户端。
♠ 套接字编程作业4：多线程Web代理服务器 - 图1

代码

您将在下面找到客户端的代码框架。 您需要完成代码框架。需要您填写代码的地方标有#Fill in start和#Fill in end。 每个地方都需要填写至少一行代码。

运行代理服务器

使用命令行模式运行您的代理服务器程序，然后从您的浏览器发送一个网页请求，将IP地址和端口号指向代理服务器。 例如：http://localhost:8888/www.google.com 为了在独立的计算机上使用浏览器和代理服务器， 因此，在运行代理服务器时，您需要将“localhost”更换为代理服务器的所在机器的IP地址。您还需要将“8888”替换您在代理服务程序中使用的端口号。

配置浏览器

您还可以直接配置您的Web浏览器以使用您的代理服务。 具体取决于您的浏览器。在Internet Explorer中，您可以在 工具 > Internet选项 > 连接选项卡 > LAN设置 中设置代理。 在Netscape（包括衍生浏览器，如Mozilla）中，您可以在 工具 > 选项 > 高级选项 > 网络选项 > 连接设置 中设置代理。 在这两种情况下你都需要给出代理服务器的地址和端口号。你首先要毫无问题地在同一台计算机上运行代理服务器和浏览器。这种方式下，使用代理服务器获取网页就只需提供页面的URL。 例如 http://www.google.com

要提交的内容

您需要提交提交完整的代理服务器代码和一张客户端屏幕截图，用于验证您是否 确实通过代理服务器获取了网页。

代理服务器的Python代码框架

  1. from socket import *
  2. import sys
  4. if len(sys.argv) <= 1:
  5.     print 'Usage : "python ProxyServer.py server_ip"\n[server_ip : It is the IP Address Of Proxy Server'
  6.     sys.exit(2)
  7. # Create a server socket, bind it to a port and start listening
  8. tcpSerSock = socket(AF_INET, SOCK_STREAM)
  9. # Fill in start.
  10. # Fill in end.
  11. while 1:
  12.     # Strat receiving data from the client
  13.     print 'Ready to serve...'
  14.     tcpCliSock, addr = tcpSerSock.accept()
  15.     print 'Received a connection from:', addr
  16.     message = # Fill in start. # Fill in end.
  17.     print message
  18.     # Extract the filename from the given message
  19.     print message.split()[1]
  20.     filename = message.split()[1].partition("/")[2]
  21.     print filename
  22.     fileExist = "false"
  23.     filetouse = "/" + filename
  24.     print filetouse
  25.     try:
  26.         # Check wether the file exist in the cache
  27.         f = open(filetouse[1:], "r")
  28.         outputdata = f.readlines()
  29.         fileExist = "true"
  30.         # ProxyServer finds a cache hit and generates a response message
  31.         tcpCliSock.send("HTTP/1.0 200 OK\r\n")
  32.         tcpCliSock.send("Content-Type:text/html\r\n")
  33.         # Fill in start.
  34.         # Fill in end.
  35.             print 'Read from cache'
  36.     # Error handling for file not found in cache
  37.     except IOError:
  38.         if fileExist == "false":
  39.             # Create a socket on the proxyserver
  40.             c = # Fill in start. # Fill in end.
  41.             hostn = filename.replace("www.","",1) 
  42.             print hostn
  43.             try:
  44.                 # Connect to the socket to port 80
  45.                 # Fill in start.
  46.                 # Fill in end.
  47.                 # Create a temporary file on this socket and ask port 80
  48.                 for the file requested by the client
  49.                 fileobj = c.makefile('r', 0)
  50.                 fileobj.write("GET "+"http://" + filename + " HTTP/1.0\n\n")
  51.                 # Read the response into buffer
  52.                 # Fill in start.
  53.                 # Fill in end.
  54.                 # Create a new file in the cache for the requested file.
  55.                 # Also send the response in the buffer to client socket and the corresponding file in the cache
  56.                 tmpFile = open("./" + filename,"wb")
  57.                 # Fill in start.
  58.                 # Fill in end.
  59.             except:
  60.                 print "Illegal request"
  61.         else:
  62.             # HTTP response message for file not found
  63.             # Fill in start.
  64.             # Fill in end.
  65.     # Close the client and the server sockets
  66.     tcpCliSock.close()
  67. # Fill in start.
  68. # Fill in end.

可选练习

  1. 目前代理服务器不能处理错误。这可能会导致一些问题，当客户端请求一个不可用的对象时，由于“404 Not Found”响应通常没有响应正文，而代理服务器会假设有正文并尝试读取它。
  2. 当前代理服务器只支持HTTP GET方法。通过添加请求体来增加对POST的支持。
  3. 缓存：每当客户端发出特定请求时，典型的代理服务器会缓存网页。缓存的基本功能如下：当代理获得一个请求时，它将检查请求的对象是否已经在缓存中，如果是，则从缓存返回对象，从而不用联系服务器。如果对象未被缓存，则代理从服务器获取该对象，向客户端返回该对象，并缓存一个拷贝以备将来的请求。在实际环境下，代理服务器必须验证被缓存的响应是否仍然有效，并且它们能对客户端正确响应。您可以在RFC 2068中阅读有关缓存及其在HTTP中实现方式的更多细节。添加上述简单的缓存功能。您不需要实现任何替换或验证策略。然而您需要实现的是，将请求和响应写入磁盘（即缓存）并能从磁盘中获取它们，用于缓存被请求命中时。为此，您需要在代理中实现一些内部数据结构，以便跟踪哪些请求处于缓存中时，以及它们在磁盘上的位置。您也可以将此数据结构保存在内存中，因为没有必要关机之后持续保存这些数据。

    答案

    作业4答案

    ```python

    改为Python3格式

    from socket import * import sys import os

if len(sys.argv) <= 1: print(‘Usage : “python ProxyServer.py server_ip”\n[server_ip : It is the IP Address Of Proxy Server’) sys.exit(2)

Create a server socket, bind it to a port and start listening

tcpSerSock = socket(AF_INET, SOCK_STREAM) tcpSerPort = int(sys.argv[1]) tcpSerSock.bind((“”, tcpSerPort)) print(tcpSerPort) tcpSerSock.listen(10) while 1:

  1. # Strat receiving data from the client
  2. print('Ready to serve...')
  3. tcpCliSock, addr = tcpSerSock.accept()
  4. print('Received a connection from:', addr)
  5. message = tcpCliSock.recv(1024)
  6. message = message.decode()
  7. print("message:", message)
  8. if(message == ''):
  9.     continue
  10. # Extract the filename from the given message
  11. print("message.split()[1]:", message.split()[1])
  12. filename = message.split()[1].partition("/")[2]
  13. print("filename:", filename)
  14. fileExist = "false"
  15. filetouse = "/" + filename
  16. print("filetouse:", filetouse)
  17. try:
  18.     # Check wether the file exist in the cache
  19.     f = open("WEB/" + filetouse[1:], "rb")
  20.     outputdata = f.read()
  21.     f.close()
  22.     fileExist = "true"
  23.     # ProxyServer finds a cache hit and generates a response message
  24.     tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  25.     tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  26.     tcpCliSock.send(outputdata)
  27.     print('Read from cache')
  28. # Error handling for file not found in cache
  29. except IOError:
  30.     if fileExist == "false":
  31.         # Create a socket on the proxyserver
  32.         c = socket(AF_INET, SOCK_STREAM)
  33.         hostn = filename.replace("www.","",1)
  34.         print("hostn:", hostn)
  35.         try:
  36.             # Connect to the socket to port 80
  37.             serverName = hostn.partition("/")[0]
  38.             serverPort = 80
  39.             print((serverName, serverPort))
  40.             c.connect((serverName, serverPort))
  41.             askFile = ''.join(filename.partition('/')[1:])
  42.             print("askFile:", askFile)
  43.             # Create a temporary file on this socket and ask port 80
  44.             # for the file requested by the client
  45.             fileobj = c.makefile('rwb', 0)
  46.             fileobj.write("GET ".encode() + askFile.encode() + " HTTP/1.0\r\nHost: ".encode() + serverName.encode() + "\r\n\r\n".encode())
  47.             # Read the response into buffer
  48.             serverResponse = fileobj.read()
  49.             if serverResponse.split()[0] != b'404':
  50.                 print('404')
  51.                 tcpCliSock.send("HTTP/1.1 404 Not Found\r\n\r\n".encode())
  52.                 tcpCliSock.close()
  53.                 continue
  54.             # Create a new file in the cache for the requested file.
  55.             # Also send the response in the buffer to client socket and the corresponding file in the cache
  56.             filename = "WEB/" + filename
  57.             filesplit = filename.split('/')
  58.             for i in range(0, len(filesplit) - 1):
  59.                 if not os.path.exists("/".join(filesplit[0:i+1])):
  60.                     os.makedirs("/".join(filesplit[0:i+1]))
  61.             tmpFile = open(filename, "wb")
  62.             print(serverResponse)
  63.             serverResponse = serverResponse.split(b'\r\n\r\n')[1]
  64.             print(serverResponse)
  65.             tmpFile.write(serverResponse)
  66.             tmpFile.close()
  67.             tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  68.             tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  69.             tcpCliSock.send(serverResponse)
  70.         except:
  71.             print("Illegal request")
  72.         c.close()
  73.     else:
  74.         # HTTP response message for file not found
  75.         print("NET ERROR")
  76. # Close the client and the server sockets
  77. tcpCliSock.close()

tcpSerSock.close()

  1. <a name="uX7Hu"></a>
  2. ## 可选练习1答案
  3. ```python
  4. #改为Python3格式
  5. from socket import *
  6. import sys
  7. import os
  9. if len(sys.argv) <= 1:
  10.     print('Usage : "python ProxyServer.py server_ip"\n[server_ip : It is the IP Address Of Proxy Server')
  11.     sys.exit(2)
  12. # Create a server socket, bind it to a port and start listening
  13. tcpSerSock = socket(AF_INET, SOCK_STREAM)
  14. tcpSerPort = int(sys.argv[1])
  15. tcpSerSock.bind(("", tcpSerPort))
  16. print(tcpSerPort)
  17. tcpSerSock.listen(10)
  18. while 1:
  19.     # Strat receiving data from the client
  20.     print('Ready to serve...')
  21.     tcpCliSock, addr = tcpSerSock.accept()
  22.     print('Received a connection from:', addr)
  23.     message = tcpCliSock.recv(1024)
  24.     message = message.decode()
  25.     print("message:", message)
  26.     if(message == ''):
  27.         continue
  28.     # Extract the filename from the given message
  29.     print("message.split()[1]:", message.split()[1])
  30.     filename = message.split()[1].partition("/")[2]
  31.     print("filename:", filename)
  32.     fileExist = "false"
  33.     filetouse = "/" + filename
  34.     print("filetouse:", filetouse)
  35.     try:
  36.         # Check wether the file exist in the cache
  37.         f = open("WEB/" + filetouse[1:], "rb")
  38.         outputdata = f.read()
  39.         f.close()
  40.         fileExist = "true"
  41.         # ProxyServer finds a cache hit and generates a response message
  42.         tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  43.         tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  44.         tcpCliSock.send(outputdata)
  45.         print('Read from cache')
  46.     # Error handling for file not found in cache
  47.     except IOError:
  48.         if fileExist == "false":
  49.             # Create a socket on the proxyserver
  50.             c = socket(AF_INET, SOCK_STREAM)
  51.             hostn = filename.replace("www.","",1)
  52.             print("hostn:", hostn)
  53.             try:
  54.                 # Connect to the socket to port 80
  55.                 serverName = hostn.partition("/")[0]
  56.                 serverPort = 80
  57.                 print((serverName, serverPort))
  58.                 c.connect((serverName, serverPort))
  59.                 askFile = ''.join(filename.partition('/')[1:])
  60.                 print("askFile:", askFile)
  61.                 # Create a temporary file on this socket and ask port 80
  62.                 # for the file requested by the client
  63.                 fileobj = c.makefile('rwb', 0)
  64.                 fileobj.write("GET ".encode() + askFile.encode() + " HTTP/1.0\r\nHost: ".encode() + serverName.encode() + "\r\n\r\n".encode())
  65.                 # Read the response into buffer
  66.                 serverResponse = fileobj.read()
  67.                 if serverResponse.split()[0] != b'404':
  68.                     print('404')
  69.                     tcpCliSock.send("HTTP/1.1 404 Not Found\r\n\r\n".encode())
  70.                     tcpCliSock.close()
  71.                     continue
  72.                 # Create a new file in the cache for the requested file.
  73.                 # Also send the response in the buffer to client socket and the corresponding file in the cache
  74.                 filename = "WEB/" + filename
  75.                 filesplit = filename.split('/')
  76.                 for i in range(0, len(filesplit) - 1):
  77.                     if not os.path.exists("/".join(filesplit[0:i+1])):
  78.                         os.makedirs("/".join(filesplit[0:i+1]))
  79.                 tmpFile = open(filename, "wb")
  80.                 print(serverResponse)
  81.                 serverResponse = serverResponse.split(b'\r\n\r\n')[1]
  82.                 print(serverResponse)
  83.                 tmpFile.write(serverResponse)
  84.                 tmpFile.close()
  85.                 tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  86.                 tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  87.                 tcpCliSock.send(serverResponse)
  88.             except:
  89.                 print("Illegal request")
  90.             c.close()
  91.         else:
  92.             # HTTP response message for file not found
  93.             print("NET ERROR")
  94.     # Close the client and the server sockets
  95.     tcpCliSock.close()
  96. tcpSerSock.close()

可选练习2答案

  1. # 找不到网站测试
  2. #改为Python3格式
  3. from socket import *
  4. import sys
  5. import os
  7. if len(sys.argv) <= 1:
  8.     print('Usage : "python ProxyServer.py server_ip"\n[server_ip : It is the IP Address Of Proxy Server')
  9.     sys.exit(2)
  10. # Create a server socket, bind it to a port and start listening
  11. tcpSerSock = socket(AF_INET, SOCK_STREAM)
  12. tcpSerPort = int(sys.argv[1])
  13. tcpSerSock.bind(("", tcpSerPort))
  14. print(tcpSerPort)
  15. tcpSerSock.listen(10)
  16. while 1:
  17.     # Strat receiving data from the client
  18.     print('Ready to serve...')
  19.     tcpCliSock, addr = tcpSerSock.accept()
  20.     print('Received a connection from:', addr)
  21.     message = tcpCliSock.recv(1024)
  22.     message = message.decode()
  23.     print("message:", message)
  24.     if(message == ''):
  25.         continue
  26.     # Extract the filename from the given message
  27.     print("message.split()[1]:", message.split()[1])
  28.     filename = message.split()[1].partition("/")[2]
  29.     print("filename:", filename)
  30.     fileExist = "false"
  31.     filetouse = "/" + filename
  32.     print("filetouse:", filetouse)
  33.     try:
  34.         # Check wether the file exist in the cache
  35.         f = open("WEB/" + filetouse[1:], "rb")
  36.         outputdata = f.read()
  37.         f.close()
  38.         fileExist = "true"
  39.         # ProxyServer finds a cache hit and generates a response message
  40.         tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  41.         tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  42.         tcpCliSock.send(outputdata)
  43.         print('Read from cache')
  44.     # Error handling for file not found in cache
  45.     except IOError:
  46.         if fileExist == "false":
  47.             # Create a socket on the proxyserver
  48.             c = socket(AF_INET, SOCK_STREAM)
  49.             hostn = filename.replace("www.","",1)
  50.             print("hostn:", hostn)
  51.             try:
  52.                 # Connect to the socket to port 80
  53.                 serverName = hostn.partition("/")[0]
  54.                 serverPort = 80
  55.                 print((serverName, serverPort))
  56.                 c.connect((serverName, serverPort))
  57.                 askFile = ''.join(filename.partition('/')[1:])
  58.                 print("askFile:", askFile)
  59.                 # Create a temporary file on this socket and ask port 80
  60.                 # for the file requested by the client
  61.                 fileobj = c.makefile('rwb', 0)
  62.                 if(message.split()[0] == 'GET'):
  63.                     fileobj.write("GET ".encode() + askFile.encode() + " HTTP/1.0\r\nHost: ".encode() + serverName.encode() + "\r\n\r\n".encode())
  64.                 else: #POST
  65.                     fileobj.write(
  66.                         "POST ".encode() + askFile.encode() + " HTTP/1.0\r\nHost: ".encode() + serverName.encode() + "\r\n\r\n".encode())
  67.                     fileobj.write(message.split("\r\n\r\n")[1].encode())
  68.                 # Read the response into buffer
  69.                 serverResponse = fileobj.read()
  70.                 if serverResponse.split()[0] != b'404':
  71.                     print('404')
  72.                     tcpCliSock.send("HTTP/1.1 404 Not Found\r\n\r\n".encode())
  73.                     tcpCliSock.close()
  74.                     continue
  75.                 # Create a new file in the cache for the requested file.
  76.                 # Also send the response in the buffer to client socket and the corresponding file in the cache
  77.                 filename = "WEB/" + filename
  78.                 filesplit = filename.split('/')
  79.                 for i in range(0, len(filesplit) - 1):
  80.                     if not os.path.exists("/".join(filesplit[0:i+1])):
  81.                         os.makedirs("/".join(filesplit[0:i+1]))
  82.                 tmpFile = open(filename, "wb")
  83.                 print(serverResponse)
  84.                 serverResponse = serverResponse.split(b'\r\n\r\n')[1]
  85.                 print(serverResponse)
  86.                 tmpFile.write(serverResponse)
  87.                 tmpFile.close()
  88.                 tcpCliSock.send("HTTP/1.1 200 OK\r\n".encode())
  89.                 tcpCliSock.send("Content-Type:text/html\r\n\r\n".encode())
  90.                 tcpCliSock.send(serverResponse)
  91.             except:
  92.                 print("Illegal request")
  93.             c.close()
  94.         else:
  95.             # HTTP response message for file not found
  96.             print("NET ERROR")
  97.     # Close the client and the server sockets
  98.     tcpCliSock.close()
  99. tcpSerSock.close()