Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.
Suppose if the system has 4 CPU's, I need to run eight workers on It. On starting the server, I want to initialize the eight workers. I want to forward the data to available workers on the new WebSocket connection and change that worker's status to occupied. If all of the workers are occupied, then the server should not accept any new connections.
I don't want to exchange data between the processes. But I need to send data to the correct Process when new data is received in the WebSocket. Whenever the WebSocket connection drops, then I need to change the status of the worker to available.
# Something like this,
# FYI : The following is not a python code, just a representation of the requirement
process_manager = ProcessManager()
WebSocket
onNewConnection:
available_process = process_manager.get_available_process()
if available_process:
available_process.set_state(state.Occupied)
else:
ws.write("Server is on full load")
onNewMessage(_data):
available_process.sendData(_data)
OnSocketClose:
available_process.set_state(state.Available)
__main__
process_pool = ProcessPool (8)
for loop 0 to 7 as i
process_manager.put(i,Process)
#-- start all process
#-- join all process
The text was updated successfully, but these errors were encountered:
Suppose if the system has 4 CPU's, I need to run eight workers on It. On starting the server, I want to initialize the eight workers. I want to forward the data to available workers on the new WebSocket connection and change that worker's status to occupied. If all of the workers are occupied, then the server should not accept any new connections.
I don't want to exchange data between the processes. But I need to send data to the correct Process when new data is received in the WebSocket. Whenever the WebSocket connection drops, then I need to change the status of the worker to available.
The text was updated successfully, but these errors were encountered: