English 中文(简体)
Dividing file to be processed partly, then batching results [closed]
原标题:

We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.

Closed 4 years ago.

Please,

We have the following situation:

Component X that divides a request file into parts, sending each part to an independent processing component Y -through a network- that will reply with a result to component Z , component Z collects all the results of the file parts into a batch result file.

Note:- Request file: file contains N number of data records that need to be processed.

What is the best practice for this situation? Is there any protocol for that?, Is there any library that can help? design patterns ??

thanx in advance.

最佳回答

We use the command pattern to queue up units of work to process very large files.

The commands themselves are stored in a database (command entries are things like "process File X, line 1-100", "process File X, line 101-200", etc.). Any number of servers from a server farm can pick up one command, indicate they are working on it, and write back their result. A controller looks for abandoned work (picked up but no result written within X minutes) and can reset the work to be eligible for pickup again.

问题回答

A message service queue/bus like RabbitMQ might be of help. Using this service, you can connect all the distributed components together & dispatch/collect results in a reliable manner.

Whilst the service bus won t solve all your "problems", it probably would address the distributed & reliable communication bits.





相关问题
Choosing the right subclass to instantiate programmatically

Ok, the context is some serialization / deserialization code that will parse a byte stream into an object representation that s easier to work with (and vice-versa). Here s a simplified example ...

Design pattern for managing queues and stacks?

Is there a design pattern for managing a queue or a stack? For example, we are looking to manage a list of tasks. These tasks will be added to a group queue, users will then be able to pull off the ...

Organizing classes using the repository design pattern

I have started upgrading one of our internal software applications, written in ASP.NET Web Forms, and moving to ASP.NET MVC. I am trying to leverage the Repository design pattern for my classes, ...

Misuse of Observer Pattern?

I have a Car object which contains a latitude field and a longitude field. I use the observer pattern so that any time either of these fields change in my application, my car object is notified. I ...

How are Models (in MVC) and DAOs supposed to interact?

How are models and DAOs supposed to interact? I m in the process of putting together a simple login module and I m unsure where to put the "business logic." If I put the logic with the data in the ...

热门标签