python - Twisted / perform asynchronous http requests -

i have twisted reactor listening incoming data. have second reactor performing http requests in time intervals sending results first reactor. both run fine.

now bring run in 1 reactor, not know how achieve this. - perform http requests every 60 sec. in asynchrounous way within first listening "main" reactor.

what have @ moment is:

# main reactor listening incoming data forever ... reactor.listentcp(8123, tcpeventreceiverfactory()) 

the http reactor uses twisted.internet.defer.deferredsemaphore() perform several http checks:

# create semaphore manage deferreds semaphore = twisted.internet.defer.deferredsemaphore(2)  # create list urls check dl = list() # append deferreds list url in self._urls:     # returns deferred     dl.append(semaphore.run(self._getpage, url))  # defferedlist dl = twisted.internet.defer.deferredlist(dl) # add callbacks error handling dl.addcallbacks(lambda x: reactor.stop(), self._handleerror)  # start reactor     reactor.run() 

how can add timed http checks "main" reactor, performed in asynchronous way? how deferredsemaphore work?

can me this?

[this kind of lightweight monitoring system processing http checkresults. new twisted , asynchronous programming. on xubuntu 12.04 running python 2.7]

you don't need multiple reactors. perform of different actions using same reactor.

if you're calling reactor.stop(), you're doing wrong, lets rid of that, , tie single function (that use callback); since doing asynchronous work, should return deferred, we'll use deferredlist you're using.

def thing_that_does_http():     # create semaphore manage deferreds     semaphore = twisted.internet.defer.deferredsemaphore(2)      # create list urls check     dl = deferredlist()     # append deferreds list     url in self._urls:         # returns deferred         dl.append(semaphore.run(self._getpage, url))      # defferedlist     dl = twisted.internet.defer.deferredlist(dl)     # add callbacks error handling     dl.adderrback(self._handleerror)     return dl 

the natural way "perform x in time intervals" looping call. callback function don't need much

reactor.listentcp(8123, tcpeventreceiverfactory()) loop_http = twisted.intertnet.task.loopingcall(thing_that_does_http) # run once per minute, starting now. loop_http.start(60) 

---

the reactor loopingcall , getpage use own purposes twisted.internet.reactor, if using different reactor, instance if doing unit testing, you'll need override default.

in case of loopingcall, it's quite simple, after construction, (but before calling start() method), set clock attribute:

from twisted.internet.task import clock fake_reactor = clock() loop_http.clock = fake_reactor fake_reactor.advance(120)  # move time forward 2 minutes... 

unfortunately, situation getpage() less nice. cannot use other reactor interface; you'll need use newer, shinier t.w.c.agent. in many ways agent superior, it's not quite convenient when want raw response body string.

aside requiring explicit reactor passed constructor, it's more fine grained control on request/response cycle convenience provided getpage. such it's implemented in terms of producers , protocols. in case of former, can pass convenience helper, filebodyproducer send request bodies minimal fuss; in latter, we'll need simple protocol buffer of chunks of data until we've gotten of it.

here's chunk of code replace getpage, same interface, taking instance of agent first argument

from cstringio import stringio twisted.internet.defer import deferred twisted.internet.protocol import protocol twisted.web.client import responsedone twisted.web.client import filebodyproducer   class getpageprotocol(protocol):     def __init__(self):         self.deferred = deferred()         self.data = []      def datareceived(self, data):         self.data.append(data)      def connectionlost(self, reason):         reason.trap(responsedone)         data = ''.join(self.data)         del self.data         self.deferred.callback(data)   def agentgetpage(agent, url,                  method="get",                  headers=none,                  postdata=none):     if postdata not none:         bodyproducer = filebodyproducer(stringio(postdata))     else:         bodyproducer = none      def _getpageresponded(response):         if response.length != 0:             proto = getpageprotocol()             response.deliverbody(proto)             return proto.deferred         else:             return none      d = agent.request(method, url, headers, bodyproducer)     d.addcallback(_getpageresponded)     return d 

which, in unit test, sort of like:

from twisted.test.proto_helpers import memoryreactor twisted.web.client import agent fake_reactor = memoryreactor() agent = agent(fake_reactor) d = agentgetpage(agent, "http://example.com")  assert fake_reactor.tcpclients  # or such, exercise code manipulating reactor 

_{edit: wanted skim on give ectomorph, less confused about; it's pretty idea drum in proper handling of reactors early, , avoid needless pain later.}