multithreading - Python: Can't start new thread. Can I stagger or delay some threads? -


not sure how ask question since beginning learn python here goes:

i have web scrapper uses threading grab info. looking pricing , stock 900 products. when test script half of that, there no problem. when try scrap 900 products can't start new thread error.

i imagine memory constraint or because asking server many requests

i know if there way slow down threads or stagger requests.

error code:

traceback (most recent call last):   file "c:\python27\tests\dxpriceupdates.py", line 78, in <module>     t.start() error: can't start new thread >>>  traceback (most recent call last):exception in thread thread-554: traceback (most recent call last):   file "c:\python27\lib\urllib.py", line 346, in open_http     errcode, errmsg, headers = h.getreply()   file "c:\python27\lib\httplib.py", line 1117, in getreply     response = self._conn.getresponse()   file "c:\python27\lib\httplib.py", line 1045, in getresponse     response.begin()   file "c:\python27\lib\httplib.py", line 441, in begin     self.msg = httpmessage(self.fp, 0)   file "c:\python27\lib\mimetools.py", line 25, in __init__     rfc822.message.__init__(self, fp, seekable)   file "c:\python27\lib\rfc822.py", line 108, in __init__     self.readheaders()   file "c:\python27\lib\httplib.py", line 308, in readheaders     self.addheader(headerseen, line[len(headerseen)+1:].strip()) memoryerror  <bound method thread.__bootstrap of <thread(thread-221, stopped 9512)>>traceback (most recent call last): traceback (most recent call last): traceback (most recent call last):  traceback (most recent call last): unhandled exception in thread started unhandled exception in thread started ...

here python (the skulist.txt text file 12345, 23445, 5551,...):

from threading import thread import urllib import re import json import math      def th(ur):     site = "http://dx.com/p/getproductinforealtime?skus="+ur     htmltext = urllib.urlopen(site)     data = json.load(htmltext)     htmlrates = urllib.urlopen("http://rate-exchange.appspot.com/currency?from=usd&to=aud")     datarates = json.load(htmlrates)     if data['success'] == true:         if data['data'][0]['discount'] 0:             price = float(data['data'][0]['price'])             rate = float(datarates['rate']) + 0.12             cost = price*rate             if cost <= 5:                 saleprice = math.ceil(cost*1.7) - .05             elif (cost >5) , (cost <= 10):                 saleprice = math.ceil(cost*1.6) - .05             elif (cost >10) , (cost <= 15):                 saleprice = math.ceil(cost*1.55) - .05             else:                 saleprice = math.ceil(cost*1.5) - .05             if data['data'][0]['issoldout']:                 soldout = "out of stock"                 enabled = "disable"                 qty = "0"             else:                 soldout = "in stock"                 enabled = "enabled"                 qty = "9999"              #print model, saleprice, soldout, qty, enabled             myfile.write(str(ur)+","+str(saleprice)+","+str(soldout)+","+str(qty)+","+str(enabled)+"\n")         else:             price = float(data['data'][0]['listprice'])             rate = float(datarates['rate']) + 0.12             cost = price*rate             if cost <= 5:                 saleprice = math.ceil(cost*1.7) - .05             elif (cost >5) , (cost <= 10):                 saleprice = math.ceil(cost*1.6) - .05             elif (cost >10) , (cost <= 15):                 saleprice = math.ceil(cost*1.55) - .05             else:                 saleprice = math.ceil(cost*1.5) - .05             if data['data'][0]['issoldout']:                 soldout = "out of stock"                 enabled = "disable"                 qty = "0"             else:                 soldout = "in stock"                 enabled = "enabled"                 qty = "9999"              #print model, saleprice, soldout, qty, enabled             myfile.write(str(ur)+","+str(saleprice)+","+str(soldout)+","+str(qty)+","+str(enabled)+"\n")     else:         qty = "0"         print ur, "error \n"         myfile.write(str(ur)+","+"0.00"+","+"out of stock"+","+str(qty)+","+"disable\n")   skulist = open("skulist.txt").read() skulist = skulist.replace(" ", "").split(",")  myfile = open("prices/price_update.txt", "w+") myfile.close()  myfile = open("prices/price_update.txt", "a") threadlist = []  u in skulist:     t = thread(target=th,args=(u,))     t.start()     threadlist.append(t)  b in threadlist:     b.join()  myfile.close()

don't fire 900 threads @ once, pc literally choke! instead, use pool , distribute activity on number of workers. use multiprocessing this:

from multiprocessing import pool  workers = 10 p = pool(workers) p.map(tr, skulist)

find right value workers experimenting bit.


Comments

Post a Comment

Popular posts from this blog

php - cannot display multiple markers in google maps v3 from traceroute result -

i'm newbie in php mysql , google maps v3. have problems in google maps v3. made traceroute web based , map traceroute's result in google maps v3 final project of college education. have try many tutorials didn't work. have tried tutorials in forum nothing worked. problem cannot display multiple markers got location data database. maybe because i'm newbie , not know how do. confuse looping show markers. here code <?php error_reporting(e_all ^ (e_notice)); ini_set('max_execution_time', 360); $enable_log_user = false; global $ip, $host_name, $host_ip; $host = @$_post['host']; $trace = @$_post['trace']; $self = $_server['php_self']; include("phpsqlajax_dbinfo.php"); $connection = mysql_connect ('127.0.0.1', $username, $password); if (!$connection) { die('not connected : ' . mysql_error());} $db_selected = mysql_select_db($database, $connection); if (!$db_selected) { die ('can\'t use db : ...
Read more

c# - DetailsView in ASP.Net - How to add another column on the side/add a control in each row? -

i have detailsview control on page used edit various fields of record, works in respect. i looking way add 1 column (and if works, why not more) right, absolutely read-only, show same fields of record comparison purposes. i aware there no obvious way such thing out of box detailsview . have looked other controls (transposed gridview , recommended formview , listview ), nothing satisfies. have special data binding setup using detailsview , can't out of without losing features. anyone on how "hack in" additional columns (for display only) on detailsview ? the solution have now, use second detailsview , visible set false in aspx. in code, make sure databind hidden detailsview hosts data third column first, initial detailsview named itemdetails . and in item created event, pass third column html rendering of hidden controls (in last code block) : protected void itemdetails_itemcreated(object sender, eventargs e) { if (dataite...
Read more

javascript - firefox memory leak -

i have following script (i use raphaeljs also) draws circle, waits 1 second, draws circle again, etc (you can copy paste - functional, need include raphael raphaeljs.com): <div id="holder"> </div> <script src="raphael.js" type="text/javascript"></script> <script> var paper = raphael(10, 50, 320, 200); var rad = math.pi / 180; var finangle = 0; function sector(cx, cy, r, startangle, endangle, params) { var x1 = cx + r * math.cos(-startangle * rad), x2 = cx + r * math.cos(-endangle * rad), y1 = cy + r * math.sin(-startangle * rad), y2 = cy + r * math.sin(-endangle * rad); return ["m", cx, cy, "l", x1, y1, "a", r, r, 0, +(endangle - startangle > 180), 0, x2, y2]; } var path = sector(100,100,50,0,180); var arc = paper.path(path); function draw() { path = sector(100,100,50,0,finangle); ...
Read more