Need inputs on optimising web service calls from Perl -


current implementation- divide original file files equal number of servers. ensure each server picks 1 file processing. each server splits file 90 buckets. use forkmanager fork 90 processes, each operating on bucket. child processes make api calls. merge output of child processes. merge output of each server.

stats- size of content downloaded using api call 40kb. on 2 servers, above process 225k user file runs in 15 minutes. aim finish 10 million file in 30 minutes. (hope doesn't sound absurd!)

i contemplated using berkeleydb but, couldn't find how convert berkeleydb file normal ascii file.

this sounds one-time operation me. although don't understand 30 minute limit, have few suggestions know experience.

first of all, said in comment, bottleneck not reading data files. not writing results harddrive. bottleneck in transfer between machines , remote machines. setup sounds sophisticated, might not in situation.

if hitting webservice, running service. there servers can handle ammount of load. have brought down dev environment servers of big logistics company small load test ran @ night. often, these things equipped long-term load, not short, heavy load.

since talking each other through various protocols, web services or other apis, should consider talking people run service. if have business-relationship, easy. if not, try find way reach them , ask if service able handle many requests @ all. end them excluding permanently because admins looks tried ddos them.

i'd ask them if send them files (or excerpt of data, cut down relevant processing) can operations in batch on side. way, remove load processing web requests, , time takes these requests.


Comments

Post a Comment

Popular posts from this blog

php - mySql Join with 4 tables -

select * web_user e inner join web_audio ce on ce.uid = e.uid inner join web_videos c on c.uid = e.uid inner join web_images d on d.uid = e.uid e.uid = '1' we have scenario have 4 tables , want fetch records on basis of uid web_user table rest of tables fetch records on basis of web_user uid. have 6 records database. returning 8 time. below response of mysql query. uid ufname ulname uemail username upassword uaddress ucity ustate uzipcode uphone uimage usertype ugenre website biography uverifycode featured activate ustatus upaid aid uid a_filename a_uploadname a_genre a_singername a_approve a_price vid uid v_filename v_uploadname v_short_desc v_singername v_approve v_price iid uid upload_file i_approve ========================== 1 muhammad mohsin mohsin@balianti.com mohsin 213123 test addresss 0 1 40100 333-2335-5555 fds_896_autumn-mist.jpg 1 6 null null 202cb962ac59075b964b07152d234b70 1 1 1 1 1 1 asd muhammad_709.mp3 5 asd 0 0.00 1 1 test video muhammad_578.flv ...
Read more

css - Text drops down with smaller window -

i've adapted css use in website , can't quite figure out why 1 aspect of page acting way acting. basically, when shrink window below size horizontally, of text "drops down". becomes issue if user viewing website through ipad vertical orientation. can view issue here . i'm hoping make if window gets smaller, placement of text remains intact. the problem area not have enough space put both side navigation , content on 1 line. make automatically adjust page's width when exceeds decrements of width. this code example, , highly unlikely work. if see css more clearly, provide exact code. @media , (max-width:960px) { /* assuming #wrapper 960px across */ #content {width:560px;} /* opposed 720px */ } edit the content drops down @ 980px, have shrink width after there. @media , (max-width:980px) { #container {width:520px;} }
Read more

c# - DetailsView in ASP.Net - How to add another column on the side/add a control in each row? -

i have detailsview control on page used edit various fields of record, works in respect. i looking way add 1 column (and if works, why not more) right, absolutely read-only, show same fields of record comparison purposes. i aware there no obvious way such thing out of box detailsview . have looked other controls (transposed gridview , recommended formview , listview ), nothing satisfies. have special data binding setup using detailsview , can't out of without losing features. anyone on how "hack in" additional columns (for display only) on detailsview ? the solution have now, use second detailsview , visible set false in aspx. in code, make sure databind hidden detailsview hosts data third column first, initial detailsview named itemdetails . and in item created event, pass third column html rendering of hidden controls (in last code block) : protected void itemdetails_itemcreated(object sender, eventargs e) { if (dataite...
Read more