Local Ruby/rails server for web scraping -


i run web scraping routines on local mac, default rails server consumes more 1 gig of memory (only 1 request running 5 hours). there server better suited this?

this web bot. rails heavy.

a script using nokigiri enough. can set cron run script.

if want higher level things, capybara do, test mainly.


Comments

Post a Comment

Popular posts from this blog

php - cannot display multiple markers in google maps v3 from traceroute result -

i'm newbie in php mysql , google maps v3. have problems in google maps v3. made traceroute web based , map traceroute's result in google maps v3 final project of college education. have try many tutorials didn't work. have tried tutorials in forum nothing worked. problem cannot display multiple markers got location data database. maybe because i'm newbie , not know how do. confuse looping show markers. here code <?php error_reporting(e_all ^ (e_notice)); ini_set('max_execution_time', 360); $enable_log_user = false; global $ip, $host_name, $host_ip; $host = @$_post['host']; $trace = @$_post['trace']; $self = $_server['php_self']; include("phpsqlajax_dbinfo.php"); $connection = mysql_connect ('127.0.0.1', $username, $password); if (!$connection) { die('not connected : ' . mysql_error());} $db_selected = mysql_select_db($database, $connection); if (!$db_selected) { die ('can\'t use db : ...
Read more

php - Boolean search on database with 5 million rows, very slow -

i have script when user searches "apples", there coloumn related searches "shiny apples", , "rotten apples", etc. problem searches database has 5,600,000 rows , mysql lagging badly. here code: $get = mysql_query("select keyword searches match (keyword) against ('".mysql_real_escape_string(preg_replace('#([\s]+)#', '$1', $title))."' in boolean mode) limit 0,15"); this taking long time process , have active website searches getting performed every second, can recommend me tips? thanks! you need indexes. have had experiences in past creating indexes got significant performance gain.
Read more

css - Text drops down with smaller window -

i've adapted css use in website , can't quite figure out why 1 aspect of page acting way acting. basically, when shrink window below size horizontally, of text "drops down". becomes issue if user viewing website through ipad vertical orientation. can view issue here . i'm hoping make if window gets smaller, placement of text remains intact. the problem area not have enough space put both side navigation , content on 1 line. make automatically adjust page's width when exceeds decrements of width. this code example, , highly unlikely work. if see css more clearly, provide exact code. @media , (max-width:960px) { /* assuming #wrapper 960px across */ #content {width:560px;} /* opposed 720px */ } edit the content drops down @ 980px, have shrink width after there. @media , (max-width:980px) { #container {width:520px;} }
Read more