mysql - Bulk INSERTs from multiple hosts preformace optimization -


i have 15 amazon aws ec2 t1.micro inctances simultaneusly populate amazon rds mysql d2.m2.xlarge database data using large inserts (40000 rows in query).

the queries send continuously. table innodb, 2 int columns, there index both columns. cpu utilization of rds instance 30% during data receiving.

when have 1 ec2 instance, speed in orders faster run 15 instances simultaneusly. , 15-instances group work slower , slower until speed becomes totally unsatisfactory.

how can optimize performance of process?

upd: show create table results following:

create table `userdata` (  `uid` int(11) not null,  `data` int(11) not null,  primary key (`uid`,`data`),  key `uid` (`uid`),  key `data` (`data`) ) engine=innodb default charset=latin1

i need 2 indexes cause nessecary me fetch data uid , data value.

i insert data insert userdata (uid, data) values (1,2),(1,3),(1,10),... 40000 (uid,data) pairs.

15 parallel instances insert ~121 000 000 rows in 2 hours, sure can more faster.

i can sense problem in these of areas :

point 1) t1.micro burst instance , spikey compute workloads. since had mentioned using continuously db queries/transactions, can show reduced overall performance. please change instance type better predictable ones.

point 2) m2.xlarge , t1.micro comes limited nw bandwidth capacity. in case if volume of data (in mb/sec) transacting/querying db above nw bandwidth capacity can observe reduced throughput. imagine if 15 t1.micro trying push 100mb/sec , more nw pipe exists between t1.micros , db instance type


Comments

Post a Comment

Popular posts from this blog

c# - DetailsView in ASP.Net - How to add another column on the side/add a control in each row? -

i have detailsview control on page used edit various fields of record, works in respect. i looking way add 1 column (and if works, why not more) right, absolutely read-only, show same fields of record comparison purposes. i aware there no obvious way such thing out of box detailsview . have looked other controls (transposed gridview , recommended formview , listview ), nothing satisfies. have special data binding setup using detailsview , can't out of without losing features. anyone on how "hack in" additional columns (for display only) on detailsview ? the solution have now, use second detailsview , visible set false in aspx. in code, make sure databind hidden detailsview hosts data third column first, initial detailsview named itemdetails . and in item created event, pass third column html rendering of hidden controls (in last code block) : protected void itemdetails_itemcreated(object sender, eventargs e) { if (dataite
Read more

javascript - firefox memory leak -

i have following script (i use raphaeljs also) draws circle, waits 1 second, draws circle again, etc (you can copy paste - functional, need include raphael raphaeljs.com): <div id="holder"> </div> <script src="raphael.js" type="text/javascript"></script> <script> var paper = raphael(10, 50, 320, 200); var rad = math.pi / 180; var finangle = 0; function sector(cx, cy, r, startangle, endangle, params) { var x1 = cx + r * math.cos(-startangle * rad), x2 = cx + r * math.cos(-endangle * rad), y1 = cy + r * math.sin(-startangle * rad), y2 = cy + r * math.sin(-endangle * rad); return ["m", cx, cy, "l", x1, y1, "a", r, r, 0, +(endangle - startangle > 180), 0, x2, y2]; } var path = sector(100,100,50,0,180); var arc = paper.path(path); function draw() { path = sector(100,100,50,0,finangle);
Read more

Trying to import CSV file to a SQL Server database using asp.net and c# - can't find what I'm missing -

i student , quite new programming, , given task using asp.net , c#, without being taught either. plan learn teach ourselves. the task stuck on making website import csv file sql server database of last year's room bookings around campus. the table has following columns: request_id; priority; module_id; day; start_time; length; park; students; room_code; status; semester_id; linked_request; week_1; week_2; week_3; week_4; week_5; week_6; week_7; week_8; week_9; week_10; week_11; week_12; week_13; week_14; week_15; ) in code, try make work 3 columns (request_id,priority,module) starters. when press import button, want read .csv file (with fixed name @ fixed directory) eg "1,2,3" - comma separating fields [delimiter] - , import sql server datab
Read more