memory - Errors with PCoA in R due to large dataset -


for workproject have perform pcoa (principal coordinate analysis aka multidimensional scaling). when using r perform analysis run few problems.

the function cmdscale accepts matrix or dist input, dist function gives error:

error: cannot allocate vector of size 4.2 gb in addition: warning messages: 1: in dist(mydata[c(3, 4)], method = "euclidian", diag = false, upper = false) :   reached total allocation of 4020mb: see help(memory.size) 2: in dist(mydata[c(3, 4)], method = "euclidian", diag = false, upper = false) :   reached total allocation of 4020mb: see help(memory.size) 3: in dist(mydata[c(3, 4)], method = "euclidian", diag = false, upper = false) :   reached total allocation of 4020mb: see help(memory.size) 4: in dist(mydata[c(3, 4)], method = "euclidian", diag = false, upper = false) :   reached total allocation of 4020mb: see help(memory.size)

and when use matrix changes input this:

     [,1]          [1,] integer,33741 [2,] integer,33741

the contents of dataset cannot posted online can give dimensions: dataset 33741 rows long , 11 columns wide first column being id , other 10 values need used pcoa.

as can see in error use 2 columns , memory error.

now questions:
possible either manipulate data in such way can manage memory limit dist function?
doing wrong matrix function changes vectors 2 column 2 row output?

what have tried: clearing garbage collection, restarting gui, restarting system.

system: windows 7 x64 i7 920qm 1.8ghz 4gb ddr3 ram

code used:

mydata <- read.table(file, header=true)  mydist <- dist(mydata[c(3,4)], method="euclidian", diag=false, upper=false) mymatrix <- matrix(mydata[c(3,4)], byrow=false) mymatrix <- matrix(cbind(mydata[c(3,4)]))  mycmdscale <- cmdscale(mydist, k=2, eig=false, add=false, x.ret=false) mycmdscale <- cmdscale(mymatrix, k=2, eig=false, add=false, x.ret=false)  plot(mycmdscale)

of course did not run code in order code contains methods have tried load data.

thanks in advance replies.

you have far little memory operation in r, holds objects in memory. may not have exact calculation quite correct (i forget size of r's objects) hold dissimilarity matrix you'll need ~9gb of ram.

> print(object.size(matrix(0, ncol = 34000, nrow = 34000)), units = "gb") 8.6 gb

dist away less in internal representation storing 0.5 * (nr * (nr - 1)) doubles (nr number of rows in input data):

> print(object.size(numeric(length = 0.5 * 34000 * 33999)), units = "gb") 4.3 gb

[which error seeing coming from]

realistically you'll need upwards of 20-30gb of ram useful dissimilarity matrix once you've computed it. if compute them, eigenvectors of pcoa solution need ~ 9gb of ram, on own.

so more pertinent question is; hope c. 34000 samples/observations?

to matrix mydata[3:4] can use

as.matrix(mydata[3:4])

or, if have factors , want preserve numeric interpretation

data.matrix(mydata[3:4])

Comments

Post a Comment

Popular posts from this blog

php - mySql Join with 4 tables -

select * web_user e inner join web_audio ce on ce.uid = e.uid inner join web_videos c on c.uid = e.uid inner join web_images d on d.uid = e.uid e.uid = '1' we have scenario have 4 tables , want fetch records on basis of uid web_user table rest of tables fetch records on basis of web_user uid. have 6 records database. returning 8 time. below response of mysql query. uid ufname ulname uemail username upassword uaddress ucity ustate uzipcode uphone uimage usertype ugenre website biography uverifycode featured activate ustatus upaid aid uid a_filename a_uploadname a_genre a_singername a_approve a_price vid uid v_filename v_uploadname v_short_desc v_singername v_approve v_price iid uid upload_file i_approve ========================== 1 muhammad mohsin mohsin@balianti.com mohsin 213123 test addresss 0 1 40100 333-2335-5555 fds_896_autumn-mist.jpg 1 6 null null 202cb962ac59075b964b07152d234b70 1 1 1 1 1 1 asd muhammad_709.mp3 5 asd 0 0.00 1 1 test video muhammad_578.flv ...
Read more

css - Text drops down with smaller window -

i've adapted css use in website , can't quite figure out why 1 aspect of page acting way acting. basically, when shrink window below size horizontally, of text "drops down". becomes issue if user viewing website through ipad vertical orientation. can view issue here . i'm hoping make if window gets smaller, placement of text remains intact. the problem area not have enough space put both side navigation , content on 1 line. make automatically adjust page's width when exceeds decrements of width. this code example, , highly unlikely work. if see css more clearly, provide exact code. @media , (max-width:960px) { /* assuming #wrapper 960px across */ #content {width:560px;} /* opposed 720px */ } edit the content drops down @ 980px, have shrink width after there. @media , (max-width:980px) { #container {width:520px;} }
Read more

c# - DetailsView in ASP.Net - How to add another column on the side/add a control in each row? -

i have detailsview control on page used edit various fields of record, works in respect. i looking way add 1 column (and if works, why not more) right, absolutely read-only, show same fields of record comparison purposes. i aware there no obvious way such thing out of box detailsview . have looked other controls (transposed gridview , recommended formview , listview ), nothing satisfies. have special data binding setup using detailsview , can't out of without losing features. anyone on how "hack in" additional columns (for display only) on detailsview ? the solution have now, use second detailsview , visible set false in aspx. in code, make sure databind hidden detailsview hosts data third column first, initial detailsview named itemdetails . and in item created event, pass third column html rendering of hidden controls (in last code block) : protected void itemdetails_itemcreated(object sender, eventargs e) { if (dataite...
Read more