python - Pandas Shaping Data for Covariance -


i need conduct simple covariance analysis in time series. raw data comes in shape this:

week_end_date              title_short          sales   2012-02-25 00:00:00.000000 "bob" (ebk)         1                            "bob" (ebk)         1 2012-03-31 00:00:00.000000 "bob" (ebk)         1                            "bob" (ebk)         1 2012-03-03 00:00:00.000000 "sally" (ebk)          1 2012-03-10 00:00:00.000000 "sally" (ebk)          1 2012-03-17 00:00:00.000000 "sally" (ebk)          1                            "sally" (ebk)          1 2012-04-07 00:00:00.000000 "sally" (ebk)          1

as can see, there duplicates. unless i'm missing something, need data become set of vectors each title, can use numpy.cov.

question:

how find duplicates in date , name , aggregate them sum? i've been trying use pandas groupby week_end_date , tittle_short comes out indexed in way don't understand.

edit: specific, when try df.groupby(["week_end_date", "title_short"]), this:

>df.ix[0:3]  week_end_date               title_short                2012-02-04 00:00:00.000000  'salem's lot (ebk)            <pandas.core.indexing._ndframeindexer object a...                             'tis season! (ebk)        <pandas.core.indexing._ndframeindexer object a...                             (not asked) (ebk)    <pandas.core.indexing._ndframeindexer object a... dtype: object

and trying select df.ix[1,] gets error:

traceback (most recent call last):   file "<stdin>", line 1, in <module>   file "/library/python/2.7/site-packages/pandas-0.11.0rc1_20130415-py2.7-macosx-10.8-intel.egg/pandas/core/series.py", line 613, in __getitem__     return self.index.get_value(self, key)   file "/library/python/2.7/site-packages/pandas-0.11.0rc1_20130415-py2.7-macosx-10.8-intel.egg/pandas/core/index.py", line 1630, in get_value     loc = self.get_loc(key)   file "/library/python/2.7/site-packages/pandas-0.11.0rc1_20130415-py2.7-macosx-10.8-intel.egg/pandas/core/index.py", line 2285, in get_loc     result = slice(*self.slice_locs(key, key))   file "/library/python/2.7/site-packages/pandas-0.11.0rc1_20130415-py2.7-macosx-10.8-intel.egg/pandas/core/index.py", line 2226, in slice_locs     start_slice = self._partial_tup_index(start, side='left')   file "/library/python/2.7/site-packages/pandas-0.11.0rc1_20130415-py2.7-macosx-10.8-intel.egg/pandas/core/index.py", line 2250, in _partial_tup_index     raise exception('level type mismatch: %s' % lab) exception: level type mismatch: 3

i'm not entirely know what's going on, here's i'd start with. first, data (which looks fixed-width me):

>>> df = pd.read_fwf("weekend.dat", widths=(26, 20, 9), parse_dates=[0]) >>> df = df.fillna(method="ffill") >>> df         week_end_date    title_short  sales 0 2012-02-25 00:00:00    "bob" (ebk)      1 1 2012-02-25 00:00:00    "bob" (ebk)      1 2 2012-03-31 00:00:00    "bob" (ebk)      1 3 2012-03-31 00:00:00    "bob" (ebk)      1 4 2012-03-03 00:00:00  "sally" (ebk)      1 5 2012-03-10 00:00:00  "sally" (ebk)      1 6 2012-03-17 00:00:00  "sally" (ebk)      1 7 2012-03-17 00:00:00  "sally" (ebk)      1 8 2012-04-07 00:00:00  "sally" (ebk)      1

then aggregate dups:

>>> g = df.groupby(["week_end_date", "title_short"]).sum().reset_index() >>> g         week_end_date    title_short  sales 0 2012-02-25 00:00:00    "bob" (ebk)      2 1 2012-03-03 00:00:00  "sally" (ebk)      1 2 2012-03-10 00:00:00  "sally" (ebk)      1 3 2012-03-17 00:00:00  "sally" (ebk)      2 4 2012-03-31 00:00:00    "bob" (ebk)      2 5 2012-04-07 00:00:00  "sally" (ebk)      1

and whatever cov stuff need (note cov series/dataframe/groupby method too, shouldn't need call np.cov specifially).


Comments

Post a Comment

Popular posts from this blog

c# - DetailsView in ASP.Net - How to add another column on the side/add a control in each row? -

i have detailsview control on page used edit various fields of record, works in respect. i looking way add 1 column (and if works, why not more) right, absolutely read-only, show same fields of record comparison purposes. i aware there no obvious way such thing out of box detailsview . have looked other controls (transposed gridview , recommended formview , listview ), nothing satisfies. have special data binding setup using detailsview , can't out of without losing features. anyone on how "hack in" additional columns (for display only) on detailsview ? the solution have now, use second detailsview , visible set false in aspx. in code, make sure databind hidden detailsview hosts data third column first, initial detailsview named itemdetails . and in item created event, pass third column html rendering of hidden controls (in last code block) : protected void itemdetails_itemcreated(object sender, eventargs e) { if (dataite
Read more

javascript - firefox memory leak -

i have following script (i use raphaeljs also) draws circle, waits 1 second, draws circle again, etc (you can copy paste - functional, need include raphael raphaeljs.com): <div id="holder"> </div> <script src="raphael.js" type="text/javascript"></script> <script> var paper = raphael(10, 50, 320, 200); var rad = math.pi / 180; var finangle = 0; function sector(cx, cy, r, startangle, endangle, params) { var x1 = cx + r * math.cos(-startangle * rad), x2 = cx + r * math.cos(-endangle * rad), y1 = cy + r * math.sin(-startangle * rad), y2 = cy + r * math.sin(-endangle * rad); return ["m", cx, cy, "l", x1, y1, "a", r, r, 0, +(endangle - startangle > 180), 0, x2, y2]; } var path = sector(100,100,50,0,180); var arc = paper.path(path); function draw() { path = sector(100,100,50,0,finangle);
Read more

Trying to import CSV file to a SQL Server database using asp.net and c# - can't find what I'm missing -

i student , quite new programming, , given task using asp.net , c#, without being taught either. plan learn teach ourselves. the task stuck on making website import csv file sql server database of last year's room bookings around campus. the table has following columns: request_id; priority; module_id; day; start_time; length; park; students; room_code; status; semester_id; linked_request; week_1; week_2; week_3; week_4; week_5; week_6; week_7; week_8; week_9; week_10; week_11; week_12; week_13; week_14; week_15; ) in code, try make work 3 columns (request_id,priority,module) starters. when press import button, want read .csv file (with fixed name @ fixed directory) eg "1,2,3" - comma separating fields [delimiter] - , import sql server datab
Read more