2018-10-21         Noel

How to apply function on mutlple-index pandas dataframe elegantly like on panel?

Suppose I have a dataframe like:ticker MS AAPL field price volume price volume0 -0.861210 -0.319607 -0.855145 0.6355941 -1.986693 -0.526885 -1.765813 1.6965332 -0.154544 -1.152361 -1.391477 -2.0161193 0.621641 -0.109499 0.143788 -0.050672generated from following codes, please ignore the numbers which are just as examplescolumns = pd.MultiIndex.from_tuples([('MS', 'price'), ('MS', 'volume'), ('AAPL', 'price'), ('AAPL', 'volume')], names=['ticker', 'field'])data = np.random.randn(4, 4)df = pd.DataFrame(data...

 pandas                     2 answers                     96 view
 2018-10-21         Larry

reshape is deprecated issue when I pick series from pandas Dataframe

When I try to take one series from dataframe I get this issue anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py:52: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead return getattr(obj, method)(*args, **kwds)This is the code snippet for idx, categories in enumerate(categorical_columns): ax = plt.subplot(3,3,idx+1) ax.set_xlabel(categories[0]) box = [df[df[categories[0]] == atype].price for atype in categories[1]] ax.boxplot(box) For avoid chained indexing use DataFrame.loc:...

 pandas                     1 answers                     7 view
 2018-10-21         Xaviera

Pandas: filling missing values by the time occurrence of an event

I already asked a similar question (see here), but unfortunately it was not clear enough, so I decided it was better to create a new one with a better dataset for example and a new explanation of the desired output - an edit would have been really a major change. So, I have the following dataset (it's already sorted by date and player):d = {'player': ['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '2', '2', '2', '2', '2', '3', '3', '3', '3', '3', '3'],'date': ['2018-01-01 00:17:01', '2018-01-01 00:17:05','2018-01-01 00:19:05', '2018-01-01 00:21:07', '2018-01-01 00:2...

 pandas                     2 answers                     53 view
 2018-10-21         Warner

ElasticSearch Hosted on AWS by ES to pandas df and back to ES ConnectionError

I just started with ElasticSearch, and took free 14 days trial from here.Got an username, password and id, now I am passing my elasticsearch host url, username and password. But getting Error. How can I retrieve dataframe from elasticsearch to pandas from AWS hosted ES.I wan't process this and save it back to ES, i.e. to same index as well in a new Index after processing it.Also if I look at /_cat/indices?v it exists there. What I did is used logstash to push MySql (hosted on AWS) to ES (as well hosted on AWS) Now I want to get data, process it and push back to ElasticSearc...

 pandas                     1 answers                     58 view
 2018-10-21         Ethel

open and parse multiple gpx files

I am trying to parse multiple gpx files stored in a directory with gpxpy in Python and create a pandas data frame.Here is my code:import gpxpyimport os# Open the file in read mode and parse itgpx_dir = r'/Users/Gav/GPX Data/'for filename in os.listdir(gpx_dir): gpx_file = open(filename, 'r') gpx = gpxpy.parse(gpx_file)I am getting the following error:UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 3131: ordinal not in range(128)I know the gpx file is fine as I am able to open it and parse it as a single file, but as soon as I try to open m...

 pandas                     1 answers                     77 view
 2018-10-21         Megan

Top 10 Rank for Each Company for Each Month

I have a data set that sums the number of pmt_unit by Year, Month, msa and company using a groupby. I'm hoping to just get the top 10 with the most pmt_units for each company for each month. This is the code used to get the full ranking list:#For each month for each builder, provide the pmt_units for the top ten cities #Group by Month and MSASFU_grouped = SFU_2.groupby(['uyear','umonth','msa','stock_ticker']).agg({'pmt_units': 'sum'}).reset_index()Sort chronologically, and then by company and rank MSA by pmt_units from most to least. Add a column that ranks msa by pmt_unit ...

 pandas                     2 answers                     30 view
 2018-10-21         Sid

Python Pandas: Read in External Dataset Into Dataframe Only If Conditions Are Met Using Function Call

Let's say I have an Excel file called "test.xlsx" on my local machine. I can read this data set in using traditional code.df_test = pd.read_excel('test.xlsx') However, I want to conditionally read that data set in if a condition is met ... if another condition is met I want to read in a different dataset. Below is the code I tried using a function: def conditional_run(x): if x == 'fleet': eval('''df_test = pd.read_excel('test.xlsx')''') elif x != 'fleet': eval('''df_test2 = pd.read_excel('test_2.xlsx')''')conditional_run('fleet') Below is the error I ge...

 pandas                     1 answers                     5 view
 2018-10-21         August

how to convert a column of pandas series without the header

It is quite odd as I hadn't experienced the issue until now,, for conversion of data series.So I have wind speed data by date & hour at different heights, retrieved from NREL.file09 = 'wind/wind_yr2009.txt'wind09 = pd.read_csv(file09, encoding = "utf-8", names = ['DATE (MM/DD/YYYY)', 'HOUR-MST', 'AWS@20m [m/s]', 'AWS@50m [m/s]', 'AWS@80m [m/s]', 'AMPLC(2-80m)'])file10 = 'wind/wind_yr2010.txt'wind10 = pd.read_csv(file10, encoding = "utf-8", names = ['DATE (MM/DD/YYYY)', 'HOUR-MST', 'AWS@20m [m/s]', 'AWS@50m [m/s]', 'AWS@80m [m/s]', 'AMPLC(2-80m)'])I merge the two reading...

 pandas                     1 answers                     6 view
 2018-10-21         Julius

Calling specific dates and values in stock prices

How do i call data for the dates and values for any shares? Example: I want to call the stock price and date for apple shares only for dec 2016, dec 2017.Here is what I've tried:import pandas as pdpd.core.common.is_list_like = pd.api.types.is_list_likeimport pandas_datareader.data as webimport numpy as npfrom matplotlib import styleimport matplotlib.pyplot as pltimport datetime as dtstart = dt.datetime(2013,10,1)end= dt.datetime(2018,4,30)AAPL_data=[]AAPL= web.DataReader('AAPL','iex', start, end)AAPL_data.append(AAPL) AAPL.loc['2016-12-01':'2016-12-31']AAPL.loc['2017-12-...

 pandas                     1 answers                     19 view
 2018-10-21         Dana

Equivalent of secondary_y for barh in Pandas/Matplotlib?

Pandas offers a nice, simple way to plot a (vertical) bar chart on two y axes. Is there an equivalent for the horizontal bar chart?To adapt from the linked example:import matplotlib.pyplot as pltimport numpy as npimport pandas as pdfrom io import StringIOs = StringIO(""" amount priceA 40929 4066443B 93904 9611272C 188349 19360005D 248438 24335536E 205622 18888604F 140173 12580900G 76243 6751731H 36859 3418329I 29304 2758928J 39768 3201269K 30350 2867059""")df = pd.read_csv(s, index_col=0, delimiter=' ', skipi...

 pandas                     1 answers                     23 view
 2018-10-21         Glenn

Converting a pandas DataFrame to read file?

I've been tasked with automating access to an API of a third party vendor.the third party vendor wants data in the format:data = open(fname, 'rb').read()yet I have data in a pandas DataFrame. What is the easiest way to go from a DataFrame to this 'data' value? I spent a long time on this and I can literally not believe the best answer:csv_string = df.to_csv()literally just omit the filename and the output of to_csv will not write to a file. It was in the documentation. [XXX]If your dataframe is foo, thenfoo.to_csv('filename') will work wonders. [XXX]

 pandas                     2 answers                     38 view
 2018-10-21         Kent

read pandas colum with number values and missing data as string

I have an Id column in my data frame like this:a = pandas.DataFrame([12673, 44, 847])This data has some missing values. If I Keep_default_NA = True, then the missing value is filled by NaN, and the data is read as float, and therefore the values will change to 12673.0 , 44.0, 847.0which is not desired ( I want to drop NA values and convert to str/obj because the id can be of any length).If I keep_default_NA = False, then other columns (such as booleans) all become object and I have to compare string values to find out true/false values. If you want NaN values, you have ...

 pandas                     2 answers                     50 view
 2018-10-21         Basil

Pandas fill missing values of a column based on the datetime values of another column

Python newbie here, this is my first question. I tried to find a solution on similar SO questions, like this one, this one, and also this one, but I think my problem is different. Here's my situation: I have a quite large dataset with two columns: Date (datetime object), and session_id (integer). The timestamps refer to the moment where a certain action occurred during an online session. My problem is that I have all the dates, but I am missing some of the corresponding session_id values. What I would like to do is to fill these missing values using the date column: If the...

 pandas                     1 answers                     50 view
 2018-10-22         Burke

Error from Importing Pandas

I use pandas regularly. I restarted my computer recently and now I get the following error every time I try to import pandas. This does not happen with other packages. Any suggestions?import pandas as pdTraceback (most recent call last):File "C:\Users\UserName\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code exec(code_obj, self.user_global_ns, self.user_ns)File "", line 1, in import pandas as pdFile "C:\Users\UserName\Anaconda3\lib\site-packages\pandas__init__.py", line 42, in from pandas.core.api import *File "C:\Users\Use...

 pandas                     1 answers                     13 view
 2018-10-22         Wade

Group and count the number of rows based on the week

I have the following sample dataframe: Date count_row Work_Type2018-07-31 00:00:00 1 logpage2018-08-01 00:00:00 1 logpage2018-08-01 00:00:00 1 logpage2018-08-01 00:00:00 1 logpage2018-08-01 00:00:00 1 logpage2018-09-02 00:00:00 1 logpage2018-09-02 00:00:00 1 logpage2018-09-03 00:00:00 1 logpage2018-09-04 00:00:00 1 logpage2018-09-04 00:00:00 1 logpage2018-09-06 00:00:00 1 logpage2018-09-07 00:00:00 1 logpage2018-09-08 00:00:00 1 ...

 pandas                     2 answers                     91 view
 2018-10-22         Clyde

Pandas: Save Dict/Json using python

Following is the code I used to store dataframe as csv into Azure blob in python. output = df.to_csv(index_label="idx", encoding="utf-8") blob_service.create_blob_from_text('test', folderName + 'abc.csv', output)Is there any way to store python dictionary as json the similar way as above?Essentially I want to get the 'output' object of dict to_json. You can convert your python dict to json string using this:import jsonoutput = json.dumps(your_dict_here)And then you can store it just like before:blob_service.create_blob_from_text('test', folderName + 'abc.json', out...

 pandas                     1 answers                     75 view

Page 1 of 49  |  Show More Pages:  Top Prev Next Last