在一个DataFrame上对另一个DataFrame的组进行t测试。

The Goal:

我们的目标:

Perform t-test on DataFrame (df_rna) using group found in another DataFrame (df_cnv). Reduce the test DataFrame (df_rna) the row indices with the most significant scores from the t-test.

使用另一个DataFrame (df_cnv)中发现的组对DataFrame (df_rna)进行t-test。减少测试数据(df_rna)的行索引，最重要的分数来自t检验。

Code Sample:

代码示例:

# Dataframe (df_cnv) that forms groups of columns (cells) either\ belonging to True or False for t-test.
cnv = {'gene': ['x','y','z','n'],
        'cell_a': [0,-1,0,-1],
        'cell_b': [0,-1,-1,-1],
        'cell_c': [-1,0,-1,0],
        'cell_d': [-1,0,-1,0],
        'cell_e': [-1,0,0,0]
       }
df_cnv = pd.DataFrame(cnv)
df_cnv.set_index('gene', inplace=True)
cnv_mask = df_cnv < 0
cnv_mask  # True values are negative (gene loss is True)

# DataFrame for t-test and subsequent reduction to most significant rows
 rna = {'gene': ['x','y','z','n'],
            'cell_a': [1, 5, 8,9],
            'cell_b': [8, 5, 4,9],
            'cell_c': [8, 6, 1,1],
            'cell_d': [1, 2, 7,1],
            'cell_e': [5, 7, 9,1],
           }
    df_rna = pd.DataFrame(rna)
    df_rna.set_index('gene')

# Manually computed T-Tests, save results in DataFrame df_report
x = scipy.stats.ttest_ind([8,1,5],[1,8])
y = scipy.stats.ttest_ind([5,5], [6,2,7])
z = scipy.stats.ttest_ind([4,1,7], [8,9])
n = scipy.stats.ttest_ind([9,9], [1,1,1])

tStat = [gene.statistic for gene in [x,y,z,n]]
pVal = [gene.pvalue for gene in [x,y,z,n]]

report = {'gene':['x','y','z','n'],
         't_stat':tStat,
         'p_val':pVal}
df_report = pd.DataFrame(report)
df_report.set_index('gene', inplace=True)

# Create reduced version of test DataFrame (df_rna) to contain only rows (genes
df_pass = df_report.loc[df_report['p_val'] < 0.05]
passed_genes = set(df_pass.index)
passed_genes

df_rna_pass = df_rna.loc[df_rna['gene'].isin(passed_genes)]
df_rna_pass.set_index('gene')

The Question:

一个问题:

Manually setting the t-test groups is not feasible for my large dataset. How do I compute all the t-test statistics across the whole DataFrame df_rna when the groups of cells being either True and False changes for every row?

手动设置t测试组对于我的大数据集是不可行的。我如何计算整个DataFrame df_rna的所有t测试统计数据，当这些单元格对每一行都是正确的和错误的变化时?

Mystery Hang: (happens if your don't cache the results of rnadf_all[~cnv_mask])

悬念:(如果你不缓存rnadf_all[~cnv_mask]的结果)

C:\Users\test\Anaconda3\lib\site-packages\numpy\core\_methods.py:82: RuntimeWarning: Degrees of freedom <= 0 for slice
  warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-16-ccabe33b2612> in <module>()
     34 
     35 for r in rnadf_all[cnv_mask].iterrows():
---> 36     df_report.at[r[0], 't_stat'], df_report.at[r[0], 'p_val'] = scipy.stats.ttest_ind(r[1].dropna(), rnadf_all[~cnv_mask].loc[r[0]].dropna())
     37 
     38 df_pass = df_report.loc[df_report['p_val'] < 0.05]

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
   1963             return self._getitem_array(key)
   1964         elif isinstance(key, DataFrame):
-> 1965             return self._getitem_frame(key)
   1966         elif is_mi_columns:
   1967             return self._getitem_multilevel(key)

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_frame(self, key)
   2036         if key.values.dtype != np.bool_:
   2037             raise ValueError('Must pass DataFrame with boolean values only')
-> 2038         return self.where(key)
   2039 
   2040     def query(self, expr, **kwargs):

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\generic.py in where(self, cond, other, inplace, axis, level, try_cast, raise_on_error)
   3931         # try to align
   3932         try_quick = True
-> 3933         if hasattr(other, 'align'):
   3934 
   3935             # align with me

KeyboardInterrupt: 

2 个解决方案

#1

 from scipy import stats

 # Create empty DF for t-test results
 df_report = pd.DataFrame(index=df_rna.index, columns=['p_val', 't_stat'])

 not_df_rna = df_rna[~cnv_mask]

 # Iterate through df_rna rows, apply mask, drop NaN values, run ttest_ind and save result to df_report
 for r in df_rna[cnv_mask].iterrows():
     df_report.at[r[0], 't_stat'], df_report.at[r[0], 'p_val'] = stats.ttest_ind(r[1].dropna(), not_df_rna.loc[r[0]].dropna())

Result:

结果:

df_report

         p_val     t_stat
gene                     
x     0.966863  0.0450988
y            1          0
z     0.141358   -1.98508
n            0        inf

#2

I'd start by transposing the two DFs, and setting up a new DF for the t-test results:

我先把两个DFs调换一下，然后为t检验结果建立一个新的DF:

cnv_mask_t = cnv_mask.transpose()
df_rna_t = df_rna.transpose()
df_tres = pd.dataframe(index=df_rna.index, columns=['pval', 'stat'])

Then you can iterate over the genes, which are now columns, and filter for values in which the mask contains True:

然后您可以遍历那些现在是列的基因，并过滤掉包含True的值的值:

for gene in df_rna_t:
    col_mask = cnv_mask_t[gene]
    tres = scipy.stats.ttest_ind(df_rna_t[gene][col_mask], df_rna_t[gene][~col_mask])
    df_tres.loc[gene] = [tres.pvalue, tres.statistic]

I assume you can take it from here.

我想你可以从这里拿走。