我在pandas数据框中有数据，其中两列包含数字序列（开始和停止）。我想确定哪些行具有与下一行重叠的停止值＆＃39;开始值。然后我需要将它们连接成一行，这样我就只有一个非重叠的数字序列，由每行中的start和stop值表示。
我已将数据加载到pandas数据帧中：
chr start stop geneID 0 chr13 32889584 32889814 BRCA2 1 chr13 32890536 32890737 BRCA2 2 chr13 32893194 32893307 BRCA2 3 chr13 32893282 32893400 BRCA2 4 chr13 32893363 32893466 BRCA2 5 chr13 32899127 32899242 BRCA2
我想比较数据框中的行。检查每行的停止值是否小于下一行的起始值，然后在新数据框中使用正确的开始值和停止值创建一行。理想情况下，当有几行全部重叠时，这将一次性连接，但我怀疑我将不得不迭代我的输出，直到这不再发生。
到目前为止，我的代码可以识别是否存在重叠（改编自this post）：
import pandas as pd
import numpy as np
columns = ['chr','start','stop','geneID']
bed = pd.read_table('bedfile.txt',sep='\s',names=['chr','start','stop','geneID'],engine='python')
def bed_prepare(inp_bed):
inp_bed['next_start'] = inp_bed['start'].shift(periods=-1)
inp_bed['distance_to_next'] = inp_bed['next_start'] - inp_bed['stop']
inp_bed['next_region_overlap'] = inp_bed['next_start'] < inp_bed['stop']
intermediate_bed = inp_bed
return intermediate_bed
这给了我这样的输出：
print bed_prepare(bed)
chr start stop geneID next_start distance_to_next next_region_overlap 0 chr13 32889584 32889814 BRCA2 32890536 722 False 1 chr13 32890536 32890737 BRCA2 32893194 2457 False 2 chr13 32893194 32893307 BRCA2 32893282 -25 True 3 chr13 32893282 32893400 BRCA2 32893363 -37 True 4 chr13 32893363 32893466 BRCA2 32899127 5661 False
我想将这个中间数据帧放入以下函数中以获得所需的输出（如下所示）：
new_bed = pd.DataFrame(data=np.zeros((0,len(columns))),columns=columns)
def bed_collapse(intermediate_bed, new_bed,columns=columns):
for row in bed.itertuples():
output = {}
if row[7] == False:
# If row doesn't overlap next row, insert into new dataframe unchanged.
output_row = list(row[1:5])
if row[7] == True:
# For overlapping rows take the chromosome and start coordinate
output_row = list(row[1:3])
# Iterate to next row
bed.itertuples().next()
# Append stop coordinate and geneID
output_row.append(row[3])
output_row.append(row[4])
#print output_row
for k, v in zip(columns,output_row): otpt[k] = v
#print output
new_bed = new_bed.append(otpt,ignore_index=True)
output_bed = new_bed
return output_bed
int_bed = bed_prepare(bed)
print bed_collapse(int_bed,new_bed)
期望的输出：
chr start stop geneID 0 chr13 32889584 32889814 BRCA2 1 chr13 32890536 32890737 BRCA2 2 chr13 32893194 32893466 BRCA2 5 chr13 32899127 32899242 BRCA2
然而，当我运行该函数时，我将原始数据帧保持不变。我知道问题是当我尝试调用bed.itertuples（）。next（）时，因为这显然不是调用的正确语法/位置。但我不知道纠正这种情况的正确方法。
一些指针会很棒。
SB：）
这是BED file，其中每行指的是具有起点和终点坐标的扩增子（基因组区域）。一些扩增子重叠;即起始坐标位于前一行的停止坐标之前。因此，我需要确定哪些行重叠并连接正确的开始和停止，以便每行代表并且完全独特的扩增子不会与任何其他行重叠。
答案 0 :(得分：1)
我会尽力给你一些指示。
一个指针是你想要根据一个由被移位的布尔组成的系列得到行。也许您可以使用以下方式获得新的移位系列：
Boolean_Series = intermediate_bed.loc[:,'next_region_overlap'].shift(periods=1, freq=None, axis=0, **kwds)
有关此功能的更多背景信息： http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.shift.html
第二个指针是通过使用此移位的系列，您可以通过以下方式获取Dataframe：
int_bed = bed.loc[Boolean_Series, :]
有关索引的更多信息，请访问： http://pandas.pydata.org/pandas-docs/dev/indexing.html
现在这些只是指针，我不知道这是否是一个实际可行的解决方案。
答案 1 :(得分：1)
我不确定我理解你为什么要做你正在做的事情，但你可以通过简单地使用索引来获得你想要的输出。例如
# assume your data is stored in <df>
# call the temporary dataframe <tmp>
tmp = df[ ['chr','start','stop','geneID'] ][(df.stop - df.start.shift(-1))>0]
这是你最终要做的事情吗？
UPDATE 好的，我明白你在做什么。请记住，我从未使用任何基因组数据，因此我不知道您的列中有多少行如此简单的“循环”可能会非常慢（如果您有几十亿行可能需要一段时间），但这是我想到的唯一解决方案。这是首先要考虑的事情（注意：这不是成品，因为您需要确定如何处理引入的NaN以及如何处理循环终止）。
import pandas as pd
df = pd.DataFrame(index = [0,1,2,3,4,5],columns=['chr','start','stop','geneID'])
df['chr'] = np.array( ['chr13']*6 )
df['start'] = np.array( [32889584,32890536,32893194,32893282,32893363,32899127] )
df['stop'] = np.array( [32889814,32890737,32893307,32893400,32893466,32899242] )
df['geneID'] = np.array( ['BRCA2']*6 )
# calculate difference between start/stop times for adjacent rows
# this will effectively "look into the future" to see if the upcoming row has
# a start time that is greater than the current stop time
df['tdiff'] = (df.start - df.stop.shift(1)).shift(-1)
# create new dataframe
df_cut = df.copy()*0
r = 0
while r < df.shape[0]:
if df.tdiff[r] > 0:
df_cut.iloc[r] = df.iloc[r]
r+=1
elif df.tdiff.iloc[r] < 0: # have to determine how you will handle the NaN's later
df_cut.chr.iloc[r] = df.chr.iloc[r]
df_cut.start.iloc[r] = df.start.iloc[r]
df_cut.geneID.iloc[r] = df.geneID.iloc[r]
# get the next-valid row and put "stop" value into <df_cut>
df_cut.stop.iloc[r] = df.ix[r:][df.tdiff>0].stop.iloc[0]
# determine new index location for <r>
r = df.ix[r:][df.tdiff>0].index[0] + 1
# eliminate empty rows
df_cut = df_cut[df_cut.start<>0]
跑完后：
>>> df_cut
chr start stop geneID tdiff
0 chr13 32889584 32889814 BRCA2 722
1 chr13 32890536 32890737 BRCA2 2457
2 chr13 32893194 32893466 BRCA2 -0
答案 2 :(得分：1)
我修改了bed_prepare函数以检查上一个和下一个基因组区域的重叠：
def bed_prepare(inp_bed):
''' Takes pandas dataframe bed file and identifies which regions overlap '''
inp_bed['next_start'] = inp_bed['start'].shift(periods=-1)
inp_bed['distance_to_next'] = inp_bed['next_start'] - inp_bed['stop']
inp_bed['next_region_overlap'] = inp_bed['next_start'] <= inp_bed['stop']
inp_bed['previous_stop'] = inp_bed['stop'].shift(periods=1)
inp_bed['distance_from_previous'] = inp_bed['start'] - inp_bed['previous_stop']
inp_bed['previous_region_overlap'] = inp_bed['previous_stop'] >= inp_bed['start']
intermediate_bed = inp_bed
return intermediate_bed
然后我使用这些布尔输出来为写入步骤执行变量存储：
# Create empty dataframe to fill with parsed values
new_bed = pd.DataFrame(data=np.zeros((0,len(columns))),columns=columns,dtype=int)
def bed_collapse(intermediate_bed, new_bed,columns=columns):
''' Takes a pandas dataframe bed file with overlap information and returns
genomic regions without overlaps '''
output_row = []
for row in bed.itertuples():
output = {}
if row[7] == False and row[10] == False:
# If row doesn't overlap next row, insert into new dataframe unchanged.
output_row = list(row[1:5])
elif row[7] == True and row[10] == False:
# Only next region overlaps; take the chromosome and start coordinate
output_row = list(row[1:3])
elif row[7] == True and row[10] == True:
# Next and previous regions overlap. Skip row.
pass
elif row[7] == False and row[10] == True:
# Only previous region overlaps; append stop coordinate and geneID to output_row variable
output_row.append(row[3])
output_row.append(row[4])
if row[7] == False:
#Zip columns and output_row values together to form a dict for appending
for k, v in zip(columns,output_row): output[k] = v
#print output
new_bed = new_bed.append(output,ignore_index=True)
output_bed = new_bed
return output_bed
这已解决了我的问题，并提供了问题中指定的所需输出。 ：）
答案 3 :(得分：1)
pyranges将使您可以在一行代码中快速完成此操作：
import pyranges as pr
c = """Chromosome Start End geneID
chr13 32889584 32889814 BRCA2
chr13 32890536 32890737 BRCA2
chr13 32893194 32893307 BRCA2
chr13 32893282 32893400 BRCA2
chr13 32893363 32893466 BRCA2
chr13 32899127 32899242 BRCA2"""
gr = pr.from_string(c)
# +--------------+-----------+-----------+------------+
# | Chromosome | Start | End | geneID |
# | (category) | (int32) | (int32) | (object) |
# |--------------+-----------+-----------+------------|
# | chr13 | 32889584 | 32889814 | BRCA2 |
# | chr13 | 32890536 | 32890737 | BRCA2 |
# | chr13 | 32893194 | 32893307 | BRCA2 |
# | chr13 | 32893282 | 32893400 | BRCA2 |
# | chr13 | 32893363 | 32893466 | BRCA2 |
# | chr13 | 32899127 | 32899242 | BRCA2 |
# +--------------+-----------+-----------+------------+
# Unstranded PyRanges object has 6 rows and 4 columns from 1 chromosomes.
# For printing, the PyRanges was sorted on Chromosome.
m = gr.merge(by="geneID")
# +--------------+-----------+-----------+------------+
# | Chromosome | Start | End | geneID |
# | (category) | (int32) | (int32) | (object) |
# |--------------+-----------+-----------+------------|
# | chr13 | 32889584 | 32889814 | BRCA2 |
# | chr13 | 32890536 | 32890737 | BRCA2 |
# | chr13 | 32893194 | 32893466 | BRCA2 |
# | chr13 | 32899127 | 32899242 | BRCA2 |
# +--------------+-----------+-----------+------------+
# Unstranded PyRanges object has 4 rows and 4 columns from 1 chromosomes.
# For printing, the PyRanges was sorted on Chromosome.
请注意，by="geneID"使得间隔仅在它们重叠且对geneID具有相同值时才被合并。如果要将区间元数据与自定义函数合并，请参见方法集群。