Question

我试图加速以下代码, 给它一个字符串列表 < code> > list I m, 试图将字符串转换为数字 ( unpack ), 并将此数字指定到嵌套列表 data 的正确位置。 data 的维度大致是 data[4][20][1024] 。不幸的是, 此函数运行非常慢。这里的代码是 :

for abs_idx in range(nbr_elements):

    # get string
    this_element = str_list[abs_idx]

    # convert into number
    this_element = unpack( d , this_element)[0]

    # calculate the buffer number
    buffer_nbr = abs_idx / NBR_DATA_POINTS_PER_BUFFER_INT

    # calculate the position inside the buffer
    index_in_buffer = abs_idx % NBR_DATA_POINTS_PER_BUFFER_INT

    # write data into correct position
    data[file_idx][buffer_nbr][index_in_buffer] = this_element

我还尝试了以下更慢的替代解决办法:

# convert each string into a number
unpacked_values = [unpack( d , str_list[j])[0] for j in range(nbr_elements)]
for abs_idx in range(nbr_elements):

    # calculate the buffer number
    buffer_nbr = abs_idx / NBR_DATA_POINTS_PER_BUFFER_INT

    # calculate the position inside the buffer
    index_in_buffer = abs_idx % NBR_DATA_POINTS_PER_BUFFER_INT

    # write data into correct position
    data[file_idx][buffer_nbr][index_in_buffer] = unpacked_values[abs_idx]

令我惊讶的是,下一个执行最慢(我希望它最快):

# convert each string into a number
unpacked_values = [unpack( d , str_list[j])[0] for j in range(nbr_elements)]

# calculate all buffer numbers at once
buffer_ids = np.arange(nbr_elements) / NBR_DATA_POINTS_PER_BUFFER_INT

# calculate all positions inside the buffer at once
index_in_buffer_id = np.arange(nbr_elements) % NBR_DATA_POINTS_PER_BUFFER_INT

for abs_idx in range(nbr_elements):
    data[file_idx][buffer_ids[abs_idx]][index_in_buffer_id[abs_idx]] = unpacked_values[abs_idx]

为什么连续实施的情况更差? 个别的瓶颈在哪里? 我怎样才能加快我的初始代码?

EDIT: from my 剖析测试,以下两个步骤是瓶颈:运行 unpack 并给 data 指定值。我不知道如何加快这些步骤。

EDIT2: 我需要使用 unpack ,因为我的字符串在十六进制中。

< 加强> EDIT3 : values = unfack ("d" * n, "".join(str_list)) 解决了拆解速度缓慢的问题。然而,使用三重(原始)或双重(修改后的)嵌套环的数据分配会消耗50%的时间。是否有办法缩短这一时间?

Answer 1

一些优化 :

Unpack all stings at once
Get item data[file_idx] before loop

试试看:

n = len(str_list)
values = unpack("d" * n, "".join(str_list))

a = data[file_idx]

# Just to shorten this code sample
q = NBR_DATA_POINTS_PER_BUFFER_INT

for i in xrange(n):
    a[i / q][i % q] = values[i]

Btw,你有没有描述代码的哪个部分最需要时间?

<强 > UPATE:

n = len(str_list)
values = unpack("d" * n, "".join(str_list))

# Just to shorten this code sample
q = NBR_DATA_POINTS_PER_BUFFER_INT

data[file_idx] = [values[i:i+q] for i in xrange(0, n, q)]

Answer 2

这样可以更快吗? 它会减少一些检查, 不需要用所有字符串的编号来做中间列表吗?

df = data[file_idx]
index = 0
for value in str_list:
    # not sure what unpack does... is there a faster function 
    # that does the same?
    number = unpack( d , value)[0]

    # calculate the buffer number
    buffer_nbr = index / NBR_DATA_POINTS_PER_BUFFER_INT

    # calculate the position inside the buffer
    index_in_buffer = index % NBR_DATA_POINTS_PER_BUFFER_INT

    # write data into correct position
    df[buffer_nbr][index_in_buffer] = number

    index += 1

Answer 3

不如这样:

df = data[file_idx]
index = 0
bufnr = 0
buf = df[0]
for value in str_list:
    # not sure what unpack does... is there a faster function 
    # that does the same?
    number = unpack( d , value)[0]

    buf[index] = number

    index += 1

    if index >= NBR_DATA_POINTS_PER_BUFFER_INT:
        index = 0
        bufnr += 1
        buf = df[bufnr]

数据是否是一个字典而不是一个列表?

友情链接