Conversion code from "implementation details" character index to word index

Example

In the reading comprehension task , The answer is often to give the index of the starting position of the character , Such as SQuAD1.1 An example of a dataset is as follows .

context：
Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the “golden anniversary” with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as “Super Bowl L”), so that the logo could prominently feature the Arabic numerals 50.

answer：

however , This answer_start It is often not the index we need , So we need to turn it into Denver Broncos In this context Word index position in .

Code

context:

she loves this puppy.

question:

what does this girl love?

answer:

puppy

answer_start:

15

tokens:

[‘she’,‘loves’,‘this’,‘puppy’,‘.’]

punctuations:

[’ ‘,’\n’]

idx = 0
flag = False
s_idx = answer_start
e_idx = s_idx + len(answer)

for i,t in enumerate(tokens):
	#  Calculation contains spaces 、 Characters including line breaks 
	while idx < len(context):
	    if context[idx] in punctuations:
	        idx += 1
	    else:
	        break
	#  Count the length of the word 
	idx += len(t)
	if idx > s_idx and flag == False:
	    #  The starting position of the word sequence 
	    s_idx = i
	    flag = True
	if idx >= e_idx:
	    #  End position of word sequence 
	    e_idx = i
	    break