Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[download fail]: bilibili download failed: panic: runtime error: index out of range [1] with length 0 #1213

Closed
scegg opened this issue Mar 11, 2023 · 15 comments · Fixed by #1227
Labels
bilibili bug Something isn't working

Comments

@scegg
Copy link

scegg commented Mar 11, 2023

Website name: Bilibili

OS:: Windows

Video URL:: https://www.bilibili.com/bangumi/play/ep704485

Stack overflow

panic: runtime error: index out of range [1] with length 0

Screenshots

none

Additional context

lux -c bl.txt -debug https://www.bilibili.com/bangumi/play/ep704485

lux: version v0.16.0, A fast and simple video downloader.


URL:         https://www.bilibili.com/bangumi/play/ep704485
Method:      GET
Headers:     http.Header{
    "Accept":          {"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
    "Accept-Charset":  {"UTF-8,*;q=0.5"},
    "Accept-Encoding": {"gzip,deflate,sdch"},
    "Accept-Language": {"en-US,en;q=0.8"},
    "Cookie":          {"_uuid=D9B1AADB-DDF4-10BBA-CF58-676253CDDE9A78747infoc; b_lsid=108EAD7AA_186CEA2D804; b_nut=1663404979; bili_jct=303538e45b28a6bbed35cfb62adb1133; bp_video_offset_26749913=766518492787638345; buvid_fp=950e665fa77caae27ca72c832f0cfeec; buvid_fp_plain=undefined; buvid3=1024B78F-A3CB-BFEF-9B57-E56D1B11145779434infoc; buvid4=3EA21217-05C0-B28E-33A3-2402ACF7E85F88375-022020721-Qd0NZsQY6NFvAP9XHGIwcQ%3D%3D; CURRENT_FNVAL=4048; DedeUserID=26749913; DedeUserID__ckMd5=b94757e248393f8e; fingerprint=950e665fa77caae27ca72c832f0cfeec; LIVE_BUVID=AUTO8416697171148738; nostalgia_conf=-1; PVID=1; rpdid=|(k||RRkYmmJ0J'uY~||llukl; SESSDATA=dc233144%2C1694054793%2C88608%2A31; sid=e3u29g0y"},
    "Referer":         {"https://www.bilibili.com"},
    "User-Agent":      {"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36"},
}
Status Code: 200
panic: runtime error: index out of range [1] with length 0

goroutine 1 [running]:
github.com/iawia002/lux/extractors/bilibili.extractBangumi({0xc0000d4030, 0x2e}, {0xc000a4e000, 0x1903cd}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/bilibili/bilibili.go:95 +0x825
github.com/iawia002/lux/extractors/bilibili.(*extractor).Extract(0x0?, {0xc0000d4030, 0x2e}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, {0xc0000b2078, ...}, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/bilibili/bilibili.go:298 +0x19d
github.com/iawia002/lux/extractors.Extract({0xc0000d4030?, 0x7ff8965304f0?}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0xa, {0xc0000b2078, 0x6}, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/extractors.go:52 +0x391
github.com/iawia002/lux/app.download(0xc0002e6000?, {0xc0000d4030, 0x2e})
        /Users/iawia002/go/src/github.com/iawia002/lux/app/app.go:270 +0x2f2
github.com/iawia002/lux/app.New.func1(0xc000265e40)
        /Users/iawia002/go/src/github.com/iawia002/lux/app/app.go:247 +0x69d
github.com/urfave/cli/v2.(*App).RunContext(0xc0002744e0, {0xaffc48?, 0xc0000b2140}, {0xc0000de000, 0x5, 0x8})
        /Users/iawia002/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:322 +0x953
github.com/urfave/cli/v2.(*App).Run(...)
        /Users/iawia002/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:224
main.main()
        /Users/iawia002/go/src/github.com/iawia002/lux/main.go:13 +0x53
@scegg scegg added the bug Something isn't working label Mar 11, 2023
@scegg
Copy link
Author

scegg commented Mar 11, 2023

in version 17:

lux -c bl.txt -debug https://www.bilibili.com/bangumi/play/ep704485

lux: version v0.17.0, A fast and simple video downloader.


URL:         https://www.bilibili.com/bangumi/play/ep704485
Method:      GET
Headers:     http.Header{
    "Accept":          {"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
    "Accept-Charset":  {"UTF-8,*;q=0.5"},
    "Accept-Encoding": {"gzip,deflate,sdch"},
    "Accept-Language": {"en-US,en;q=0.8"},
    "Cookie":          {"_uuid=D9B1AADB-DDF4-10BBA-CF58-676253CDDE9A78747infoc; b_lsid=108EAD7AA_186CEA2D804; b_nut=1663404979; bili_jct=303538e45b28a6bbed35cfb62adb1133; bp_video_offset_26749913=766518492787638345; buvid_fp=950e665fa77caae27ca72c832f0cfeec; buvid_fp_plain=undefined; buvid3=1024B78F-A3CB-BFEF-9B57-E56D1B11145779434infoc; buvid4=3EA21217-05C0-B28E-33A3-2402ACF7E85F88375-022020721-Qd0NZsQY6NFvAP9XHGIwcQ%3D%3D; CURRENT_FNVAL=4048; DedeUserID=26749913; DedeUserID__ckMd5=b94757e248393f8e; fingerprint=950e665fa77caae27ca72c832f0cfeec; LIVE_BUVID=AUTO8416697171148738; nostalgia_conf=-1; PVID=1; rpdid=|(k||RRkYmmJ0J'uY~||llukl; SESSDATA=dc233144%2C1694054793%2C88608%2A31; sid=e3u29g0y"},
    "Referer":         {"https://www.bilibili.com"},
    "User-Agent":      {"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36"},
}
Status Code: 200
panic: runtime error: index out of range [1] with length 0

goroutine 1 [running]:
github.com/iawia002/lux/extractors/bilibili.extractBangumi({0xc00013e060, 0x2e}, {0xc00072c000, 0x1903d7}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/bilibili/bilibili.go:95 +0x806
github.com/iawia002/lux/extractors/bilibili.(*extractor).Extract(0x208002e?, {0xc00013e060, 0x2e}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, {0xc000126078, ...}, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/bilibili/bilibili.go:298 +0x19d
github.com/iawia002/lux/extractors.Extract({0xc00013e060?, 0xb9361b?}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0xa, {0xc000126078, 0x6}, ...})
        /Users/iawia002/go/src/github.com/iawia002/lux/extractors/extractors.go:52 +0x391
github.com/iawia002/lux/app.download(0xc0002d8000?, {0xc00013e060, 0x2e})
        /Users/iawia002/go/src/github.com/iawia002/lux/app/app.go:270 +0x2f2
github.com/iawia002/lux/app.New.func1(0xc000255ac0)
        /Users/iawia002/go/src/github.com/iawia002/lux/app/app.go:247 +0x689
github.com/urfave/cli/v2.(*App).RunContext(0xc000105380, {0x15673e8?, 0xc000126130}, {0xc000154000, 0x5, 0x8})
        /Users/iawia002/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:322 +0x97c
github.com/urfave/cli/v2.(*App).Run(0xc00003c0c8?, {0xc000154000?, 0x101c000070088?, 0x1f142ba1cc8?})
        /Users/iawia002/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:224 +0x38
main.main()
        /Users/iawia002/go/src/github.com/iawia002/lux/main.go:13 +0x45

@Hypnodance
Copy link

Same problem here, it may be possible to download after a few more attempts. Generally, the download can be successful within 10 attempts, but it is really troublesome to encounter this problem repeatedly.

@scegg
Copy link
Author

scegg commented Mar 12, 2023

Same problem here, it may be possible to download after a few more attempts. Generally, the download can be successful within 10 attempts, but it is really troublesome to encounter this problem repeatedly.

Tried 10 hrs with no luck at all.

@cj3617
Copy link

cj3617 commented Mar 14, 2023

请更改 bilibi页面获取aid方式 python 实例
initial_state_text = match1(html_content,
r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>') # FIXME
initial_state_data = json.loads(initial_state_text)
initial_state = initial_state_data['props']['pageProps']['dehydratedState']['queries'][0]['state']['data']
initial_map = initial_state['epMap'][ep_id]
aid = initial_map['aid']
cid = initial_map['cid']

@Nathaniel-Wu
Copy link
Contributor

I'm not familiar with Golang and surely there are more elegant and complete solutions to the issue, but here's a rudimentary implementation (in diff format) for the discovery posted by @cj3617 above. This fix does not purge all the problems so I'm not opening a pull request. It enables lux to download bangumi video files just fine, but the episode names are not retrieved correctly, and files are named by bangumi names only.
If you need to download things right now like me, edit extractors/bilibili/bilibili.go accordingly and compile main.go yourself.

-       dataString := utils.MatchOneOf(html, `window.__INITIAL_STATE__=(.+?);\(function`)[1]
-       var data bangumiData
-       err := json.Unmarshal([]byte(dataString), &data)
+       dataString := utils.MatchOneOf(html, `<script\s+id="__NEXT_DATA__"\s+type="application/json"\s*>(.*?)</script\s*>`)[1]
+       epMapString := utils.MatchOneOf(dataString, `"epMap"\s*:\s*(.+?)\s*,\s*"initEpList"`)[1]
+       videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ep(\d+)"`)[1]
+       var epMap map[string]json.RawMessage
+       err := json.Unmarshal([]byte(epMapString), &epMap)
        if err != nil {
                return nil, errors.WithStack(err)
        }
+       var data bangumiData
+       for idString, jsonByte := range epMap {
+               var EpInfo bangumiEpData
+               err := json.Unmarshal(jsonByte, &EpInfo)
+               if err != nil {
+                       return nil, errors.WithStack(err)
+               }
+               EpID, err := strconv.ParseInt(idString, 10, 0);
+               if err != nil {
+                       return nil, errors.WithStack(err)
+               }
+               EpInfo.EpID = int(EpID)
+               if idString == videoIdString {
+                       data.EpInfo = EpInfo
+               }
+               data.EpList = append(data.EpList, EpInfo)
+       }

@scegg
Copy link
Author

scegg commented Mar 18, 2023

After changing source code, the error is still with new edited lines:

panic: runtime error: index out of range [1] with length 0

goroutine 1 [running]:
github.com/iawia002/lux/extractors/bilibili.extractBangumi({0xc000038190, 0x45}, {0xc000580000, 0x1c2590}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, ...})
        d:/Downloaded/DownTool/lux/extractors/bilibili/bilibili.go:97 +0xc86
github.com/iawia002/lux/extractors/bilibili.(*extractor).Extract(0x208002e?, {0xc000038190, 0x45}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, {0xc00002e0d8, ...}, ...})
        d:/Downloaded/DownTool/lux/extractors/bilibili/bilibili.go:317 +0x19d
github.com/iawia002/lux/extractors.Extract({0xc000038190?, 0x14361b?}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0xa, {0xc00002e0d8, 0x6}, ...})
        d:/Downloaded/DownTool/lux/extractors/extractors.go:52 +0x391
github.com/iawia002/lux/app.download(0xc0002ba500?, {0xc000038190, 0x45})
        d:/Downloaded/DownTool/lux/app/app.go:270 +0x2f2
github.com/iawia002/lux/app.New.func1(0xc000239b00)
        d:/Downloaded/DownTool/lux/app/app.go:247 +0x689
github.com/urfave/cli/v2.(*App).RunContext(0xc0002621a0, {0xb175c8?, 0xc00002e180}, {0xc00008a040, 0x4, 0x4})
        C:/Users/Allen/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:322 +0x97c
github.com/urfave/cli/v2.(*App).Run(0xc00003e0c8?, {0xc00008a040?, 0x100c000072088?, 0x1c0c8759a38?})
        C:/Users/Allen/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:224 +0x38
main.main()
        d:/Downloaded/DownTool/lux/main.go:13 +0x45

@jjit
Copy link

jjit commented Mar 18, 2023

I can download it based on Nathaniel-Wu's change
just download the Source code(zip) from here
https://github.com/iawia002/lux/releases/tag/v0.17.0
unzip it and edit extractors/bilibili/bilibili.go
you need to delete the first 3 lines code
then use golang docker image to compile
docker run --rm -v /path/to/lux:/usr/src/myapp -w /usr/src/myapp -e GOOS=windows -e GOARCH=amd64 golang go build -v

bl

@scegg
Copy link
Author

scegg commented Mar 18, 2023

It failed on my Windows. I dont have docker installed on my Windows.

the function edited is:

func extractBangumi(url, html string, extractOption extractors.Options) ([]*extractors.Data, error) {
      dataString := utils.MatchOneOf(html, `<script\s+id="__NEXT_DATA__"\s+type="application/json"\s*>(.*?)</script\s*>`)[1]
      epMapString := utils.MatchOneOf(dataString, `"epMap"\s*:\s*(.+?)\s*,\s*"initEpList"`)[1]
      videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ep(\d+)"`)[1]
      var epMap map[string]json.RawMessage
      err := json.Unmarshal([]byte(epMapString), &epMap)
	if err != nil {
		return nil, errors.WithStack(err)
	}
      var data bangumiData
      for idString, jsonByte := range epMap {
              var EpInfo bangumiEpData
              err := json.Unmarshal(jsonByte, &EpInfo)
              if err != nil {
                      return nil, errors.WithStack(err)
              }
              EpID, err := strconv.ParseInt(idString, 10, 0);
              if err != nil {
                      return nil, errors.WithStack(err)
              }
              EpInfo.EpID = int(EpID)
              if idString == videoIdString {
                      data.EpInfo = EpInfo
              }
              data.EpList = append(data.EpList, EpInfo)
      }
	if !extractOption.Playlist {
		aid := data.EpInfo.Aid
		cid := data.EpInfo.Cid
		bvid := data.EpInfo.BVid
...

Error:

panic: runtime error: index out of range [1] with length 0

goroutine 1 [running]:
github.com/iawia002/lux/extractors/bilibili.extractBangumi({0xc000038190, 0x45}, {0xc000680000, 0x1c25a0}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, ...})
        d:/Downloaded/DownTool/lux/extractors/bilibili/bilibili.go:97 +0xc86
github.com/iawia002/lux/extractors/bilibili.(*extractor).Extract(0x208002e?, {0xc000038190, 0x45}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0x1, {0xc00002e0d8, ...}, ...})
        d:/Downloaded/DownTool/lux/extractors/bilibili/bilibili.go:317 +0x19d
github.com/iawia002/lux/extractors.Extract({0xc000038190?, 0xa3361b?}, {0x0, {0x0, 0x0}, 0x1, 0x0, 0xa, {0xc00002e0d8, 0x6}, ...})
        d:/Downloaded/DownTool/lux/extractors/extractors.go:52 +0x391
github.com/iawia002/lux/app.download(0xc0002ba500?, {0xc000038190, 0x45})
        d:/Downloaded/DownTool/lux/app/app.go:270 +0x2f2
github.com/iawia002/lux/app.New.func1(0xc00023bb00)
        d:/Downloaded/DownTool/lux/app/app.go:247 +0x689
github.com/urfave/cli/v2.(*App).RunContext(0xc0002641a0, {0x14075c8?, 0xc00002e180}, {0xc00008a040, 0x4, 0x4})
        C:/Users/Allen/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:322 +0x97c
github.com/urfave/cli/v2.(*App).Run(0xc00003e0c8?, {0xc00008a040?, 0x100c000072088?, 0x1dfa8d49a38?})
        C:/Users/Allen/go/pkg/mod/github.com/urfave/cli/v2@v2.6.0/app.go:224 +0x38
main.main()
        d:/Downloaded/DownTool/lux/main.go:13 +0x45

the bilibili.go:97 is

videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ep(\d+)"`)[1]

Is there anyone who want to fork this project for a quick fix?

@scegg
Copy link
Author

scegg commented Mar 18, 2023

After I dig the text dataString, I found that there is no such a text can match the condition of the line 97.

The only one "videoId" (with quotes) is near the end of dataString. The context is:

...,"page":"/bangumi/play/[videoId]","query":{"spm_id_from":"333.337.0.0","videoId":"ss26257"},"buildId":"UjsRaNzhtNURzCjvPY8PC","assetPrefix":...

No such an ep at all.

@jjit
Copy link

jjit commented Mar 18, 2023

compiled exe based on Nathaniel-Wu's change
win x64: lux.zip

@scegg
Copy link
Author

scegg commented Mar 18, 2023

compiled exe based on Nathaniel-Wu's change win x64: lux.zip

Thanks a lot.

By providing the url including ep other than ss, the exe you compiled as well as mine works.

@fgr178707
Copy link

fgr178707 commented Mar 30, 2023

根据Nathaniel-Wu的更改Win x64:lux编译的exe.zip

你这个 Title: 只会显示这个番的名字 这一集叫啥啥也没有
正常的话应该叫 Title: 幸运星:第1话 狂奔的女人
image
image
英文:
Your Title: Only the name of this episode will be displayed
Normally it should be called Title: Lucky Star: Chapter 1 The Running Woman
@jjit

@jjit
Copy link

jjit commented Mar 30, 2023

@fgr178707 是基于这个帖子里 Nathaniel-Wu 发的修改编译的
他在原帖里有说,可以往上面翻看他发的帖

but the episode names are not retrieved correctly
and files are named by bangumi names only

番集名字没有正确提取,文件只能以番的名字命名

@ChrisVicky
Copy link

compiled exe based on Nathaniel-Wu's change win x64: lux.zip

Thanks a lot.

By providing the url including ep other than ss, the exe you compiled as well as mine works.

By Replacing

videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ep(\d+)"`)[1]

with

videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ss(\d+)"`)[1]

you'll be able to download videos with ss in the url.

And you could modify the Regular Expression to satisfy both ss and ep conditions.

@baiyufei
Copy link
Contributor

I'm not familiar with Golang and surely there are more elegant and complete solutions to the issue, but here's a rudimentary implementation (in diff format) for the discovery posted by @cj3617 above. This fix does not purge all the problems so I'm not opening a pull request. It enables lux to download bangumi video files just fine, but the episode names are not retrieved correctly, and files are named by bangumi names only. If you need to download things right now like me, edit extractors/bilibili/bilibili.go accordingly and compile main.go yourself.

-       dataString := utils.MatchOneOf(html, `window.__INITIAL_STATE__=(.+?);\(function`)[1]
-       var data bangumiData
-       err := json.Unmarshal([]byte(dataString), &data)
+       dataString := utils.MatchOneOf(html, `<script\s+id="__NEXT_DATA__"\s+type="application/json"\s*>(.*?)</script\s*>`)[1]
+       epMapString := utils.MatchOneOf(dataString, `"epMap"\s*:\s*(.+?)\s*,\s*"initEpList"`)[1]
+       videoIdString := utils.MatchOneOf(dataString, `"videoId"\s*:\s*"ep(\d+)"`)[1]
+       var epMap map[string]json.RawMessage
+       err := json.Unmarshal([]byte(epMapString), &epMap)
        if err != nil {
                return nil, errors.WithStack(err)
        }
+       var data bangumiData
+       for idString, jsonByte := range epMap {
+               var EpInfo bangumiEpData
+               err := json.Unmarshal(jsonByte, &EpInfo)
+               if err != nil {
+                       return nil, errors.WithStack(err)
+               }
+               EpID, err := strconv.ParseInt(idString, 10, 0);
+               if err != nil {
+                       return nil, errors.WithStack(err)
+               }
+               EpInfo.EpID = int(EpID)
+               if idString == videoIdString {
+                       data.EpInfo = EpInfo
+               }
+               data.EpList = append(data.EpList, EpInfo)
+       }

根据这个回答做了些修改,可以正确处理 ep 或者 ss 开头的id,可以识别分段标题,PR在这里 #1227

这是打包好的文件 lux.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bilibili bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants