Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize reddit/PRAW #1021

Closed
snarfed opened this issue Mar 17, 2021 · 3 comments
Closed

optimize reddit/PRAW #1021

snarfed opened this issue Mar 17, 2021 · 3 comments
Labels

Comments

@snarfed
Copy link
Owner

snarfed commented Mar 17, 2021

Our use of the Reddit API, via PRAW, is pretty badly inefficient right now. Not usually a big deal, but got a lot more noticeable recently due to https://brid.gy/reddit/lgats, which backfeeds a few big sites that get linked and mentioned on Reddit a lot: https://sec.report/, https://uspto.report/, https://lei.report/, https://fccid.io/.

Two obvious problems right now. First, PRAW repeats user lookups a lot. Example poll log:

D 2021-03-10 15:46:13.897607+00:00 Params: [('source_key', 'agdicmlk...'), ('last_polled', '2021-03-10-09-24-49')]
I 2021-03-10 15:46:13.915966+00:00 Source: schnarfed (Reddit) schnarfed, https://brid.gy/reddit/schnarfed
I 2021-03-10 15:46:13.916355+00:00 Last poll: https://brid.gy/log?start_time=1615368289&key=agdicmlkLWd5chULEgZSZWRkaXQiCXNjaG5hcmZlZAw
I 2021-03-10 15:46:13.931726+00:00 Updating schnarfed (Reddit) /reddit/schnarfed : {'poll_status': 'polling', 'last_poll_attempt': datetime.datetime(2021, 3, 10, 15, 46, 13, 916446), 'rate_limited': False}
D 2021-03-10 15:46:14.007187+00:00 Using ETag None, last activity id uouxi
D 2021-03-10 15:46:14.128411+00:00 Fetching: GET https://oauth.reddit.com/r/all/search/
D 2021-03-10 15:46:14.128634+00:00 Data: None
D 2021-03-10 15:46:14.128711+00:00 Params: {'q': 'site:"snarfed.org" OR selftext:"snarfed.org"', 'restrict_sr': False, 'sort': 'new', 'syntax': 'lucene', 't': 'all', 'limit': 100, 'raw_json': 1}
...
D 2021-03-10 15:46:14.861554+00:00 Fetching: GET https://oauth.reddit.com/user/milkbone20/about/
D 2021-03-10 15:46:14.861680+00:00 Data: None
D 2021-03-10 15:46:14.861738+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:14.950963+00:00 Response: 200 (656 bytes)
D 2021-03-10 15:46:14.951954+00:00 Fetching: GET https://oauth.reddit.com/user/gst/about/
D 2021-03-10 15:46:14.952096+00:00 Data: None
D 2021-03-10 15:46:14.952163+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:15.057583+00:00 Response: 200 (655 bytes)
D 2021-03-10 15:46:15.058516+00:00 Fetching: GET https://oauth.reddit.com/user/gst/about/
D 2021-03-10 15:46:15.058632+00:00 Data: None
D 2021-03-10 15:46:15.058688+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:15.152530+00:00 Response: 200 (655 bytes)

eg in that poll, https://oauth.reddit.com/user/gst/about/ was fetched five times.

Second, we're fetching every comment in big threads, even in source mf2 handlers for individual comments, which seems excessive. Example log from /comment/reddit/lgats/m74e3g/gr9rmye:

2021-03-17 13:36:07.204 PDT Fetching: GET https://oauth.reddit.com/comments/m74e3g/
2021-03-17 13:36:07.204 PDT Data: None
2021-03-17 13:36:07.204 PDT Params: {'limit': 2048, 'sort': 'confidence', 'raw_json': 1}
2021-03-17 13:36:10.915 PDT Response: 200 (138240 bytes)
2021-03-17 13:36:11.192 PDT Fetching: GET https://oauth.reddit.com/user/Pesa2w/about/
2021-03-17 13:36:11.192 PDT Data: None
2021-03-17 13:36:11.192 PDT Params: {'raw_json': 1}
2021-03-17 13:36:11.280 PDT Response: 200 (868 bytes)
2021-03-17 13:36:11.282 PDT Fetching: GET https://oauth.reddit.com/comments/m74e3g/
2021-03-17 13:36:11.283 PDT Data: None
2021-03-17 13:36:11.283 PDT Params: {'limit': 2048, 'sort': 'confidence', 'raw_json': 1}
2021-03-17 13:36:14.329 PDT Response: 200 (138198 bytes)
2021-03-17 13:36:15.197 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
2021-03-17 13:36:15.197 PDT Data: [('api_type', 'json'), ('children', 'gr9vsp0,gr9vtqu,gr9diip,gr9l349,gr9cx6y,gr9i6tp,gr9wa32,gr9zcpc,gr9r6s4,gr9et5k,gr9klqy,gra0n1r,gr9fwny,gr9lkhx,gr9tc80,gr9ruht,gr97kfa,gr9pauh,gr9sq3u,gr9p113,gr9ttm4,gra03la,gr9e5gv,gr9izao,gra0d2p,gr9mjar,gr9oiew,gr9f2nw,gr9vubl,gr9tqh3,gr9sle3,gr9xdn0,gr9uixa,gr9obso,gr9xljq,gr9qb88,gr9tduk,gr9h84d,gr9gx29,gra0hur,gr9emw4,gr9tili,gr9j8tn,gr9aagg,gr9sbxu,gr9i5bn,gr9n71x,gr9f5va,gr9h4zj,gr9s2gw,gr9zm9w,gr9ia2t,gr9vr7q,gr9xifc,gr9t7kc,gr9faml,gr9gscx,gr9q1sn,gr9igex,gr9hcwt,gr9m9we,gr9ox51,gr9ynj5,gr9fdly,gr9o90p,gr9x12j,gr9qct9,gr9psbz,gr9x2nk,gr9x12q,gr9suyb,gr9rbnc,gr9jbv9,gr9w2bl,gr9sjwi,gr9z7pn,gr9ni6g,gr9tozt,gr9fikq,gr9m21a,gr9zj67,gr9u1n5,gr9x903,gr9yh91,gr9x49j,gr9vv38,gr9zgen,gr9wezg,gr9ejdi,gr9cvry,gr9gqgv,gr9csm9,gr9jm7a,gr9v8bq,gr9n0tj,gr9l3ag,gr9a12h,gr9db2p,gr9zcvg,gr9lnu6,gr9mrcf,gr9oqgm,gr9xlo9,gr9v0fv,gr9fsgl,gr9euwx,gr9di4b,gr9bnk3,gr9vuh5,gr9irx3,gr9i8lp,gr9hnmg,gr9pfrm,gr9g6c3,gr9mwq7,gr9sqal,gra0273,gr9t4io,gr9zvvl,gr9cxer,gra0a3q,gr9pe0h,gr9ni96,gr9wt9o,gr9kffw,gr9nacz,gr9j5tk,gr9esdv,gr9hpof,gr9ml34,gr9uu6c,gr9cmdh,gr9i2bq,gr9sq27,gr9vgaw,gr9ynnt,gr9j03b,gr9l6j0,gr9o18t,gr9j13k,gr9e0yc,gr9uxcl,gr9euze,gr9l4yj,gr9n42m,gr9zuch,gr9ufz6,gr9h3kz,gr9sk15,gr9qlh9,gr9wf3m,gr9qjcj,gr9zek6,gr9v248,gr9xyex,gr9qp8i,gr9legv,gr9cg3l,gr9tymd,gr9ye86,gr9djm2,gr9x2ts,gr9wbyr,gr9yx7i,gra0k79,gr9vw6c,gr9ccyt,gr9kvjb,gr97hk8,gr9ii7e,gr9ef8t,gr9e46n,gr9yhf4,gr9z6pe,gr9ttx0,gra0ddk,gr9zp4k,gr9uk8l,gr9p4uf,gr9nxvw,gr9r8g0,gr9wv7b,gr9e11o,gr9lbd9,gr9yqxb,gr9mfw8,gr9pknm,gr9o637,gr9eifo,gr9u09i,gr9v3rt,gr9da6y,gr9tivw,gr9sdst,gr9k9gh,gr9da78,gr9kqud,gr9j5ye,gr9khd5,gr9r0tm,gr9z9wr,gr9vung,gr9o1d9,gr9rmye,gr9nqbc,gr9u516,gr9o4qn,gr9y39c,gr9bx8m,gr9ny87,gr9wah4,gr9ql1l,gr9t9fu,gr9z56z,gr9lngi,gr9pnv3,gr9nqc1,gr9ubdf,gr9s93p,gr9pxcu,gr9hd7j,gr9r2rx,gr9rjtw,gr9fkfb,gr9l00t,gr9p9o0,gr9h5bu,gr9onjv,gra0dh9,gr9sh1d,gr9mrbp,gr9pr2e,gr9qsm9,gr9he6k,gr9nsdb,gr9z0i5,gra05lc,gr9yhjl,gr9s191,gr9ffq2,gr9lz6j,gr9mrmq,gr9qt0j,gr9swv9,gr9hd9i,gr9x65j,gr9va7e,gr9xso7,gr9swvh,gr9f76s,gr9za0i,gr9vrli,gr9z6ux,gr9kqyi,gr9yb98,gr9ur9a,gr9kfwh,gr9tqwy,gr9qetg,gr9z8g6,gr9sozt,gr9tklm,gr9gmfh,gr9z0jy,gr9nfql,gr9f7v6,gr9rjtd,gr9xlzq,gr9qw7t,gr9hjmj,gr9hvf2,gr9faca,gr9ql5q,gr9h4g5,gr9xyn7,gr9avfp,gr9kpev,gr9e17t,gr9bp0y,gr9nfe6,gr9tpdb,gr9cer7,gr9epsu,gr9q27s,gr9wtkp,gr9vwe5,gr9ynya,gr9z26d,gr9nfyd,gr9iguh,gr9hrka,gr9i47f,gr9kma4,gr9rtfx,gr9v8pa,gr9ndn1,gr9f5jo,gr9nc9h,gr9y507,gr9l595,gr9gkwq,gr9rdni,gr9yr4z,gr9y86f,gr9vq3t,gr9wqga,gr9z0mj,gr9txbg,gr9xx4e,gr9lhwz,gr9ugad,gr9xx4n,gra0gsa,gr9fm67,gr9ykcg,gr9q3us,gr9lukm,gr9iakn,gr9donf,gr9qqjq,gr9slxv,gr9vgno,gr9ocnp,gr9i2ft,gr9ik2g,gr9tvs0,gr9v0v6,gr9mq78,gr9zu6e,gr9znbw,gr9wv8r,gr9f5t8,gr9avjm,gr9z4mr,gr9schk,gr9w94q,gr9opbw,gr9v8j0,gr98y0w,gr9mrsy,gr9tefd,gr9ws3u,gr9ox8r,gr9nxxp,gr9tyz4,gr9rc5x,gr9xmwp,gr9vq6y,gr9jhb0,gr9dy72,gr9igyn,gra0as0,gr9y3j6,gr9y89w,gr9oyjq,gr9x6co,gr9e99l,gr9zmuk,gr9p3la,gr9j0fc,gr9tu9e,gr9ljlf,gr9m2k4,gr9jsdx,gr9oa21,gr9gd4q,gr9jhc2,gr9dc3n,gr9iao5,gr9rtl3,gr9vdl5,gr9xsi5,gr9qi6m,gr9sx3j,gr9nybo,gr9jb16,gr9vgrl,gr9uun6,gr9lew5,gr9v0yx,gr9tmej,gr9p8tu,gr9yo4x,gr9l8l0,gr9naut,gr9la63,gr9wwxp,gr9e9by,gr9kbud,gr9eke8,gr9okox,gr9wwxw,gr9nqo5,gr9m3kx,gr9eqq1,gr9wcej,gr9b1zh,gr9htce,gr9u8k0,gr9xvpt,gr9m11v,gr9ggd1,gr9llo9,gr9iaqj,gr9f84d,gra0an4,gr9afvm,gr9vopv,gr9gt0k,gr9s4pz,gr9iine,gr9em0d,gr9vlke,gr9t3hr,gr9n1fa,gr9vjzu,gr9jkl7,gr9yrhm,gr9xb7j,gr9mk1t,gr9svm4,gr9e68d,gr9honj,gr9kd1s,gr9jqxh,gr9x4ww,gr9l58t,gr9uxwk,gr9sheu,gr9y6uf,gr9rs4o,gr9shey,gr9pxc7,gr9rs4v,gr9hdma,gr9y243,gr9pz6m,gr9lrnb,gra02t4,gr9pedm,gr9vakg,gr9x83o,gr9vy9x,gra0llt,gr9txl8,gr9fubx,gr9yzbo,gr9mc7q,gr9ydjj,gr9mqii,gr9spcu,gr9z2hr,gr9r7mo,gr9votf,gr9gup1,gr9hief,gr9vfcd,gr9kxny,gr9z8ts,gr9vn8x,gr9te6r,gr9amb8,gr9x1tj,gr9jb7d,gr9dau2,gr9ympn,gr9oj98,gr9wqrh,gr9u2da,gr9lgyd,gr9g70j,gr9d7k4,gr9hn5x,gr9sb61,gr9u8p6,gr9rp1n,gr9shhr,gr9t8cw,gr9jnv5,gr9kd5e,gr9xuac,gr9vyc5,gr9hora,gr9u2dz,gr9scre,gr9c5y2,gr9xcwv,gr9o0bk,gr9tswy,gr9r4ir,gr9h6by,gr9pvon,gr99h9i,gr9wkh6,gr9pzfv,gr9lzwv,gr9i65t,gr9milw,gr9uuuv,gr9ko8p,gr9p0oq,gr9fc4x,gr9d613,gr9z8w8,gra0cen,gr9rykx,gr9wixe,gr9bt1w,gr9kevx,gr9pa6v,gr9pof0,gr9htk0,gr9l0wy,gr9k0s9,gr9vilg,gr9ltd3,gr9yt48,gra099n,gr9n6cw,gr9x3ht,gr9grn7,gr9vdvg,gr9b1ed,gr9sphc,gr9l0xt,gr9ketg,gr9hqym,gr9g2dl,gr9uy2z,gr9plak,gr9dfkg,gr9ymtt,gr9gjrm,gr9z47o,gr9xxjp,gr9z8yg,gr9tjiy,gr9ickd,gr9x89s,gr9neap,gr9vwvb,gr9qt8n,gr9trg1,gr9w4s2,gr9w6cz,gr9sqrr,gr9cndd,gr9llze,gr9t8hq,gr9fsxt,gr9z2o3,gr9uwjo,gr9qybp,gr9r4ng,gr9mgvb,gr9f9zm,gr9jxhw,gr9tw7q,gr9x0ex,gr9zfbx,gr9s1ut,gr9ti08,gr9lai3,gr9k29b,gr9rm28,gr9jcz5,gr9zn8y,gr9rm2d,gr9iu0o,gr9n3ay,gr9hapo,gr9x9xd,gr9bbsy,gr9koa3,gr9uq9x,gr9poju,gr9zfd7,gr9iu18,gr9f5aq,gr9jmmb,gr9e3dh,gr9uahk,gr9yzkp,gr9sja2,gr9qbcf,gr9ms9o,gr9zfdp,gr9qv8f,gr9obky,gr99rce,gr9pcp1,gr9riey,gr9v99q,gra0hbj,gr9sud4,gr9zc8r,gr9xl1h,gra0m2u,gr9ryrt,gr9nxt9,gr9udos,gr9shql,gr9co5p,gr9ez4e,gr9yok9,gr9ztng,gr9nyzp,gr9zox2,gra0cmf,gr9u8yw,gr9e69b,gr9wpgu,gr9q19r,gra09gw,gr9ve2a,gr9l90z,gr9qs51,gr9vqpi,gr9n3ef,gr9suex,gr9tzi2,gr9p3mz,gr9e1vp,gr9tzie,gr9ti4n,gr9xmof,gr9kbv6,gr9pic3,gr9wcus,gr9w6j9,gr9yzoa,gr9sg7z,gr9xo9v,gr9xjj9,gr9gvch,gr9zwvd,gra0j8q,gr9n0a7,gr9r5e1,gr9w08o,gr9so57,gra0fuq,gr9dtsb,gr9omrw,gr9vp72,gr9qtsf,gr9wmds,gr9xgep,gr9xetv,gr9eudw,gr9e1xy,gr9k0uy,gr9kmzf,gr9fi59,gr9f5gj,gr9m975,gr9qpuv,gr9lqi4,gr9u7hp,gr9mvul,gr9wsq8,gr9o1dn,gr9pvhv,gr9ggxf,gr9xt2u,gr9ugb8,gr9yago,gr9l5p0,gr9lfga,gr9wmez,gr9argd,gr9qkce,gr9utmp,gr9spsc,gr9b79r,gr9qtjs,gr9z992,gr9ltp4,gr9mdok,gr9owbg,gra00hy,gr9gk2n,gr9r069,gr9kzix,gr9jmoz,gr9f0rp,gr9t5m1,gr9f4xz,gr9u5yl,gr9pjq5,gr9kn1t,gr9nhrn,gr9c7n8,gr9mm60,gr9rjmn,gr9qv4w,gr9n6nq,gr9wudn,gr9ccek,gr9yaj1,gr9xzgv,gr9tjv8,gr9dh9q,gr9v36g,gra0b89,gr9kyw7,gr9wjbz,gr9sd7k,gr9qtwh,gr9dtj4,gr9g8o4,gr9fdgo,gr9aewa,gr9hdt3,gr9wr8t,gr9x8ms,gr9vx85,gr9q1an,gr9ultl,gr9r5ey,gr9h69r,gr9i3gg,gr9njep,gr9u1aa,gra0kqs,gr9m1oo,gr9qbx3,gr9s5cc,gr9go04,gr9y49m,gr9ticb,gr9ze3i,gr9wz6s,gr9st2j,gr9bkpi,gr9ty5s,gr9ptma,gr9ii43,gr9rz1w,gr9sxth,gr9iud2,gr9u2wn,gr9rz20,gr9toop,gr9mevx,gr9lcfz,gr9zqrk,gr9la74,gr9ttfm,gr9m81v,gr9pydg,gr9n0hy,gr9n0cs,gr9ucel,gr9k47i,gr9nt0p,gr9t45u,gr9f2hf,gr9gz2m,gr9tv13,gr94hnc,gr9octg,gr9ist7,gr9ejj4,gr9tahz,gr9sgh6,gr9hiav,gr9mpgv,gr9jddf,gr9oxaf,gr9k2ns,gr9ywsb,gr9ei23,gr9o8sk,gr9y939,gr9vr0n,gr9hklk,gr9wgay,gr9ibgv,gr9kosw,gr9k5dv,gr9vrvy,gr9nmon,gr9ey4k,gr9emqj,gr9zsf4,gr9pga3,gr9zvdo,gr9i0fh,gra052m,gr9uylr,gr9vj5c,gr9vuiw,gr9rz3o,gr9y5k2,gr9ytoe,gr9z003,gr9wl39,gr9lcjr,gr9j8p5,gr9wd70,gr9hu4y,gr9id3n,gr9bqhh,gr99d5i,gr9wpug,gr9qqxm,gr9homy,gr9x0x1,gr9h1pi,gr9t49z,gr9edbe,gr9qa3r,gr9y19p,gr9rsvj,gr9qz4k,gr9gh6k,gr9rpke,gr9nolx'), ('link_id', 't3_m74e3g'), ('sort', 'confidence')]
2021-03-17 13:36:15.197 PDT Params: {'raw_json': 1}
2021-03-17 13:36:15.609 PDT Response: 200 (23434 bytes)
2021-03-17 13:36:15.677 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
2021-03-17 13:36:15.678 PDT Data: [('api_type', 'json'), ('children', 'gr9vsp0,gr9lkhx,gr9diip,gr9l349,gr9cx6y,gr9i6tp,gr9wa32,gr9zcpc,gr9r6s4,gr9et5k,gr9klqy,gra0n1r,gr9fwny,gr9tc80,gr9ruht,gr9kvjb,gr97kfa,gr9pauh,gr9sq3u,gr9kffw,gr9sgh6,gr9e5gv,gr9izao,gr9fc4x,gr9mjar,gr9oiew,gr9f2nw,gr9vubl,gr9tqh3,gr9sle3,gr9nxvw,gr9uixa,gr9xljq,gr9tduk,gr9l00t,gr9gx29,gra0hur,gr9emw4,gr9tili,gr9j8tn,gr9aagg,gr9bp0y,gr9n71x,gr9f5va,gr9vtqu,gr9h4zj,gr9ykcg,gr9zm9w,gr9ia2t,gr9vr7q,gr9xifc,gr9faml,gr9gscx,gr9q1sn,gr9igex,gr9hcwt,gr9m9we,gr9z26d,gr9m3kx,gr9o90p,gr9x12j,gr9x2nk,gr9x12q,gr9suyb,gr9rbnc,gr9w2bl,gr9sjwi,gr9wudn,gr9tozt,gr9fikq,gr9m21a,gr9l58t,gr9u1n5,gr9x903,gr9mrbp,gr9x49j,gr9vv38,gr9wezg,gr9ejdi,gr9cvry,gr9csm9,gr9jm7a,gr9v8bq,gr9l3ag,gr9a12h,gr9db2p,gr9zcvg,gr9lnu6,gr9qbcf,gr9oqgm,gr9xlo9,gr9v0fv,gr9nc9h,gr9euwx,gr9bnk3,gr9f5t8,gr9irx3,gr9i8lp,gr9v36g,gr9pfrm,gr9g6c3,gr9e69b,gr9sqal,gr9nfyd,gr9t4io,gr9cxer,gra0a3q,gra0d2p,gr9r5ey,gr9wt9o,gr9ii43,gr9j5tk,gr9hpof,gr9ml34,gr9ey4k,gr9cmdh,gr9i2bq,gr9vgaw,gr9ynnt,gr9j03b,gr9l6j0,gr9o18t,gr9j13k,gr9e0yc,gr9uxcl,gr9euze,gr9l4yj,gr9r5e1,gr9n42m,gr9zuch,gr9h3kz,gr9sk15,gr9wf3m,gr9qjcj,gr9zek6,gr9xyex,gr9legv,gr9cg3l,gr9tymd,gr9ye86,gr9djm2,gr9x2ts,gr9yx7i,gr9vw6c,gr9ccyt,gr9pvon,gr97hk8,gr9ii7e,gr9ef8t,gr9e46n,gr9z6pe,gr9ttx0,gra0ddk,gr9x8ms,gr9vx85,gr9p4uf,gr9xdn0,gr9r8g0,gr9wv7b,gr9e11o,gr9lbd9,gr9yqxb,gr9pknm,gr9o637,gr9eifo,gr9v3rt,gr9da6y,gr9tivw,gr9sdst,gr9k9gh,gr9da78,gr9kqud,gr9j5ye,gr9khd5,gr9z9wr,gr9v8j0,gr9o1d9,gr9rmye,gr9u516,gr9o4qn,gr9y39c,gr9bx8m,gr9ny87,gr9wah4,gr9ql1l,gr9t9fu,gr9z56z,gr9u1aa,gr9pnv3,gr9nqc1,gr9ubdf,gr9s93p,gr9ez4e,gr9pxcu,gr9hd7j,gr9rjtw,gr9fkfb,gr9h84d,gr9p9o0,gr9h5bu,gr9onjv,gra0dh9,gr9sh1d,gr9pr2e,gr9qsm9,gr9s5cc,gr9co5p,gr9s191,gr9ffq2,gr9lz6j,gr9mrmq,gr9swv9,gr9hd9i,gr9qpuv,gr9va7e,gr9hief,gr9swvh,gr9f76s,gr9za0i,gr9vrli,gr9z6ux,gr9kqyi,gr9kfwh,gr9tqwy,gr9go04,gr9z8g6,gr9tklm,gr9gmfh,gr9gz2m,gr9nfql,gr9f7v6,gr9bkpi,gr9qw7t,gr9hjmj,gr9faca,gr9ql5q,gr9h4g5,gr9avfp,gr9qa3r,gr9e17t,gr9sbxu,gr9tpdb,gr9cer7,gr9epsu,gr9q27s,gr9wtkp,gr9vwe5,gr9ynya,gr9ox51,gr9iguh,gr9hrka,gr9i47f,gr9kma4,gr9rtfx,gr9f5jo,gr9fsgl,gr9y507,gr9l595,gr9gkwq,gr9rdni,gr9yr4z,gr9y86f,gr9vq3t,gr9wqga,gr9z0mj,gr9xx4e,gr9lhwz,gr9ugad,gr9xx4n,gr9riey,gr9fm67,gr9s2gw,gr9q3us,gr9lukm,gr9iakn,gr9donf,gr99rce,gr9slxv,gr9ugb8,gr9ocnp,gr9i2ft,gr9ik2g,gr9tvs0,gr9v0v6,gr9mq78,gr9rs4v,gr9wv8r,gr9mevx,gr9avjm,gr9schk,gr9w94q,gr9opbw,gr9lzwv,gr9mrsy,gr9iud2,gr9tefd,gr9ws3u,gr9nxxp,gr9tyz4,gr9rc5x,gr9toop,gr9vq6y,gr9jhb0,gr9dy72,gr9igyn,gra0llt,gr9y3j6,gr9y89w,gr9oyjq,gr9x6co,gr9e99l,gr9zmuk,gr9p3la,gr9j0fc,gr9tu9e,gr9ljlf,gr9m2k4,gr9jsdx,gr9gd4q,gr9jhc2,gr9dc3n,gr9iao5,gr9qt8n,gra0gsa,gr9vdl5,gr9xsi5,gr9qi6m,gr9nybo,gr9jb16,gr9vgrl,gr9uun6,gr9lew5,gr9v0yx,gr9pga3,gr9yo4x,gr9l8l0,gr9naut,gr9la63,gr9wwxp,gr9e9by,gr9kbud,gr9eke8,gr9mdok,gr9nqo5,gr9f4xz,gr9eqq1,gr9wcej,gr9b1zh,gr9htce,gr9u8k0,gr9xvpt,gr9m11v,gr9ggd1,gr9k0s9,gr9iaqj,gr9f84d,gr9la74,gr9afvm,gr9o1dn,gr9gt0k,gr9hiav,gr9iine,gr9em0d,gr9vrvy,gr9t3hr,gr9f2hf,gr9homy,gr9jkl7,gr9wkh6,gr9xb7j,gr9mk1t,gr9svm4,gr9e68d,gr9honj,gr9kd1s,gr9jqxh,gr9v99q,gr9x4ww,gr9ist7,gr9uxwk,gr9sheu,gr9y6uf,gr9rs4o,gr9shey,gr9pxc7,gr9mwq7,gr9hdma,gr9esdv,gr9pz6m,gr9lrnb,gra02t4,gr9qlh9,gr9vakg,gr9x83o,gr9vy9x,gr9txl8,gr9fubx,gr9mc7q,gr9xmof,gr9p0oq,gr9spcu,gr9z2hr,gr9votf,gr9gup1,gr9rjtd,gr9vfcd,gr9kxny,gr9z8ts,gr9vn8x,gr9te6r,gr9amb8,gr9x1tj,gr9jb7d,gr9ympn,gr9oj98,gr9wqrh,gr9u2da,gr9lgyd,gr9g70j,gr9d7k4,gr9hn5x,gr9sb61,gr9u8p6,gr9rp1n,gra0as0,gr9jnv5,gr9kd5e,gr9vyc5,gr9hora,gr9u2dz,gr9scre,gr9c5y2,gr9xcwv,gr9o0bk,gr9tswy,gr9h6by,gr99h9i,gr9yrhm,gr9pzfv,gr9xlzq,gr9i65t,gr9milw,gr9uuuv,gr9ko8p,gr9mqii,gr9z8w8,gra0cen,gr9rykx,gr9wixe,gr9bt1w,gr9kevx,gr9pa6v,gr9pof0,gr9htk0,gr9l0wy,gr9vilg,gr9ltd3,gr9yt48,gr9koa3,gr9n6cw,gr9x3ht,gr9grn7,gr9vdvg,gr9b1ed,gr9sphc,gr9l0xt,gr9yh91,gr9ketg,gr9g2dl,gr9dtsb,gr9plak,gr9dfkg,gr9ymtt,gr9gjrm,gr9l5p0,gr9xxjp,gr9z8yg,gr9tjiy,gr9ickd,gr9x89s,gr9hdt3,gr9oxaf,gr9kpev,gr9trg1,gr9w4s2,gr9w6cz,gr9cndd,gr9llze,gr9t8hq,gr9fsxt,gr9k5dv,gr9uwjo,gr9qybp,gr9r4ng,gr9mgvb,gr9f9zm,gr9jxhw,gr9tw7q,gr9x0ex,gr9zfbx,gr9s1ut,gr9ti08,gr9lai3,gr9bbsy,gr9k29b,gr9rm28,gr9jcz5,gr9zn8y,gr9rm2d,gr9iu0o,gr9bqhh,gr9hapo,gr9x9xd,gr9pedm,gra099n,gr9uq9x,gr9zfd7,gr9iu18,gr9f5aq,gr9e3dh,gr9uahk,gr9yzkp,gr9sja2,gr9mrcf,gr9qv8f,gr9obky,gr9h69r,gr9dh9q,gra0hbj,gr9zc8r,gr9xl1h,gra0m2u,gr9ryrt,gr9udos,gr9shql,gr9z0i5,gr9hvf2,gr9yok9,gr9ztng,gr9nyzp,gr9zox2,gra0cmf,gr9u8yw,gr9oa21,gr9wpgu,gra09gw,gr9ve2a,gr9l90z,gr9ydjj,gr9qs51,gr9vqpi,gr9llo9,gr9tzi2,gr9p3mz,gr9e1vp,gr9ti4n,gr9q1an,gr9kbv6,gr9pic3,gr9wcus,gr9w6j9,gr9yzoa,gr9uylr,gr9sg7z,gr9xo9v,gr9xjj9,gr9vuh5,gr9w08o,gr9so57,gr9jmmb,gr9omrw,gr9vp72,gr9wmds,gr9xgep,gr9xetv,gr9eudw,gr9k0uy,gr9kmzf,gr9f5gj,gr9m975,gr9x65j,gr9lqi4,gr9tmej,gr9wsq8,gr9nhrn,gr9n6nq,gr9p113,gr9yago,gr9z47o,gr9lfga,gr9wmez,gr9argd,gr9dtj4,gr9n0cs,gr9u7hp,gr9qtjs,gr9z992,gr9ltp4,gr9wwxw,gr9owbg,gr9dau2,gr9gk2n,gr9r069,gr9kzix,gr9jmoz,gr9f0rp,gr9t5m1,gr9u5yl,gr9pjq5,gr9kn1t,gr9ggxf,gr9c7n8,gr9mm60,gr9rjmn,gr9vung,gr9z7pn,gr9ccek,gr9xzgv,gr9tjv8,gr9hnmg,gra0b89,gr9wjbz,gr9sd7k,gr9qtwh,gr9g8o4,gr9fdgo,gr9aewa,gr9vwvb,gr9wr8t,gr9zp4k,gr9uk8l,gr9mfw8,gr9ultl,gr9ni96,gr9gqgv,gr9i3gg,gr9lngi,gra0kqs,gr9utmp,gr9m1oo,gr9t8cw,gr9qbx3,gr9he6k,gr9qetg,gr9y49m,gr9ticb,gr9ze3i,gr9wz6s,gr9rpke,gr9ty5s,gr9nacz,gr9rz1w,gr9sxth,gr9vgno,gr9u2wn,gr9uy2z,gr9xmwp,gr9lcfz,gr9zqrk,gr9ttfm,gr9qp8i,gr9m81v,gr9n0hy,gr9spsc,gr9y19p,gr9ucel,gr9k47i,gr9t45u,gr9vopv,gr9z0jy,gr9tv13,gr9octg,gr9b79r,gr9ejj4,gr9tahz,gra0j8q,gr9s4pz,gr9mpgv,gr9jddf,gr9k2ns,gr9ywsb,gr9ei23,gr9o8sk,gr9y939,gr9vr0n,gr9hklk,gr9wgay,gr9ibgv,gr9kosw,gr9z2o3,gr9vlke,gr9nmon,gr9uu6c,gr9emqj,gr9zsf4,gr9y243,gr9fdly,gr9i0fh,gra052m,gr9fi59,gr9gvch,gr9mvul,gr9hqym,gr9z003,gr9wl39,gr9lcjr,gr9j8p5,gr9wd70,gr9hu4y,gr9id3n,gr9jbv9,gr99d5i,gr9wpug,gr9vjzu,gr9x0x1,gr9h1pi,gr9t49z,gr9ynj5,gr9edbe,gr9kyw7,gr9nxt9,gr9rsvj,gr9rtl3,gr9qz4k,gr9gh6k,gr9di4b'), ('link_id', 't3_m74e3g'), ('sort', 'confidence')]
2021-03-17 13:36:15.678 PDT Params: {'raw_json': 1}
2021-03-17 13:36:16.051 PDT Response: 200 (17413 bytes)
2021-03-17 13:36:16.064 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
...

That poll includes over a dozen requests to https://oauth.reddit.com/api/morechildren/, many of which ask for >800 children each.

PRAW does some caching, but evidently not enough, or we haven't configured it correctly. Related issues: praw-dev/praw#1140, praw-dev/praw#131. And from praw-dev/praw#627: "Also the client side cache has been removed in PRAW4." (We're on PRAW 7.)

@snarfed snarfed changed the title optimize reddit optimize reddit/PRAW Mar 17, 2021
@snarfed
Copy link
Owner Author

snarfed commented Mar 21, 2021

Found the culprit for the comment handler. We fetch the post so we can do OPD, and we also fetch comments (with fetch_replies=True as an optimization for some sources, but that's clearly harmful for Reddit.

Relevant commits:

  • 7bd3a6d: handlers: bug fix for f823e41: pass fetch_likes, etc when prefetching posts
  • f823e41: handlers: use optimization in snarfed/granary@4d32457 to avoid double fetches
  • snarfed/granary@4d32457: add optional activity kwarg to Source.get_*; move Facebook.get_rsvp to Source. the requested object in get_* can sometimes be extracted from the optional activity, which avoids an unnecessary API call.

snarfed added a commit to snarfed/granary that referenced this issue Mar 21, 2021
snarfed added a commit that referenced this issue Mar 22, 2021
for #1021, avoids massive inefficiency with big Reddit threads. uses new granary OPTIMIZED_COMMENTS constant from snarfed/granary@6cf8dd9.
@snarfed
Copy link
Owner Author

snarfed commented Mar 26, 2021

this last change, 241a4da, had a big impact. we shouldn't have been fetching replies to posts that just link to the user's site; now we don't. no more morechildren fetches in polls, since reddit currently only does link search.

i should probably also find someplace to put logic to drop those replies even if they do get fetched, since they shouldn't result in wms.

i still want to get rid of the duplicate user fetches though!

snarfed added a commit to snarfed/granary that referenced this issue Apr 1, 2021
...when fetching multiiple comments or posts from the same author. snarfed/bridgy#1021
@snarfed
Copy link
Owner Author

snarfed commented Apr 1, 2021

added user profile API request caching in snarfed/granary@85ccb5a. pretty happy with reddit now, i'm going to close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant