summaryrefslogtreecommitdiff
path: root/cgithub/src/scraper.test.ts
AgeCommit message (Collapse)Author
2026-09-03Handle route errors in middlewareJoe Mou
Every page route wrapped its scrape in tryRender, which awaited the promise, rendered the template on success, and turned a failure into a redirect or an error page. Hono's onError does that mapping for all of them at once, so handlers now await their scrape inline and both wrappers are gone. Fragments under /api/ still fail as an empty body rather than a page; onError checks for them first, since a fragment turns a redirect into a 404 instead of following it. RedirectError becomes an abstract base with InternalRedirectError and ExternalRedirectError under it, so the handler dispatches on the type instead of sniffing the location for an https:// prefix. One behavior change: a thrown value that is not an Error no longer renders error.eta, because Hono only routes Error instances to onError. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QERwoRJhuqMhKzzCUW4Skv
2026-09-03Link blobs to their history and backJoe Mou
Nothing in a commits payload says whether the path it lists is a file or a directory: currentBlobPath, despite the name, is only the path filter, and is set for a directory just the same. GitHub's own UI cannot tell either, and points its browse button at /blob/<oid>/<path> so that the redirect resolves a file to its blob. We do the same, and the blob route already turns that redirect into one of our own tree URLs. Breadcrumbs now stay in the view they were clicked from, so walking up from a history lands in the history of each parent directory rather than in its listing. The last crumb remains the page you are already on. A path with no commits now says so rather than rendering an empty page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7pRKdmTvNDgT4ZXrwkbH8
2026-09-03Fix drifted testsJoe Mou
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWDWkWbj5oZCGQEYoybMrV
2026-09-03Render repository wikisJoe Mou
Wikis are one of the last parts of GitHub still served as plain Rails HTML, with no embedded JSON to read, so this scrapes the markup: a page and any older revision of it, the _Sidebar and _Footer a wiki can define, and the page index. Every page is already in the sidebar of every page, since the "Show N more pages" button only unhides the rest, so linking the whole wiki costs no second request. Routes mirror GitHub's and resolve in its order, so an underscore name reserved for its own actions (_pages, _history, _new, _edit) can never be mistaken for a page; the ones we don't serve fall through to the usual redirect. A slug reaches us decoded and has to be encoded again rather than interpolated the way a ref is, or a page named C# would truncate at the fragment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rQG1io1i4GJQW5RFcXNGd
2026-08-27Display PDF blobsJoe Mou
GitHub renders a PDF with a viewer of its own and so ships no content for one, leaving the blob view with nothing to show. Its raw bytes come as application/octet-stream, which browsers download rather than display, so serve them from a new embed route that retypes them and point an <object> at that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BErPR92xppzGFXkPqXPYj2
2026-08-12Handle blobs GitHub cannot previewJoe Mou
Binary files with no preview (a .ttf, say) have no rawLines, no colorizedLines and no richText, so the blob view's final branch threw on a null textLines and the page 500ed. Offer the raw file instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-11Read the About box and latest release from JSONJoe Mou
GitHub rebuilt the repository sidebar, so the selectors parseRepoInfo used matched nothing and the About box rendered empty. Everything but the latest release is in the page's embedded payload as sidebarAbout; the latest release is only in the _sidebar JSON already fetched for languages, so it arrives with that fragment and is moved beside the release count.
2026-08-11Add language percentages to repo pagesJoe Mou
GitHub no longer server-renders the repository sidebar; its React UI lazily fetches /:owner/:repo/_sidebar as JSON, which carries the language breakdown (name, percentage, color). getGitHubLanguages() uses that endpoint, and a new /api/:owner/:repo/languages fragment route renders the bar and legend the same way refs and latest-commit already work, so repo page loads still cost a single scrape. The latest-commit placeholder loader in refs.js is generalized over .placeholder[data-src] so both placeholders share it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-22Add async branch/tag dropdown and latest-commit barJoe Mou
Adds two lazily-fetched UI pieces to match vanilla GitHub while keeping initial page loads plain server-rendered HTML: a branch/tag selector (native <dialog>, CSS anchor-positioned to its button, light-dismissed via closedby="any") and a latest-commit bar above the file listing. Both are populated by new /api/:owner/:repo/refs/:type and /api/:owner/:repo/latest-commit/:branch/:path routes; the latter calls GitHub's own /latest-commit JSON endpoint (the same one the React UI uses) rather than scraping the full commit history page.
2026-07-15Replace null overview files with HTML commentsJoe Mou
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15Owner page nav menu and layout restructureJoe Mou
Add a menu bar of profile tab links, split the org repository list from pinned repos, and restructure the owner template to match repo pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15Add organization support to owner pagesJoe Mou
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15Remove watchers, no longer rendered by GitHubJoe Mou
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15User profile pagesJoe Mou
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-05-27Pull request indexJoe Mou
2026-05-27Add repo-scoped search support for issues and PRsJoe Mou
We do not support hybrid issue/PR search. Any searches on the issues endpoint will redirect to an explicit issue-only search.
2026-05-27Update unit testsJoe Mou
2026-04-28Support redirectsJoe Mou
2026-04-20Apply overlapping styling drectives instead of duplicating contentJoe Mou
2026-04-20Update for unexpected actions/deploy-pages releaseJoe Mou
2026-03-14Fix tree overviewHtml: use tree.readme instead of overview.overviewFilesJoe Mou
2026-03-13Fix GitHub scraping for new JSON structure, update TypeScript interfacesJoe Mou
GitHub restructured their embedded JSON: tree/blob data is now split across codeViewRepoRoute, codeViewTreeRoute, codeViewBlobLayoutRoute, codeViewBlobLayoutRoute.StyledBlob, codeViewBlobRoute, and the shared codeViewLayoutRoute (repo, refInfo, path). Replace RawPayload and OverviewFile with typed interfaces for each route, add AppPayload as the top-level union, and simplify parsePayload to always extract react-app.embeddedData.
2026-02-21Release assetsJoe Mou
Also fix release content parsing.
2026-02-21Add latest release detailsJoe Mou
By Gemini
2026-02-21Add single release detail pages and refactor templatesJoe Mou
Refactored releases display into reusable partial template (_release.eta) and added support for viewing individual releases at /:owner/:repo/releases/tag/:tag with navigation back to all releases. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-21Parse release page HTMLJoe Mou
Fixes reading complex release descriptions that had unbalanced tags. Written with Gemini.
2026-02-11Add GitHub releases supportJoe Mou
Scrapes releases page HTML to display version tags, publish dates, and changelog content. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-28Commits listJoe Mou
2026-01-28Image supportJoe Mou
2026-01-28Syntax highlightingJoe Mou
Sometimes GitHub returns colorizedLines and sometimes it returns a terser stylingDirectives, which needs to be applied to rawLines.
2026-01-28Render Markdown blobsJoe Mou
2026-01-27List issuesJoe Mou
2026-01-27Propagate GitHub HTTP errorsJoe Mou
Render `error.eta` on HTTP error from GitHub. Implemented with Gemini.
2026-01-27GitHub scraper testJoe Mou