Skip to content
This repository has been archived by the owner on Jun 3, 2020. It is now read-only.

00F - Internal crawler, database, and API #17

Open
wants to merge 249 commits into
base: master
Choose a base branch
from
Open

00F - Internal crawler, database, and API #17

wants to merge 249 commits into from

Conversation

pfrazee
Copy link
Member

@pfrazee pfrazee commented Dec 8, 2018 •
edited
Loading

Todos

  • Web APIs
  • SQL deltas
  • User management

@pfrazee
Copy link
Member Author

pfrazee commented Dec 8, 2018

This PR introduces the "user" concept as well as a semantic crawler that indexes content followed by the user. You can find the semantics being used at https://github.com/beakerbrowser/unwalled.garden.

This PR will add two new (currently internal) Web APIs: beaker.posts, beaker.followgraph. It will also include some functions for managing the local user, perhaps as another internal Web API (beaker.users).

pfrazee added 27 commits February 4, 2019 08:47
@pfrazee
Reorg profile-data.sql.js
d12aa3d
@pfrazee
Add the beaker.posts and beaker.followgraph API skeletons
0a4e01f
@pfrazee
Implement the majority of the crawler (WIP)
428568f
@pfrazee
Implement post and follows crawlers
36e1f9e
@pfrazee
Add write methods to crawler posts and followgraph
f592ca2
@pfrazee
Implement remaining crawler logic
bd11231
@pfrazee
Implement followgraph web api
8af6f4f
@pfrazee
Add posts web api
cfd8a6a
@pfrazee
Add crawler sqlite delta
1f22b4d
@pfrazee
Add userSessionAPI interface
d7b3fa2
@pfrazee
Add getUserSession/setUserSession to beaker.browser api
3f3400b
@pfrazee
Fixes
1eb42a6
@pfrazee
Fixes
99d71ec
@pfrazee
Fixes
462427d
@pfrazee
Crawler fixes, particularly around the changes watcher
a79a5f6
@pfrazee
Give better author info in posts responses
c08bcc3
@pfrazee
Correctly handle get() that doesnt have content
fc8a52a
@pfrazee
Normalize the URL provided by the users api
6252f9b
@pfrazee
More consistent URL normalization
85c9f3c
@pfrazee
Create folders as needed
b23ef93
@pfrazee
Fixes
e9b33ba
@pfrazee
Fixes to followgraph crawler
bc4bbbb
@pfrazee
Fixes to crawler
3495ba1
@pfrazee
Add authors param to beaker.posts.list
6b6dab7
@pfrazee
More consistent URL normalization in followgraph
1461a5f
@pfrazee
Correctly get last ID of inserts
2848a52
@pfrazee
Add readFile and showEditProfileModal to beaker.browser
b155908
pfrazee added 30 commits July 15, 2019 10:07
@pfrazee
Small crawler fixes
fcc18a0
@pfrazee
Tolerate DNS names in more dat library codepaths
ad8b256
@pfrazee
Correctly handle timeouts during crawling
1889f4f
@pfrazee
Use dns name on users
2de515d
@pfrazee
Track the 'primary url' (assigned domain in manifest) more closely an…
1b0d96b 
…d use in all internal records
@pfrazee
Expose an applications API
db3000c
@pfrazee
Update listSuggestions for current types
82c5371
@pfrazee
Add beaker.browser.toggleSidebar
b5100e2
@pfrazee
Remove quad9 from dns providers (it was failing unwalled.garden lookups)
2127825
@pfrazee
Add beaker.browser. moveWindow and maximizeWindow
0e28de5
@pfrazee
Add beaker.browser.setWindowDragModeEnabled
bf42113
@pfrazee
Factor out dat-serve-resolve-path and remove markdown styles
2be4dc1
@pfrazee
Add beaker.browser toggleLiveReloading() and gotoUrl()
d8957d1
@pfrazee
Fix handling of visiting +preview on a site that doesnt have it enabled
d1b1cd1
@pfrazee
Fix: ug web apis for user with a domain name now work
b97a59d
@pfrazee
Bump deps, temporarily use sqlite fork that builds on electron 6
d6daf79
@pfrazee
Go back to sqlite3 main package
5109f2e
@pfrazee
Fix syntax error in profile-data.sql.js
5e61d56
@pfrazee
Add simple theme/template.html serving
28df9f9
@pfrazee
Fix misnamed fn
faccc14
@pfrazee
Fix: catch failed reads when confirming domain names
59d9f08
@pfrazee
Update search API suggestions to use themes and remove apps & templates
ff9db2f
@pfrazee
Drop the beaker* prefix on navigator methods
a7fb77d
@pfrazee
Disable navigator.getApplicationState
e07681f
@pfrazee
Move from app perms to session perms, add navigator.session.* web api
f5927e4
@pfrazee
Change theme-serving to a simple js/css injection
1947298
@pfrazee
Add tools for sidebar fetching of site sessions
53e039a
@pfrazee
Add beaker.browser.openSidebar
7b8eb62
@pfrazee
Allow writes via DatArchive to unsaved archives
7360723
@pfrazee
Bump package-lock.json
88f9018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pfrazee