diff --git a/README.md b/README.md index 9ad1157..15de015 100644 --- a/README.md +++ b/README.md @@ -140,6 +140,7 @@ Webcmd Cloud can run supported commands and browser sessions on hosted infrastru | Plugin | Description | Author | | --- | --- | --- | +| [`omnisearch`](./plugins/omnisearch/) | Scrape & research 12 platforms — X, Reddit, LinkedIn, Instagram, YouTube, Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, Bluesky | [Rishet Mehra](https://github.com/Rishet11) | | [`pypi`](./plugins/pypi/) | Inspect public Python package metadata, downloads, and releases from PyPI | [Kemal Kaya](https://github.com/yoldaolmak) | | [`skyscanner`](./plugins/skyscanner/) | Skyscanner flight search commands for Webcmd | [Rishabh](https://github.com/rishabhraj36) | diff --git a/plugins/omnisearch/README.md b/plugins/omnisearch/README.md new file mode 100644 index 0000000..8db2bdf --- /dev/null +++ b/plugins/omnisearch/README.md @@ -0,0 +1,138 @@ +# 🔎 OmniSearch + +**OmniSearch is a research assistant that reads the internet for you.** + +Ask it what people on **X, Reddit, LinkedIn, Instagram, YouTube, Hacker News, Stack Overflow, GitHub, and more** are saying, asking, and complaining about — and it brings back a clear answer with links. + +**You never need to touch the command line.** This tool is for your AI agent (Claude Code, Codex, Cursor, ChatGPT), not for you. Just tell your agent what you want to learn, and it handles the rest. + +--- + +## How to use OmniSearch (no coding needed) + +There are three steps — and you only do the first one yourself. + +**Step 1 — Install once (5 minutes).** You add OmniSearch to your AI assistant, like installing an app once. Your agent runs this for you: + +``` +webcmd plugin install github:Rishet11/webcmd-plugin-omnisearch +``` + +**Step 2 — Tell your agent what you want to know.** + +You don't type commands. You just talk. Say something like: + +> "Before I build this, find out what people actually think about it." + +Your agent recognizes OmniSearch and runs it for you. + +**Step 3 — Read the answer.** + +You get a plain-language summary — what people say, where they say it, and links to the actual posts. No folders, no files, nothing to interpret. + +--- + +## What you say. What your agent does. + +Here are real conversations, ready to copy-paste. Swap the words in **{braces}** for your topic. + +| You say | What your agent finds | +|---|---| +| "**{My idea}** — do people actually want this?" | What people say across X, Reddit, Hacker News, GitHub | +| "Find what people complain about when using **{a tool}**." | The real problems people report on Stack Overflow and GitHub | +| "Is **{this idea}** already done? Who's winning?" | The competition and reception across platforms | +| "Is **{this idea}** even technically possible yet?" | What research says is proven vs. still speculative | +| "Before I launch **{product}**, what could kill it?" | The risks, past failures, and strongest arguments for it | +| "What's the reaction to **{a product / a launch / the news}**?" | What people are saying on X, Reddit, and Hacker News right now | + +### Ready-to-paste prompts for your agent + +Pick one, replace **{the brackets}**, and paste it to your agent: + +1. "Research **{topic/competitor/product}**. Use OmniSearch to check what people say across Reddit, X, Hacker News, and GitHub. Give me a two-paragraph summary and the 5 most-talked-about posts with links." + +2. "Find what's broken or frustrating about **{topic}**. Look at Stack Overflow questions and GitHub issues, and list the 10 most common problems people mention, each with a source link." + +3. "Check if **{my idea}** is technically realistic yet. Scan recent research papers and tech discussions, and tell me what's proven to work vs. still speculative." + +4. "Validate my idea: **{one-line pitch}**. Gather what people are saying about it, rate how much real interest there is, and quote 3 real people with links." + +5. "Do a pre-mortem on **{my startup/product}**. Find how similar things failed before, the top 3 reasons this could fail, and the strongest argument it could succeed — each with a source." + +6. "What do developers complain about when using **{a tool/language}**? Summarize the biggest recurring pain points with links." + +7. "Check the reaction to **{a new release / launch}**. What are people on X, Reddit, and Hacker News saying right now?" + +--- + +## What you can research + +OmniSearch reads **12 places people talk** — the big social networks (through your own accounts, quietly in the background) plus a bunch of no-login communities: + +| Platform | Best at telling you | +|---|---| +| **X / Twitter** 🐦 | what people think right now | +| **Reddit** 🧑‍🤝‍🧑 | real threads and discussion | +| **LinkedIn** 💼 | professional opinions, finding people | +| **Instagram** 📸 | visual and chat culture | +| **YouTube** ▶️ | video + what's being discussed | +| **Hacker News** 🟠 | what the tech world thinks | +| **Stack Overflow** 📚 | what's actually broken | +| **GitHub** 🐙 | real problems people file | +| **Dev.to** 💜 | what developers are writing | +| **arXiv** 📄 | the latest research | +| **Lobsters** 🦞 | smart developer discussion | +| **Bluesky** 🦋 | fresh public posts | + +**No login needed for most.** Only X, Reddit, LinkedIn, Instagram, and YouTube use your own account — the agent uses it quietly in the background; you never log in again. + +--- + +## Start now + +**Install OmniSearch once, then ask your agent:** + +> *"Before I build this, find out what people actually think about it."* + +That's the whole product. + +--- + +## For developers + +Everything below is for the agent (and for people who want to dig in). A normal user never reads this. + +### The commands + +```bash +# Aggregate everything about a topic +webcmd omnisearch research "" -f json + +# 🏆 The community's verdict, distilled +webcmd omnisearch verdict "" -f json + +# One source at a time +webcmd omnisearch hackermind "" -f json +webcmd omnisearch stackoverflow "" -f json +webcmd omnisearch github "" -f json +webcmd omnisearch arxiv "" -f json +webcmd omnisearch devto -f json +webcmd omnisearch lobsters -f json +webcmd omnisearch bluesky-posts -f json + +# Logged-in social (one-time login: webcmd --profile social login) +webcmd --profile social twitter search "" -f json +webcmd --profile social reddit search "" -f json +webcmd --profile social linkedin people-search "" -f json +webcmd --profile social instagram search "" -f json +``` + +### Output + +Every command returns the same shape: `platform`, `title`, `author`, `score`, `commentCount`, `createdAt`, `url`, `text`. The `score` + `commentCount` let an agent rank results by real traction. + +### How it works + +All platforms live in `sources.js` behind one signature (`search(query, limit)`), so adding a new platform to `research` is about 20 lines. + +Ships `SKILL.md` so agent harnesses auto-discover OmniSearch. diff --git a/plugins/omnisearch/SKILL.md b/plugins/omnisearch/SKILL.md new file mode 100644 index 0000000..fd1f25a --- /dev/null +++ b/plugins/omnisearch/SKILL.md @@ -0,0 +1,26 @@ +--- +name: omnisearch +description: Read-only, no-login web research across Hacker News, Lobste.rs, Stack + Overflow, GitHub issues, Dev.to, arXiv, and Bluesky. Use to answer "what does the + internet think / ask / struggle with" about any topic. Returns JSON via webcmd. +--- + +# OmniSearch skill + +1. Invoke every query as `webcmd omnisearch -f json`. The site name is + `omnisearch` — never call `webcmd research` or omit the site prefix. +2. Choose the command by intent: + - `research ""` — whole-space scan across all sources. Add + `--sources hn,lobsters,stackoverflow,github,arxiv` to narrow. + - `github` / `stackoverflow` — real problems people report/ask. + - `hackermind ""` — HN opinions and discussions. + - `arxiv ""` — research papers. + - `devto ` — Dev.to articles. **Takes a TAG, not free text** (e.g. `saas`). + - `lobsters --sort active|newest|hot` — live discussion (no free query). + - `bluesky-posts ` — one public account's feed. +3. `research` distributes `--limit` across sources, so use `--limit` >= 24 for + coverage. `title` and `url` are always populated; `text` may be empty (HN, + StackOverflow) or truncated (GitHub 300 chars, arXiv 200). Never fabricate a + summary from an empty `text` — cite `url` and rank by `score`. +4. GitHub search is rate-limited (~10 req/min). Space out parallel calls; do not + fan out 6 GitHub calls at once. \ No newline at end of file diff --git a/plugins/omnisearch/arxiv.js b/plugins/omnisearch/arxiv.js new file mode 100644 index 0000000..f7487b4 --- /dev/null +++ b/plugins/omnisearch/arxiv.js @@ -0,0 +1,43 @@ +/** + * OmniSearch arxiv — search arXiv research papers. + * No login. Uses the public arXiv Atom API. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { arxivSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'arxiv', + tags: ['search'], + access: 'read', + description: "Search arXiv research papers (no login)", + domain: 'export.arxiv.org', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Research topic' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await arxivSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/arxiv', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/bluesky-posts.js b/plugins/omnisearch/bluesky-posts.js new file mode 100644 index 0000000..06232ea --- /dev/null +++ b/plugins/omnisearch/bluesky-posts.js @@ -0,0 +1,89 @@ +/** + * omnisearch bluesky-posts — read what a public Bluesky account is saying. + * + * No login. Uses the public Bluesky API (public.api.bsky.app) author feed + * endpoint, which returns a user's recent posts without authentication. + * This is the closest no-login analog to reading someone's public X/Twitter + * timeline. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { + ArgumentError, + CommandExecutionError, + EmptyResultError, +} from '@agentrhq/webcmd/errors'; + +const API = 'https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed'; +const HANDLE = /^[a-zA-Z0-9][a-zA-Z0-9.-]*\.[a-zA-Z]{2,}$/; + +function requireHandle(value) { + const s = String(value ?? '').trim().toLowerCase(); + if (!s || !HANDLE.test(s)) { + throw new ArgumentError('bluesky handle is required, e.g. "bsky.app" or "user.bsky.social"'); + } + return s; +} + +cli({ + site: 'omnisearch', + name: 'bluesky-posts', + access: 'read', + description: "Recent posts from a public Bluesky account (no login)", + domain: 'public.api.bsky.app', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'handle', required: true, positional: true, help: 'Bluesky handle (e.g. bsky.app)' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of posts' }, + ], + columns: ['rank', 'uri', 'createdAt', 'text', 'likeCount', 'replyCount', 'repostCount', 'url'], + func: async (kwargs) => { + const handle = requireHandle(kwargs.handle); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + + const url = new URL(API); + url.searchParams.set('actor', handle); + url.searchParams.set('limit', String(limit)); + + let json; + try { + const res = await fetch(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; webcmd-omnisearch/0.1)' }, + }); + if (!res.ok) { + throw new CommandExecutionError(`Bluesky API request failed: HTTP ${res.status}`); + } + json = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`Bluesky API request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const feed = Array.isArray(json?.feed) ? json.feed : []; + if (!feed.length) { + throw new EmptyResultError('omnisearch/bluesky-posts', `no posts found for "${handle}"`); + } + + return feed.slice(0, limit).map((entry, index) => { + const post = entry?.post ?? {}; + const author = post.author ?? {}; + const record = post.record ?? {}; + const uri = String(post.uri ?? ''); + const rkey = uri.split('/').pop() ?? ''; + return { + rank: index + 1, + uri, + createdAt: String(record.createdAt ?? post.indexedAt ?? ''), + text: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim(), + likeCount: post.likeCount ?? 0, + replyCount: post.replyCount ?? 0, + repostCount: post.repostCount ?? 0, + url: `https://bsky.app/profile/${author.handle ?? handle}/post/${rkey}`, + }; + }); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/github.js b/plugins/omnisearch/github.js new file mode 100644 index 0000000..acb9f93 --- /dev/null +++ b/plugins/omnisearch/github.js @@ -0,0 +1,43 @@ +/** + * OmniSearch github — search GitHub issues & PRs (people reporting real problems). + * No login. Uses the public GitHub search API (rate-limited to ~10/min). + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { githubSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'github', + tags: ['search'], + access: 'read', + description: "Search GitHub issues & PRs for real problems (no login)", + domain: 'api.github.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Problem or feature to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await githubSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/github', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/hackermind.js b/plugins/omnisearch/hackermind.js new file mode 100644 index 0000000..ff92ab6 --- /dev/null +++ b/plugins/omnisearch/hackermind.js @@ -0,0 +1,80 @@ +/** + * omnisearch hackermind — search Hacker News stories & comments for omnisearch. + * + * No login. Uses the public HN Algolia API, which indexes stories and + * comments. Good for researching what the tech/startup community is saying + * about a topic, product, or problem. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; + +const API = 'https://hn.algolia.com/api/v1/search'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'hackermind', + tags: ['search'], + access: 'read', + description: "Search Hacker News stories & comments (no login)", + domain: 'hn.algolia.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Topic, product, or problem to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + { + name: 'scope', + default: 'story', + help: 'What to search: story (headlines) or comment (reply text)', + choices: ['story', 'comment'], + }, + ], + columns: ['rank', 'id', 'objectType', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + const scope = String(kwargs.scope ?? 'story'); + + const url = new URL(API); + url.searchParams.set('query', query); + url.searchParams.set('tags', scope === 'comment' ? 'comment' : 'story'); + url.searchParams.set('hitsPerPage', String(limit)); + + let json; + try { + const res = await fetch(url); + if (!res.ok) throw new CommandExecutionError(`HN Algolia request failed: HTTP ${res.status}`); + json = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`HN Algolia request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const hits = Array.isArray(json?.hits) ? json.hits : []; + if (!hits.length) { + throw new EmptyResultError('omnisearch/hackermind', `no results for "${query}"`); + } + + return hits.slice(0, limit).map((h, index) => ({ + rank: index + 1, + id: h.objectID, + objectType: scope, + title: String(h.title ?? h.story_title ?? h.comment_text ?? '').replace(/<[^>]+>/g, '').trim(), + author: String(h.author ?? ''), + score: h.points ?? 0, + commentCount: h.num_comments ?? 0, + createdAt: String(h.created_at ?? ''), + url: String(h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`), + })); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/lobsters.js b/plugins/omnisearch/lobsters.js new file mode 100644 index 0000000..c793332 --- /dev/null +++ b/plugins/omnisearch/lobsters.js @@ -0,0 +1,73 @@ +/** + * omnisearch lobsters — Lobste.rs newest / active discussions. + * + * No login. Uses the public lobste.rs JSON endpoint. Lobste.rs is a + * Reddit/HN-style community; good for surfacing what developers are + * discussing about a topic right now. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; + +const SORTS = { + newest: 'https://lobste.rs/newest.json', + active: 'https://lobste.rs/active.json', + hot: 'https://lobste.rs/hottest.json', +}; + +function requireSort(value) { + const s = String(value ?? 'newest').toLowerCase(); + if (s === 'new') s = 'newest'; + if (!SORTS[s]) throw new ArgumentError(`sort must be one of: ${Object.keys(SORTS).join(', ')}`); + return s; +} + +cli({ + site: 'omnisearch', + name: 'lobsters', + access: 'read', + description: "Lobste.rs newest / active / hot discussions (no login)", + domain: 'lobste.rs', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'limit', type: 'int', default: 20, help: 'Number of stories' }, + { name: 'sort', default: 'newest', help: 'Sort order: newest, active, hot', choices: ['newest', 'active', 'hot'] }, + ], + columns: ['rank', 'id', 'title', 'author', 'score', 'commentCount', 'createdAt', 'tags', 'url'], + func: async (kwargs) => { + const sort = requireSort(kwargs.sort); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 100); + + let rows; + try { + const res = await fetch(SORTS[sort], { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; webcmd-omnisearch/0.1)' }, + }); + if (!res.ok) throw new CommandExecutionError(`lobste.rs request failed: HTTP ${res.status}`); + rows = await res.json(); + } catch (err) { + if (err instanceof CommandExecutionError) throw err; + throw new CommandExecutionError(`lobste.rs request failed: ${err instanceof Error ? err.message : String(err)}`); + } + + if (!Array.isArray(rows) || !rows.length) { + throw new EmptyResultError('omnisearch/lobsters', 'lobste.rs returned no stories'); + } + + return rows.slice(0, limit).map((s, index) => ({ + rank: index + 1, + id: s.short_id, + title: String(s.title ?? '').trim(), + author: String(s.submitter_user ?? ''), + score: s.score ?? 0, + commentCount: s.comment_count ?? 0, + createdAt: String(s.created_at ?? ''), + tags: Array.isArray(s.tags) ? s.tags.join(', ') : '', + url: String(s.comments_url ?? s.short_id_url ?? ''), + })); + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/package.json b/plugins/omnisearch/package.json new file mode 100644 index 0000000..499d584 --- /dev/null +++ b/plugins/omnisearch/package.json @@ -0,0 +1,9 @@ +{ + "name": "webcmd-plugin-omnisearch", + "version": "0.1.0", + "type": "module", + "description": "Universal search across public platforms (Bluesky, Hacker News, Lobsters, Product Hunt) — no login", + "peerDependencies": { + "@agentrhq/webcmd": ">=0.5.3" + } +} diff --git a/plugins/omnisearch/research.js b/plugins/omnisearch/research.js new file mode 100644 index 0000000..55307dc --- /dev/null +++ b/plugins/omnisearch/research.js @@ -0,0 +1,87 @@ +/** + * OmniSearch research — aggregate results about a topic across all public + * platforms into one feed. + * + * No login. Combines Hacker News, Lobste.rs, Stack Overflow, Dev.to, GitHub + * issues, and arXiv, tagging each row with its platform. Use for quick + * universal reconnaissance on a topic, product, or problem. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { + hnSearch, + lobstersSearch, + stackoverflowSearch, + devtoSearch, + githubSearch, + arxivSearch, +} from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a research topic/query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'research', + tags: ['search'], + access: 'read', + description: "Aggregate results about a topic across all public platforms (Hacker News, Lobsters, Stack Overflow, Dev.to, GitHub, arXiv)", + domain: 'multiple', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Topic, product, or problem to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results per platform' }, + { + name: 'sources', + default: 'hn,lobsters,stackoverflow,devto,github,arxiv', + help: 'Comma-separated sources to query (default: all)', + }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) { + throw new ArgumentError('limit must be a positive integer'); + } + const limit = Math.min(raw, 50); + const perPlatform = Math.ceil(limit / 6); + + const wanted = String(kwargs.sources ?? '') + .split(',') + .map((s) => s.trim().toLowerCase()) + .filter(Boolean); + + const fetchers = { + hn: () => hnSearch(query, perPlatform), + lobsters: () => lobstersSearch(query, perPlatform), + stackoverflow: () => stackoverflowSearch(query, perPlatform), + devto: () => devtoSearch(query, perPlatform), + github: () => githubSearch(query, perPlatform), + arxiv: () => arxivSearch(query, perPlatform), + }; + + const selected = wanted.length ? wanted.filter((s) => fetchers[s]) : Object.keys(fetchers); + + let rows; + try { + // Failure isolation: one rate-limited/erroring source must not wipe out the rest. + const outcomes = await Promise.allSettled(selected.map((key) => fetchers[key]())); + rows = outcomes + .filter((o) => o.status === 'fulfilled') + .flatMap((o) => o.value); + } catch (err) { + throw new CommandExecutionError(`research aggregation failed: ${err instanceof Error ? err.message : String(err)}`); + } + + if (!rows.length) { + throw new EmptyResultError('omnisearch/research', `no results found across platforms for "${query}"`); + } + + return rows.slice(0, limit); + }, +}); diff --git a/plugins/omnisearch/sources.js b/plugins/omnisearch/sources.js new file mode 100644 index 0000000..5fedf25 --- /dev/null +++ b/plugins/omnisearch/sources.js @@ -0,0 +1,194 @@ +/** + * OmniSearch — shared source fetchers for public platforms. + * + * Every search command aggregates across these. Each fetcher returns + * normalized rows: { platform, title, author, score, commentCount, createdAt, url, text } + */ +import { CommandExecutionError } from '@agentrhq/webcmd/errors'; + +/** Fetch helper: isolates transport errors into webcmd's typed error. */ +async function get(url, init, { source } = {}) { + let res; + try { + res = await fetch(url, init); + } catch (err) { + throw new CommandExecutionError( + `OmniSearch: ${source} request failed: ${err instanceof Error ? err.message : String(err)}`, + ); + } + if (!res.ok) { + throw new CommandExecutionError(`OmniSearch: ${source} HTTP ${res.status}`); + } + return res; +} + +// --- Hacker News (Algolia) --- +export async function hnSearch(query, limit) { + const url = new URL('https://hn.algolia.com/api/v1/search'); + url.searchParams.set('query', query); + url.searchParams.set('tags', 'story'); + url.searchParams.set('hitsPerPage', String(limit)); + const res = await get(url, {}, { source: 'Hacker News' }); + const json = await res.json(); + return (json?.hits ?? []).slice(0, limit).map((h) => ({ + platform: 'hackernews', + title: String(h.title ?? h.story_title ?? '').trim(), + author: String(h.author ?? ''), + score: h.points ?? 0, + commentCount: h.num_comments ?? 0, + createdAt: String(h.created_at ?? ''), + url: String(h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`), + text: '', + })); +} + +// --- Lobste.rs --- +export async function lobstersSearch(query, limit) { + const res = await get('https://lobste.rs/newest.json', { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Lobste.rs' }); + const rows = await res.json(); + if (!Array.isArray(rows)) return []; + const q = query.toLowerCase(); + return rows + .filter((s) => + String(s.title ?? '').toLowerCase().includes(q) || + String(s.description_plain ?? '').toLowerCase().includes(q) || + (Array.isArray(s.tags) && s.tags.some((t) => t.toLowerCase().includes(q))), + ) + .slice(0, limit) + .map((s) => ({ + platform: 'lobsters', + title: String(s.title ?? '').trim(), + author: String(s.submitter_user ?? ''), + score: s.score ?? 0, + commentCount: s.comment_count ?? 0, + createdAt: String(s.created_at ?? ''), + url: String(s.comments_url ?? ''), + text: String(s.description_plain ?? ''), + })); +} + +// --- Stack Overflow (public StackExchange API) --- +export async function stackoverflowSearch(query, limit) { + const url = new URL('https://api.stackexchange.com/2.3/search/advanced'); + url.searchParams.set('order', 'desc'); + url.searchParams.set('sort', 'relevance'); + url.searchParams.set('q', query); + url.searchParams.set('site', 'stackoverflow'); + url.searchParams.set('pagesize', String(limit)); + const res = await get(url, {}, { source: 'Stack Overflow' }); + const json = await res.json(); + return (json?.items ?? []).slice(0, limit).map((q) => ({ + platform: 'stackoverflow', + title: String(q.title ?? '').trim(), + author: String(q.owner?.display_name ?? ''), + score: q.score ?? 0, + commentCount: q.answer_count ?? 0, + createdAt: String(new Date((q.creation_date ?? Date.now()) * 1000).toISOString()), + url: String(q.link ?? ''), + text: '', + })); +} + +// --- Dev.to (tag-based public articles) --- +export async function devtoSearch(query, limit) { + const url = new URL('https://dev.to/api/articles'); + url.searchParams.set('tag', query.replace(/[^a-zA-Z0-9-]/g, '')); + url.searchParams.set('per_page', String(limit)); + const res = await get(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Dev.to' }); + const json = await res.json(); + if (!Array.isArray(json)) return []; + return json.slice(0, limit).map((a) => ({ + platform: 'devto', + title: String(a.title ?? '').trim(), + author: String(a.user?.username ?? a.user?.name ?? ''), + score: a.positive_reactions_count ?? 0, + commentCount: a.comments_count ?? 0, + createdAt: String(a.published_at ?? ''), + url: String(a.url ?? ''), + text: String(a.description ?? ''), + })); +} +// --- GitHub issues (people reporting problems) --- +export async function githubSearch(query, limit) { + const url = new URL('https://api.github.com/search/issues'); + url.searchParams.set('q', `${query} in:title,body`); + url.searchParams.set('per_page', String(limit)); + const res = await get(url, { headers: { 'User-Agent': 'OmniSearch/0.1' } }, { source: 'GitHub' }); + const json = await res.json(); + return (json?.items ?? []).slice(0, limit).map((i) => ({ + platform: 'github', + title: `[${i.state ?? ''}] ${String(i.title ?? '').trim()}`.trim(), + author: String(i.user?.login ?? ''), + score: i.reactions?.total_count ?? 0, + commentCount: i.comments ?? 0, + createdAt: String(i.created_at ?? ''), + url: String(i.html_url ?? ''), + text: String(i.body ?? '').slice(0, 300), + })); +} + +// --- arXiv (research papers) --- +export async function arxivSearch(query, limit) { + const url = new URL('https://export.arxiv.org/api/query'); + url.searchParams.set('search_query', `all:${query.split(' ').join('+')}`); + url.searchParams.set('max_results', String(limit)); + const res = await get(url, {}, { source: 'arXiv' }); + const xml = await res.text(); + const entryRe = /([\s\S]*?)<\/entry>/g; + const rows = []; + let m; + while ((m = entryRe.exec(xml)) !== null && rows.length < limit) { + const e = m[1]; + const title = (e.match(/([\s\S]*?)<\/title>/) || [])[1] + ?.replace(/\s+/g, ' ').trim() ?? ''; + const author = ((e.match(/<name>([\s\S]*?)<\/name>/) || [])[1] ?? '').trim(); + const id = (e.match(/<id>([\s\S]*?)<\/id>/) || [])[1] ?? ''; + const updated = ((e.match(/<updated>([\s\S]*?)<\/updated>/) || [])[1] ?? '').trim(); + const summary = ((e.match(/<summary>([\s\S]*?)<\/summary>/) || [])[1] ?? '') + .replace(/\s+/g, ' ').trim(); + rows.push({ + platform: 'arxiv', + title, + author, + score: 0, + commentCount: 0, + createdAt: updated, + url: id, + text: summary.slice(0, 200), + }); + } + return rows; +} + +// --- Bluesky (public author feed) --- +export async function blueskyPosts(handle, limit) { + const url = new URL('https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed'); + url.searchParams.set('actor', handle); + url.searchParams.set('limit', String(limit)); + const res = await get(url, { + headers: { 'User-Agent': 'Mozilla/5.0 (compatible; OmniSearch/0.1)' }, + }, { source: 'Bluesky' }); + const json = await res.json(); + const feed = Array.isArray(json?.feed) ? json.feed : []; + return feed.slice(0, limit).map((entry) => { + const post = entry?.post ?? {}; + const author = post.author ?? {}; + const record = post.record ?? {}; + const uri = String(post.uri ?? ''); + const rkey = uri.split('/').pop() ?? ''; + return { + platform: 'bluesky', + title: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim().slice(0, 200), + author: String(author.handle ?? handle), + score: post.likeCount ?? 0, + commentCount: post.replyCount ?? 0, + createdAt: String(record.createdAt ?? post.indexedAt ?? ''), + url: `https://bsky.app/profile/${author.handle ?? handle}/post/${rkey}`, + text: String(record.text ?? '').replace(/\s*\n+/g, ' ').trim(), + }; + }); +} diff --git a/plugins/omnisearch/stackoverflow.js b/plugins/omnisearch/stackoverflow.js new file mode 100644 index 0000000..60c082b --- /dev/null +++ b/plugins/omnisearch/stackoverflow.js @@ -0,0 +1,43 @@ +/** + * OmniSearch stackoverflow — search Stack Overflow for real problems & questions. + * No login. Uses the public StackExchange API. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { stackoverflowSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a search query is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'stackoverflow', + tags: ['search'], + access: 'read', + description: "Search Stack Overflow questions & problems (no login)", + domain: 'api.stackexchange.com', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'query', required: true, positional: true, help: 'Problem or question to research' }, + { name: 'limit', type: 'int', default: 20, help: 'Number of results' }, + ], + columns: ['platform', 'title', 'author', 'score', 'commentCount', 'createdAt', 'url', 'text'], + func: async (kwargs) => { + const query = requireQuery(kwargs.query); + const raw = Number(kwargs.limit ?? 20); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('limit must be a positive integer'); + const limit = Math.min(raw, 50); + let rows; + try { + rows = await stackoverflowSearch(query, limit); + } catch (err) { + throw new CommandExecutionError(err instanceof Error ? err.message : String(err)); + } + if (!rows.length) throw new EmptyResultError('omnisearch/stackoverflow', `no results for "${query}"`); + return rows; + }, +}); diff --git a/plugins/omnisearch/verdict.js b/plugins/omnisearch/verdict.js new file mode 100644 index 0000000..c1674e4 --- /dev/null +++ b/plugins/omnisearch/verdict.js @@ -0,0 +1,80 @@ +/** + * OmniSearch verdict — synthesize what the community thinks about a topic. + * + * Runs the multi-source research sweep, then returns an opinionated summary: + * per-platform top results + the single highest-traction item + which platforms + * are most engaged. This is "signal, not search" — a verdict, not a link dump. + */ +import { cli, Strategy } from '@agentrhq/webcmd/registry'; +import { ArgumentError, CommandExecutionError, EmptyResultError } from '@agentrhq/webcmd/errors'; +import { hnSearch, lobstersSearch, stackoverflowSearch, githubSearch, arxivSearch, devtoSearch } from './sources.js'; + +function requireQuery(value) { + const s = String(value ?? '').trim(); + if (!s) throw new ArgumentError('a topic is required'); + return s; +} + +cli({ + site: 'omnisearch', + name: 'verdict', + tags: ['search'], + access: 'read', + description: "Synthesize the community's verdict on a topic across all public platforms (signal, not search)", + domain: 'multiple', + strategy: Strategy.PUBLIC, + browser: false, + args: [ + { name: 'topic', required: true, positional: true, help: 'Topic to synthesize' }, + { name: 'perSource', type: 'int', default: 3, help: 'Top results per source to consider' }, + ], + columns: ['verdict', 'topResult', 'topSource', 'topScore', 'platforms', 'totalResults'], + func: async (kwargs) => { + const topic = requireQuery(kwargs.topic); + const raw = Number(kwargs.perSource ?? 3); + if (!Number.isInteger(raw) || raw <= 0) throw new ArgumentError('perSource must be a positive integer'); + const perSource = Math.min(raw, 10); + + const fetchers = [ + () => hnSearch(topic, perSource), + () => stackoverflowSearch(topic, perSource), + () => githubSearch(topic, perSource), + () => arxivSearch(topic, perSource), + () => devtoSearch(topic, perSource), + () => lobstersSearch(topic, perSource), + ]; + + let results; + try { + const outcomes = await Promise.allSettled(fetchers.map((f) => f())); + results = outcomes.filter((o) => o.status === 'fulfilled').map((o) => o.value); + } catch (err) { + throw new CommandExecutionError(`verdict failed: ${err instanceof Error ? err.message : String(err)}`); + } + + const all = results.flat(); + if (!all.length) { + throw new EmptyResultError('omnisearch/verdict', `no results for "${topic}"`); + } + + // Highest-traction single result across all sources (by score, fallback commentCount) + const top = all.reduce((a, b) => { + const as = (a.score ?? 0) + (a.commentCount ?? 0); + const bs = (b.score ?? 0) + (b.commentCount ?? 0); + return bs > as ? b : a; + }); + + const platformCounts = {}; + for (const r of all) platformCounts[r.platform] = (platformCounts[r.platform] ?? 0) + 1; + const platforms = Object.entries(platformCounts).map(([p, n]) => `${p}(${n})`).join(', '); + + return [{ + verdict: `Community signal on "${topic}": strongest result is "${top.title.slice(0, 100)}" on ${top.platform} (score ${top.score}, ${top.commentCount} comments).`, + topResult: top.title, + topSource: top.platform, + topScore: (top.score ?? 0) + (top.commentCount ?? 0), + platforms, + totalResults: all.length, + }]; + }, +}); \ No newline at end of file diff --git a/plugins/omnisearch/webcmd-plugin.json b/plugins/omnisearch/webcmd-plugin.json new file mode 100644 index 0000000..85c0ff0 --- /dev/null +++ b/plugins/omnisearch/webcmd-plugin.json @@ -0,0 +1,10 @@ +{ + "name": "omnisearch", + "version": "0.1.0", + "description": "Scrape & research 12 platforms — X, Reddit, LinkedIn, Instagram, YouTube, Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, Bluesky", + "webcmd": ">=0.5.3", + "author": { + "name": "Rishet Mehra", + "handle": "Rishet11" + } +} diff --git a/webcmd-plugin.json b/webcmd-plugin.json index 33ed848..5f6af42 100644 --- a/webcmd-plugin.json +++ b/webcmd-plugin.json @@ -774,6 +774,16 @@ "handle": "agentrhq" } }, + "omnisearch": { + "path": "plugins/omnisearch", + "version": "0.1.0", + "description": "Scrape & research 12 platforms — X, Reddit, LinkedIn, Instagram, YouTube, Hacker News, Stack Overflow, GitHub, arXiv, Dev.to, Lobsters, Bluesky", + "webcmd": ">=0.5.3", + "author": { + "name": "Rishet Mehra", + "handle": "Rishet11" + } + }, "openalex": { "path": "plugins/openalex", "version": "0.1.0",