# Community Archive

> Community Archive is a public, searchable collection of Twitter/X archives. Use this file as the canonical starting point for agents, researchers, and developers.

Canonical site: https://www.community-archive.org
Source repository: https://github.com/TheExGenesis/community-archive

## Start here

- [Docs and access guide](https://www.community-archive.org/docs): Human-readable overview of the bulk dump, API, and raw archives.
- [Agent and schema guide](https://raw.githubusercontent.com/TheExGenesis/community-archive/main/docs/agents.md): Tables, relationships, fields, and common query patterns.
- [API guide](https://raw.githubusercontent.com/TheExGenesis/community-archive/main/docs/api-doc.md): REST and JavaScript examples, filters, and pagination.
- [Interactive API reference](https://www.community-archive.org/api/reference): Browse the PostgREST API.
- [OpenAPI specification](https://www.community-archive.org/openapi.json): Machine-readable API schema.
- [Examples](https://github.com/TheExGenesis/community-archive/tree/main/docs/examples): Python examples and a canonical notebook.

## Choose the right data access method

1. Filtered or application queries: use the Supabase REST API.
2. Bulk analysis: download the daily consent-safe Parquet package from the [latest release](https://github.com/TheExGenesis/community-archive/releases/latest).
3. One user's original processed archive: authenticated owners may use the policy-aware web endpoint for their own archive.

## Bulk dump

The daily package contains enriched `tweets.parquet`, separate `profiles.parquet`,
and `manifest.json` with row counts, schemas, and checksums. It includes eligible
members and checks current PostgreSQL membership and opt-outs before publication,
including consent for referenced authors.

Start from the [latest GitHub release](https://github.com/TheExGenesis/community-archive/releases/latest).
Scripts can read the stable [latest.json pointer](https://fabxmporizzqflnftavs.supabase.co/storage/v1/object/public/community-archive-public-export/latest.json)
and follow its `manifest_url` to the current package. GitHub lists download links;
the Parquet files remain in consent-managed storage. Superseded packages are
removed, so do not bookmark versioned file URLs. Downloads may be temporarily
withdrawn when consent changes.

## API

API base URL: https://fabxmporizzqflnftavs.supabase.co
REST base URL: https://fabxmporizzqflnftavs.supabase.co/rest/v1
Public anon key: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImZhYnhtcG9yaXp6cWZsbmZ0YXZzIiwicm9sZSI6ImFub24iLCJpYXQiOjE3MjIyNDQ5MTIsImV4cCI6MjAzNzgyMDkxMn0.UIEJiUNkLsW28tBHmG-RQDW-I5JNlJLt62CSk9D_qG8

The anon key is intentionally public. Send it as both `apikey` and `Authorization: Bearer <key>`. Never expose or request a service-role key.

cURL example:

```bash
export CA_API_URL='https://fabxmporizzqflnftavs.supabase.co'
export CA_ANON_KEY='eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImZhYnhtcG9yaXp6cWZsbmZ0YXZzIiwicm9sZSI6ImFub24iLCJpYXQiOjE3MjIyNDQ5MTIsImV4cCI6MjAzNzgyMDkxMn0.UIEJiUNkLsW28tBHmG-RQDW-I5JNlJLt62CSk9D_qG8'

curl --get "$CA_API_URL/rest/v1/enriched_tweets" \
  -H "apikey: $CA_ANON_KEY" \
  -H "Authorization: Bearer $CA_ANON_KEY" \
  --data-urlencode "select=tweet_id,username,created_at,full_text" \
  --data-urlencode "username=ilike.defenderofbasic" \
  --data-urlencode "order=created_at.desc" \
  --data-urlencode "limit=5"
```

Useful public resources:

- `enriched_tweets`: tweets joined with username, display name, conversation, quote, and avatar fields.
- `tweets`: core tweet records.
- `all_account`: account IDs, usernames, display names, and counts.
- `all_profile`: bios, websites, locations, and profile media.
- `user_directory`: directory-ready account and profile data.
- `followers`, `following`, `likes`, `liked_tweets`, `tweet_media`, `tweet_urls`, `user_mentions`, `mentioned_users`, `retweets`, `quote_tweets`, `conversations`: relationship and supporting tables.

Important API rules:

- API username casing is preserved. Use a case-insensitive `ilike` filter when the original casing is unknown. Raw archive storage paths use lowercase usernames.
- Treat Twitter IDs as strings; JavaScript numbers cannot safely represent every Twitter ID.
- Use PostgREST filters such as `username=eq.name`, `created_at=gte.2025-01-01`, and `order=created_at.desc`.
- Select only needed columns.
- Responses are capped at 1,000 rows. Paginate with `limit` and `offset` or HTTP Range headers, using a stable order.
- Bulk exports include eligible members, not every account observed in the archive.

## Individual raw archives

Raw archives are private and policy-checked at download time. Use the web
endpoint `/api/archive/<username>` while authenticated as that archive owner;
blocked owners receive no download URL.

Raw archive structure: https://raw.githubusercontent.com/TheExGenesis/community-archive/main/docs/archive_data.md

## Optional deeper context

- [Docs index](https://github.com/TheExGenesis/community-archive/tree/main/docs)
- [Database types](https://github.com/TheExGenesis/community-archive/blob/main/src/database-types.ts)
- [Declarative database schemas](https://github.com/TheExGenesis/community-archive/tree/main/supabase/schemas)
- [Project README](https://raw.githubusercontent.com/TheExGenesis/community-archive/main/README.md)