Add External Competitions feature

.env_sample

@@ -88,6 +88,13 @@ ENABLE_SIGN_UP=True
 ENABLE_SIGN_IN=True
 
 
+# -----------------------------------------------------------------------------
+# Enable or disable the External Competitions feature (button, page, API,
+# and the daily fetch task). Off by default - intended for the main instance only.
+# -----------------------------------------------------------------------------
+EXTERNAL_COMPETITIONS_ENABLED=False
+
+
 # # S3 storage example
 # STORAGE_TYPE=s3
 # AWS_ACCESS_KEY_ID=12312312312312312331223

documentation/docs/Developers_and_Administrators/External-Competitions.md

@@ -0,0 +1,63 @@
+External Competitions lets a Codabench instance show a browsable list of public competitions hosted on *other* Codabench and CodaLab instances, fetched and synced automatically once a day. It's off by default and intended for the main `codabench.org` instance rather than self-hosted deployments.
+
+## For Codabench administrators
+
+### Enabling the feature
+
+Set the following in your `.env` file:
+
+```
+EXTERNAL_COMPETITIONS_ENABLED=True
+```
+
+This turns on the External Competitions page, a banner/link to it on the public benchmarks and competitions page, the two read-only API endpoints, and the daily Celery beat task that fetches and syncs competitions. Leaving it `False` (the default) disables all of it - the API endpoints return a 404, and the page and the banner linking to it aren't shown.
+
+### Adding a platform to sync from
+
+Platforms are managed from the Django admin, under **External Competitions -> External platforms -> Add**. Each platform needs:
+
+| Field | Description |
+|---|---|
+| Name | Display name shown on the competition tiles (e.g. "Codabench @ LISN") |
+| Platform type | `Codabench instance` or `CodaLab instance` |
+| Competitions fetch URL | The API endpoint to get that platform's list of public competitions |
+| Competition base URL | The base URL used to create complete link for each competition |
+| Active | Unchecking this skips the platform in the daily fetch (already-synced competitions stay visible - see note below) |
+
+Once saved, the platform is picked up by the next scheduled fetch (or trigger one manually, see below).
+
+### How the sync works
+
+`fetch_external_competitions` (`src/apps/external_competitions/fetch_sync.py`) runs once a day via Celery beat. For each active platform it calls the fetcher matching its platform type (`codabench_fetcher.py` or `codalab_fetcher.py`) and then diffs the result against what's already stored:
+
+- Competitions present in the fetch are created or updated (matched by `competition_url`).
+- Competitions no longer present in the fetch are deleted.
+
+Each run writes an `ExternalFetchLog` entry (visible in the admin) recording the outcome (`SUCCESS`/`FAILURE`), counts (`new_count`/`updated_count`/`deleted_count`), and, on failure, an error message - check there first if a platform's competitions look stale or missing.
+
+!!! note
+    Deactivating a platform (`is_active=False`) only stops it from being fetched going forward - competitions already synced from it stay visible on the public list until manually removed.
+
+To trigger a fetch immediately instead of waiting for the daily schedule:
+
+```bash
+docker compose exec django ./manage.py shell -c "from external_competitions.fetch_sync import fetch_external_competitions; fetch_external_competitions()"
+```
+
+## For platform administrators
+
+If you run your own Codabench or CodaLab instance and would like your public competitions to be discoverable on `codabench.org`'s External Competitions page, you can request to be added.
+
+### Registering your platform
+
+Send an email to **info@codabench.org** with the subject **"Platform Registration for External Competitions"**, including:
+
+- **Your platform's name and type** (Codabench, CodaLab instance, or other).
+- **Your competitions fetch URL** - the API endpoint we'll use once a day to retrieve your list of public competitions (for example, a Codabench instance's `.../api/competitions/public/`). This endpoint must be publicly reachable without authentication, since the fetch runs unattended.
+- **Your competition base URL** - the base URL used to build the link back to each competition on your site, so visitors clicking a competition on `codabench.org` land on the right page on yours.
+
+Please share as much detail as you can, **especially about the fetch URL** - pagination behavior, response format, expected size of the response, rate limits, or anything else likely to affect an automated daily fetch. The more we know upfront, the more reliably we can keep your competitions in sync.
+
+### Unregistering your platform
+
+To have your platform's competitions removed or paused, email **info@codabench.org** asking us to deactivate (or delete) fetching for your platform, including your platform's name and/or URL so we can identify it.

documentation/docs/Developers_and_Administrators/How-to-deploy-Codabench-on-your-server.md

@@ -243,6 +243,8 @@ You can update these by:
 1. Replacing the logos in `src/static/img/` folder
 2. Updating the code in `src/templates/pages/home.html` to point to the right websites of your organizations
 
+!!! tip
+    Now that your instance is up and running, consider registering it with `codabench.org` as an [External Competitions](External-Competitions.md) platform. Once registered, your instance's public competitions are also listed on `codabench.org`'s own External Competitions page, giving them more visibility. See the [registration instructions](External-Competitions.md#registering-your-platform) for details.
 
 ## Frequently asked questions (FAQs)
 

documentation/zensical.toml

@@ -54,6 +54,7 @@ nav = [
     {"Self-Hosters" = [
         {"How to Deploy a Server" = "Developers_and_Administrators/How-to-deploy-Codabench-on-your-server.md"},
         {"Administrative Procedures" = "Developers_and_Administrators/Administrator-procedures.md"},
+        {"External Competitions" = "Developers_and_Administrators/External-Competitions.md"},
         {"Backups - Automating Creation and Restoring" = "Developers_and_Administrators/Creating-and-Restoring-from-Backup.md"},
         {" Upgrading Codabench" = [
             "Developers_and_Administrators/Upgrading_Codabench/index.md",

src/apps/api/serializers/external_competitions.py

@@ -0,0 +1,30 @@
+from rest_framework import serializers
+
+from external_competitions.models import ExternalCompetition, ExternalPlatform
+
+
+class ExternalCompetitionSerializer(serializers.ModelSerializer):
+    platform_name = serializers.CharField(source='platform.name', read_only=True)
+    platform_type = serializers.CharField(source='platform.platform_type', read_only=True)
+
+    class Meta:
+        model = ExternalCompetition
+        fields = (
+            'id',
+            'name',
+            'description',
+            'image_url',
+            'organizer_name',
+            'competition_url',
+            'competition_created_when',
+            'competition_started_when',
+            'platform',
+            'platform_name',
+            'platform_type',
+        )
+
+
+class ExternalPlatformFilterSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = ExternalPlatform
+        fields = ('id', 'name', 'platform_type')

src/apps/api/tests/test_external_competitions.py

@@ -0,0 +1,212 @@
+import importlib
+
+from django.test import TestCase, override_settings
+from django.urls import clear_url_caches
+from rest_framework.test import APIClient
+
+from external_competitions.models import ExternalPlatform
+from factories import ExternalCompetitionFactory, ExternalPlatformFactory
+
+
+def _reload_api_urls():
+    # api/urls.py only adds the external_competitions paths to urlpatterns when
+    # it first runs, based on the setting's value at that moment. To pick up a
+    # changed setting, we need Django to re-run that file - reloading the module
+    # does that. But reloading api.urls by itself isn't enough: the root urls.py
+    # did `path('api/', include('api.urls'))`, and that include() already built a
+    # URLResolver object which cached the old urlpatterns list from api.urls the
+    # first time it ran. So we also reload the root urls module, which re-runs
+    # include('api.urls') and builds a fresh resolver pointing at the new list.
+    # clear_url_caches() then drops Django's cached lookup of the whole urlconf,
+    # so the next request resolves routes against these freshly reloaded modules.
+    import api.urls
+    importlib.reload(api.urls)
+    import urls
+    importlib.reload(urls)
+    clear_url_caches()
+
+
+class ExternalCompetitionsApiFunctionalTests(TestCase):
+    """
+    Covers the endpoints' behavior with the feature flag on. `api/urls.py` only
+    registers these paths at import time when EXTERNAL_COMPETITIONS_ENABLED is True,
+    so the flag is forced on and the urlconf reloaded for the lifetime of this class.
+    """
+
+    @classmethod
+    def setUpClass(cls):
+        super().setUpClass()
+        # override_settings only patches the setting value - it doesn't touch
+        # api/urls.py's already-built urlpatterns, since those were compiled once
+        # at import time (before any test ran). We store the override on cls (not
+        # self) because setUpClass/tearDownClass are classmethods that run once for
+        # the whole class, outside of any test instance, and both need to reference
+        # the same override object to enable/disable it.
+        cls._settings_override = override_settings(EXTERNAL_COMPETITIONS_ENABLED=True)
+        cls._settings_override.enable()
+        # Force api/urls.py (and the root urlconf that includes it) to re-run now
+        # that the setting is True, so these endpoints actually get registered.
+        _reload_api_urls()
+
+    @classmethod
+    def tearDownClass(cls):
+        cls._settings_override.disable()
+        # Reload again with the real setting restored, so api.urls doesn't leak
+        # the forced-True urlpatterns into whatever test module runs next.
+        _reload_api_urls()
+        super().tearDownClass()
+
+    def setUp(self):
+        self.client = APIClient()
+
+        self.codabench_platform = ExternalPlatformFactory(
+            name='Some Codabench',
+            platform_type=ExternalPlatform.PLATFORM_TYPE_CODABENCH,
+        )
+        self.codalab_platform = ExternalPlatformFactory(
+            name='Some CodaLab',
+            platform_type=ExternalPlatform.PLATFORM_TYPE_CODALAB,
+        )
+        self.inactive_platform = ExternalPlatformFactory(
+            name='Inactive Platform',
+            platform_type=ExternalPlatform.PLATFORM_TYPE_CODABENCH,
+            is_active=False,
+        )
+
+        self.competition1 = ExternalCompetitionFactory(
+            platform=self.codabench_platform,
+            name='AI Challenge',
+            description='An AI competition',
+            organizer_name='Jane Doe',
+        )
+        self.competition2 = ExternalCompetitionFactory(
+            platform=self.codalab_platform,
+            name='Vision Contest',
+        )
+        self.inactive_platform_competition = ExternalCompetitionFactory(
+            platform=self.inactive_platform,
+            name='Old Contest',
+        )
+
+    def test_list_returns_expected_fields(self):
+        """
+        Calls the list endpoint and checks the response is a 200 containing one
+        of the competitions we created, with every field holding the right value.
+        """
+        response = self.client.get('/api/external_competitions/')
+
+        self.assertEqual(response.status_code, 200)
+        result = next(r for r in response.data['results'] if r['id'] == self.competition1.id)
+        self.assertEqual(result['name'], 'AI Challenge')
+        self.assertEqual(result['description'], 'An AI competition')
+        self.assertEqual(result['organizer_name'], 'Jane Doe')
+        self.assertEqual(result['competition_url'], self.competition1.competition_url)
+        self.assertEqual(result['platform'], self.codabench_platform.id)
+        self.assertEqual(result['platform_name'], 'Some Codabench')
+        self.assertEqual(result['platform_type'], ExternalPlatform.PLATFORM_TYPE_CODABENCH)
+        self.assertIn('image_url', result)
+        self.assertIn('competition_created_when', result)
+        self.assertIn('competition_started_when', result)
+
+    def test_list_pagination_shape(self):
+        """
+        Calls the list endpoint and checks the response has the pagination fields
+        (count, next, previous, page_size, results), not just a plain list.
+        """
+        response = self.client.get('/api/external_competitions/')
+
+        self.assertEqual(response.status_code, 200)
+        for key in ('count', 'next', 'previous', 'page_size', 'results'):
+            self.assertIn(key, response.data)
+
+    def test_search_filter(self):
+        """
+        Searches for "vision" and checks only the competition matching that term
+        comes back in the results, and the other competition is left out.
+        """
+        response = self.client.get('/api/external_competitions/?search=vision')
+
+        self.assertEqual(response.status_code, 200)
+        ids = [r['id'] for r in response.data['results']]
+        self.assertIn(self.competition2.id, ids)
+        self.assertNotIn(self.competition1.id, ids)
+
+    def test_platform_filter(self):
+        """
+        Filters by one platform's id and checks only that platform's competition
+        comes back, while the other platform's competition is left out.
+        """
+        response = self.client.get(f'/api/external_competitions/?platform={self.codabench_platform.id}')
+
+        self.assertEqual(response.status_code, 200)
+        ids = [r['id'] for r in response.data['results']]
+        self.assertIn(self.competition1.id, ids)
+        self.assertNotIn(self.competition2.id, ids)
+
+    def test_platform_filter_accepts_comma_separated_ids(self):
+        """
+        Filters by both platforms' ids joined with a comma, and checks that both
+        platforms' competitions come back in the results.
+        """
+        response = self.client.get(
+            f'/api/external_competitions/?platform={self.codabench_platform.id},{self.codalab_platform.id}'
+        )
+
+        self.assertEqual(response.status_code, 200)
+        ids = [r['id'] for r in response.data['results']]
+        self.assertIn(self.competition1.id, ids)
+        self.assertIn(self.competition2.id, ids)
+
+    def test_competitions_from_inactive_platform_still_listed(self):
+        """
+        Checks that a competition from a deactivated platform still shows up in
+        the list - being inactive only skips future fetches, not visibility.
+        """
+        response = self.client.get('/api/external_competitions/')
+
+        self.assertEqual(response.status_code, 200)
+        ids = [r['id'] for r in response.data['results']]
+        self.assertIn(self.inactive_platform_competition.id, ids)
+
+    def test_platforms_endpoint_returns_all_platforms_unpaginated(self):
+        """
+        Calls the platforms endpoint and checks it returns every platform, active
+        or not, as a plain list (no pagination), sorted alphabetically by name.
+        """
+        response = self.client.get('/api/external_competitions/platforms/')
+
+        self.assertEqual(response.status_code, 200)
+        names = [p['name'] for p in response.data]
+        self.assertIn('Some Codabench', names)
+        self.assertIn('Some CodaLab', names)
+        self.assertIn('Inactive Platform', names)
+        self.assertEqual(names, sorted(names))
+
+
+class ExternalCompetitionsUrlGatingTests(TestCase):
+    """
+    Covers that the endpoints only exist when EXTERNAL_COMPETITIONS_ENABLED is True,
+    isolated from the functional tests above so each test controls its own flag state.
+    """
+
+    def setUp(self):
+        self.client = APIClient()
+
+    def tearDown(self):
+        # Restore api.urls to match the real (non-overridden) settings, so later
+        # test modules in the same run aren't affected by our reloads.
+        _reload_api_urls()
+
+    def test_urls_return_404_when_disabled(self):
+        with override_settings(EXTERNAL_COMPETITIONS_ENABLED=False):
+            _reload_api_urls()
+
+            self.assertEqual(self.client.get('/api/external_competitions/').status_code, 404)
+            self.assertEqual(self.client.get('/api/external_competitions/platforms/').status_code, 404)
+
+    def test_urls_return_200_when_enabled(self):
+        with override_settings(EXTERNAL_COMPETITIONS_ENABLED=True):
+            _reload_api_urls()
+
+            self.assertEqual(self.client.get('/api/external_competitions/').status_code, 200)
+            self.assertEqual(self.client.get('/api/external_competitions/platforms/').status_code, 200)

src/apps/api/urls.py

@@ -1,3 +1,4 @@
+from django.conf import settings
 from django.conf.urls import include
 from django.urls import path
 
@@ -13,6 +14,7 @@
     analytics,
     competitions,
     datasets,
+    external_competitions,
     profiles,
     leaderboards,
     submissions,
@@ -76,3 +78,9 @@
     # Include this at the end so our URLs above run first, like /datasets/completed/<pk>/ before /datasets/<pk>/
     path('', include(format_suffix_patterns(router.urls, allowed=['html', 'json', 'csv', 'zip']))),
 ]
+
+if settings.EXTERNAL_COMPETITIONS_ENABLED:
+    urlpatterns += [
+        path('external_competitions/', external_competitions.ExternalCompetitionListView.as_view(), name='external_competition_list'),
+        path('external_competitions/platforms/', external_competitions.ExternalPlatformListView.as_view(), name='external_platform_list'),
+    ]