From Terra Incognita to API Incognita: What Captain Cook Teaches Us About API Discovery

 

 

When I was young, my parents often took me on UK seaside holidays, including a trip to Whitby,  which was known for its kippers (smoked herring), jet (a semi-precious stone), and the then vibrant fishing industry. Whitby was also where Captain Cook, a famous British explorer, learned seamanship.

During the 1760s, Cook surveyed Newfoundland and Labrador's coasts, creating precise charts of harbors, anchorages, and dangerous waters, such as Trinity Bay and the Grand Banks. His work, crucial for fishing and navigation, demonstrated exceptional skill and established his reputation, leading to his Pacific expeditions. Cook's charts, featuring detailed observations and navigational hazards, were so accurate they remained in use well into the 20th century.

This made me think about how cartography and API Discovery are very similar. API discovery is about mapping and finding unknown and undocumented APIs. When you know which APIs you're actually putting out there, you're much less likely to run into problems that could sink your application rollout like a ship that runs aground, or leaves you dead in the water like a vessel that's lost both mast and rudder in stormy seas.

This inspired me to show F5’s process for finding cloud SaaS APIs with a simple REST API. I honored Captain Cook and his Newfoundland trip by using this.

Roughly speaking...this is my architecture.

 

 

 

I thought about a simple rest API running in AWS. I call it the “Cook API”.  

An overview of the API interface is as follows.

 

"title": "Captain Cook's Newfoundland Mapping API",

        "description": "Imaginary Rest API for testing API Discovery",

        "available_endpoints": [

            {"path": "/api/charts", "methods": ["GET", "POST"], "description": "Access and create mapping charts"},

            {"path": "/api/charts/<chart_id>", "methods": ["GET"], "description": "Get details of a specific chart"},

            {"path": "/api/hazards", "methods": ["GET"], "description": "Get current navigation hazards"},

            {"path": "/api/journal", "methods": ["GET"], "description": "Access Captain Cook's expedition journal"},

            {"path": "/api/vessels", "methods": ["GET"], "description": "Get information about expedition vessels"},

            {"path": "/api/vessels/<vessel_id>", "methods": ["GET"], "description": "Get status of a specific vessel"},

            {"path": "/api/resources", "methods": ["GET", "POST"], "description": "Manage expedition resources"}

        ],

 

I set up a container running on a server in AWS that hosts my REST API.

To test my API, I create a partial swagger file that represents only a subset of the APIs that the container is advertising.

 

openapi: 3.0.0
info:
  title: Captain Cook's Newfoundland Mapping API (Simplified)
  description: >
    A simplified version of the API. This documentation shows only the Charts and Vessels endpoints.
  version: 1.0.0
  contact:
    name: API Support
    email: support@example.com
  license:
    name: MIT
    url: https://opensource.org/licenses/MIT

servers:
  - url: http://localhost:1234
    description: Development server
  - url: http://your-instance-ip
    description: Instance

tags:
  - name: Charts
    description: Mapping charts created by Captain Cook
  - name: Vessels
    description: Information about expedition vessels

paths:
  /:
    get:
      summary: API welcome page and endpoint documentation
      description: Provides an overview of the available endpoints and API capabilities
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                type: object
                properties:
                  title:
                    type: string
                    example: "Captain Cook's Newfoundland Mapping API"
                  description:
                    type: string
                  available_endpoints:
                    type: array
                    items:
                      type: object
                      properties:
                        path:
                          type: string
                        methods:
                          type: array
                          items:
                            type: string
                        description:
                          type: string

  /api/charts:
    get:
      summary: Get all mapping charts
      description: Retrieves a list of all mapping charts created during the Newfoundland expedition
      tags:
        - Charts
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    example: "success"
                  data:
                    type: object
                    additionalProperties:
                      $ref: '#/components/schemas/Chart'
    post:
      summary: Create a new chart
      description: Adds a new mapping chart to the collection
      tags:
        - Charts
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ChartInput'
      responses:
        '201':
          description: Chart created successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    example: "success"
                  message:
                    type: string
                    example: "Chart added"
                  id:
                    type: string
                    example: "6"
        '400':
          description: Invalid input
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

  /api/charts/{chartId}:
    get:
      summary: Get a specific chart
      description: Retrieves a specific mapping chart by its ID
      tags:
        - Charts
      parameters:
        - name: chartId
          in: path
          required: true
          description: ID of the chart to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    example: "success"
                  data:
                    $ref: '#/components/schemas/Chart'
        '404':
          description: Chart not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

  /api/vessels:
    get:
      summary: Get all expedition vessels
      description: Retrieves information about all vessels involved in the expedition
      tags:
        - Vessels
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    example: "success"
                  data:
                    type: array
                    items:
                      $ref: '#/components/schemas/VesselBasic'

  /api/vessels/{vesselId}:
    get:
      summary: Get a specific vessel
      description: Retrieves detailed information about a specific vessel by its ID
      tags:
        - Vessels
      parameters:
        - name: vesselId
          in: path
          required: true
          description: ID of the vessel to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Successful operation
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    example: "success"
                  data:
                    $ref: '#/components/schemas/VesselDetailed'
        '404':
          description: Vessel not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

components:
  schemas:
    Chart:
      type: object
      properties:
        name:
          type: string
          example: "Trinity Bay"
        completed:
          type: boolean
          example: true
        date:
          type: string
          format: date
          example: "1763-06-14"
        landmarks:
          type: integer
          example: 12
        risk_level:
          type: string
          enum: [Low, Medium, High]
          example: "Medium"

    ChartInput:
      type: object
      required:
        - name
      properties:
        name:
          type: string
          example: "St. Mary Bay"
        completed:
          type: boolean
          example: false
        date:
          type: string
          format: date
          example: "1767-05-20"
        landmarks:
          type: integer
          example: 7
        risk_level:
          type: string
          enum: [Low, Medium, High, Unknown]
          example: "Medium"

    VesselBasic:
      type: object
      properties:
        id:
          type: string
          example: "HMS_Grenville"
        type:
          type: string
          example: "Survey Sloop"
        crew:
          type: integer
          example: 18
        status:
          type: string
          enum: [Active, In-port, Damaged, Repairs]
          example: "Active"

    VesselDetailed:
      allOf:
        - $ref: '#/components/schemas/VesselBasic'
        - type: object
          properties:
            current_position:
              type: string
              example: "LAT: 48.2342°N, LONG: -53.4891°W"
            heading:
              type: string
              example: "Northeast"
            weather_conditions:
              type: string
              enum: [Favorable, Challenging, Dangerous]
              example: "Favorable"

    Error:
      type: object
      properties:
        status:
          type: string
          example: "error"
        message:
          type: string
          example: "Resource not found"

 

The process to set up API discovery in F5 Distributed Cloud is very simple.

• I create an origin pool that points to my upstream REST API.
• I then create a load balancer in distributed cloud and

o   Associate the origin pool with the load balancer

o   Enable API definition and import my partial swagger file as my API inventory. Some Screenshots below.

 

 

 

 

 

 

 

 

o   Enable API Discovery

• Select Enable from Redirect Traffic
• Run a shell script that tests my API.
• Take a break or do something else for the API Discovery capabilities to populate the dashboard. The process of API Discovery to show up in the XC Security Dashboard can take several hours.

 

 

Results

 

Well, as predicted, API discovery has found that my Swagger file is only representing a subset of my APIs.

 

 

 

API discovery has found an additional 4 APIs that were not included in the swagger file. Distributed cloud describes these as “Shadow” APIs, or APIs that you may not have known about.

 

 

 

 

 

API Discovery has also discovered that sensitive data is being returned by couple of the APIs

 

 

 

What Now?

If this were a real-world situation, you would have to review what was found, paying special attention to APIs that may be returning sensitive data.

Each of what we call “shadow” APIs could pose a security risk, so you should review each of these APIs.

The good thing is that we are now using distributed cloud, and we can use distributed cloud to protect our APIs.

It is very easy to allow only those APIs that your project team might be using.

For the APIs that you are exposing through the platform, you can:-

• Implement JWT Authentication if none exists and authentication is required.
• Configure Rate Limiting
• Add a WAF Policy 
• Implement Bot Protection Policy
• Continually log and monitor your API traffic.
• Also, you should update your API inventory to include the entirety of the APIs that  the Application provides
• You should only expose the APIs that are being used

All of these things are simple to set up in the F5 Distributed Cloud Application

 

Conclusion

You need effective API management and discovery. Detecting "Shadow" APIs is crucial to preventing sensitive data exposure. Much like Captain Cook charting unknown territories, the process of uncovering APIs previously hidden in the system reveals a need for precision and vigilance. Cook’s expeditions needed detailed maps and careful navigation to avoid hidden dangers. Modern API management needs tools that can accurately map and monitor every endpoint. By embracing this meticulous approach, we can not only safeguard sensitive data but also steer our digital operations toward a more secure and efficient future.

To quote a pirate who happens to be an API security expert and likes a Haiku.

Know yer API seas,
Map each endpoint 'fore ye sail—
Blind waters sink ships.