From Terra Incognita to API Incognita: What Captain Cook Teaches Us About API Discovery
When I was young, my parents often took me on UK seaside holidays, including a trip to Whitby, which was known for its kippers (smoked herring), jet (a semi-precious stone), and the then vibrant fishing industry. Whitby was also where Captain Cook, a famous British explorer, learned seamanship. During the 1760s, Cook surveyed Newfoundland and Labrador's coasts, creating precise charts of harbors, anchorages, and dangerous waters, such as Trinity Bay and the Grand Banks. His work, crucial for fishing and navigation, demonstrated exceptional skill and established his reputation, leading to his Pacific expeditions. Cook's charts, featuring detailed observations and navigational hazards, were so accurate they remained in use well into the 20th century. This made me think about how cartography and API Discovery are very similar. API discovery is about mapping and finding unknown and undocumented APIs. When you know which APIs you're actually putting out there, you're much less likely to run into problems that could sink your application rollout like a ship that runs aground, or leaves you dead in the water like a vessel that's lost both mast and rudder in stormy seas. This inspired me to show F5’s process for finding cloud SaaS APIs with a simple REST API. I honored Captain Cook and his Newfoundland trip by using this. Roughly speaking...this is my architecture. I thought about a simple rest API running in AWS. I call it the “Cook API”. An overview of the API interface is as follows. "title": "Captain Cook's Newfoundland Mapping API", "description": "Imaginary Rest API for testing API Discovery", "available_endpoints": [ {"path": "/api/charts", "methods": ["GET", "POST"], "description": "Access and create mapping charts"}, {"path": "/api/charts/<chart_id>", "methods": ["GET"], "description": "Get details of a specific chart"}, {"path": "/api/hazards", "methods": ["GET"], "description": "Get current navigation hazards"}, {"path": "/api/journal", "methods": ["GET"], "description": "Access Captain Cook's expedition journal"}, {"path": "/api/vessels", "methods": ["GET"], "description": "Get information about expedition vessels"}, {"path": "/api/vessels/<vessel_id>", "methods": ["GET"], "description": "Get status of a specific vessel"}, {"path": "/api/resources", "methods": ["GET", "POST"], "description": "Manage expedition resources"} ], I set up a container running on a server in AWS that hosts my REST API. To test my API, I create a partial swagger file that represents only a subset of the APIs that the container is advertising. openapi: 3.0.0 info: title: Captain Cook's Newfoundland Mapping API (Simplified) description: > A simplified version of the API. This documentation shows only the Charts and Vessels endpoints. version: 1.0.0 contact: name: API Support email: support@example.com license: name: MIT url: https://opensource.org/licenses/MIT servers: - url: http://localhost:1234 description: Development server - url: http://your-instance-ip description: Instance tags: - name: Charts description: Mapping charts created by Captain Cook - name: Vessels description: Information about expedition vessels paths: /: get: summary: API welcome page and endpoint documentation description: Provides an overview of the available endpoints and API capabilities responses: '200': description: Successful operation content: application/json: schema: type: object properties: title: type: string example: "Captain Cook's Newfoundland Mapping API" description: type: string available_endpoints: type: array items: type: object properties: path: type: string methods: type: array items: type: string description: type: string /api/charts: get: summary: Get all mapping charts description: Retrieves a list of all mapping charts created during the Newfoundland expedition tags: - Charts responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: type: object additionalProperties: $ref: '#/components/schemas/Chart' post: summary: Create a new chart description: Adds a new mapping chart to the collection tags: - Charts requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ChartInput' responses: '201': description: Chart created successfully content: application/json: schema: type: object properties: status: type: string example: "success" message: type: string example: "Chart added" id: type: string example: "6" '400': description: Invalid input content: application/json: schema: $ref: '#/components/schemas/Error' /api/charts/{chartId}: get: summary: Get a specific chart description: Retrieves a specific mapping chart by its ID tags: - Charts parameters: - name: chartId in: path required: true description: ID of the chart to retrieve schema: type: string responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: $ref: '#/components/schemas/Chart' '404': description: Chart not found content: application/json: schema: $ref: '#/components/schemas/Error' /api/vessels: get: summary: Get all expedition vessels description: Retrieves information about all vessels involved in the expedition tags: - Vessels responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: type: array items: $ref: '#/components/schemas/VesselBasic' /api/vessels/{vesselId}: get: summary: Get a specific vessel description: Retrieves detailed information about a specific vessel by its ID tags: - Vessels parameters: - name: vesselId in: path required: true description: ID of the vessel to retrieve schema: type: string responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: $ref: '#/components/schemas/VesselDetailed' '404': description: Vessel not found content: application/json: schema: $ref: '#/components/schemas/Error' components: schemas: Chart: type: object properties: name: type: string example: "Trinity Bay" completed: type: boolean example: true date: type: string format: date example: "1763-06-14" landmarks: type: integer example: 12 risk_level: type: string enum: [Low, Medium, High] example: "Medium" ChartInput: type: object required: - name properties: name: type: string example: "St. Mary Bay" completed: type: boolean example: false date: type: string format: date example: "1767-05-20" landmarks: type: integer example: 7 risk_level: type: string enum: [Low, Medium, High, Unknown] example: "Medium" VesselBasic: type: object properties: id: type: string example: "HMS_Grenville" type: type: string example: "Survey Sloop" crew: type: integer example: 18 status: type: string enum: [Active, In-port, Damaged, Repairs] example: "Active" VesselDetailed: allOf: - $ref: '#/components/schemas/VesselBasic' - type: object properties: current_position: type: string example: "LAT: 48.2342°N, LONG: -53.4891°W" heading: type: string example: "Northeast" weather_conditions: type: string enum: [Favorable, Challenging, Dangerous] example: "Favorable" Error: type: object properties: status: type: string example: "error" message: type: string example: "Resource not found" The process to set up API discovery in F5 Distributed Cloud is very simple. I create an origin pool that points to my upstream REST API. I then create a load balancer in distributed cloud and o Associate the origin pool with the load balancer o Enable API definition and import my partial swagger file as my API inventory. Some Screenshots below. o Enable API Discovery Select Enable from Redirect Traffic Run a shell script that tests my API. Take a break or do something else for the API Discovery capabilities to populate the dashboard. The process of API Discovery to show up in the XC Security Dashboard can take several hours. Results Well, as predicted, API discovery has found that my Swagger file is only representing a subset of my APIs. API discovery has found an additional 4 APIs that were not included in the swagger file. Distributed cloud describes these as “Shadow” APIs, or APIs that you may not have known about. API Discovery has also discovered that sensitive data is being returned by couple of the APIs What Now? If this were a real-world situation, you would have to review what was found, paying special attention to APIs that may be returning sensitive data. Each of what we call “shadow” APIs could pose a security risk, so you should review each of these APIs. The good thing is that we are now using distributed cloud, and we can use distributed cloud to protect our APIs. It is very easy to allow only those APIs that your project team might be using. For the APIs that you are exposing through the platform, you can:- Implement JWT Authentication if none exists and authentication is required. Configure Rate Limiting Add a WAF Policy Implement Bot Protection Policy Continually log and monitor your API traffic. Also, you should update your API inventory to include the entirety of the APIs that the Application provides You should only expose the APIs that are being used All of these things are simple to set up in the F5 Distributed Cloud Application Conclusion You need effective API management and discovery. Detecting "Shadow" APIs is crucial to preventing sensitive data exposure. Much like Captain Cook charting unknown territories, the process of uncovering APIs previously hidden in the system reveals a need for precision and vigilance. Cook’s expeditions needed detailed maps and careful navigation to avoid hidden dangers. Modern API management needs tools that can accurately map and monitor every endpoint. By embracing this meticulous approach, we can not only safeguard sensitive data but also steer our digital operations toward a more secure and efficient future. To quote a pirate who happens to be an API security expert and likes a Haiku. Know yer API seas, Map each endpoint 'fore ye sail— Blind waters sink ships.
