From Terra Incognita to API Incognita: What Captain Cook Teaches Us About API Discovery
When I was young, my parents often took me on UK seaside holidays, including a trip to Whitby, which was known for its kippers (smoked herring), jet (a semi-precious stone), and the then vibrant fishing industry. Whitby was also where Captain Cook, a famous British explorer, learned seamanship. During the 1760s, Cook surveyed Newfoundland and Labrador's coasts, creating precise charts of harbors, anchorages, and dangerous waters, such as Trinity Bay and the Grand Banks. His work, crucial for fishing and navigation, demonstrated exceptional skill and established his reputation, leading to his Pacific expeditions. Cook's charts, featuring detailed observations and navigational hazards, were so accurate they remained in use well into the 20th century. This made me think about how cartography and API Discovery are very similar. API discovery is about mapping and finding unknown and undocumented APIs. When you know which APIs you're actually putting out there, you're much less likely to run into problems that could sink your application rollout like a ship that runs aground, or leaves you dead in the water like a vessel that's lost both mast and rudder in stormy seas. This inspired me to show F5’s process for finding cloud SaaS APIs with a simple REST API. I honored Captain Cook and his Newfoundland trip by using this. Roughly speaking...this is my architecture. I thought about a simple rest API running in AWS. I call it the “Cook API”. An overview of the API interface is as follows. "title": "Captain Cook's Newfoundland Mapping API", "description": "Imaginary Rest API for testing API Discovery", "available_endpoints": [ {"path": "/api/charts", "methods": ["GET", "POST"], "description": "Access and create mapping charts"}, {"path": "/api/charts/<chart_id>", "methods": ["GET"], "description": "Get details of a specific chart"}, {"path": "/api/hazards", "methods": ["GET"], "description": "Get current navigation hazards"}, {"path": "/api/journal", "methods": ["GET"], "description": "Access Captain Cook's expedition journal"}, {"path": "/api/vessels", "methods": ["GET"], "description": "Get information about expedition vessels"}, {"path": "/api/vessels/<vessel_id>", "methods": ["GET"], "description": "Get status of a specific vessel"}, {"path": "/api/resources", "methods": ["GET", "POST"], "description": "Manage expedition resources"} ], I set up a container running on a server in AWS that hosts my REST API. To test my API, I create a partial swagger file that represents only a subset of the APIs that the container is advertising. openapi: 3.0.0 info: title: Captain Cook's Newfoundland Mapping API (Simplified) description: > A simplified version of the API. This documentation shows only the Charts and Vessels endpoints. version: 1.0.0 contact: name: API Support email: support@example.com license: name: MIT url: https://opensource.org/licenses/MIT servers: - url: http://localhost:1234 description: Development server - url: http://your-instance-ip description: Instance tags: - name: Charts description: Mapping charts created by Captain Cook - name: Vessels description: Information about expedition vessels paths: /: get: summary: API welcome page and endpoint documentation description: Provides an overview of the available endpoints and API capabilities responses: '200': description: Successful operation content: application/json: schema: type: object properties: title: type: string example: "Captain Cook's Newfoundland Mapping API" description: type: string available_endpoints: type: array items: type: object properties: path: type: string methods: type: array items: type: string description: type: string /api/charts: get: summary: Get all mapping charts description: Retrieves a list of all mapping charts created during the Newfoundland expedition tags: - Charts responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: type: object additionalProperties: $ref: '#/components/schemas/Chart' post: summary: Create a new chart description: Adds a new mapping chart to the collection tags: - Charts requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ChartInput' responses: '201': description: Chart created successfully content: application/json: schema: type: object properties: status: type: string example: "success" message: type: string example: "Chart added" id: type: string example: "6" '400': description: Invalid input content: application/json: schema: $ref: '#/components/schemas/Error' /api/charts/{chartId}: get: summary: Get a specific chart description: Retrieves a specific mapping chart by its ID tags: - Charts parameters: - name: chartId in: path required: true description: ID of the chart to retrieve schema: type: string responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: $ref: '#/components/schemas/Chart' '404': description: Chart not found content: application/json: schema: $ref: '#/components/schemas/Error' /api/vessels: get: summary: Get all expedition vessels description: Retrieves information about all vessels involved in the expedition tags: - Vessels responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: type: array items: $ref: '#/components/schemas/VesselBasic' /api/vessels/{vesselId}: get: summary: Get a specific vessel description: Retrieves detailed information about a specific vessel by its ID tags: - Vessels parameters: - name: vesselId in: path required: true description: ID of the vessel to retrieve schema: type: string responses: '200': description: Successful operation content: application/json: schema: type: object properties: status: type: string example: "success" data: $ref: '#/components/schemas/VesselDetailed' '404': description: Vessel not found content: application/json: schema: $ref: '#/components/schemas/Error' components: schemas: Chart: type: object properties: name: type: string example: "Trinity Bay" completed: type: boolean example: true date: type: string format: date example: "1763-06-14" landmarks: type: integer example: 12 risk_level: type: string enum: [Low, Medium, High] example: "Medium" ChartInput: type: object required: - name properties: name: type: string example: "St. Mary Bay" completed: type: boolean example: false date: type: string format: date example: "1767-05-20" landmarks: type: integer example: 7 risk_level: type: string enum: [Low, Medium, High, Unknown] example: "Medium" VesselBasic: type: object properties: id: type: string example: "HMS_Grenville" type: type: string example: "Survey Sloop" crew: type: integer example: 18 status: type: string enum: [Active, In-port, Damaged, Repairs] example: "Active" VesselDetailed: allOf: - $ref: '#/components/schemas/VesselBasic' - type: object properties: current_position: type: string example: "LAT: 48.2342°N, LONG: -53.4891°W" heading: type: string example: "Northeast" weather_conditions: type: string enum: [Favorable, Challenging, Dangerous] example: "Favorable" Error: type: object properties: status: type: string example: "error" message: type: string example: "Resource not found" The process to set up API discovery in F5 Distributed Cloud is very simple. I create an origin pool that points to my upstream REST API. I then create a load balancer in distributed cloud and o Associate the origin pool with the load balancer o Enable API definition and import my partial swagger file as my API inventory. Some Screenshots below. o Enable API Discovery Select Enable from Redirect Traffic Run a shell script that tests my API. Take a break or do something else for the API Discovery capabilities to populate the dashboard. The process of API Discovery to show up in the XC Security Dashboard can take several hours. Results Well, as predicted, API discovery has found that my Swagger file is only representing a subset of my APIs. API discovery has found an additional 4 APIs that were not included in the swagger file. Distributed cloud describes these as “Shadow” APIs, or APIs that you may not have known about. API Discovery has also discovered that sensitive data is being returned by couple of the APIs What Now? If this were a real-world situation, you would have to review what was found, paying special attention to APIs that may be returning sensitive data. Each of what we call “shadow” APIs could pose a security risk, so you should review each of these APIs. The good thing is that we are now using distributed cloud, and we can use distributed cloud to protect our APIs. It is very easy to allow only those APIs that your project team might be using. For the APIs that you are exposing through the platform, you can:- Implement JWT Authentication if none exists and authentication is required. Configure Rate Limiting Add a WAF Policy Implement Bot Protection Policy Continually log and monitor your API traffic. Also, you should update your API inventory to include the entirety of the APIs that the Application provides You should only expose the APIs that are being used All of these things are simple to set up in the F5 Distributed Cloud Application Conclusion You need effective API management and discovery. Detecting "Shadow" APIs is crucial to preventing sensitive data exposure. Much like Captain Cook charting unknown territories, the process of uncovering APIs previously hidden in the system reveals a need for precision and vigilance. Cook’s expeditions needed detailed maps and careful navigation to avoid hidden dangers. Modern API management needs tools that can accurately map and monitor every endpoint. By embracing this meticulous approach, we can not only safeguard sensitive data but also steer our digital operations toward a more secure and efficient future. To quote a pirate who happens to be an API security expert and likes a Haiku. Know yer API seas, Map each endpoint 'fore ye sail— Blind waters sink ships.
REST API Equivalent for "tmsh show cm device-group Sync-Failover"
Hi! I found the REST API for the TMSH command mentioned in the subject line: /mgmt/tm/cm/device-group/~Common~Sync-Failover?expandSubcollections=true But I can't find the following values that comes with the CLI: Time Since Last Sync (HH:MM:SS) CID Originator CID Time (UTC) LSS Originator LSS Time (UTC) Please advise Thanks J.PowerShell - How to modify system iFile?
I use PowerShell to upload a text file containing a number of parameters that I want to use in an existing iRule. Through the web-gui I have already created an iFile named 'iFileApiKeys'. In PowerShell I use this procedure to upload the text file: $length = "0-" + ($fileContent.Length - 1) + "/" + $fileContent.Length $headers = @{ "Content-Range" = $length} $uploadResult = Invoke-WebRequest $URL -Method POST -Headers $headers -InFile $filePathPlusTextfile - ContentType "multipart/form-data" -TimeoutSec 20 -Credential $mycreds | ConvertFrom-Json Write-Host "Upload Result:" Write-Host $uploadResult According to the 'uploadResult' the file ends up in the folder:'var/config/rest/downloads/iFileApiKeys.txt' on my F5 LTM Using 'SuperPutty' I can via tmos (tmsh) modify the existing system iFile by executing: `tmos> modify /sys file ifile iFileApiKeys source-path file:///var/config/rest/downloads/iFileApiKeys.txt My problem is that I can't seem to find the correct PowerShell command to achieve the same result as the tmos (tmsh) command does. I want to use and actually think I should use: `Invoke-Webrequest -Method Put` I've been trying to emulate an example taken from a Jason Rahm post on this site: https://devcentral.f5.com/articles/getting-started-with-icontrol-working-with-the-system-20592 Like this: $sysIfilePath = "/mgmt/tm/sys/file/ifile/iFileApiKeys" `$sysPath = "https://" + $host_address + $sysIfilePath $updateresult = Invoke-WebRequest -Method Put -Uri $sysPath -Headers $headers -Credential $mycreds -Body $body But this command fails unfortunately, My assumption is that I don't fill $headers and/or $body with the correct values. When executing a GET for my sys iFile object the result is: ``{"kind":"tm:sys:file:ifile:ifilestate","name":"iFileApiKeys","fullPath":"iFileApiKeys","generation":10970077,"selfLink":"https://localhost/mgmt/tm/sys/file/ifile/iFileApiKeys?ver=13.1.0.2","chec ksum":"SHA1:878:52a261b5a113db5c9421a54e1e8b5685e7da7a4d","createTime":"2018-11-26T22:52:08Z","createdBy":"per.eriksson","lastUpdateTime":"2018-12-08T19:49:15Z","mode":33188,"revision":24,"size" :878,"sourcePath":"file:///var/config/rest/downloads/iFileApiKeys.txt","updatedBy":"per.eriksson"} Anyone out there that can point me in the right direction on how to update my sys iFile using PowerShell? Thank you! /Per
How can I list distributed applications using the Python SDK?
I am trying to get a list of distributed applications using the Python SDK. I can do this via REST/Postman by drilling down to /mgmt/tm/gtm/distributed-app. However when I drill down to gtm using the Python SDK, there is no "distributed-app" class to call. I see the following... {'_meta_data': {'container': , 'bigip': , 'icr_session': , 'icontrol_version': '', 'uri': 'https://beiglb1:443/mgmt/tm/gtm/', 'minimum_version': '11.5.0', 'allowed_commands': [], 'required_command_parameters': set(), 'exclusive_attributes': [], 'object_has_stats': True, 'minimum_additional_parameters': set(), 'allowed_lazy_attributes': [, , , , , , , , , ]}} As you can see I can access my wide ips/pools, servers, data_centers, etc... from here just fine, but I cannot see my distributed apps. Any ideas on how to access them via Python SDK, or do I have to use REST?REST API access right
Hi all, We're running F5 LTM (version 14.1.4.x) and sorry for any newbie question. We're trying to perform some tasks through REST API to the LTM: 1. Check system status, e.g. curl -ks -H "Content-Type: application/json" -u admin:admin "https://192.168.1.1/mgmt/tm" -d "{\"command\":\"show\", \"utilCmdArgs\":\"-c 'cm failover-status'\"}" 2. Perform failover, e.g. curl -ks -H "Content-Type: application/json" -u admin:admin "https://192.168.1.1/mgmt/tm" -d "{\"command\":\"run\", \"utilCmdArgs\":\"-c 'sys failover standby'\"}" We don't want to expose admin password in any script, while authentication token seems only valid for 10 hours at most and not suitable to be used in cronjobs. We create user for this specific purpose with user role Resource Administrator who can perform the task through ssh or GUI. However, both commands can run as that specific user (testuser1) in tmsh interactively, but fail when running through curl with error: {"code":401,"message":"Authorization failed: user=https://localhost/mgmt/shared/authz/users/testuser1 resource=/mgmt/tm verb=POST uri:http://localhost:8100/mgmt/tm referrer:192.168.1.8 sender:192.168.1.8","referer":"192.168.1.8","restOperationId":2225072,"kind":":resterrorresponse"} I'm afraid we've something missed or setting up based on wrong concept... Would anyone please help? Thanks and Rgds /st wong
