Conversation
Pull Request Test Coverage Report for Build 20110929781Details
💛 - Coveralls |
Merge branch 'main' into 495-failed-to-recover-vlan-when-deleting-l2vpn-after-controller-restarts
Collaborator
Author
After the sdx-controller restarted, deleting the connection returns the used vlan. The sdx-controller log shows that the 'delete' calls to OXPs: |
Collaborator
Author
italovalcy
requested changes
Dec 1, 2025
Contributor
There was a problem hiding this comment.
@YufengXin tested here with a few scenarios and regarding the VLAN state, it seems fine, but regarding bw state, it seems there is a problem. See the test below:
- Create a few L2VPNs without bw requirements (I'm not sure this is actually necessary):
docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn1", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "999"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet01:50", "vlan": "999"}]}' | tee /tmp/l2vpn1
docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn2", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet01:50", "vlan": "any"}]}' | tee /tmp/l2vpn2
docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn3", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath1:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "any"}]}' | tee /tmp/l2vpn3
docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn4", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath2:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "vlan": "any"}]}' | tee /tmp/l2vpn4
- Check that all L2VPN were created:
~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID STATUS ENDPOINT-1 VLAN-1 ENDPOINT-2 VLAN-2
-- ------ ---------- ------ ---------- ------
1bab50e7-3e28-4818-ab7d-7c37f71aad5c up urn:sdx:port:ampath.net:Ampath2:50 4094 urn:sdx:port:tenet.ac.za:Tenet02:50 4094
1c5e6c1e-8771-4ebb-b7b8-8241bad5679b up urn:sdx:port:ampath.net:Ampath3:50 999 urn:sdx:port:tenet.ac.za:Tenet01:50 999
56af9336-019e-4b4f-b211-b459e293519d up urn:sdx:port:ampath.net:Ampath3:50 4094 urn:sdx:port:tenet.ac.za:Tenet01:50 4094
fd1b56c0-ba70-4fd4-9a36-04a3e9ad6574 up urn:sdx:port:ampath.net:Ampath1:50 4094 urn:sdx:port:tenet.ac.za:Tenet03:50 4094
- check the BW to make sure we have no reservations (ie. residual bw is 100%):
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s http://sdx-controller:8080/SDX-Controller/topology | jq -r '.links[]|.id + " " + (.residual_bandwidth|tostring)'
urn:sdx:link:interdomain:sax.net:Sax01:41:tenet.ac.za:Tenet01:41 100
urn:sdx:link:interdomain:sax.net:Sax02:41:tenet.ac.za:Tenet02:41 100
urn:sdx:link:interdomain:ampath.net:Ampath1:40:sax.net:Sax01:40 100
urn:sdx:link:interdomain:ampath.net:Ampath2:40:sax.net:Sax02:40 100
urn:sdx:link:ampath.net:Ampath1/1_Ampath2/1 100
urn:sdx:link:ampath.net:Ampath2/3_Ampath3/3 100
urn:sdx:link:ampath.net:Ampath1/2_Ampath3/2 100
urn:sdx:link:sax.net:Sax01/1_Sax02/1 100
urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1 100
urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2 100
- Create two L2VPN with bw 9Gbps (links are 10GE):
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn5", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "any"}], "qos_metrics": {"min_bw": {"value": 9}}}' | tee /tmp/l2vpn5
{
"reason": "Connection published",
"service_id": "cbcec3b4-264e-43af-960b-cf23d840c8f2",
"status": "under provisioning"
}
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn6", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet01:50", "vlan": "any"}], "qos_metrics": {"min_bw": {"value": 9}}}' | tee /tmp/l2vpn6
{
"reason": "Connection published",
"service_id": "1c5acfc1-c09d-4f83-a490-1a23efe314c3",
"status": "under provisioning"
}
- Check that link BW was updated correctly:
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s http://sdx-controller:8080/SDX-Controller/topology | jq -r '.links[]|.id + " " + (.residual_bandwidth|tostring)'
urn:sdx:link:interdomain:sax.net:Sax01:41:tenet.ac.za:Tenet01:41 10
urn:sdx:link:interdomain:sax.net:Sax02:41:tenet.ac.za:Tenet02:41 10
urn:sdx:link:interdomain:ampath.net:Ampath1:40:sax.net:Sax01:40 10
urn:sdx:link:interdomain:ampath.net:Ampath2:40:sax.net:Sax02:40 10
urn:sdx:link:ampath.net:Ampath1/1_Ampath2/1 100
urn:sdx:link:ampath.net:Ampath2/3_Ampath3/3 10
urn:sdx:link:ampath.net:Ampath1/2_Ampath3/2 10
urn:sdx:link:sax.net:Sax01/1_Sax02/1 100
urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1 10
urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2 10
- Try to create more L2VPN with BW requirements (it should fail and it is failing correctly):
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn-fail", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "vlan": "any"}], "qos_metrics": {"min_bw": {"value": 9}}}'
{
"reason": "Could not solve the request",
"service_id": "a5c6f94d-002a-4c24-8a78-a999c26d77ac",
"status": "error"
}
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn-fail", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath2:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "vlan": "any"}], "qos_metrics": {"min_bw": {"value": 9}}}'
{
"reason": "Could not solve the request",
"service_id": "aaf75403-33ed-4f3d-b977-2ce4ca7da5fc",
"status": "error"
}
- Then restart the controller and wait for it to UP and running:
~/sdx-end-to-end-tests$ docker compose up -d sdx-controller --force-recreate
- Check the BW info on the links (seems correct):
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s http://sdx-controller:8080/SDX-Controller/topology | jq -r '.links[]|.id + " " + (.residual_bandwidth|tostring)'
urn:sdx:link:interdomain:sax.net:Sax01:41:tenet.ac.za:Tenet01:41 10
urn:sdx:link:interdomain:sax.net:Sax02:41:tenet.ac.za:Tenet02:41 10
urn:sdx:link:interdomain:ampath.net:Ampath1:40:sax.net:Sax01:40 10
urn:sdx:link:interdomain:ampath.net:Ampath2:40:sax.net:Sax02:40 10
urn:sdx:link:ampath.net:Ampath1/1_Ampath2/1 100
urn:sdx:link:ampath.net:Ampath2/3_Ampath3/3 10
urn:sdx:link:ampath.net:Ampath1/2_Ampath3/2 10
urn:sdx:link:sax.net:Sax01/1_Sax02/1 100
urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1 10
urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2 10
- However, if we try to create a L2VPN with 9Gbps again (it should fail) it is actually passing:
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "l2vpn-fail", "endpoints": [{"port_id": "urn:sdx:port:ampath.net:Ampath2:50", "vlan": "any"}, {"port_id": "urn:sdx:port:tenet.ac.za:Tenet02:50", "vlan": "any"}], "qos_metrics": {"min_bw": {"value": 9}}}'
{
"reason": "Connection published",
"service_id": "fac38663-1065-4dfa-a90a-b3cf2ac487a1",
"status": "under provisioning"
}
- and BW information now is super weird (sax.net:Sax01:41:tenet.ac.za:Tenet01:41 100 become 100%!!! and also ampath.net:Ampath1:40:sax.net:Sax01:40 became 100%):
~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s http://sdx-controller:8080/SDX-Controller/topology | jq -r '.links[]|.id + " " + (.residual_bandwidth|tostring)'
urn:sdx:link:sax.net:Sax01/1_Sax02/1 100
urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1 10
urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2 10
urn:sdx:link:interdomain:sax.net:Sax01:41:tenet.ac.za:Tenet01:41 100
urn:sdx:link:interdomain:sax.net:Sax02:41:tenet.ac.za:Tenet02:41 10
urn:sdx:link:ampath.net:Ampath1/1_Ampath2/1 100
urn:sdx:link:ampath.net:Ampath2/3_Ampath3/3 10
urn:sdx:link:ampath.net:Ampath1/2_Ampath3/2 10
urn:sdx:link:interdomain:ampath.net:Ampath1:40:sax.net:Sax01:40 100
urn:sdx:link:interdomain:ampath.net:Ampath2:40:sax.net:Sax02:40 10
Tests above were executed using the end-to-end repo and a custom sdx-controller docker image built with this PR applied:
$ git clone https://github.com/atlanticwave-sdx/sdx-end-to-end-tests
$ cd sdx-end-to-end-tests
$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 8b430f2..083ff00 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -152,8 +152,7 @@ services:
python3 -m uvicorn sdx_lc.app:asgi_app --host 0.0.0.0 --port 8080
command: [""]
sdx-controller:
- image: awsdx/sdx-controller:latest
- pull_policy: always
+ image: sdx-controller:498
env_file:
- .env
volumes:
$ docker compose up -d
$ ./wait-mininet-ready.sh
$ ./scripts/run-mininet-interactive.sh
Merge remote-tracking branch 'origin/main' into 495-failed-to-recover-vlan-when-deleting-l2vpn-after-controller-restarts
Collaborator
Author
|
Did a quick test with one QoS connection provisioned, then re-start the sdx-controller, the BW accounting was correct. Will do more testing |
Collaborator
Author
|
The issue was on the inter-domain link residual bw, which was reset by the OXP domain topology re-population after the controller restart. I added a few lines of code to refresh them with the maintained info out of DB. In your above test, Steps 4, 7, 9, 10 are sufficient. Here is my test results, after Step 4, with the new commit, that shows both bw and vlans are correct after restart and connection deletion. |
Contributor
|
@YufengXin thanks for patching this! Indeed after your changes, the creation of new L2VPNs are more consistent and working fine after restart. However, if I delete all L2VPN and try to create again, it seems inconsistent. I followed exactly the same steps as my previous comment, and added the following ones:
It seems that after deleting the L2VPNs, the residual_bandwidth is inconsistent: |
Collaborator
Author
|
@italovalcy I added one more patch. Please re-test. |
italovalcy
approved these changes
Dec 10, 2025
Contributor
italovalcy
left a comment
italovalcy
left a comment
There was a problem hiding this comment.
awesome @YufengXin ! great work. I've re-executed the end-to-end tests and also the manual tests documented on this PR and they all seems fine!
YufengXin
added a commit
to atlanticwave-sdx/pce
that referenced
this pull request
Dec 10, 2025
…-details separate unreserve_vlan and update_bw in delete_connection atlanticwave-sdx/sdx-controller#498
YufengXin
deleted the
495-failed-to-recover-vlan-when-deleting-l2vpn-after-controller-restarts
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.