Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/2020-09-30-checkin

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh

Agenda:

• PoC Updates
• Proposal Updates/Feedback

Dallas

• Pending vlan changes being worked on
• Almost ready
• Need to test the NATs involved: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh#/media/File:WMCS_network-NAT.png

Feedback:

• Document is much clearer now :-)
• clarify firewalling aspect. Having a perimetral firewall is a new thing for cloudvps
• How could we prioritize security concerns of both SRE and WMCS? Can we close the dmz_cidr loophole this year?

Questions:

• Can the existing technical debt be quantified?
  • Code in neutron limits WMCS, prevents upgrades. Maintaining own patches. Applying the patch isn't a ton of work, but have to verify patch still works, nothing broken, etc
  • Caused outages? Routing has broken, services broken in the past. Typically we find issues before releasing. Requires hand validation. ~1-2 weeks?
  • inability to have network isolation between tenants is a bigger longterm issue than the dmz patches.
  • current network topology is flat, upstream uses tenant networking
  • overall, maintenance, improvements, and security concerns with existing patches

• NAT exceptions, dmz_cidr, we want to remove these right?
  • Yes, we want to deprecate overall. And the existing exceptions we want to get out of neutron

• option 3 doesn't move floating ip, it moves NAT for pool of tenants between world and prod?
  • Yes. Also firewalling. The only firewall today is the core router, so want to move firewall as well.

• What kind of firewalling? What loads?
  • Core router policy firewall. Allow contacting supporting services (nfs, wikireplicas). Managing policy has proven difficult. Policies that exist in core routers protect prod from tenants. Stateless firewall today.
  • If firewall is protecting production, it can't move outside of production right?
  • WMCS can't block unwanted services from crossing
  • VM specific policy could be relocated
  • until dmz_cidr is gone, core routers must retain firewalling policies (or at extension of prod network)

• What's preventing us today from limiting network under dmz_cidr? Limiting subnets, limiting network, etc
  • We tried to do this specifically in the past, but has implications on wiki communities. Rate limits. Unclear how this would impact communities. For NFS, we have to know which server
  • time and expertise?
  • last time we tried, we had to rollback. Agreed it was a hard task.
    • But is a dependency a requirement? Could we work things in a different order?
    • Yes, it's possible. Doing this work, it unblocks other things. Freeing the NAT clarifies how to remove it.

• Is it possible to delay reducing technical debt within WMCS without puytting things at risk?

• What timing concerns does SRE have with the current proposal?
  • Want to secure infra as much as possible, and do so under Frontline Defenses this year
    • Big network security issues from cloud into production for this reason
  • Work has been delayed for this larger effort. But the loophole needs to be closed this year

• This NAT has been a longstanding issue, and needs resolved

• Can we move forward on Arhzel's goals?
  • Yes. UIsing BGP, stop exposing core router. In

• Can we move forward on option 3?
  • Still TBD