Live blog of operational systems track at NSDI’14

Operational Experiences with Disk Imaging in a Multi-Tenant Datacenter

Overview

  • Looking at disk images in a datacenter to determine characteristics
  • Logged every single request to provision a virtual machine in Emulab for for four years, 180k requests.  1,301 individuals, 368 orbs, 714 unique images.  600 machines are included.
  • No idea how representative emu lab is of usage in a ‘standard datacenter.’  But, it’s a datacenter for which data can be easily collected.

Results

  • Facility vs. user images: Most people in the tail use facility images, whereas most frequent users use their own custom images.
  • Image popularity (requests for images vs. rank).  Facility images follow an exponential distribution.   Small number of images that are used a lot.  User images have a heavier tail.
  • scaling (number of images used vs. percentage of user base).  All facility images are used by just 10% of user base.  User images grows linearly with percent of user phase.
  • daily working set size (frequency vs. images used per day).  Follows a normal distribution with mean 11.98.  So, working set size/day is relatively small, so caching maybe possible
  • Block-level similarity of images — measured by how many blocks would need to be written to transform the base image into a derived image.  Found that there is a large amount of similarity.  So, de-duplication is attractive.
  • pre-loading rate (probability of satisfying request vs. rate at which facility can reload images normalized to arrival rate).  As rate approaches arrival rate, probability of satisfying image load requests increases dramatically.

Conclusion

  • Treating user and facility images makes sense
  • De-duplication is attractive
  • Differential loading requires new strategies
  • Can explore data at http://aptlab.net/p/tbres/nsdi14

Questions

  • How many people chose the default image?  Answer: Not many, only 8% or so.  Maybe because the default isn’t that good.  Might be able to influence this imaging and convince more people to choose default(?)
  • If disk images are sparse, can you preload lots of images onto disks so you don’t have to fetch them on-demand?  Answer: Yes, we already do that
  • If you had to design emu lab from scratch, what you still have users have custom images?  Answer: Yes, would absolutely use same model in the future.

VPN Gate: A Volunteer-Organized Public VPN Relay System with Blocking Resistance for Government Censorship Firewalls

Overview

  • Circumventing firewall blocking is difficult.  Authors of this work is to make a public VPN relay that circumvent’s China’s firewall.
  • Approach is to organize many volunteers by creating a server that lists computers running VPN Gate (VPN servers).
  • Problem with the above is that authorities can ask for the server list.  To overcome this, the authors use a technique called ‘innocent IP mixing.’  This seems to involve adding random servers to the server list.   This might cause problems for real clients.  Also, authors can probe each IP address in the server list.
  • An additional technique is ‘collaborative spy detection.’  VPN Gate servers collaborate collaborate to determine whether they are being probed by authorities and return invalid information in such cases (how do the servers know authorities are trying to access them instead of regular users?)
  • All implementation and research on this work done in Japan 🙂

Results

  • Response to VPN software.  Initially there was nothing because those days were holidays!   Afterward, they ran an automated program to extract all IP address in VPN software list and added it to their firewall.  Authorities completely trusted the server list initially, so they also added the randomly added IP addresses to their firewall.  These IP addresses included DNS servers!  So, the presenter pointed out that china’s firewall was basically under their control (in that they could influence it directly) for a while.
  • Next, authorities improved their crawler to avoid certain IP addresses.  They started probing individual VPN machines in the sever list using static IPs.  The VPN machines logged the IP addresses that were accessing them. An analysis on the logs revealed the authorities’ IP addresses and started to return invalid information to them.
  • Collaborative spy detection has worked effectively since deployment.  Before deployment, the average chinese user had to try five times to find a server that circumvented the firewall.  After deployment, this number has decreased — at one point, the number was 1.2 tries.
  • Number of active VPN machines are increasing linearly at a fairly large rate.  Currently, there are over 7000 servers participating.
  • Currently 32,000 daily unique client IPs from China and 16,000 daily from Iran

Questions

  • How many authority IPs were seen?  Seemingly 2000-3000, a lot of them are very long lived
  • How do you prevent the firewall blocking the server list?  Answer: mirroring?  [I missed the rest of this answer]
  • What’s next for the firewall?  Can authorities start sending packets from lots of machines, most of which are not usually controlled by them?  Will this cause your spy mechanism to stop blocking actual valid users?  Answer: Hard to predict future
  • Does firewall do packet inspection?  Answer: No.

Leave a Reply

Your email address will not be published. Required fields are marked *