Take the hypothetical conversation between two network engineer colleagues.

Engineer 1: What’s our strategy for IPv6?

Engineer 2: We posted a press release. Check it out on our media relations page.

Engineer 1: <Reads press release> Hmm. OK. I see that we’re going to provide our customers access the IPv6 Internet by end-of-year 2013. How are we going to do that?

Engineer 2: You know…I’m not sure who’s working on that.

A press release–or any other high-level assertions about IPv6 enablement– is not an IPv6 strategy. It a goal and only that. You can’t roll out a scalable, production-quality service without focusing on crucial aspects of the deployment. Some examples follow.

• IPv6 addressing schemes
• Address assignment mechanisms
• Internal and 3rd party applications/services
• Security
• IP infrastructure tools and systems
• Transition technologies
• Remote access
• End system and network IPv4/IPv6 protocol interaction
• Routing protocol selection

The carriers write detailed network design documents for new services and infrastructure components. The design document is shared with all stakeholders and revised based on feedback. This practice should be adopted by enterprises and other entities that operate large IP networks. There is necessary complexity in IP networks that can’t be informally passed along in organizational memory. Experience tells us that writing documentation is time-consuming and that engineers don’t like doing it. I believe a formalized design document is an absolute requirement for network changes on the scale of IPv6 introduction.

If the IPv6 design document is thorough and communicated throughout the organization, you’ll be positioned to avoid issues such as a department continuing to invest in IPv4-only hosts, applications, and network infrastructure. The lack of last-minute surprises helps you meet your IPv6 goals and breaks your dependence on the rapidly depleting IPv4 space.

Get those IPv6 design documents written. When you have an actionable plan, then you can issue the press release.

VMware explains why Teams was removed in a blog post.

In VMware Workstation 8 the concept of a Team has been removed from the user interface, but all of the functionality has been preserved. We did this because we believe that our implementation had serious limitations that both our users and our developers were going to great lengths to work around.

For instance, thumbnail views, bulk power operations, bandwidth throttling, packet loss, and LAN segments were all artificially restricted to only work within a team. It was also becoming increasingly difficult to add features to our product because we had to write conditional code to handle when a virtual machine was operating in a Team.

Leaving the functionality but removing it from the user interface makes little sense to me. It seems to me like VMware is discouraging the use of Teams in Workstation 8 by making their creation more time-consuming. But what do I know? I don’t have experience with massive software efforts.

I tried VMware 8. There are no features or enhancements that would convince me to give up the ability to create subnets for a team from the Team Settings menu. If you are using VMware Workstation for testing networking code, stick with Workstation 7.

The tech industry’s market leaders offer evidence of Layer 3′s superior scaling properties. The world’s largest ISPs use Layer 3 oriented designs as do content providers Google and Facebook. Amazon’s AWS, the undisputed leader of public cloud providers, is built on Layer 3. Wonder why you can’t get broadcast or multicast in your EC2 instance? Now you know.

Layer 2 still has too much focus in the network designs of several segments of the industry. The mobile broadband providers–with their history of walled-garden environments–will need to re-architect their networks on a Layer 3 foundation. Keeping up with the massive influx of bandwidth (at ~$50 USD/month per subscriber!) forces this change. These Layer 2 relics contribute negatively to the cost per bit and customer satisfaction.

Layer 2 oriented designs are also dominant in corporate data centers and corporate networks in general. I’ve probably heard every reason why the Layer 2 focus is necessary. I’d argue that over 90 percent of Layer 2 requirements stem from old assumptions about building corporate IT infrastructure. Of course, overhauling these networks is non-trivial, and I don’t underestimate the massive effort required. But it can be done. An excellent 2008 Network World article called The Google-ization of Bechtel describes Bechtel’s IT revamp. If Bechtel can fundamentally change its IT architecture, why can’t your organization?

Before investing in hacks to make Layer 2 scale, consider how a Layer 3 oriented design can reduce outages, simplify new service introduction, and scale existing services to meet business needs.

IPv6–with its long hexadecimal addresses–can be intimidating for engineers who have built a career on IPv4 networking. I recall my hesitation to get involved in turning up customer tunnels to Sprint’s IPv6 overlay network in the early 2000s. I felt that I could fix any problem thrown my way on IPv4. Why invest time to learn IPv6 when its adoption was so limited? Clearly, my sentiment my short-sighted, and I corrected my thinking.

I take the reassuring approach in talking to engineers who are new to IPv6. Gaurab Upadhaya of Limelight Networks put it well: only 96 more bits, no magic. After engineers understand IPv6 basics such as addressing, SLAAC, and neighbor discovery, they’ll begin to understand that the simple, connectionless packet service provided by IP is the same for both versions. Routing is routing (albeit with OSPFv3 for IPv6). The best practices for building scalable IPv4 networks carry over to IPv6 largely intact.

I tend to emphasize the differences between IPv4 and IPv6 in talking about network strategy, design (including migration and transition mechanisms), security, and business continuity. In these areas, the protocol deviations drive the discussion. Let’s take security.  IPv6 implementation introduces new attack vectors. Neighbor Discovery and Router Advertisements, not present in IPv4, can be   subject to denial of service and spoofing. Also, there is new way for malcontents to communicate with your infrastructure. Does your security policy align for IPv4/IPv6 and does it account for IPv6-specific security issues?

IPv4 exhaustion is imminent; engineers and IT leaders will be forced to make critical decision about IPv6 in 2012. Their discussions will likely include aspects of both IPv4/IPv6 similarities and differences.  As long as organizations are having these talks with the intent to act in the near term, they are better off than they would be by ignoring IPv6.

That’s enough on me.

This post is about thanking my professional and social network for advice, feedback, referrals, and business. I am thankful to have met so many people during my ten years at Sprint and my time working independently who have been gracious with their time. Although I have a small budget for online advertising, my engagements will almost always stem from personal/business ties. I hope to continue to finding ways to give back to my network. I will be in the 51% of businesses that survive the first five years, and I am confident that I’ll have many more people to thank in 2014.

I design IP networks for my clients. I know my way around basic Linux system administration and can probably figure most things out with patience and Google. I respect talented sys admins who understand the service that the IP network provides to their systems and can communicate simple network conditions (e.g., “I can’t ping the default gateway”). Who will be integrating the VPC and the corporate network? Clearly, both network engineers and sys admins will be involved. You wouldn’t want a sys admin making critical IP design decisions any more than you’d want me standing up a hadoop cluster.

Network engineers will have to adapt their thinking to the virtualized environment. This is a new way of thinking about moving packets. Networking components in the physical world are about as un-elastic resources as possible. I would argue more so than servers. Getting to a point in which network engineers can grasp the flexibility in VPC is going to require investment on their part in learning–the same way learning IS-IS would for an engineer who knows OSPF.

Educating network engineers in VPC networks is in Amazon’s best interests. It’s going to be guys like me who will get calls from potential clients wanting to tie their VPC into their network. The existing documentation does little to further that goal. I had to reach the VPC guide several times before obtaining a degree of comfort. Elastic Network Interfaces? Implied routers? Subnet routing tables? These concepts are not intuitive for network engineers.

Here’s how I recommend that Amazon could educate my networking brethren.

1. Write a guide on the VPC intended for network engineers. Think about how Juniper write JUNOS documentation for engineers with an Cisco background. This is a very effective way to quickly get smart folks up-to-speed.
2. Document use cases & recommended architectures for VPC that involve VPC to VPC and VPC to data center connectivity. Cisco excels in this area with its Cisco Validated Designs. Mimic their approach. Today, the documentation is limited to connecting a VPC gateway to a router with IPsec. This barely scratches the surface of how customers will use the networking capabilities of the VPC.
3. Create online training that steps through the configuration of a VPC. Adding a hands-on component with “actual” VPCs shouldn’t be that difficult for a company that does virtualization at a massive scale.
4. Talk to internal and external networking savvy engineers. I’ve met some sharp engineers who work on Amazon’s backbone. By engaging them and engineers outside of Amazon, the company could gain valuable insight on networking.

Migrating to the VPC should be as frictionless as possible for businesses. The accelerated set-up of a stable and scalable VPC will translate into more revenue for Amazon.

Recently I moved my company’s page to Amazon’s AWS. I needed IPv6 support, and the hosting company I was using kept promising IPv6 in 2 to 3 months but never delivered. I used the process I outlined in a previous post to make my site reachable via IPv6 using Elastic Load Balancers. I recommend reading that post before continuing if you don’t know how to do this.

In implementing IPv6 connectivity for my site, I stumbled on a problem that I had not considered. The URL for my company is http://brooksconsulting-llc.com. The URL is already long; I don’t want to put http://www.brooksconsulting-llc.com on company material, email signature, and business card. The “naked” domain, meaning the top of the zone, is called the zone apex. Per RFC 1034, CNAMEs cannot co-exist with required NS and SOA records. The IPv6 hack using AWS Elastic Load Balancers needs a CNAME. Fortunately, AWS does some proprietary magic and accommodates CNAMEs at the zone apex (see announcement here).

You must use AWS’s Route 53 tool for your zone. This wasn’t a problem for me. I prefer Route 53′s zone management GUI over GoDaddy’s. I realized that the Route 53 GUI appears not to support AWS’s on-the-fly conversion from CNAME to A/AAAA record. I had to use the CLI tools to add the records. I used the elb-associate-route53-hosted-zone command twice–once with the –rr-type A (the default) and once with the –rr-type AAAA flag–to add the entry. For more information, check out this section of the Elastic Load Balancing Developer Guide.

I posted a question to ServerFault to see if there was a way to perform the association in the Route 53 GUI. Jesper Mortensen provided a very helpful response. He believes the association can’t be made in the GUI.

Does all of this sound daunting? Well, I probably took a more difficult path than necessary. I’ve read that DNS30 has a GUI to manage Route 53 that includes a method to instruct Route 53 to do the CNAME to A/AAAA record conversion. You may want to take this approach, especially if you don’t already have the EC2 and Load Balancing API tools installed on your system.

In responding to my question at ServerFault, Jesper pointed out that there is an effort underway to standardize the use of CNAMEs at the zone apex. The Internet Draft is here.

——————————————————————————————-

UPDATE (1/5/2012) – A friendly engineer from the AWS Route 53 team contacted me and provided instructions for creating alias resource record sets in the Route 53 console. I confirmed that these work.

Here are the steps.

1. click create record set
2. for zone apex record just leave the name field blank
3. select the type of alias you want to make A or AAAA (all steps after this are the same for both types)
4. Select the yes radio button.
5. Open the EC2 console in another tab and navigate to the list of your load balancers.
6. Click on the load balancer and look at the description tab in the pane below the list. Sample output below

DNS Name:
new-balancer-751654286.us-east-1.elb.amazonaws.com (A Record)
ipv6.new-balancer-751654286.us-east-1.elb.amazonaws.com (AAAA Record)
dualstack.new-balancer-751654286.us-east-1.elb.amazonaws.com (A or AAAA Record)

Note: Because the set of IP addresses associated with a LoadBalancer can change over time,
you should never create an “A” record with any specific IP address. If you want to use a friendly
DNS name for your LoadBalancer instead of the name generated by the Elastic Load Balancing
service, you should create a CNAME record for the LoadBalancer DNS name, or use Amazon Route 53
to create a hosted zone. For more information, see the Using Domain Names With Elastic Load Balancing

Status: 0 of 0 instances in service

Port Configuration: 80 (HTTP) forwarding to 80 (HTTP)

Stickiness: Disabled(edit)

Availability Zones:
us-east-1b

Source Security Group:
amazon-elb-sg

Owner Alias: amazon-elb

Hosted Zone ID:
Z3DZXD0Q79N41H

7. Now copy the Hosted zone ID in the above case ‘ Z3DZXD0Q79N41H’ and paste it into the field labeled ‘Alias Hosted Zone ID:’
8. Now copy the DNS Name in the above case ‘ new-balancer-751654286.us-east-1.elb.amazonaws.com‘ and paste into the field ‘ Alias DNS Name:’
-Just an FYI this DNS name is the same for both A and AAAA alias records. (do not use ‘ ipv6.new-balancer-751654286.us-east-1.elb.amazonaws.com‘)
9. Click create record set or at this time you can select yes to weight the record and provide a weight between 0-255 and a setID such as ‘my load balancer’

Check it out at http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_14-3/ipj_14-3.pdf.

Instance Set-up

1. Install a Linux-based Amazon Machine Instance. If you want to follow along with this tutorial, use a Ubuntu 10.04 LTS instance that Canonical uploaded to the Community AMIs (AMI ID ami-63be790a). Use US East or EU (Ireland) servers. If this is your first time setting up an instance, I recommend viewing Greg Wilson’s tutorial on Youtube.
2. Log in using the “ubuntu” user name. Use the ssh private key as described in the video.
3. Install the packages required for a LAMP server. A simple way to do this is to “sudo tasksel --section server”. Select “LAMP server” in the graphical installer. Strangely, the LAMP selection does not install PHP. I did this manually with “sudo apt-get install php5-cli”.

Load Balancer Set-up

1. Click on “Load Balancer” in the “Network & Security” left panel of the AWS Console. Click the “Create Load Balancer” button.
2. Give your load balancer a name. I used the default HTTP entry. For the health check, I used the default settings.
3. Add your instance to the load balancer.
4. Now that the load balancer is created, place a check next to its entry so that detailed information appears in the bottom panel.
5. Write down your IPv4, IPv6, and dual stack DNS names.
6. Click on the Instances tab in the bottom panel. Make sure the instance’s status indicates “In Service”. Note:  I’ve noticed that the time required for the health check to add the instance into service can be 20 – 45 minutes.

Testing DNS and Load Balancer

1. Use dig or nslookup to verify that you get A (IPv4) and AAAA (IPv6) records. This verification step is primarily for your information.

ubuntu@ip-10-244-171-28:~$ nslookup
> Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com
Server: 172.16.0.23
Address: 172.16.0.23#53
Non-authoritative answer:
Name: Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com
Address: 50.19.220.184
> set type=AAAA
> ipv6.Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com
Server: 172.16.0.23
Address: 172.16.0.23#53
Non-authoritative answer:
ipv6.Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com has AAAA address 2406:da00:ff00::3213:dcb8
Authoritative answers can be found from:
> dualstack.Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com
Server: 172.16.0.23
Address: 172.16.0.23#53
Non-authoritative answer:
dualstack.Jeff-LB-Test-1796974432.us-east-1.elb.amazonaws.com has AAAA address 2406:da00:ff00::3213:dcb8
Authoritative answers can be found from:
>

2. Create a script called test.php with the following text.

   <?php
   
   $headers = apache_request_headers();
   $ip = $headers["X-Forwarded-For"];
   
   if($ip) {
     print "X-Forwarded-For header is $ip";
   }
   else {
     $ip =  getenv('REMOTE_ADDR');
     print "IP is $ip";
   }
   
   ?>
   

   Amazon’s Elastic Load Balancers will set the X-Forwarded-For header to the IPv6 source address. If the connection is made via IPv4, the X-Forwarded-For variable is undefined. Put this script in /var/www.

3. Using your web browser, access http://yourIPv4DNS/test.php, http://yourIPv6DNS/test.php, and http://yourDualstackDNS/test.php. Assuming you are accessing from a dual stack IPv4/IPv6 end host that prefers IPv6, you will see an IPv4 address, an IPv6 address, and an IPv6 address respectively.

Congratulations! Your content is now available over IPv6. Now you can set the CNAME record for your domain to the dual stack DNS name so that users can type in your domain and reach your site via IPv4 or IPv6. For more information on how to use CNAME’s with Amazon EC2, see Using Domain Names with Elastic Load Balancing.

I hope this post encourages people to make content available over IPv6. The days of assuming all end hosts are reachable via IPv4 are over. Amazon’s EC2 and Elastic Load Balancers make transitioning content to IPv6 simple.

Natty does not activate IPv6 by default and requires special configuration. Windows 7 has better IPv6 support out of the box than Ubuntu Natty Narwhal. Surprising? An average user can obtain connectivity to the IPv6 Internet using default configuration in Windows 7.

Activating IPv6 in Natty can be done using NetworkManager  (GUI tool) or  manually editing /etc/network/interfaces. There are limitations. Even though the ISC DHCP client supports DHCPv6, the end station will not request a nameserver. For /etc/network/interfaces, you’d think that obtaining an IPv6 address through DHCPv6 would be simple: add an “iface ethx inet6 dhcp” line. Unfortunately, this won’t work in Natty. I am using a hack. I assign a ULA address and then use post-up to execute dhclient.

iface eth0 inet6 static
address fc00::1
netmask 64
post-up /sbin/dhclient -6 eth0

I figured this out through trial and error; this method is not fitting for the average user.

The next Ubuntu release is Oneiric Ocelot. IPv6 should be enabled by default in it. I am testing an alpha build that doesn’t yet have a fix for enabling DHCPv6 in /etc/network/interfaces. Let’s hope this gets fixed prior to release.

After encountering problems, I found that Tore Anderson opened bugs in April 2011. I recommend reading Tore’s exchange with the developer on enabling IPv6 by default. Tore tries to correct the mindset that IPv6 is only needed for power users.

The thread is here.

The bug for the missing DNS server in DHCPv6 is here.