KERNEL REQUIREMENTS
===================

The linux kernel has had various major regressions, performance
issues and subtle bugs (especially in pmtu). Here is a short list
of some -stable kernels and the first point release that is supposedly
working well with opennhrp/dmvpn:
  3.12.8 or later
  3.14.54 or later
  3.18.22 or later[1]

[1] But you need to apply the following two backported commits:
    3cdaa5be9e ipv4: Don't increase PMTU with Datagram Too Big message
    cb6ccf09d6 route: Use ipv4_mtu instead of raw rt_pmtu

See below for list of known issues in various kernel versions.

Kernels earlier than 3.12 need CONFIG_ARPD enabled in the configuration.
Many distributions do not enable it by default, and you may need to
compile your own kernel.

KERNEL BUGS
===========

DMVPN and mGRE support in the kernel has been brittle. There are various
regressions in multiple kernel versions.

This list tries to collect them to one source of information:

- forward pmtu is disabled intentionally (but tunnel devices rely on it)
  Broken since 3.14-rc1:
    commit "ipv4: introduce ip_dst_mtu_maybe_forward and protect forwarding path against pmtu spoofing"
  Workaround:
    Set sysctl net.ipv4.ip_forward_use_pmtu=1
    See: https://marc.info/?t=143636239500003&r=1&w=2 for details
    (Should fix kernel to have this by default on for tunnel devices)

- subtle path mtu mishandling issues
  Broken since (uncertain)
  Fixed in 4.1-rc2:
    commit "ipv4: Don't increase PMTU with Datagram Too Big message."
    commit "route: Use ipv4_mtu instead of raw rt_pmtu"

- fragmentation of large packets inside tunnel not working
  Broken since 3.11-rc1
    commit "ip_tunnels: Use skb-len to PMTU check."
  Fixed in 3.14.54, 3.18.22, 4.1.9, 4.2-rc3
    commit "ip_tunnel: fix ipv4 pmtu check to honor inner ip header df"

- ipsec will crash during xfrm gc
  Broke since 3.15-rc1
    commit "flowcache: Make flow cache name space aware"
  Fixed in 3.18.10, 4.0
    commit "flowcache: Fix kernel panic in flow_cache_flush_task"

- TSO on GRE tunnels failed, and resulted in very slow performance
  Broke since 3.14.24, 3.18-rc3
    commit "gre: Use inner mac length when computing tunnel length"
  Fixed in 3.14.30, 3.18.4
    commit "gre: fix the inner mac header in nbma tunnel xmit path"
    commit "gre: Set inner mac header in gro complete"

- NAPI GRO handling was broken; causing immediate crash (32-bit only?)
  Broken since 3.13-rc1
    commit "net: gro: allow to build full sized skb"
  Fixed 3.14.5, 3.15-rc7
    commit "net: gro: make sure skb->cb[] initial content has not to be zero"

- ip_gre dst caching broke NBMA GRE tunnels
  Broken since 3.14-rc1
  Fixed in 3.14.5, 3.15-rc6
    commit "ipv4: ip_tunnels: disable cache for nbma gre tunnels"

- Few packets can be lost when neighbor entry is in NUD_PROBE state,
  and there is continuous traffic to it.
  Broken since dawn of time
  Fixed in 3.15-rc1
    commit "neigh: probe application via netlink in NUD_PROBE"

- GRO was implemented for GRE, but the hw capabilities were not updated
  correctly. In practice forwarding from non-GRE (physical) interface
  to GRE interface with gro/gso/tx offloads enabled (also on the target
  interface) does not work properly.
  Broken around 3.9 to 3.11, need to check details.

- recvfrom() returned incorrect NBMA address, breaking NAT detection
  Broken since 3.10-rc1
    commit "GRE: Refactor GRE tunneling code."
  Fixed in 3.10.27, 3.12.8, 3.13-rc7
    commit "ip_gre: fix msg_name parsing for recvfrom/recvmsg"

- sendto() was broken causing opennhrp not work at all
  Broken since 3.10-rc1
    commit "GRE: Refactor GRE tunneling code."
  Fixed in 3.10.12, 3.11-rc6
    commit "ip_gre: fix ipgre_header to return correct offset"

- PMTU was broken due to GRE driver rewrite
  Broken since 3.10-rc1
    commit "GRE: Refactor GRE tunneling code."
  Fixed in 3.11-rc1
    commit "ip_tunnels: Use skb-len to PMTU check."

- PMTU was broken due to routing cache removal
  Broken since 3.6-rc1
    commit "ipv4: Cache input routes in fib_info nexthops"
  Fixed in 3.11-rc1
    commit "ipv4: use next hop exceptions also for input routes"
    + 3 other commits
    Patches exist for 3.10, but they were not approved to 3.10-stable.

- Race condition during bootup: changing ARP flag did not flush
  existing neighbor entries, causing problems if traffic was routed
  to gre interface before opennhrp was running.
  Broken since dawn of time
  Fixed in 3.11-rc1
    commit "arp: flush arp cache on IFF_NOARP change"

- Crash in IPsec
  Broken since 3.9-rc1
    commit "xfrm: removes a superfluous check and add a statistic"
  Fixed in 3.10-rc3
    commit "xfrm: properly handle invalid states as an error"

- An incorrect ip_gre change broke NHRP traffic over GRE
  Broken since 3.8-rc2
    commit "ip_gre: make ipgre_tunnel_xmit() not parse network header as IP unconditionally"
  Fixed in 3.8.5, 3.9-rc4
    commit "Revert "ip_gre: make ipgre_tunnel_xmit() not parse network header as IP unconditionally""

- Multicast traffic over mGRE was broken.
  Broken since 2.6.34-rc2
    commit "gre: fix hard header destination address checking"
  Fixed in 2.6.39-rc2
    commit "net: gre: provide multicast mappings for ipv4 and ipv6"

- Serious performance issues causing small throughput on medium to large DMVPN networks
  Broken since dawn of time
  Fixed in 2.6.35
    multiple commits rewriting ipsec caching

- Even though around 2.6.24 is the first version where opennhrp started
  to work, there has been various PMTU, performance, and functionality
  bugs before 2.6.34. That's one of the first version I consider stable
  wrt. to opennhrp functionality.