diff options
Diffstat (limited to 'todo')
| -rw-r--r-- | todo/david.txt | 42 | ||||
| -rw-r--r-- | todo/djalal.txt | 146 | ||||
| -rw-r--r-- | todo/dmiller.txt | 12 | ||||
| -rw-r--r-- | todo/done.txt | 3711 | ||||
| -rw-r--r-- | todo/gorjan.txt | 66 | ||||
| -rw-r--r-- | todo/henri.txt | 41 | ||||
| -rw-r--r-- | todo/nmap.txt | 638 | ||||
| -rw-r--r-- | todo/nping.txt | 799 | ||||
| -rw-r--r-- | todo/patrick.txt | 77 | ||||
| -rw-r--r-- | todo/paulino.calderon.txt | 4 | ||||
| -rw-r--r-- | todo/sctp.txt | 49 | ||||
| -rw-r--r-- | todo/shinnok.txt | 150 |
12 files changed, 5735 insertions, 0 deletions
diff --git a/todo/david.txt b/todo/david.txt new file mode 100644 index 0000000..a9ce685 --- /dev/null +++ b/todo/david.txt @@ -0,0 +1,42 @@ +* Make improvements to the irc-unrealircd-backdoor script. +* Brandon says: "Sometime -sV goes just a little too fast and gets a connect + error. It should back off and try again a few times before giving up trying + to fingerprint the service." It looks like + Got nsock CONNECT response with status ERROR - aborting this service + Add a delay of 500 ms? +Summer of coder: +* Add a library function to test the randomness of a string. Use it to make + version scripts for services that send random or encrypted data, for example + cccam on port 12000 which sends 16 bytes. + +Zenmap: +* Do a memory audit of loading a large scan file. +* Figure out what licensing notices are required in the Mac package for GTK+, + Glib, Python, and anything else we use. +Summer of Coder: +* Merge a scan aggregation into one XML file. +* Synthesize text Nmap output from an XML file. + +Ncat: +* Make Ncat send one line at a time when --delay is in effect. This is + cumbersome to do until Nsock supports buffered reading. +* Make the HTTP proxy support the chunked transfer encoding, then change it to + be HTTP/1.1 and support pipelining. +* See if we can make Ncat drop privileges on startup. + +Nsock: +* Add a buffer to each iod, so that you can ask for a certain number of bytes + or lines and get exactly that many, no more. Venkat wrote a proposal at + http://seclists.org/nmap-dev/2009/q3/0600.html. + +Web site: +* Look for a good online respository viewer. + +Done: +* Handle multiple targets with the same address. +* Check necessity of mswin32 pcap includes. +* Try removing the call to PacketSetReadTimeout in readip_pcap, so that Windows + uses the short 2 ms timeout like some other platforms without selectable pcap + fds do. Measure difference in time and CPU time. +* Do JavaScript magic to expand/contract NSEDoc sidebar. +* Check out compression options for the NSIS installer. diff --git a/todo/djalal.txt b/todo/djalal.txt new file mode 100644 index 0000000..b674d16 --- /dev/null +++ b/todo/djalal.txt @@ -0,0 +1,146 @@ +== + +GSoC 2011 TASKS: + +o Work on my GSoC vulnerability and exploitation script ideas: + https://secwiki.org/w/Nmap/Script_Ideas#Djalal_Harouni + +o Review all the "Improve NSE HTTP architecture" proposal suggetions + and comments, and try to include them and update the proposal. + http://seclists.org/nmap-dev/2011/q2/967 + +o Start a thread on Nmap-dev about users favorite Nmap and NSE commands, + and create a special page for it in the secwiki.org site. + This will also let us to create more scan profiles for Zenmap. + +== + +1) Nmap Scripting Engine Infrastructure: + +o [High priority] + Take a look at Dan's NSE XML output patch and try to commit it. + http://seclists.org/nmap-dev/2011/q2/1230 + +o NSE Version Numbering. + http://seclists.org/nmap-dev/2010/q4/693 + +[Other tasks] +o Propose a better duplicate scanned IPs filtering engine. + + +2) NSE Scripts: + +[Priorities tasks] +o NFS/RPC features: +- add NFS READLINK support to let nfs-ls show symbolic files. + +o Review NSE scripts and libs, and fixing bugs: + - Document all the new NFS procedures. + +[Other tasks] +o NFS/RPC features: +- Add more authentication support: Unix authentication. +- NFSv4 support. +- Add recursion support to nfs-ls.nse + + +== + +MAYBE: + +o Create a new rule "versionrule" which will be used by version + category scripts. + http://seclists.org/nmap-dev/2010/q3/551 + +o NSE debugger. + +o Add more NSE control for long running scripts: one option will be a +boolean expression filter (like: tcpdump) which will change NSE scripts +arguments or behaviour according to previous results, this will be +really useful for big networks. Another option will be a generic NSE +(Lua) script with an easy and readable code that includes expressions or +filters selection to let us change NSE arguments according to previous +results. +Note: this option will be useful on big networks. however for the moment +this is a simple idea and it needs further discussion on the nmap-dev. + +o Privileges dropping for NSE scripts [nmap TODO list]. + +o NSE security review [nmap TODO list]. + + +o Fixing bugs. +- NSE not honoring the source port flag when doing version scan. + http://seclists.org/nmap-dev/2010/q2/576 + + David said that it will not be easy to support setting the source port + http://seclists.org/nmap-dev/2010/q3/331 + + +== + +DONE: + +1) Nmap Scripting Engine Infrastructure: + +o Submitted the "Improve NSE HTTP architecture" proposal + http://seclists.org/nmap-dev/2011/q2/967 + +o Make NSE scripts able to retrieve the interface network + information. + +o LuaFileSystem directory iterator [1] port. +[1] http://keplerproject.github.com/luafilesystem/ + +o New class of scripts which use two new script rules: + - Script Pre-scanning and Script Post-scanning rules: "prerule" and + "postrule". Documented these new phases. + - Update scripts to use these new rules: + dns-zone-transfer now uses "prerule" and "portrule". + +o Update other parts of Nmap book to show the new Script scan phases. + +o Fixing bugs: + - NSE not honoring the Exclude directive bug fixed and committed + as r18467. + +o Let NSE "prerule", "portrule" and "hostrule" scripts to add new +discoverd targets to Nmap. + +o Update scripting.xml to show the new script scan phases. + + +2) NSE Scripts: + +o smtp-vuln-cve2011-1764 script to check Exim DKIM Format String + vulnerability (CVE-2011-1764). + +o Updated and Improved ftp-vsftpd-backdoor to detect the vsFTPd backdoor + (CVE-2011-2523). + +o ftp-vuln-cve2010-4221.nse script to check the ProFTPD Telnet IAC stack + overflow (CVE-2010-4221). + +o smtp-vuln-cve2010-4344 script to check and exploit Exim SMTP Server: + heap overflow (CVE-2010-4344) and privileges escalation (CVE-2010-4345) + +o SMTP library. + +o Rewritten SMTP scripts to use the smtp library: + - smtp-commands + - smtp-open-relay + - smtp-enum-users + +o smtp-vuln-cve2011-1720 script to check for CVE-2011-1720 + +o broadcast-avahi-dos script to check for CVE-2011-1002 + +o NFS/RPC features: + - New script: nfs-ls which combines nfs-dirlist and nfs-acls and try to + emulates some features of the old "ls" unix tool. The script support + NFSv2 and NFSv3. + - Readapted the RPC and NFS library code with a new re-design with new + high level functions. + - Added NFS procedures support: + NFSv2: LOOKUP + NFSv3: FSSTAT, FSINFO, READDIRPLUS, PATHCONF, ACCESS, LOOKUP diff --git a/todo/dmiller.txt b/todo/dmiller.txt new file mode 100644 index 0000000..0216a3c --- /dev/null +++ b/todo/dmiller.txt @@ -0,0 +1,12 @@ +* Make Zenmap unit tests work. Guessing lots don't, since r32569 fixed real code + that matched some unit tests, too. + +* Make sure Ndiff, Zenmap are 2to3 compatible with python -3 + +* Script to check for updated versions of included libs. Have shell for libpcap, + but should convert to python. + +* NSE stuff + * broadcast-srvloc-info - test + * broadcast-rpcbind - write, test + * Consolidate utility functions diff --git a/todo/done.txt b/todo/done.txt new file mode 100644 index 0000000..ccb0a1c --- /dev/null +++ b/todo/done.txt @@ -0,0 +1,3711 @@ +DONE: + +o Change Ncat so that it does SSL certificate trust checking by + default (even without --ssl-verify) and provides a warning and the key + fingerprint if there is no valid trusted chain or the cert is + expired, etc. The warning should happen (to STDERR) even if -v is + not specified. We should add a new option to force Ncat to quit if + cert not valid, and --ssl-verify should become an undocumented alias + for that. [GH#30] + +o Augment the configure script to list unmet dependencies. Currently, configure + works just fine without a C++ compiler installed, but make generates an + error. The configure script should be able to detect this. Also, a list of + features that are/are-not available would be nice at the end of the script, + so folks can see that they've e.g. missed the OpenSSL dependency. + +o Add parallel IPv6 reverse DNS support (right now we use the system + functions). + +o [Ncat] This may sound ridiculous, but I'm starting to think that + Ncat should offer a very simple built-in http server (e.g. for simply + sharing files, etc.) And maybe a simple client too. (Done via --lua-exec and + the httpd.lua script shipped with Ncat) + +o INFRASTRUCTURE: Add IPv6 support to secwiki + - We probably just have to designate a new IPv6 address for it and + add it to Apache config. + +o [INFRASTRUCTURE] Improve our main web server http configuration to + better handle high load situations and DoS attacks. As part of + this, we may have to raise the max client limits. But then there is + a risk of running out of RAM, which can be even worse. So we need + to figure out a good balance. + +o Migrate web.insecure.org to a RHEL-6 derived distro (probably CENTOS + 6, since Linode doesn't currently offer ScientificLinux images). + o Actually, if we can wait until "second half of 2013", we might be + able to jump straight to RHEL 7. And RHEL 5 support looks like it + will go on for many more years for critical/security patches. + o Maybe start with svn server, since we've had reports of our + current one giving people unexpected password prompts. There is a + thread about that at http://seclists.org/nmap-dev/2012/q2/17 + o UPDATE on this - adding read-only rights (rather than no rights) + to the root of the svn repo seems to have solved this problem. + +o Make Windows 8.1 VM with VS 2013 and do more testing of Nmap compilation/running + +o Make and test build on a newer OS X than 10.6 (10.10 was recently released) + +o Adopt an issue tracking system for Nmap and related tools. We + should probably look at our needs and options and then decide on and + either install it on our own infrastructure or use it hosted elsewhere. + - David notes that Trac seems to work well for Tor -- see + https://trac.torproject.org/projects/tor + - One thing which can be nice is being able to interact with the + system through email. Like for bugs people file on the Nmap package + in Debian, I can just reply to the mail and it gets added in the tracker. + - This is now live at http://issues.nmap.org/ + +o Update OpenSSL library to 1.0.1j + +o Our "make uninstall" should uninstall ndiff if it was installed too. + We should probably do it in pretty much the same way we handle + Zenmap (configure.ac, Makefile.in, and ndiff/setup.pl) + +o Web: We should probably distribute RapidSSL intermediate certificate + on SecWiki so it is trusted even if browsers don't have that cert + cached. Here's a page nothing the issue: + https://www.ssllabs.com/ssltest/analyze.html?d=secwiki.org + - We probably need to add an entry in apache conf after + SSLCertificateFile which looks something like: + SSLCertificateChainFile /etc/apache2/rapidssl.pem + +o The XML version of Nmap lists and describes the six port states + recognized by Nmap near the top of the "Port Scanning Basics" + section. That can be seen in the HTML rendering at + https://nmap.org/book/man-port-scanning-basics.html. But in the man + page (nroff) rendering, the list is missing and it just gives the + title: "The six port states recognized by Nmap". UPDATE: Now the + descriptions for each state appear in the man page, but the headings + ("open", etc.) are missing. We should figure out + why, and fix it. + - The bug in the stylesheets means that (From Daniel): "if you have an <indexterm> + element and it's followed by anything other than whitespace+CDATA + (like "</indexterm> foo") then the remaining cdata or element until + the next new element will be nroff-commented so this + <indexterm>blah</indexterm> is ok, but this <indexterm>blah</indexterm>, is not ok because of the commaand this <indexterm>blah</indexterm> <command>nmap -A</command> is bad no matter how much whitespace intervenes" + + +o Fix a segmentation fault in Ncat when scanned with the SSL NSE + scripts. I was able to reproduce this on 2013-09-27 with latest SVN + by running: + Ncat: ncat -v -k --ssl -l localhost + Nmap: ./nmap --script-trace --script '+ssl*' localhost -p 31337 + This was initially reported by Timo Juhani Lindfors on the Debian + bug tracker: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724580 + Henri notes: "I traced the latter back to openssl and opened a + ticket there, which never got any reply... https://rt.openssl.org/Ticket/Display.html?id=2885&user=guest&pass=guest" + +o Investigate how we're ending up with OS fingerprints in nmap-os-db + with attribute names like W0 and W8 when according to the docs they + are only supposed to be W1 - W6 (and plain W). + https://nmap.org/book/osdetect-methods.html#osdetect-w. See also + http://seclists.org/nmap-dev/2013/q4/68. Need to determine how + these are getting into the file (from Nmap itself or our + integration/merge tools) and fix that then remove them from the + file. + +o Integrate latest IPv4 OS detection submissions and corrections + +o We should improve the Windows build process for Ndiff, since it + works differently now that it is modularized. To build the Nmap + 6.45 release, we (as a temporary hack, not in SVN): + - Added 'ndiff' to zenmap/setup.py 'packages' list in + COMMON_SETUP_ARGS + - Created a zenmap/ndiff subdir (empty) and copy ndiff/ndiff.py into zenmap/ before build. + We should find a more elegant solution and check it into SVN. The + fundamental issue is that the ndiff.exe we generate needs to be + able to access the new ndiff.py module. + Also, we need to make sure the -win32.zip Nmap distribution works + properly. + +o [Zenmap] Combine parallel timed-out hops into one node in the + topology view. http://seclists.org/nmap-dev/2012/q1/82 has a patch, + however it doesn't handle the case of two or more consecutive + timeouts. + +o If Nmap uses a "tcpwrapped" port to do fingerprinting on, OS detection + might give false matches/results. Since it doesn't really matter which + open port gets chosen, we should move onto another open port if we + notice "tcpwrapped". + +o Implement an --exclude-ports option. See + http://seclists.org/nmap-dev/2012/q1/275 + +o In an ideal world, Zenmap would not run out of memory and crash. + And we already have an entry for improving Zenmap's memory + consumption. But in the meantime, we should catch the error and + present a more useful error message/explanation so the user + understands the problem. This should reduce the number of + out-of-memory "crash reports" we get too. See + http://seclists.org/nmap-dev/2014/q2/298 + +o Provide an option to send a comment in scan packet data for target + network. Examples: --data-string "Scan conducted by Marc Reis from + SecOps, extension 2147" or --data-string "pH33r my l3eT + s|<iLLz! I'll 0wN UR b0x!" + +o We should probably update our included libpcap. We currently + include version 1.2.1 (we upgraded to that in April 2012) while the + latest version on tcpdump.org is 1.5.3. We make minor changes to + libpcap that we ship, and instructions for upgrading are in + libpcap/NMAP_MODIFICATIONS. + +o Investigate report of Nmap ARP discovery using the wrong target MAC + address field in ARP requests (it is correct in the ethernet frame + itself). See this thread: http://seclists.org/nmap-dev/2011/q3/547 + +o Add randomizer to configure script so that a random ASCII art from + docs/leet-nmap-ascii-art*.txt is printed. I think I'll start naming + them leet-nmap-ascii-art-submittername.txt. + +o Add IPv6 subnet/pattern support like we offer for IPv4. + o OK, we now have the subnet/pattern support, but not the two-stage + model discussed below. So we added a separate task for that. + o Obviously we can't go scanning a /48 in IPv6, but small subnets do + make sense in some cases. For example, the VPS hosting company + Linode assigns only one IPv6 address per user (unless they pay) + and you can find many Linode machines by scanning certain /112's. + And patterns might be useful because people assigned /64's might + still put their machines at ::1, ::2, etc. + o David says: "We need to design a new way to iterate over host + specifications (i.e., different than nexthost). Because the new + host discovery code is sometimes going to want whole netblocks + and sometimes individual hosts. So I'm thinking of a two-stage + model, where the iterator will received (parsed) specifications + like AAAA::1/48, and then it can decide whether to further + iterate that into individual addresses, or pass the block off + to some specialized discovery routine." + + +o Consider implementing RPC scan with ultra_scan or something else. + Right now it is the only program using pos_scan. On the other hand, + I'm not sure TCP RPC scanning is appropriate for ultra_scan. + +o When Ncat is compiled without OpenSSL, we should still accept the + --ssl argument and just give an error message noting that SSL was not + compiled in. This reduces confusion for users + (e.g. http://seclists.org/nmap-dev/2013/q3/579) + +o We should update our OpenSSL Windows binaries from version 1.0.1c to + something newer, like 1.01f + +o Web: figure out why autogeneration of nmap.org/nsedoc/ doesn't seem + to be working. I think we had a cron job which was supposed to be + doing it. + - hb system was still running crontab files from old web vm in its + rc.local. Fixed. + +o Add a W3C XML Schema Definition (XSD) for Nmap XML output. Keeping the DTD + around is also helpful, but XSD is widely supported and could help improve + support for Nmap XML in other tools. + o We're going to discuss this on mailing list before deciding + whether to 1) switch from DTD to XSD, 2) stick with just a DTD, or + 3) try to support both. + +o Update copyright year to 2013 in the Nmap copyright header files + +o Update CHANGELOG for new release + +o New Nmap Release + +o Nping in ICMP mode (default) must not be checking the icmp IDs or + returned packets or something, because if I have two separate 'nping + scanme.nmap.org' running at the same time, each nping sees the replies + from the other nping (as well as its own) and it screws up the timing + stats too. + +o Process Nmap OS service detection submissions + - New fingerprints + corrections + - Last done November 2012: http://seclists.org/nmap-dev/2012/q4/222 + +o Process Nmap IPv6 OS detection submissions + - New fingerprints + corrections + +o Process Nmap IPv4 OS detection submissions + - New fingerprints + corrections + - Last done in November 2012: http://seclists.org/nmap-dev/2012/q4/221 + +o Make Ncat reset the signal handler for SIGPIPE to SIG_DFL before + execing a program with --exec and friends. A "broken pipe" error in + a subprocess should kill the subprocess. Lack of default SIGPIPE + handling is what prevents a trivial Lua chargen script--it loops + forever after the socket disconnects because none of its writes + fail. Cf. http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2009-07-02-python-sigpipe.html. + +o [Nping] In '-q' mode, Nping should keep the line giving the min/max/avg rtt + times. That way people can avoid seeing each individual packet but + still see the stats which are similar to what normal ping gives + them. + +o [Nping] Remove the lines starting with 'Tx time' and 'Rx time' by + default (and of course quieter modes), but leave them for cases at + least one level of -v. + +o Nping/Nmap should probably show ICMP ping sequence values by default + in packet trace mode. This would be nice for Nping since that is + the default ping it sends and is the main way to distinguish the + packets since the IPIDs are the same. + +o Complete migration away from Syn colocated machine + - [Done - actually was already on web] Move submission CGIs to web + - Make sure notification still works + - [Done] Mailman + - [Done] Install mailman software on web, including CGIs + - Migrate mailing lists to web + +o Remove the -q/FAKE_ARGV stuff from Nmap, since I don't think people + use that any more. + +o We should document Ron's sample script + (https://nmap.org/svn/docs/sample-script.nse) in docs/scripting.xml + so that new script writers know about it. + - Decided to remove it instead. Justification: "It is a great idea, + but nobody seems to use it (for example, there were no replies to + usage inquiry here: http://seclists.org/nmap-dev/2012/q4/379). I + think there are two main uses for this script, both of which are + being served by other resources. 1) as a template for new + scripts. Users instead seem to pick a script that is most similar + to the one they want to write and start with that. 2) As a way to + learn more about the format of an NSE script. Users instead seem + to use our documentation + (https://nmap.org/book/nse-script-format.html). So I'm deleting it + for now. But if folks miss it, they're welcome and encouraged to + say so on dev@nmap.org and we could consider putting it back + and/or improving it" + +o Upgrade Mac Mini to Mac OS X 10.8 (Mountain Lion) and test building + as well as testing usage of our normal builds (which we currently + build on 10.6). + +o Make a branch from the 6.20BETA1 release (r30266) for new stable + release, apply any important bugfix patches from the meantime and then + release it after Thanksgiving as new Stable release. + +o [NSE] We may want to consider a better exception handling method -- + one which doesn't require wrapping every I/O line in its own try + function call. David says "Lua has an internal "exception handling" + mechanism based on a function called pcall, which is implemented + with setjmp/longjmp. You can wrap a function call in it and the + function will return there whenever there's an unhandled error. + Something based on that would be better [than the current system], I + think." + - This one is obsolete as the Lua 5.2 now lets you do a Lua yield + across C function calls. + +o Add IPv6 support to Nping, including raw packet mode (hopefully + sharing as much code with Nmap as possible, though Nping's packet code + is a bit different), and also including echo mode server and client + support. + +o Make sure we update everywhere relevant (e.g. refguide, etc.) to + note the addition in Nmap of the Liblinear library for large linear + classification (http://www.csie.ntu.edu.tw/~cjlin/liblinear/). It + uses a three-clause BSD license: + http://www.csie.ntu.edu.tw/~cjlin/liblinear/COPYRIGHT + - David has added it to 3rd-party-licenses.txt + - Fyodor moved it into the refguide + +o Consider including OpenSSL in our Nmap tarball + - Need to check the size, etc. + - OK, we're counting this as done because we took all the Win + binaries out of the tarball and put them in an nmap-mswin32-aux svn + directory which users check out to compile Nmap on Windows, and + OpenSSL is included in this. + +o Update the Nmap CHANGELOG for latest improvements + +o Do an Nmap dev release. Last release was Nmap 6.01 June 22. + o Update Nmap version number and auto-generated files for release. + +o Process latest Nmap OS submissions and corrections (IPv4 and IPv6). + Last done (for IPv4 anyway) in February 2012. + +o Review and consider integrating Tomas Hozza's UNIX-domain socket + support patch for nsock/ncat: http://seclists.org/nmap-dev/2012/q4/24. + +o Improve CPE coverage in OS detection DB from 84% to 90% (see CPE + entry a ways down for more on this). + +o Process latest service detection submissions. They were last done + in February 2012. + +o Integrate Henri's new kqueue/poll nsock-engines support. + +o If it is trivial to add, it would be nice if the "New VA Module + Alert Service" also gave the Author field for NSE scripts so everyone + knows which hero(es) wrote it. + +o Clean up the Nmap repo to remove some bloat we've allowed to creep + in. Should do a more thorough search, but for now here are two + obvious candidates: + - Create publicly readable /nmap-mswin32-aux in svn + - Files not needed for compiling Nmap itself (e.g. only needed for + creating or including in Nmap packages), particularly including the + vcredist files, should be moved to new /nmap-mswin32-aux + - The /nmap-mswin32-aux files won't be included in Nmap tarballs + either + - Add the gtk, glib, etc. Windows dependencies to /nmap-mswin32-aux + so users don't have to all install those in order to compile Zenmap + and make Nmap packages. + - move the nmap-private-dev/mswin32 stuff into /nmap-mswin32-aux + - Update nmap-install.xml for new changes. Such as noting need to + checkout this new directory for building packages, removing the + need to install your own gtk, glib, etc. + - [done] Remove the 5MB of XSL in nping/docs/xsl + +o Update our mswin32/OpenSSL to newest version (previous update was + September 2010 to 1.0.0a). + +o Nmap should have a better way to handle XML script output. + o done: https://nmap.org/book/nse-api.html#nse-structured-output + o We currently just stick the current script output text into an XML tag. + o Daniel Miller is working on an implementation: + https://secwiki.org/w/Nmap/Structured_Script_Output + +o Update more web content in real time (or near real-time, or at least + on an automated basis rather than requiring manual checkin and + update). In particular: + o NSEDoc generation + o [done] SVN dir (https://nmap.org/svn/) should be removed and a redirect + added to https svn server. + o Maybe Nmap book building + o Maybe the generated files in nmap.org/data/ + +o Update web.insecure.org so that rather than requiring us to build + nsedoc on other machines, check it into svn, and then update svn on + web, it is done by a script on web which could be run through cron + (and potentially from a simple svn commit hook) to build them on the + web server directly. + - There are other similar things we might want to automate later, + such as book rebuilding when the XML files are changed. + +o Investigate/fix potential routing-related issue. See emails from + Djalal and others: http://seclists.org/nmap-dev/2012/q3/116, + http://seclists.org/nmap-dev/2012/q3/4, + http://seclists.org/nmap-dev/2012/q2/449 + +o Even without the --osscan-guess flag, Nmap should show the closest + matches (if they pass our threshold) in the XML output. We omit + them from the normal output in large part to encourage people to + submit fingerprints, but that argument doesn't apply so well to XML + output users. Normal output users who really want to see the Nmap + guesses could still use --osscan-guess as before. + +o Change the interface of nmap.ip_send to take an explicit + destination address. It currently extracts the destination from + the packet buffer, which does not have enough information to + reconstruct link-local addresses. See r26621 for a similar change + that was made to Nmap internals. + +o [Zenmap] Install higher-resolution icons (at least 64x64 and maybe + up to 512x512). Here is a screenshot of the current 48x48 icon on + GNOME 3: http://seclists.org/nmap-dev/2012/q2/395. + o Sean did Windows and Linux icons, and David did the Mac + one. + + +o [NPING] At least on my (Fyodor) system, I get errors like "READ-PCAP + killed: Resource temporarily unavailable" with some commands. + Example: + # nping --tcp -p80 -c1 scanme.nmap.org + + Starting Nping 0.5.61TEST4 ( https://nmap.org/nping ) at 2012-02-16 17:52 PST + SENT (0.3307s) TCP 192.168.0.5:42005 > 74.207.244.221:80 S ttl=64 id=23109 iplen=40 seq=1015357225 win=1480 + RCVD (0.3524s) TCP 74.207.244.221:80 > 192.168.0.5:42005 SA ttl=51 id=0 iplen=44 seq=3197025741 win=14600 <mss 1460> + nping_event_handler(): READ-PCAP killed: Resource temporarily unavailable + nping_event_handler(): TIMER killed: Resource temporarily unavailable + [...] + +o [NPING] Nping should probably give you an error or warning when you + do: "nping -p80 google.com" since it is ignoring the port specifier. + The user probably wants to add --tcp. + +o Investigate why http pipelining so often doesn't work in NSE + scripts, and often NSE ends up reverting to one request at a time. + Scripts may not be using it correctly, and also we wish it were more + transparent and there wasn't this big API divide between pipeline + and non-pipeline. We just want it send requests as fast as it can, + and get a callback when there's a response. Maybe the http library + buffers them, or pipelines them, or blocks the http.get call until + there's more room. It just seems to always degenerate to 1 request + at a time. For example: + sudo nmap --script=http-enum bamsoftware.com -p80 -d2 + quickly (within a few seconds) gives: + NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument) + NSE: Total number of pipelined requests: 2081 + NSE: Number of requests allowed by pipeline: 100 + NSE: Received only 41 of 100 expected responses. + Decreasing max pipelined requests to 41. + NSE: Received only 1 of 41 expected responses. + Decreasing max pipelined requests to 1. + 100 may a wildly high number of requests to attempt to pipeline. + And then something else probably goes wrong after it decides 41 is okay. + - Related: Does caching work with pipeleined requests? We should + make sure it does. + [ OK, the main part of this todo item is done. Though there is a + patch pending from Piotr which changes how pipelining works that + is worth considering. We did fix the underlying pipelining bug, but + (just as with most browsers), it isn't enabled by default. Also, it + doesn't support caching. See + http://seclists.org/nmap-dev/2012/q3/616. ] + +o Make Nmap from a clean start (e.g. after make clean or whatever, so + it compiles everything) and research all the compile warnings to see + which ones can be fixed/removed. Of course caution is needed to + make sure we don't cause problems. For example, an unused variable + on one platform might not be unused on another, so we can't just + remove it. May have to surround it by ifdefs though. + +o Solve "spurious closed port detection" issue discovered by David: + http://seclists.org/nmap-dev/2012/q1/62 . So we need to figure out + what is going on here and then how to fix it. Note that this + doesn't seem to happen when you do ICMP host discovery first (-PE), + so it probably relates to the ACK packet that Nmap sends to port 80 + on the target by default. + +o Add real headers for more protocol types in -6 -sO scan. Dario + Ciccarone provided some packet captures for + 0x00: hop-by-hop + 0x2b: routing + 0x2c: fragment + 0x3c: destination + (http://seclists.org/nmap-dev/2011/q2/1003). We also have examples + of crafting some of these in FPEngine.cc. [Sean and David] + + +o Investigate increasing FD_SETSIZE on Windows to allow us to + multiplex more sockets. See Henri's email: + http://seclists.org/nmap-dev/2012/q1/267 + [James Rogers did some investigative work on this in July 2012, but + we weren't able to find a great solution. Maybe we should + investigate this more in the future, and also investigate other + Windows socket APIs such as completion ports. ] + +o Finish sv-tidy - a program to canonicalize and tidy nmap-service-probes. + o Check for the same reference (like $1) being used in unrelated fields + (where related fields are the pairs (p, cpe:), (v, cpe:), (i, cpe:), + (o, cpe:)). + For example if we have v/$1/ h/$1/ it is a bug. + o Check a list of common product names that should only appear in p//, + not in i//. We still have entries that are like this: + p/Foobar 2000 ADSL router/ i/micro_httpd web server/ + that should rather be written this way: + p/micro_httpd/ i/Foobar 2000 ADSL router/ + o [Done] Check for e.g. i/French/ without :fr in cpe:/a, and vice versa. + [Sean and David?] + +o Remove Nmap's --log-errors feature and make its behavior the + default. A few notes: + - Nmap should just ignore --log-errors if it sees it + - Remember to remove it from the documentation + +o We should probably sort script output (for port output and host + output) by script name or something so that it comes in a + deterministic order. If the same three scripts produce output in + two different scans, they should be listed in the same order. Right + now the order can vary, at least for host output. + [Sean] + +o Add a function such as --disable-arp-ping which prevents hosts from + being automatically detected as 'up' just because they responded to + ARP. Instead, Nmap will actually send the requested host discovery + probes (ICMP ping packets, SYN packets, etc.) and only mark the host + as up if it responds on an IP level. This is how machines are + already treated if they're not on the local network (e.g. if ARP + discovery is unavailable). This technique is a bit slower and more + likely to miss hosts (e.g. if they're heavily firewalled) than ARP + discovery, but the option is needed to handle local networks which use + proxy ARP, which would otherwise cause all IPs to appear to be up. + +o We should add fields to the service submitter [James is working on this] + (http://insecure.org/cgi-bin/submit.cgi?new-service) for the + application name and version. + o We also need to ensure all fields of /cgi-bin/submit.cgi have + proper escapting to prevent possible reflected XSS attacks + reported by Maxim Rupp (@mmrupp). The risk is low, if any, since + we don't give authentication cookies for bad guys to steal, but is + still better to properly escape. + o If we get a chance, would be interesting to run our XSS-testing + NSE scripts against this and see if they locate the problems. + o Also, need to change the font family in there from "Lucida Grand" + to "Lucida Grande"? Just a typo. And fix "WIkipedai". We should + just spell-check all the output + +o Make Nmap 6.01 release containing (among possibly other little +fixes) + - Python upgrade + - [done] Zenmap 10.7 hang fix (done in trunk) + - [done] Zenmap crash when filtering hosts (done in trunk) + - [done] get_srcaddr fix (done in trunk) + +o Upgrade Python on build machines to try and resolve Python 2.7 + security warning (it doesn't affect us, but can worry users). See + this thread: http://seclists.org/nmap-dev/2012/q2/621 + +o Fix get_srcaddr error happening on Windows XP + +o [Web] Add a page with the Nmap related videos we do have already + - We have a page on Secwiki now: https://secwiki.org/w/Nmap/Presentations + +o Zenmap hang on OS X 10.7 + +o For many years, the Nmap man page and online documentation has had + an "Inappropriate Usage" section which notes that "Nmap should never + be installed with special privileges (e.g. suid root) for security + reasons". And of course Nmap's official installer would never + install Nmap that way. While one would thinks that would be enough, + we might want to go even further and have Nmap detect when it is run + suid and print a security warning. + +o Prepare release notes, web page, etc. + +o Do private beta release + +o Make the release + +o In Nmap XML output, osclass (OS Classification) tags should be + children of osmatch (the human readable OS name line) rather than + having Nmap deduplicate all the osclasses and put them in as + siblings. But this change might break some systems which utilize + Nmap XML output, so, along with this change, we need to introduce an + option such as --deprecated-osclass-xml to return the old behavior. + That option only needs to be documented in the CHANGELOG entry + referring to this change, and it should note that we're likely to + remove this option in a year or two. + +o Right now, when an IPv4 or IPv6 address seems bogus (such as 1.2.3 + or 2001::0 in IPv4 mode), we give a fatal error and abort the scan. + But since that might just be one bad target in a long list of hosts to + be scanned, it is probably better to just print a warning and + continue. Some sort of warning or host element should be included in + the XML to explain what happened too. This should also happen if + we're unable to resolve a DNS name. + +o In sv-tidy, check that used references start at 1 and are + contiguous. If $1 and $3 are used but not $2, it's probably a bug. + Maybe you can even find out how many there should be by inspecting + the regular expression. + +o Raw scans from Mac OS X seems not to retrieve the MAC address or do + ARP ping, except when scanning the router on an interface. For + example, scanning 192.168.0.1-5 sends ARP pings to 192.168.0.1, but + the normal four-probe combination to the other addresses. The "MAC + address:" line appears in the output for .1 but not for the others. + +o To avoid Nmap memory usage bloat, find a way for NSE scripts to + store information about a host which expires after Nmap is done + scanning that host (e.g. when the hostgroup containing that host is + finished). Right now scripts store such information in the registry + and it persists forever. For example, a web spidering + script/library could store information about the web structure and + even page contents so that other scripts can use that information + without spidering the target again, but ensuring that the memory + will be freed after the hostgroup finishes so there is room to store + the web information for the next group of systems. One idea would + be to make a host.registry member which contains a registry specific + to a specific target. Scripts could store temporary information + there, but still use the global registry for information which must + persist (e.g. to be used by postrules, etc.) + +o Add CPE support to IPv6 OS detection + +o Use BPF libpcap logic on Solaris 11, otherwise packet capture doesn't + work at all. http://seclists.org/nmap-dev/2012/q1/613 + +o [NSE] host.os should not just be a list of strings which can contain + human-readible strings and/or CPE info. It should probably be list + of host.os tables which can contain: + host.os[].name <-- human readible name + host.os[].class[].vendor + host.os[].class[].osfamily + host.os[].class[].osgen + host.os[].class[].devicetype + host.os[].class[].cpe[] <-- array of cpe:/ strings + So host.os[1].class[1].cpe[1] is the first CPE entry for the first + classification of the first OS match for the target system. + The host.os entry docs/scripting.xml would have to be updated too. + +o We should probably go through the nmap-os-db (and IPv6 version) + entries and, where the fingerprint line specifies a service pack + number (or even two of them), ensure that we have sp-qualified CPE + entries like "cpe:/o:microsoft:windows_xp::sp2". Right now we + sometimes include the qualification, and sometimes not. + o This is best done with cpeify-os.py, if possible. + +o Zenmap no longer ads the installed module directory to its module + search path because some distributors first install in a world + writeable directory (like /tmp) and then put those files into their + packages which they distribute to users. But this change can lead + to Zenmap not working for users who install in nonsystem areas like + their home directory (e.g. --prefix /home/fyodor) unless they have + their PYTHONPATH set to find them. We should implement a solution, + such as making sure Zenmap catches the missing modules error and + suggest that the user set their PYTHONPATH or something. + +o Scans from Mac OS X tend to use raw IP packets rather than ethernet + frames even on the local network because Dnet does not seem to be + retrieving the routing table properly -- so the LAN doesn't even + show up in --iflist. Patrik can reproduce this on all 3 of his + MACs (OS X versions 10.7.3). Comparing the code in DNet route-bsd.c + to Apple's own routing table code discovered by Patrik suggests that + the Dnet code may be incorrect. + +o ssl-google-cert-catalog should not require that the user specify + ssl-cert in order to run. Instead, they should probably both call a + library which obtains the certificate (and caches it so that it + doesn't happen twice if both scripts are run). In general, we want + to avoid having any scripts tell the user "this script only works if + you specify this other script too". If we really find we need that + functionality, we should add a "strong dependencies" feature so that + scripts can tell Nmap what other scripts they require. + [Patrik did this by adding an ssl cert library] + +o Our targets-ipv6-multicast-slaac.nse should probably send the router + advertisements with low priority to reduce the chances of any + negative impacts on clients, if we're not doing that already. See + http://lists.si6networks.com/pipermail/ipv6hackers/2012-March/000503.html. + - Actually, I think we already do this. Marking as done. + +o Deal with the issue of timeouts happening too soon due to global + congestion control in some cases. For example, if Nmap sends host + discovery probes to two hosts, and one comes back extremely quickly, + it can cause the global congestion control to use a very low timeout + and cause the 2nd host (which doesn't have any host-based congestion + control values yet) to timeout arguably too quickly. We should look + at potential algorithm changes to improve this. + David: I think I was wrong about the cause of this. Even when + replies come back very quickly, the timeout is by default limited + to 100000 microseconds, much higher than the straightforward + calculation would give. What I think is really happening is that + select is not working reliably on this platform (Solaris 10 x86). + In the loop in read_arp_reply_pcap, pcap_select returns 1, then a + pcap_next is done. Then pcap_select returns 0, but if I insert + another pcap_next after that, the pcap_next finds another packet + without blocking (the first time, anyway; after that it blocks). + +o Create CHANGELOG + +o Make stable release candidate branch + +o Make at least one more test release from the candidate branch + +o Write and send GSoC 2011 results email + +o Document the nsearg format changes made by Paulino (how you can + preface an argument with a script to make it more specific, or make it + general to apply to multiple scripts) + o Rough drafts: + o nmap-exp/calderon/refguide.xml + o nmap-exp/calderon/scripting.xml + o Relates to: + o We should probably modify stdnse.get_script_args so that it first + checks [scriptname].[argname] and then (if that fails) looks for + [argname] by itself. This way people who are only running one + script or who want to use the same value for multiple scripts that + take the same argument can just give [argname]. But those who want + an argument to only apply to a specific script can give + [scriptname].[argname]. + +o Make the nmap.header.tmpl wording a little more generic so it more + clearly applies to Ncat, Zenmap, Nping, etc. Then use + templatereplace.pl to apply those changes to the code. [Fyodor] + +o Change Nmap copyright dates (in the file headers, etc.) from 2011 to + 2012. + +o Get RPM staticly linking to libsvn (rather than dynamic linking) so + that it isn't a requirement for installing the RPM. + - We decided to just make nmap-update its own separate RPM so that + it can dynamically link to libsvn without forcing that dependency on + the whole nmap RPM package. + - since the libsvn-devel package apparently only installs dynamic + libs, we'll probably have to install it ourselves on the CentOS + build machines. + +o Fix "BOGUS! Can't parse supposed IP packet" in packet trace of IPv6 + packets. + +o Integrate latest IPv6 OS detection fingerprint submissions + - In addition to the submission CGI submissions, some were emailed to Fyodor and David on Oct 21 + +o Integrate new service fingerprint submissions (we have more than + 2,531 submissions in two files since 11/30/10) + +o Integrate new OS detection submissions (1,893 since 6/22/11) + +o Add options in configure script for users to specify where to find + subversion lib/include dirs (like we do with our other library + dependencies). See this mail: + http://seclists.org/nmap-dev/2012/q1/37 + -- David added --with-apr and --with-subversion + +o We need to fix the svn server so that Nmap committers can make + branches from /nmap to /nmap-exp. We may need to add some sort of + OPTIONS permission to the root directory or something, because + they're getting errors like: + $ svn cp https://svn.nmap.org/nmap https://svn.nmap.org/nmap-exp/branchname + svn: Server sent unexpected return value (403 Forbidden) in response + to OPTIONS request for 'https://svn.nmap.org' + - Patrick also reported some other funny business related to svn + mv'ing directories in email to Fyodor and David. + +o Give CPE visibility to NSE. + - done by Henri + +o Document the new IPv6 OS detection novelty system in os-detection.xml + +o Do more thinking/researching/investigating the way our machine + learning IPv6 OS detection system decides whether a match is perfect + and/or how close the match is. Maybe our current system works well + enough, we'll need to watch how it performs as we increase the DB + size and collect/integrate more signatures. The goal is to: + o Producing fewer way-off matches since it would have a way (like our + current system) to decide how close the match really is + o Doing a better job about printing fingerprints for matches with + aren't close enough + +o Improve the "run Zenmap as root" menu item to work on distributions + without su-to-root. We might even want to improve Zenmap so that it + itself does not have to run as root, and just executes Nmap that + way. Rather than not showing Zenmap as root on the Menu of + non-working systems, it might be better to have it but let it give + an error message (and then, perhaps, run as nonroot) so that users + of those distributions are more likely to contribute a fix. We also + might want to look at how the distributions themselves package Zenmap. + +o Consider changing Nsock so that it is able to take advantage of more + modern interfaces to dealing with large sockets, rather than just + select. Perhaps we should look at poll(), Windows completion ports, + and some of the advanced Linux APIs. Select() limits us to + descriptors no higher than FD_SETSIZE, and it may not performa all + that well. We should do some benchmarking and decide on the + interface to use for each platform. May want to take a look at + libevent (http://www.monkey.org/~provos/libevent/) for inspiration. + The libevent home page has some interesting benchmark graphs too. + [Josh implemented poll as a SoC student, but it had problems with + Nsock's architecture. O(1) lookups were becoming O(n) because of + the nature of the data structures. It was slower in his benchmarks. + Nsock would have change from a model of "loop over the event list, + and check to see if the fd for each event is set," to one of "loop + over the fd list, and see if there is a corresponding event for + each. It is the "see if the fd is set" operation that's O(1) with + select (it's FD_ISSET) and O(n) with poll (it's a traversal of a + linked list).] + o Henri added nsock-engines + +o Consider an update feed system for Nmap which let's people obtain + the latest Nmap data files, such as NSE scripts/libs, nmap-os-db, + nmap-service-probes, etc. + o Note that some scripts require updated compiled libraries. We + will need some sort of compatability system. + o One approach is "svn up". Note that Metasploit uses that approach + even for Windows by shipping .svn directories and an svn executable + with the Windows installer. In taht case we might need to have a + separate branch for each release that gets updated version/OS + databases and scripts. + o Another approach is a special feed system as is used by Nessus and + OpenVAS. OpenVAS uses a script wrapper around rsync, or an HTTP + download if that fails. + o Colin's analysis of different methods: + http://seclists.org/nmap-dev/2011/q2/821 + +o [NSE] Consider using .idl files rather than manually coding all the + MSRPC stuff. The current idea, if we do this, is to have an + application in nmap-private-dev which converts .idl files to LUA + code for nmap/nselib. Consider adapting the pidl utility from Samba. + o Drazen did some work on this during SoC. + https://svn.nmap.org/nmap-exp/drazen/nmap-msrpc could get someone + started. + o We moved this out of the active section of the TODO because, while + it is still a good idea and we'd welcome the change if someone wants + to take it on, it isn't something that we are likely to make + progress on unless someone steps forward. + +o Implement a solution for people who want NIST CPE OS detection + results (we'll save version detection for a 2nd phase). Notes: + David report on CPE for OS Detection: + http://seclists.org/nmap-dev/2010/q3/278 + David report on CPE for version detection: + http://seclists.org/nmap-dev/2010/q3/303 + Nessus has described their integration of CPE: + http://blog.tenablesecurity.com/2010/05/common-platform-enumeration-cpe-with-nessus.html. + Older messages about it: + http://seclists.org/nmap-dev/2008/q4/627 + http://seclists.org/nmap-dev/2010/q2/788 + +o [NSE] HTTP spidering library/script + +o We should probably modify stdnse.get_script_args so that it first + checks [scriptname].[argname] and then (if that fails) looks for + [argname] by itself. This way people who are only running one + script or who want to use the same value for multiple scripts that + take the same argument can just give [argname]. But those who want + an argument to only apply to a specific script can give + [scriptname].[argname]. + o The code is in place now, we just need to document the feature. + +o Script review + o Martin Swende patch to force script run + http://seclists.org/nmap-dev/2010/q4/567 + o applied + o irc-info patch. http://seclists.org/nmap-dev/2011/q2/289. + o applied + o http-slowloris. http://seclists.org/nmap-dev/2011/q1/916. + o Had some issues--never got to a state ready for integration + o http-phpself-xss + - Would need to be rewritten to use newer spider.lua. Added an item + to incoming section of Nmap Script Ideas secwiki page. + +o Make new SecTools.Org site with the 2010 survey results. + +o Collect many more IPv6 OS detection training samples from users + - Can start with nmap-dev, but will probably have to do an Nmap + release too. + +o Integrate more NSE scripts, I think our review queue is getting + pretty long. + +o Decide what to do with Henri's nsock-engines branch + (/nmap-exp/henri/nsock-engines). + +o finish making nmap-update part of the nmap windows compile-time + infrastructure + o See if we can build just one project within a solution, rather + than having special "with nmap-update" configuration. + +o Add homedir support to Nmap for the updater + +o Fix expiration date parsing on Nmap Windows for the updater + +o Updater: Make a missing nmap-update.conf nonfatal (perhaps doesn't + even need to mention it). + +o Updater: Clean up the output messages (e.g. only print what user needs to see + unless debugging is specified) + +o [Nping] The --safe-payloads option should be default (though we + should keep it for backward compatability). We could then introduce + --include-payloads for cases where they are desired. + +o A program to canonicalize and tidy nmap-service-probes. + o Order of fields: m p v i d o h cpe:/a cpe:/h cpe:/o. + o Check for duplicate templates (except cpe:). + o Check for unknown templates. + o Canonicalize delimiters (use // first, otherwise try in order + | % = @ #). + o Retain line breaks and comments. + +o Document IPv6 OS detection at https://nmap.org/book/osdetect.html + +o Script review: + - New scripts from Paulino: http-wordpress-brute and http-joomla-brute, + http-majordomo2-dir-traversal.nse, http-trace, http-waf-detect + - http-methods patch. http://seclists.org/nmap-dev/2011/q1/936. + - quake3-info. http://seclists.org/nmap-dev/2011/q2/172. + - smb-os-discovery additional + information. http://seclists.org/nmap-dev/2011/q2/276. + - Outlook web + address. http://seclists.org/nmap-dev/2011/q2/296. [probably not + going to merge to Nmap trunk at this point, though it is good that + the script is available for d/l for those who need it. ] + +o Fix reported (by many people) crash when trying to launch Zenmap on + Mac OS X 10.7 (Lion). + +o Unless we get good arguments for keeping it, we should remove Mac OS + X PowerPC support from our binaries. Apple stopped selling PowerPC + machines in 2006 and they stopped making new OS releases available + for PowerPC as of Snow Leopard (10.6) in August 2009. See this + thread: http://seclists.org/nmap-dev/2011/q3/430 + +o Improvements to the Nmap multicast IPv6 host discovery scripts + - Note that we hope to move them into core Nmap at some point, but + would be good to improve them for now. + - They should probably print the discovered IPv6 addresses, otherwise + they don't actually give the user any information (despite doing + their work) unless you give the newtargets script arg. This would + be similar to the current behavior of broadcast-ping. + - It might be nice if they gave the target MAC address and vendor + when printing the discovered IPv6 information too. Daniel Miller + wrote an initial patch for this (though we need to make sure it can + handle (e.g. doesn't crash for) non-ethernet + devices:http://seclists.org/nmap-dev/2011/q3/862. Our broadcast-ping script + currently prints MAC addresses. + - It is great that the scripts properly use a specific device when + given the Nmap -e option, but they shouldn't require this. They + should do something smart if no specific device name is given. + Examples include performing on all compatable devices or trying to + pick the best device. The all-devices appraoch may be the best, + IMHO. That is how our broadcast-ping script works now. + +o Add anti-spam defenses to secwiki.com to stop the current onslaught + of spam. An extention like ConfirmEdit + (http://www.mediawiki.org/wiki/Extension:ConfirmEdit) may be a good choice. + +o Collect a bunch of IPv6 OS detection signatures from users, + integrate them, and then when we have enough, re-enable OS detection + results. + +o IPv6 OS detection working (when run on) Solaris and AIX + - AIX 6.1 - iSeries / System p + - AIX 7.1 - iSeries / System p + - Solaris 10 - SPARC + +o We should consider splitting a 'brute' category out of the 'auth' + category now that we have so many brute force scripts. I suppose + users can already do "--script *-brute", but having its own category + might still be nice. + +o IPv6 OS detection merge + o [DONE] Initial branch working (nmap-exp/luis/nmap-os6) + o [DONE] Implement the 2 remaining probes + o [DONE] Disable the printing of matches (except maybe with debug on). We + want more training examples first so that results are better. + o [DONE] Merge to /nmap + +o Document Nmap CPE support in appropriate places (candidates: + refguide, os detection book chapter, version detection book chapter, + output book chapter). + +o Finish CPE support code + - Escape certain values that can be inserted into cpe string through + substitution, like cpe:/a:apache:httpd:$1 where $1 contains a + colon. + +o Add advanced IPv6 host discovery features + o Initially done using NSE by adding these scripts: + targets-ipv6-multicast-slaac, targets-ipv6-multicast-invalid-dst, and + targets-ipv6-multicast-echo + +o Initial IPv6 OS detection system (may not make it into stable + though, but we want to at least have it working in a branch first.) + - OK, it is working in nmap-exp/luis/nmap-os6 + +o Investigate a probe/response matching problem reported by QA Cafe + Matthew Stickney and Joe McEachern of QA Cafe. See this thread: + http://seclists.org/nmap-dev/2011/q3/227 + +o When our winpcap installer is run in silent mode + (e.g. "winpcap-nmap-4.12.exe /S"), it seems to execute nmap.exe if + that binary exists in the same directory. This leads to a cmd.exe + window briefly poping up as Nmap displays its console help output. + Moving the Winpcap installer into its own subdir and running it from + there seems to fix this (because it then can't find nmap.exe to + run), but it would be better to determine why this is happening in + the first place and fix it. + +o Obtain Nmap data directory information from nmaprc at runtime rather than + compiled in -- among other advantages this is needed to make + relocateable rpm. [actually we ended up doing this without needing + nmaprc for now] + +o Summer of Code feature creeper: + o Ncat should probably have an --append-output option like Nmap does + so that we can use -o without clobbering existing file. This would + at least be useful for chat.nmap.org. + o Change Zenmap bug reporter so that instead of an automatic + submission system, we print a stack trace and request that the user + send a bug report to nmap-dev. + +o [Ncat] Solve a crash that only happens on Windows when connecting + with --ssl-verify and -vvv, for example + ncat --ssl-verify -vvv www.amazon.com 443 + The crash happens in the function verify_callback, when the function + X509_NAME_print_ex_fp is called. Just commenting those two calls + avoids the problem. By trying different combinations of debug print + statements, I once got the message + OPENSSL_Uplink(10109000,08): no OPENSSL_Applink + This refers to a Windows dynamic linking issue: + http://www.openssl.org/support/faq.html#PROG2 + However I tried both including <openssl/applink.c> and changing the + linker mode to /MD, and neither changed the behavior. + Changing the flags from XN_FLAG_ONELINE to 0 seems to make the + problem go away. + +o Integrate new OS detection submissions (We have about 1,700 + submissions since 11/30/10) + +o Nmap should defer address parsing in arguments until it has read + through all the args. Otherwise you get an error if you use like -S + with an IPv6 address before you put -6 in the command line. You get + a similar problem if you do "-A -6" (but "-6 -A works properly). + This is a possible feature creeper task. + +o Ncat chat (at least in ssl mode) no longer gives the banner greeting + when I connect. This worked in r23918, but not in r24185, which is + the one running on chat.nmap.org as of 6/20/11. Verify by running + "ncat --ssl -v chat.nmap.org" + +o IPv6 Neighbor Discovery-based host discovery (analog to ARP scan). + +o Investigate and document how easy it is to drop Ncat.exe by itself + on other systems and have it work. We should also look into the + dependencies of Nmap and Zenmap. It may be instructive to look at + "Portable Firefox" + (http://portableapps.com/apps/internet/firefox_portable) which is + built using open source technology from portableapps.com, or look at + "The Network Toolkit" by Cace + (http://www.cacetech.com/products/network_toolkit.html). For Nmap + and Nping, we may want to improve our Winpcap to load as a DLL + without requiring installation. There is a separate TODO item for that. + +o The SCRIPT_NAME variable should not include the ".nse" in script + names. Currently, it omits that for scripts in the DB, but includes + it for scripts you specify based on their filename. See: + http://seclists.org/nmap-dev/2011/q2/481 + +o If possible, Ncat, in listen mode, should probably listen on the system's + IPv6 interfaces as well as IPv4. This is what servers like apache + and ssh do by default. It might now be possible to listen on IPv6 + by running a second ncat with -6, but that doesn't really work for + broker and chat modes because you want the IPv6 users to be able to + talk to IPv4 and vice versa. + - This was partially implemented, but still doesn't seem to work in + --chat mode. Can test against chat.nmap.org + - Done. Tested on scanme with David & Fyodor on 7/18/11. + +o Right before the release, we could build Ncat portable and post it + on https://nmap.org/ncat/. + - Actually we did that for 5.59BETA1, which is good enough for now. + +o CHANGELOG updates [Fyodor] + +o [Ncat] Add new certificate bundle (ca-bundle.crt) since the current + one is out of date. See http://seclists.org/nmap-dev/2011/q2/641. + +o Move these prerule/postrule script ideas to secwiki script idea page + if appropriate (with a bit more details): + o AS Number to IP ranges: http://seclists.org/nmap-dev/2010/q2/101 + In progress. + o DNS service discovery (Bonjour): http://en.wikipedia.org/wiki/Bonjour_%28software%29 + Present as dns-service-discovery.nse. + o Netbios Name Service + Already present as broadcast-netbios-master-browser.nse? + o DHCP broadcast requests + Present as dhcp-discover.nse. + o Postrules could be created which give final reports/statistics or + other useful output. Like a reverse-index, which shows all the open + port numbers individually and the hosts which had that port open + (e.g. so you can see all the ssh servers at once, etc.) + Admittedly you can do that pretty easy with Zenmap instead. + Have a few of these: ssh-hostkey and upcoming creds-summary. + o We could have a prerule sniffer script which uses pcap to sniff + traffic for some short configurable amount of time and then adds the + discovered hosts to the target list. + Already present as targets-sniffer.nse. + o We could have a script which takes traceroute results and adds them to the target list. + Already present as targets-traceroute.nse. + +o [NSE] Add these ideas to secwiki script ideas page if appropriate + (with a bit more details): + o Windows system logs (like sysinternals' psloglist) + o Services (like sysinternals' psservice) + o A script (or modification to smb-check-vulns) to + detect this MSRPC vulnerability: + http://seclists.org/fulldisclosure/2010/Aug/122 + o BasicHTML/XML parser library? For example, Sven Klemm wrote a script + which uses libxml2: http://seclists.org/nmap-dev/2008/q3/0462.html. + And here is one by Duart Silva using Expat: + http://seclists.org/nmap-dev/2009/q3/1093. + o Add detection of duplicate machines via IP.ID technique. + Maybe I should use uptime timestamps too. Oh, and MAC addresses + too. Our SSH host key script is useful for this as well. + +o Look into iplog ( http://ojnk.sourceforge.net/ ) -z option which is + supposed to fool OS detection. + o The software is no longer maintained, so we're not going to worry + about it. The page says: "I am through working on this project. I + will not be making any updates, and I will ignore just about all + email about it. If anybody wants to take it over (for whatever + reason), let me know" + +o [NSE] Consider how we compare to the Nessus Web Application Attack + scripts + (http://blog.tenablesecurity.com/2009/06/enhanced-web-application-attacks-added-to-nessus.html). + [Joao making a list of web scripts which we might find useful, + Fyodor asking HD moore for permission to use http enum dir list] + +o [NSE] HTTP persistant connections/keepalive? May make + spidering/grinding/auth cracking more efficient + +o [NSE] HTTP Pipelining support? May make spidering/grinding/auth + cracking more efficient + +o [NSE] HTTP Cookie suppport? Might be useful for spidering sites which use it + for authentication/authorization/personalization. + +o [NSE] URL grinder checks for existence of applications in common/default + paths. Scanning http paths to see if they exist is in some ways + similar to scanning to see which ports are open. + o Our http-enum does this. + +o Investigate why and whether we need mswin32/pcap-include/pcap-int.h. + This file is not included in the official WinPcap 4.1.1 developers' + pack + (http://www.winpcap.org/install/bin/WpdPack_4_1_1.zip). Presumably + it covers internal functions and structures which we aren't really + supposed to access it. If we can get rid of it, that would be + great. If we need it, we should probably upgrade to the + 4.1.1. version (presumably from the Winpcap source code + distribution). Right now it is included in tcpip.h, + nsock/src/nsock_pcap.h, and nping/common_modified.cc: o David looked + into it. He says it isn't distributed with the WinPcap developer's + pack. You have to extract it from the source file. He updated to the + 4.1.1 version. He says The entire reason we need it is so we can + peek at the definition of struct pcap, so we can access the + pcap.adapter member on Windows. In order to pass it to + PacketSetReadTimeout. Usually struct pcap is an opaque type and you + are only supposed to access it through a pcap_t *. Unfortunately I + don't think there's an easy way to manipulate the timeouts in + WInPcap like we do on other platforms. You can specify a timeout + when you do pcap_open, but we like to set a timeout on every + read. So we sort of sneak in and call PacketSetReadTimeout. In the + code there's even a comment: "BUGBUG: This is cheating." libdnet + also uses the Packet* functions, but in a more innocuous + way. It doesn't access them through a struct pcap, so it + doesn't need pcap-int.h. David tried testing whether this makes + any signficiant difference--to see if we could just remove the + PcapSetReadTimeout()--but that didn't work out. + - We're not going to worry about this for now since it isn't + important enough to pester the pcap people about, and they don't + seem to be changing their internal structure anyway. And if they + do, we can get the new pcap-int.h. + +o Further brainstorm and consider implementing more prerule/postrule + scripts: + o [Implemented] dns-zone-transfer + o [Implemented, but a joke] http-california-plates + +o Investigate this interface-matching problem on Windows: + http://seclists.org/nmap-dev/2011/q1/52. It is related to the + libdnet changes we made to allow choosing the correct physical + interface when teamed interfaces share the same MAC. + I think this is solved with the rewritten libdnet code (that uses + GetAdaptersAddresses) in my nmap-ipv6 branch. --David + +o [Ncat] When in connection brokering or chat mode with ssl support + enabled, if one client connects and doesn't complete ssl negotiation, + it hangs any other connections while that first is active. One way to + reproduce: + Run SSL chat server like: /usr/local/bin/ncat --ssl -l --chat + Window #1: Connect without ssl: ncat -v chatserverip + Window #2: Try to connect with SSL: ncat -v --ssl chatserverip + Window #2 will not work while #1 is active. If you quit #1, #2 + should work again. + +o IPv6 todo. + - Protocol scan (-sO). + +o [Ncat] Find out what RDP port forwarding apparently doesn't work on + Windows. http://seclists.org/nmap-dev/2011/q1/86 + +o Add raw packet IPv6 support, initially for SYN scan + o After that can add UDP scan, and sometime OS detection (David did + some research on what IPv6 OS detection might require). + +o When I (Fyodor) scan scanme.nmap.org with the command "nmap -sC -p80 +-Pn -n scanme.nmap.org", I get a blank http-favicon line like: + 80/tcp open http + |_http-title: Go ahead and ScanMe! + |_http-favicon: + But if I use "--script http-favicon" instead of -sC, it works fine. + +o UDP scanning with IP options causes "Received short ICMP packet" on + receipt. http://seclists.org/nmap-dev/2011/q1/82 + + +o [Zenmap] Make formerly open ports that are now closed or filtered + disappear from the "Ports / Hosts" tab. This appears to be related + to ignored states; if in the second scan I use -d2 so all ports are + included in the output, the interface is updated correctly. + http://seclists.org/nmap-dev/2010/q4/659 + +o [Zenmap] When a target is unresponsive (and its distance isn't + known), put it at the next furthest ring from the known traceroute + hosts (with a dashed line), instead of putting it at the first ring. + See http://seclists.org/nmap-dev/2011/q1/834. + +o Rewrite the portreasons code not to use parallel arrays + (reason_text, reason_pl_text) and not to require special alignment + between the enum codes and (for example) ICMP types. Instead define + one structure containing all relevant information about a reason, + and define helper functions to map ICMP types to reason codes. In + particular, code like this needs to go away: current_reason = + ping->type + ER_ICMPTYPE_MOD; if (current_reason == ER_DESTUNREACH) + current_reason = ping->code + ER_ICMPCODE_MOD; + +o Fix memory consumption problem in drda-info (see + http://seclists.org/nmap-dev/2011/q2/451) + - Fixed (turned out to affect a lot of scripts) + +o Script dispensation + - sip-enum-users and + sip-brute. http://seclists.org/nmap-dev/2011/q2/56. + o Merged + - xmpp. http://seclists.org/nmap-dev/2011/q2/239. + o Merged + +o Script review/disposition: + - Merged: DNSSEC enumeration. http://seclists.org/nmap-dev/2011/q1/406. + - Merged: quake3-master-getservers patch. http://seclists.org/nmap-dev/2011/q1/925. + - Merged: backorifice-info. http://seclists.org/nmap-dev/2011/q2/185. + - Merged: omp2-brute and omp2-enum-targets. http://seclists.org/nmap-dev/2011/q2/231. + - Merged: http-wp-plugins. http://seclists.org/nmap-dev/2011/q1/806. + +o Decide what to do about ms-sql-info slowing scans: + http://seclists.org/nmap-dev/2011/q1/913 + - patch applied: http://seclists.org/nmap-dev/2011/q1/1102 + +o Script disposition + - Patch to get interfaces by Djalal. + http://seclists.org/nmap-dev/2011/q1/291 + - Incorporated + - epmd-info. http://seclists.org/nmap-dev/2011/q1/931. + - Incorporated + - google-id. http://seclists.org/nmap-dev/2011/q1/952. + - Incorporated as http-affiliate-id + +o [Ndiff] should, in non-verbose mode, perhaps not print the changed + Nmap version and/or scan time if nothing else has changed between + two files. See http://seclists.org/nmap-dev/2011/q1/674. + +o Script review disposition: + - ssl-known_key http://seclists.org/nmap-dev/2010/q4/733 + Thread continues at http://seclists.org/nmap-dev/2011/q1/26. + - Merged + - dns-nsec-enum + - Merged + +o The file /nmap/mswin32/icon1.ico is used by the NSIS installer to + set the Nmap uninstall icon (I'm not sure if it is used for anything + else). But this is a very old icon and doesn't match the blue eye + we use now. So we should probably update that with a modern "blue + insecure eye" icon. I (Fyodor) tried simply replacing icon1.ico + with http://insecure.org/shared/images/tiny-eyeicon.ico, but that + didn't work. It must not meet the required format. + +o Add some content to https://secwiki.org and announce it. + +o Removing -sR option (but keeping the functionality as part + of -sV). See http://seclists.org/nmap-dev/2011/q1/688 + - Update Nmap documentation/book to remove it there too + + +o Script disposition: + - dns-brute by cirrus. http://seclists.org/nmap-dev/2011/q1/351 + Should share domain list with http-vhosts. + git://code.0x0lab.org/nmap-dns-brute.git + - Added by David + +o Write and post 2010 SoC Successes writeup [Fyodor] + +o Script review + - quake3-master-getservers http://seclists.org/nmap-dev/2011/q1/64 + [merged] + - dpap-brute by Patrik Karlsson. + http://seclists.org/nmap-dev/2011/q1/252. + [merged] + +o The -V option to Nmap, in addition to reporting the version number, + should give details on how Nmap was compiled and the environment it + is running on. This includes things like whether SSL is enabled, + the platform string, versions of libraries it is linked to, and + other stuff which is often useful in debugging problems. + o We want to list at least: + o Nmap version number (that line is fine as is) + o host platform string (for which it was compiled) + o Whether OpenSSL and LibSSL, NLS, and IPv6 are enabled + - Version number of OpenSSL and LibSSL if those are enabled + o Version numbers of libdnet, libpcre, and libpcap + +o Script review: + - SCADA scripts http://seclists.org/nmap-dev/2010/q4/612 + http://seclists.org/nmap-dev/2010/q4/613 + http://seclists.org/nmap-dev/2010/q4/623 + http://seclists.org/nmap-dev/2010/q4/639 + [on hold] + - servicetags http://seclists.org/nmap-dev/2010/q4/691 + needs new testing on OpenSolaris: http://seclists.org/nmap-dev/2011/q1/91 + [committed] + - firewalk-path http://seclists.org/nmap-dev/2011/q1/63 + [committed over previous firewalk script] + - snmp-ios-config http://seclists.org/nmap-dev/2011/q1/10 + Requires a TFTP server; decision was to build such server in Lua + if possible. Patrik Karlsson's beginning TFTP implementation: + http://seclists.org/nmap-dev/2011/q1/169. + [committed by Patrik] + +o Script merged: p2p-dropbox-listener + http://seclists.org/nmap-dev/2010/q4/689 + +o A trivial change: we currently print some lines about NSE + pre-scanning and post-scanning in verbose mode even when no such + scripts are being run. We should not print those in that case. For + example, nmap -A -v scanme.nmap.org gives me these superfluous lines: + NSE: Script Pre-scanning. + NSE: Starting runlevel 1 (of 2) scan. + Initiating NSE at 12:23 + Completed NSE at 12:23, 0.00s elapsed + NSE: Starting runlevel 2 (of 2) scan. + NSE: Script scanning 64.13.134.52. + NSE: Starting runlevel 1 (of 2) scan. + Initiating NSE at 12:24 + Completed NSE at 12:24, 4.14s elapsed + NSE: Starting runlevel 2 (of 2) scan. + NSE: Script Post-scanning. + NSE: Starting runlevel 1 (of 2) scan. + NSE: Starting runlevel 2 (of 2) scan. + +o Do new Nmap release with the stuff merged from SoC students and + other new developments. + +o Modify Zenmap to use the new --script-help system to enumerate + scripts and collect information such as their descriptions. This + will resolve the problem of Nmap's broadcast prerule scripts running + when you open the profile editor. + +o Document --script-help in docs/refguide.xml and docs/scripting.xml. + +o [Zenmap] Brian Krebs found a problem (which Fyodor is able to + reproduce) in the target selector on the left pane. When you select + one of the scanned targets, it is supposed to jump to that target in + the "Nmap Output" tab on the right pane. Instead, nothing seems to + happen. One of our output format changes probably broke the + feature. It still works fine if you have the "Ports / Hosts" or + "Host Details" tabs active in the right pane instead. + +o Include a --script-help system to Nmap, which provides user readable + text help and also machine parsable XML information for scripts + which match a pattern (e.g. the same sort of arguments you could use + for --script, like a category or http-* or whatever). The + --script-help ONLY provides help and quits, it does not run the + script. For some initial implementation work, see this thread: + http://seclists.org/nmap-dev/2011/q1/163 + +o [Nping] See whether --echo-client mode really requires root, and + remove that restriction if not. + Luis explanation for requiring root: + http://seclists.org/nmap-dev/2011/q1/248 + +o Script review: + - p2p-dropbox-listener http://seclists.org/nmap-dev/2010/q4/689 + +o Decide whether to include NSE console script help, decide on + implementation issues. http://seclists.org/nmap-dev/2011/q1/163 + +o [Zenmap] Use a more efficient algorithm to update the display of Nmap normal + output in live scans. + zenmapGUI.NmapOutputViewer.NmapOutputViewer.refresh_output calls + zenmapCore.NmapCommand.NmapCommand.get_output, which re-reads the + entire output file (into memory) and then puts it in the text buffer + if it has changed. So already we're storing the whole output twice in + memory. When the text field changes, update_output_colors + re-highlights the whole file. + +o Update changelog to note recent changes + +o Do final dev/test release + +o If Nping is compiled w/o SSL support, and the user specifies an + encryption key, it should fail and insist they use --no-crypto + rather than ignoring the key and omitting crypto. Otherwise the + user might think they're getting encryption when they're not. David + found this problem in the server, and we also should check how the + client behaves. + +o [Ncat] Make --exec work in conjunction with --proxy. The --proxy + code path skips the --exec code. See + http://seclists.org/nmap-dev/2010/q4/604 and the test "--exec + through proxy" in ncat-test.pl. + +o Decide what to do about Nmap static binaries failing to work on new + Fedora releases (and others?). See these threads: + http://seclists.org/nmap-dev/2011/q1/46 and + http://seclists.org/nmap-dev/2010/q1/308 + o We ended up dynamically linking system libs in the RPM rather than + statically linking them. We still statically link things like lua, + pcre, ssl, etc. + +o Fix our mac builds so that they contain SSL support again (5.35DC1 + did, but TEST1 and TEST2 didn't for some reason. + +o Add our broadcast discovery scripts to a "broadcast" category (they + should generally just be in "broadcast" and (assuming they are safe) + "safe", and not normal "discovery". Update scripting.xml to note + this new category too. + +o The latest IANA services file + (http://www.iana.org/assignments/port-numbers) has many identified + services which are still "unknown" in our files because ours is + based on a much older version of that file. We should probably take + that file and add names and comments to our nmap-services-all where + they are "unknown" in our file. An example of such a port is 3872, + oem-agent. + +o Script review: + - patch for ftp-proftpd-backdoor + http://seclists.org/nmap-dev/2010/q4/678 + - patch for hddtemp-info http://seclists.org/nmap-dev/2010/q4/676 + +o We should probably update our Windows build systems to use Python + 2.7. As of 11/8, it looks like all our dependency libraries are + available for 2.7: + o David upgraded and it worked, though Rob found a potential problem + and added vcredist 2008. Fyodor will test on the official Win7 Nmap + build system. + PyGTK: 2.22.0 IS available for 2.7 + PyCairo: 1.8.10 IS available for 2.7 + PyGObject: 2.26.0 IS available for 2.7 + Py2exe: 0.6.9 IS available for 2.7 + +o Do service/version detection submission integration (last done in + April) + +o Do os detection submission integration (last done in April) + +o Script review: + - modbus-enum http://seclists.org/nmap-dev/2010/q4/489 + +o Create Nmap wiki + o Decide on domain name + o Include insecure Chrome + o Decide on wiki software, probably just use mediawiki + o install it on a Linode, probably Web + +o [NSE] Web application fingerprinting script. Would be great to be + able to take a URL and determine things like "this is Joomla" or + "this is Plone" or "Mediawiki" or whatever. Rather than hard code + regular expressions or other tests in a script, it should use a + signature file like Nmap OS and version detection do. Might work in + combination with URL grinder to check for applications at + default/common locations. See also a script that does favicon + scanning TODO item. + - http-enum pretty much does this now. + +o Update our distribution build systems and documentation to use + Visual C++ 2010 Express rather than the 2008 version. See + http://www.microsoft.com/express/Windows/ + +o Dependency licensing issues (OpenSSL, Python, GTK+, etc.) + o Almost done! We just have some file renaming/organizing left to do. + o We should do an audit to ensure that we are in complete compliance for the + licenses of all the software we ship in any of our downloads, as some + licenses have special clauses for things like including their + license/copyright file, mentioning them in our documentation, etc. + And of course we want to credit them properly even where the license + doesn't require it. We should probably make a list of these in our + docs/ directory along with any special information/requirements of + their license. And maybe we should put the current licenses in a + subdir too. In particular, these come to mind: + o libpcre + o lua + o OpenSSL + o libpcap + o GTK+/Glib/ATK/Pango/PyGTK (Win/Mac versions of Zenmap link to + PyGTK) + o SQLite + o Python (Win/Mac versions of Zenmap link to Python) + o X.org libraries (Mac version links to them) + o libdnet + +o Small NSEDoc bug: + https://nmap.org/nsedoc/scripts/dns-zone-transfer.html contains 'id + \222\173' near the bottom. This is presumably due to misparsing this + line from the script: local req_id = '\222\173'. Given that we don't + use IDs any more, maybe we can just get rid of the functionality. + +o [NSE] We should probably enable broadcast scripts to work better by + (initial thoughts): + o Done and merged by David! + 1) Change NSE to always set nsp_setbroadcast() on new sockets + 2) Change nsock to create real sockets at time of nsi_new so you can + bind to them. + See this thread (only some of the messages involve broadcast + support): http://seclists.org/nmap-dev/2010/q3/357 + +o [NSE] Review scripts: + o Hostmap (Ange Gutek) - http://seclists.org/nmap-dev/2010/q3/159 + +o Post BH/Defcon Nmap videos + +o Let Nsock log to stderr, so its messages don't get mixed up with the + output stream when Ncat is run with -vvv. + http://seclists.org/nmap-dev/2010/q3/113 + +o [NSE] Our http-brute should probably support form POST method rather + than just GET because some forms require that. + +o Nping needs to call nsp_delete so that its socket descriptors are + not left behind. + +o [Zenmap] Add a button to select script files from the filesystem. + +o [Zenmap] Show help for individual script arguments in the Help pane, + not for all arguments at once. + +o Upgrade our Windows OpenSSL binaries from version 0.9.8j to the + newest version (1.0.0a as of Aug 12, 2010). + +o Since Libdnet files (such as ltmain.sh) are apparently only used by + libdnet (they used to be used by shared library NSE C scripts), we + should move them to the libdnet directory. + o Turned out to be a pain. See + http://seclists.org/nmap-dev/2010/q3/733 + +o [Zenmap] Consider a memory usage audit. This thread includes a claim + that a 4,094 host scan can take up 800MB+ of memory in Zenmap: + http://seclists.org/nmap-dev/2010/q1/1127 + The reporter mentioned Guppy/Heapy to debug memory use: + http://guppy-pe.sourceforge.net/ + http://www.pkgcore.org/trac/pkgcore/doc/dev-notes/heapy.rst. Many + Nmap survey respondants complained about this too. + Note: Fyodor has a 50MB scan log file named ms-vscan.xml which + demonstrates this problem. When trying to load the file, Zenmap + grows to 1150MB of RAM, pegs the CPU usage at 100% for many + minutes or maybe hours (I forgot about it, but woke up the next day + to find that it had started, was then using 2.4GB of RAM. The + hosts/services functionality seemed to work, although it would take + a minute or so to switch from say "ftp" port to view "ssh" ports. + +o [NSE] Maybe we should create a script which checks once a day + whether similar tools (Metasploit, Nessus, OpenVAS, etc.) have any + new modules, and then mails out a list of them with the description + fields. The mail could go to just interested parties, or maybe + nmap-dev. This may help prevent important vulnerabilities from + falling through the cracks. Perhaps we would include new NSEs in + there too, especially if we open it up as a public list. + +o Now that NSE has more script phases (prerule, postrule, hostrule, + portrule, and versionrule soon to come), the NSEDoc should specify + which phases a script belongs to. + +o Consider implementing a nsock_pcap_close() function or making + nsp_delete() call pcap_close() when pcap IODs are used. Currently valgrind + warns about a socket descriptor left opened (at least in Nping). + See http://seclists.org/nmap-dev/2010/q3/305. + o It turns out that the pcap descriptors are being closed properly, + but Nping isn't calling nsp_delete. + +o [NSE] High speed brute force HTTP authentication. Possibly POST and + GET/HEAD brute force cracking. [done except for form POST, adding + separate TODO item for that] + +o [NSE] Review scripts: + o New brute, vnc, and svn scripts by Patrik. This guy is a coding + machine :). http://seclists.org/nmap-dev/2010/q3/111 + o rmi-dumpregistry by Martin + Swende. http://seclists.org/nmap-dev/2010/q2/904 + o path-mtu.nse - http://seclists.org/nmap-dev/2010/q3/222 + o 15 more from Patrik :). http://seclists.org/nmap-dev/2010/q3/284 + +o [NSE] Consider modifying our brute force scripts to take advantage + of the new NSE multiple-thread parallelism features. + - We've done this with db2-brute, but the DB may have been a + bottleneck there, so we should probably do more testing after + modifying another script for this sort of parallel cracking. + +o Look into implementing security technologies such as DEP and ASLR on + Windows: http://seclists.org/nmap-dev/2010/q3/12. + +o Ncat and Nmap should probably support SSL Server Name Indication + (SNI). See this thread: http://seclists.org/nmap-dev/2010/q3/112. + We need this to talk to web servers which share one SSL IP and port + because we need to ask for the right SSL key. + +o [NSE] In the same way as our -brute scripts limit their runtime by + default, I think qscan should be less intense by default. For + example, perhaps it could run by default on no more than 8 open + ports, plus up to 1 closed port. Right now it does things like + running on 65,000+ closed ports and bloats scan time (and output). + Of course there could (probably should) still be options to enable + more intense qscanning. + +o [Web] We should see if we can easily put the Insecure chrome around + Apache directory listings and 404 pages (e.g. https://nmap.org/dist/ + and https://nmap.org/404). I think we may have had this working + before the move to Linode, so maybe check conf/httpd.conf.syn. + +o Do a serious analysis if and how we should use the NIST CPE standard + (http://cpe.mitre.org/) for OS detection and (maybe in a different + phase) version detection results. One thing to note is that they + may not have entries for many vendors we have. For example, one + person told me they couldn't find SonicWall or D-Link in the CPE + dictionary. Here are some + discussions threads on adding CPE to Nmap: + http://seclists.org/nmap-dev/2008/q4/627 and + http://seclists.org/nmap-dev/2010/q2/788. + Nessus has described their integration of CPE at + http://blog.tenablesecurity.com/2010/05/common-platform-enumeration-cpe-with-nessus.html. + +o [NSE] Create NSE scripts to scan for and/or exploit these VXWorks issues: + http://blog.metasploit.com/2010/08/vxworks-vulnerabilities.html [Ron + may be able to do this. Or others are welcome to take a shot at it.] + +o The -g (set source port) option doesn't seem to be working (at least + in Fyodor's quick tests) for version detection or connect() scan, + and apparently doesn't work for NSE either. We should fix this + where we can, and document the limitation in the refguide where it + is impractical. Also see http://seclists.org/nmap-dev/2010/q2/576. + +o [Zenmap] script selection interface for deciding which NSE scripts to + run. Ideally it would have a great, intuitive UI, the smarts to + know the scripts/categories available, display NSEdoc info, and even + know what arguments each can take. + +o Review http-xst (Eduardo Garcia Melia) - + http://seclists.org/nmap-dev/2010/q3/159 + +o [NSE] Investigate sslv2.nse falsely reporting SSLv2 as being + supported. + http://seclists.org/nmap-dev/2010/q2/754 + +o [NSE] The NSEDoc for some scripts includes large "Functions" + sections which aren't really useful to script users. For example, + see https://nmap.org/nsedoc/scripts/snmp-interfaces.html. Perhaps we + should hide these behind an expander like "Developer documentation + (show)". I don't think we need to do this for libraries, since + developers are the primary audience for those documents. + o Talked to David. We should just remove the function entries. + +o We should add a shortport.http or similar function because numerous + services use this protocol and many of our scripts already try to + detect http in their portrule in inconsistent ways. + +o [NSE] Maybe we should create a class of scripts which only run one + time per scan, similar to auxiliary modules in Metasploit. We + already have script classes which run once per port and once per + host. For example, the once-per-scan ("network script"?) class might + be useful for broadcast LAN scripts (Ron Bowes, who suggested this + (http://seclists.org/nmap-dev/2010/q1/883) offered to write a + NetBIOS and DHCP broadcast script). Another idea would be an AS to + IP ranges script, as discussed in this thread + http://seclists.org/nmap-dev/2010/q2/101 [Could be a good SoC + infrastructure project] + o David notes: "I regret saying this before I say it, because I'm + imagining implementation difficulties, we should think about + having such auxiliary scripts be able to do things like host + discovery, and then let the following phases work on the list it + discovers." + +o Analyze what sort of work would likely be required for Nmap to + support OS detection over IPv6 to a target. + o Would probably start with a way to send raw IPv6 packets + o There is a raw IPv6 patch here: + http://seclists.org/nmap-dev/2008/q1/458 + o Also it looks like Nping may be doing this already. + o Then we need to figure out if we can use our current DB and + techniques, or if we'd likely thave to have an IPv6-specific + DB. [David] + +o July Nmap releases (at least a beta version, and maybe a stable + too). Last release was 5.30BETA1 on March 29 + +o Add this patch for compilation on OpenSolaris. + http://blogs.sun.com/sdaven/entry/nmap_5_35dc1_compile_on + +o Now that we've put the ndiff, ncat, and nping man pages under the + scope of the book (e.g. https://nmap.org/book/ncat-man.html), we need + to add a redirect from the old locations and also update our links. + +o Make sure the long output lines in Nping's man page are OK for the book. + See r18829 and r18864. + +o Update "History and Future of Nmap" + (https://nmap.org/book/history-future.html) to include all the news + since September 2008. [Fyodor] + +o Fix Win7 networking issue reported by Luis which seems to have been + triggered by r17542. See this thread: + http://seclists.org/nmap-dev/2010/q3/40 + +o Upgrade to WinPcap 4.1.2 - Rob has a patch - See this thread: + http://seclists.org/nmap-dev/2010/q3/18 + +o [NSE] Review UnrealIRCd backdoor detection script + http://seclists.org/nmap-dev/2010/q2/854 + +o [Zenmap] Investigate segfault on some installs of OS X 10.6.3: + http://seclists.org/nmap-dev/2010/q2/587 + o David rebuilt with MacPorts 1.9.1 rather than 1.8.2 and the + problem went away. + +o [Zenmap] Investigate failure to start on some installations of OS X + 10.6.3. + [ We think one may just not have waited long enough as he said it + started working, and another case (the 587) seems to be a + segfault--we added a new task for that ] + http://seclists.org/nmap-dev/2010/q2/587 + http://seclists.org/nmap-dev/2010/q2/859 (He responded to David + privately and said that it was not an I7 processor.) + Nmap seems to be having problems too: + http://seclists.org/nmap-dev/2010/q2/747 + +o [NSE] Review Gutek's PHP version disclosure script. + http://seclists.org/nmap-dev/2010/q2/569 + +o Fix the IPv6 name resolution problem described in this thread: + http://seclists.org/nmap-dev/2010/q2/787 + +o [NSE] Review Gutek's libopie detection/DOS script. + http://seclists.org/nmap-dev/2010/q2/635 + +o [NSE] Review Gutek's web server directory traversal script. + http://seclists.org/nmap-dev/2010/q2/595 + - It became modifications to http-passwd + +o [NSE] Review dns-cache-snoop.nse from Eugene Alexeev. + http://seclists.org/nmap-dev/2010/q2/195 + Better attachment at: http://seclists.org/nmap-dev/2010/q2/200 + Need to decide on a domain list: http://seclists.org/nmap-dev/2010/q2/199 + +o Fix bug where multiple targets with the same IP can end up in a + hostgroup and cause port scanning and probably OS detection to + misbehave. An example is "nmap -F scanme2.nmap.org + scanme3.nmap.org". See this thread for details: + http://seclists.org/nmap-dev/2010/q2/322 + +o Need to fix our current win32.zip distribution so that .svn files + aren't included (currently they are in nselib/data). Will probably + be a simple adjustment to mswin32/Makefile. + +o Make Zenmap splash screen + +o [NSE] Add one of, or combine, ntp-peers and ntp-monlist. + http://seclists.org/nmap-dev/2010/q2/190 + http://seclists.org/nmap-dev/2010/q2/191 + +o [NSE] Reorganize nselib to allow libraries in subdirectories. + Currently, to avoid expanding the number top-level libraries, code + that is only used by one library is built into that library's file, + even if it is logically separate. For example, the mongodb library + contains a BSON-parsing library. Instead, that library could go in + mongodb/bson.lua. The msrpc and smb libraries could potentially be + broken up in this way. + UPDATE: We decided not to do this for now, given complications in + nsedoc, packaging, etc. to support the new hierarchy. Instead, we + can use prefixes like we do with scripts (e.g. mongodb-bson.lua, + msrpc-types.lua). + +o Add a configure option to our libpcap which enables an older Linux + packet capture system (David's noring patch). This is needed in + some cases for 32-bit static binaries to work on 64-bit Linux + systems. Note that it is unneccessary if both the build system and + the target system use Linux 2.6.27, as that has an architecture + independent tpacket_hdr (called tpacket2_hdr). [Added by David as + --disable-packet-ring] + +o Test Jay Fink's UDP payload prototype. + http://seclists.org/nmap-dev/2010/q1/168 + [ tested, improved, merged by David] + +o Resolve Ncat broadcast support issue (see this thread: + http://seclists.org/nmap-dev/2010/q2/422). + +o [NSE] Review and test the DB2 library and + scripts. http://seclists.org/nmap-dev/2010/q2/395 (but updated + versions may be available). + +o Move nmap/docs/TODO into its own todo directory (probably nmap/todo) + and then encourage maintainers of /status/ TODOs and any other TODOs + to migrate theirs there. Unlike the status directory, /nmap/todo + would be readible by anyone. [Fyodor] + +o Nmap should at least print (and maybe scan) all IP addresses for + hostnames specified on the command line. We will start with just + printing all the addresses. Here is a thread on the topic: + http://seclists.org/nmap-dev/2010/q2/302 + [David made it do the printing, adding a different task related to + scanning them all] + +o Integrate new service detection fingerprint submissions (we have + more than 730 since Dec. 17, 2009. + +o [Ncrack] Use our new password lists (now used by NSE) for Ncrack as + well. Ncrack can probably handle a larger list than NSE uses. + +o Consider MSRPC ideas from Ron--we might want to add some as TODO + tasks: http://seclists.org/nmap-dev/2010/q2/389 + +o Fix XML inconsistency described at + http://seclists.org/nmap-dev/2010/q2/326 + +o Integrate new OS fingerprints (we have more than 1,300 since + November 10, 2009). + +o Finish selecting GSoC 2010 projects + +o Upgrade libpcap to the new 1.1.1 version. + +o Improve the NSI installer by adding command-line options for unsetting + each of these GUI checkboxes individually (particularly useful for + silent mode): + LangString DESC_SecCore ${LANG_ENGLISH} "Installs Nmap executable, NSE scripts and Visual C++ 2008 runtime components" + LangString DESC_SecRegisterPath ${LANG_ENGLISH} "Registers Nmap path to System path so you can execute it from any directory" + LangString DESC_SecWinPcap ${LANG_ENGLISH} "Installs WinPcap 4.1 (required for most Nmap scans unless it is already installed)" + LangString DESC_SecPerfRegistryMods ${LANG_ENGLISH} "Modifies Windows registry values to improve TCP connect scan performance. Recommended." + LangString DESC_SecZenmap ${LANG_ENGLISH} "Installs Zenmap, the official Nmap graphical user interface. Recommended." + LangString DESC_SecNcat ${LANG_ENGLISH} "Installs Ncat, Nmap's Netcat replacement." + LangString DESC_SecNdiff ${LANG_ENGLISH} "Installs Ndiff, a tool for comparing Nmap XML files." + LangString DESC_SecNping ${LANG_ENGLISH} "Installs Nping, a packet generation tool." + +o We should have a standard function which takes time arguments in the + same format as Nmap does (e.g. 60s, 1m, etc.) and the scripts which + take time arguments should be modified to use it. David suggests + this here: http://seclists.org/nmap-dev/2010/q2/35. We are also + going to update the normal Nmap timing functions to take seconds by + default, as described here: http://seclists.org/nmap-dev/2010/q2/159 + +o Nmap should probably always produce a well-formed XML file, even if + it exits with a fatal() error. In that case, the error should be + included in the XML. Right now, for example, if the network is + down, the XML output will just stop (no closing tags) and Nmap will + print something to STDERR like: + nexthost: failed to determine route to 9.48.184.164 + QUITTING! + +o Get @output sections for the last remaining scripts w/o them: + [WARN] script auth-spoof missing @output + [WARN] script db2-das-info missing @output + [WARN] script db2-info missing @output + [WARN] script http-passwd missing @output + [WARN] script iax2-version missing @output + [WARN] script ms-sql-config missing @output + [WARN] script ms-sql-query missing @output + [WARN] script oracle-sid-brute missing @output + [WARN] script pop3-brute missing @output + [WARN] script pptp-version missing @output + [WARN] script skypev2-version missing @output + +o [Zenmap] Maybe it should sort IPs in an octet-aware way. And maybe + you should be able to sort by IP address (perhaps that should be the + default). Current plan is to just sort by IP by default, and maybe + we'll offer other sort techniques later if desired. See + http://seclists.org/nmap-dev/2010/q2/27 [possible SoC student task] + +o Brainstorm for GSoC 2010 ideas and fill out the org application by + Friday 3/12 4PM PST. + o NSE scripts + o Maybe a whole SoC role for http scripts + o Maybe look at other web app scanners for some inspiration + (including w3af - http://w3af.sourceforge.net/) + o Maybe a non-http developer too + o NSE infrastructure manager + o Ncrack + o Nping + o Mobile Devices? N900, iPhone, Android + o Zenmap developer + o Must have solid user interface design experience + o Zenmap script selector (subset of a Zenmap or NSE SoC role) + o Feature Creepers/Bug fixers + +o Review IDS detection scripts from Joao Correa. + http://seclists.org/nmap-dev/2010/q1/814 + +o Review mssql library and scripts from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/1000 (files) + http://seclists.org/nmap-dev/2010/q1/1014 (sample output) + +o Review DNS fuzzer script from Michael Pattrick. + http://seclists.org/nmap-dev/2010/q1/1005 + +o Our nsedoc generator should probably give a warning if a script is + missing any important fields. @output comes to mind. @usage can be + nice too, though we could consider auto-generating that for trivial + scripts. + +o [NSE] Consider pros and cons of splitting information retrieval + scripts into a bunch of small single-purpose script vs. one larger + argument-controlled script. See + http://seclists.org/nmap-dev/2010/q1/1023 + [we ended up combining three of the ms-sql scripts. If we combine + future scripts, we need to remember to add them to the deprecation + list in the Makefile] + +o Remove --interactive. It was broken for a long time and nobody + seemed to notice, and we put a call out on nmap-dev for + --interactive users and didn't get any good reasons to keep it. We + should kill it to remove the code complexity it adds and to avoid + the documentation complexity of people having to read and learn + about a feature they are unlikely to ever use. + +o Zenmanp should perhaps be able to print Nmap output on a Printer (if + not too much of a pain to implement.) + +o Review afp-serverinfo.nse from Andrew Orr. + http://seclists.org/nmap-dev/2010/q1/470 Just waiting on some bug fixes: + http://seclists.org/nmap-dev/2010/q1/665 + +o Test 64-bit pcap installer (e.g. remove old version and install new) + before next release, as we've applied a change from Rob which works on + his system (http://seclists.org/nmap-dev/2010/q1/796). + +o [NSE] Improve username/password library (the database files + themselves). We don't have very good lists at the moment. Maybe + work in combination with Ncrack dev. + o Now there are some even better lists available (f.e. RockYou)--see + this thread: http://seclists.org/nmap-dev/2010/q1/764 + o We've improved the ncrack files--we should probably either use + those for NSE or use a subset of them. + o perhaps from Solar Designer. (he sent us permission) + o perhaps add phpbb hack data (there is at least a list of 28,635 + passwords in phpbb_users.sql, and possibly more in other files. + +o [Nping] Should take the version number 0.[nmap version], such as + 0.5.22TEST + +o Review rpc.lua, nfs-showmount.nse, nfs-get-stats.nse, and + nfs-get-dirlist.nse from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/270 + +o [NSE] Look into moving packet module to C for better performance + [Patrick] + o Removing this one because it is stale (has been here for many + months with no action seen), but it is something we can consider + if/when there is a desire to implement it. A key is probably to + measure current performance and see if it is a material problem. + +o Maybe the Nmap ASCII art should come after make rather than + configure? + - We decided it would probably be annoying for developers to see it + every time they 'make'. + +o Review snmpenum.nse from William Njuguna. + http://seclists.org/nmap-dev/2009/q4/721 + http://seclists.org/nmap-dev/2010/q1/656 + o Dropping for now unless original author or someone else picks it + up and fixes the bugs. + +o Add smtp-enum-users from Duarte Silva if testing is favorable. + http://seclists.org/nmap-dev/2010/q1/699 + +o After the new -sn and -Pn options (added to SVN around 7/20, just + after the 5.00 release) have been around long enough to be in most + people's copy of Nmap (e.g. in all the versions we distribute from + download page (stable+dev)) for at least a few months, we'll document + these as the preferred version rather than -sP and -PN. These match + -n, and the main problem with -sP is that we now use it more for + "disable portscan" than ping only. For example, you can also use + NSE, traceroute, etc. [David] + +o Nmap currently selects routes based on the first matching one it + finds. But it should really take the most specific route instead. + So it should: + 1) Keep searching the routing table for the most specific match, and + 2) Use a stable sort (not qsort) so that routes with identical + netmasks aren't rearranged. + For more, see http://seclists.org/nmap-dev/2010/q1/685 + +o Review pgsql-brute.nse from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/455 + +o psexec missing (need to download yourself now) nmap_services.exe + output issue: "The function where this is detected returns a value + that is passed to stdnse.format_output. format_output takes a + parameter to decide whether it's displaying an error message, but it + is hard-coded to only display error messages with debugging >= 1. So + options are to change format_output and make it more flexible, or + somehow decouple the sensing of nmap_service.exe from the normal + output channel of the script." + +o Website: Create shared directory in svn, which will contain + directories shared between the Insecure.org network of sites + (e.g. templates, error, css). Then sites such as sectools, + nmap.org, insecure.org can just check that out via externals + declaration (or, I suppose, symlink). CSS directives will then use + /shared/css/insecdb.css etc. ). + +o Add CouchDB and JSON scripts once the JSON library is finished. + http://seclists.org/nmap-dev/2010/q1/641 + +o Review NSE raw IP from Kris Katterjohn. + http://seclists.org/nmap-dev/2010/q1/559 + +o Review sslv3-enum.nse from Mak Kolybabi. + http://seclists.org/nmap-dev/2010/q1/563 + +o [NSE] Consider LDAP library and scripts from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/70 [all merged, except David is + still reviewing ldap-search] + +o More potential improvements to http-methods: + http://seclists.org/nmap-dev/2010/q1/630 and + http://seclists.org/nmap-dev/2010/q1/640 + +o Remove smtp-open-relay.nse sometime after 9/24/09 if nobody adopts it (see + http://seclists.org/nmap-dev/2009/q3/0986.html). [It got fixed up + and we kept it.] + +o The -v and -d arguments should take the same syntax. Right now you + use -vvv vs. -d3. We should probably just make either approach work + with either of them. + +o Zenmap should be able to export normal Nmap output + +o Integrate Nping. + +o [NSE] Consider the http-methods script from Bernd Stroessenreuther. + http://seclists.org/nmap-dev/2010/q1/76. [integrated, but David is + making some improvements]. + +o The Nmap web page is beginning to show its age. Ah, who am I + kidding, it was showing its age 5 years ago :). It could do with an + upgrade to XHTML+CSS. It could also do with a whole redesign, but I + think that can be done as a second step after converting to + XHTML+CSS with roughly the same look. Though adding a few more + modern touches (like hover interaction on the menu bar) wouldn't + hurt. This is a moderatly big project, which will involve: o + Designing the new XHTML+CSS to look similar to the current HTML + pages, but be extensible enough that it can be redesigned in the + (near) future by mostly just changing the CSS and graphics. + o Converting the existing Nmap pages to the new XHTML format. + This will likely include using open source programs and likely + modifying them or creating your own scripts to help with the + process. To apply for this task, you need to have some web + development experience and an example XHTML+CSS web page you + have created online. + o We decided not to worry about XHTML for now, and we're + integrating CSS in piece by piece -- we already have the section + headers, left sidebar links. etc. + o Should not use SSI like the current pages -- should do all its + magic through CSS. That way it will work on seclists too (which + can't do SSI for security reasons). + o Maybe alpha transparency for menus, gradiants, curves, etc. But + the main goal isn't flashiness. + +o Seclists.org should maybe be fixed so that it doesn't strip quoted + text for its summaries from the IP list because that list consists + almost entirely of forwarded material which is being stripped. Look + at the summaries at http://seclists.org/interesting-people/. + +o Web site HTML improvements + - Maybe start with nmap.org. + - Find and fix HTML validation problems, bad links. I'm not sure + what tool is best for this. + - Then do the same with seclists.org, insecure.org, sectools.org + - The icon on the top-left of the screen should be for (and link + to) the root URL of current site. e.g. seclists.org, + sectools.org, nmap.org rather than always insecure.org. + +o [NSE] Consider SNMP scripts from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/162 + http://seclists.org/nmap-dev/2010/q1/174 + http://seclists.org/nmap-dev/2010/q1/178 + +o Deal with AV false positive issue RE nmap_services.exe: + - For now, David is going to apply Ron's patch which removes this, + but David will make it print output in verbose mode rather than + debug and maybe make it a little less verbose. LT plan is for Ron + to encrypt it with OpenSSL. + +o Web site improvements + - Update to use CSS, at least for header bars + - Also, if it is easy to give the header bars rounded corners, + we should probably do so. But if it is hard, it isn't + important enough to matter. + - The Nmap.Org navigation table should have a background and more + subtle lines, like we use for our calendars now. + - The first item (table) in featured news has slightly more + left/right margin than the later ones on Firefox 3.5.6, and with + IE8 it doesn't extend as far when you make the page really wide. + Plus the images on the right are problematic (extend through the + border below them) when you make the window too wide on IE8. + Having a slight margin on the left/right of entries would + actually be a bit nice. And it would be nice if it only took a + simple tag or two, controlled by CSS rather than pasting in a + whole table with font tags and the like for each entry. + +o [Ncat] Test, review, and (if appropriate) merge Venkat's HTTP Digest + proxy authentication patch. See + http://seclists.org/nmap-dev/2009/q3/773. [David] + +o [NSE] Look at new DB2 script by Tom + Sellers. http://seclists.org/nmap-dev/2009/q4/659 + +o [NSE] Consider MongoDB scripts and libraries from Martin Holst Swende. + http://seclists.org/nmap-dev/2010/q1/177 + +o [NSE] Document Patrick's worker thread patch in scripting.xml (see + http://seclists.org/nmap-dev/2009/q4/294, + https://nmap.org/nsedoc/lib/stdnse.html#new_thread, + https://nmap.org/nsedoc/lib/nmap.html#condvar) [Patrick] + +o Make Nmap 5.21 bugfix-only release + +o [NSE] Consider afp-showmount script from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/97 + [merged to trunk] + +o [NSE] Review DNS-SD script from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/87 + [merged to trunk] + +o [NSE] Consider MySQL scripts from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/163 + [merged to trunk] + +o [NSE] Consider DAAP script from Patrik Karlsson. + http://seclists.org/nmap-dev/2010/q1/164 + [merged to trunk] + +o NSEDoc left sidebar should include a link to + https://nmap.org/book/nse.html below "Index". + +o Consider enhancing the new OS Assist system to handle version + detection too. [We decided not to do this as David noted that Doug's + serviceunwrap.lisp does pretty much everything he needs.] + +o [NSE] HTTP header parsing is not very robust, and is duplicated in a + lot of places. For example, it's legal to have header fields like +Content-type:\r\n +___text/html\r\n +(with spaces in place of _, but http.lua won't parse such a header +correctly. In other words you can extend them to any number of lines +as long as each line after the first begins with whitespace. [David] + +o Investigate issue with our Pcap and Wireshark x64, as described in + this thread: http://seclists.org/nmap-dev/2009/q4/557 [Rob] + [Taking this off the list until/unless we get more reports] + +o Decide what to do about Windows 7/Vista and starting NPF. See this + thread: http://seclists.org/nmap-dev/2010/q1/20 + +o [NSE] We should do a favicon survey like the one Brandon did for + /favicon.ico files but which uses the favicons specified by the HTML + files rather than just that exact location. For example, insecure.org + sites include in the headers: + <link REL="SHORTCUT ICON" HREF="http://images.insecure.org/images/tiny-eyeicon.png" TYPE="image/png"> + Then we should update our favicon database to include the top ones, + and we should also improve our favicon script so that it either + omits checking /favicon.ico if the HTML-specified one exists, or it + should just download, interpret, and display info for both (right + now it seems to give prority to the wrong one: /favicon.ico). + + +o [Ncat] Add SSL support for --exec so you can use SSL to talk to your + remote shell, etc. See this thread: + http://seclists.org/nmap-dev/2009/q4/255, particularly the + implementation sketch at http://seclists.org/nmap-dev/2009/q4/268 [Venkat,David] + +o Look at new Kerberos script from Patrik Karlsson. + http://seclists.org/nmap-dev/2009/q4/715 . [We decided not to merge + this one since its usefulness turned out to be limited on Windows and + very limited on any other platform. ] + +o Add feature to http library to let user set the user agent to be + used. The NSEDoc for this feature should probably tell what our + current default user agent is ("Mozilla/5.0 (compatible; Nmap + Scripting Engine; https://nmap.org/book/nse.html") [David] + +o On our NSEDoc pages (e.g. https://nmap.org/nsedoc/), perhaps the link + text for scripts should not include the ".nse". Basides saving + horizontal space, this may improve the sorting so that the likes of + "citrix-enum-apps" comes before "citrix-enum-apps-xml". Also, we can + probably get away with reducing the width of the NSEDoc left-column, + especially if ".nse" is removed. + +o [NSE] Patrick's script dependency patch: + http://seclists.org/nmap-dev/2009/q4/295 + o I'm not sure if he has gone through and actually set appropriate + dependencies (and removed runlevels) yet + +o Integrate latest version detection submissions and corrections. + This was last done based on submissions until February 9, 2009. + +o Release 5.10BETA2 + +o Add --evil to set the RFC3514 evil bit. + ftp://ftp.rfc-editor.org/in-notes/rfc3514.txt + o We're not going to add this right now. + +o Talk to Libpcap folks about incorporating (at least some of) my + changes from libpcap/NMAP_MODIFICATIONS. [marking as done since the + upstream-appropriate changes are pretty minor now that we've + upgraded to 1.0] + +o Nping -- like hping3 but uses Nmap infrastructure and to a + large degree the same command-line options as Nmap. + [We now have an alpha version at https://nmap.org/nping/] + +o Further investigate SCTP functionality, as some people reported + problems (see this thread: + http://seclists.org/nmap-dev/2009/q2/0669.html) + +o [NSE] NFS query script for checking exports, etc.? [Patrik Karlsson] + +o [NSE] Attempt to reproduce and fix a deadlock reported by Brandon + when he does large-scale scanning with a new favicon script with + hostgroups as small as 8,192 (he hasn't seen it with 4096 + hostgroups). Could be a bug in internal NSE socket lock. Probably + not specific to the favicon script, but that is how Brandon + reproduces it. At the hang, stack trace is usually the threads stuck + in socket_lock function, sometimes lookup_cache mutex in http + library. David guesses that it's threads being garbage-collected + from the socket lock table. The only thing that can wake up a thread + waiting on a socket lock is if a thread that holds a lock is removed + from the table. But the table has weak keys, meaning that a thread + can be garbage collected and it will be automatically removed from + the table by the Lua runtime. Then there is no event that can wake + up a thread waiting for a lock. [David and Patrick made some commits + at end of November meant to resolve this, and we haven't seen the + problem since, so we're marking it as done for now]. + +o Look into reducing Nmap memory consumption + o UDP scans with -p- and large hostgroups are a particularly large + offender. See if there is a way to prevent them from eating up + gigs of RAM. See the message "Port memory bloat" at + http://seclists.org/nmap-dev/2009/q3/0926.html for a patch that + reduces Port memory use by about 50%. + o One idea David has been considering is a way to represent filtered + ports (or whatever the default state is) without creating a Port + object for each one. + [David] + +o Fix assertion failure with certain --exclude arguments (see + http://seclists.org/nmap-dev/2009/q4/276). [David] + +o Many people may have stale (since removed/renamed) scripts in their + Nmap scripts directory because our 'make install' does not remove + them and so they remain and can cause problems (like running twice + after being renamed). We should probably add a line to our 'make + install' which removes the scripts/lib names we have previously + used. We're doing this rather than blowing away the old directory + just in case someone has custom scripts/libs there (though that is + still a bad idea). [David] + +o Update the CHANGELOG for new 5.10BETA1 + release. [Fyodor] + +o Make the new Nmap 5.10BETA1 release + +o Ndiff man page should be built from XML source whenever a release is + done, as ncat/zenmap/nmap man pages are. [Fyodor] + +o We should package the rendered Nroff man page translations (e.g. all + 16 languages) in the tarball to make it easier for distributors to + package them. For example, see + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=358336. Including + the translations would add 2.5MB to the (currently 28MB) + uncompressed tarball and about 800KB to the (currently 9MB) bz2 + compressed tarball. [Fyodor] + +o The Nmap 5.00 tarball contains: + -rw-r--r-- fyodor/fyodor 122943 2009-06-24 14:35 nmap-5.00/docs/scripting.xml + -rw-r--r-- fyodor/fyodor 151 2009-06-24 14:35 nmap-5.00/docs/nmap-usage.xml + -rw-r--r-- fyodor/fyodor 604 2009-06-24 14:35 nmap-5.00/docs/nmap-man-enclosure.xml + -rw-r--r-- fyodor/fyodor 76918 2009-06-24 14:35 nmap-5.00/docs/nmap-install.xml + -rw-r--r-- fyodor/fyodor 10179 2009-06-24 14:35 nmap-5.00/docs/legal-notices.xml + If we're going to include the XML source files, we should include + refguide too. But rather than add that, we should probably take + these out. After all, people can easily grab them from svn or our + new http svn gateway if desired. So no need to bloat the tarball + with these files which aren't installed. [We're going to take the + XML source files out of the tarball] [Fyodor] + +o Consider converting this file to emacs org-mode + (http://orgmode.org/) format. [Fyodor] + o That format is still plain text and can be read/edited by vi + users, etc. + [Considered, but I don't think I'll change right now] + +o Windows 7 RTM Nmap testing (With particular attention to 64-bit and + our pcap installer). [Fyodor] + +o We should print host latency (when available) in the XML output, as + suggested at http://seclists.org/nmap-dev/2009/q4/215. + docs/nmap.dtd will have to be modified accordingly, and you might + even consider adding support to docs/nmap.xsl. + +o Integrate latest OS fingerprint submissions and corrections. This + was last done based on submissions up to May 8, 2009. + +o Potential OS X 10.6 problems. There are two issues reported by the + same user which may be related: + http://seclists.org/nmap-dev/2009/q3/0936.html, + http://seclists.org/nmap-dev/2009/q3/0996.html. One is that Nmap + hangs doing nothing and needs to be killed with Ctrl-C, and the + other is that it dies after printing "Initiating UDP Scan". Another + reported the same problem at + http://seclists.org/nmap-dev/2009/q3/0990.html, where it dies after + the first ARP request is sent. But Brandon has run Nmap on 10.6 + without problems. It is a bit of a mystery. [David] [Resolution: + Apple fixed the problems in 10.6.2; For users who have 10.6 and + 10.6.1, the versions David builds on 10.5 will still work for them + because they are 32-bit binaries rather than 64. Users who build + Nmap on 10.6 or 10.6.1 should compile with -m32 or update to 10.6.2] + +o [NSE] Patrick's worker thread patch: + http://seclists.org/nmap-dev/2009/q4/294 + +o Investigate get_rpc_results error (infinite loop) reported by Lionel + Cons. See these threads: http://seclists.org/nmap-dev/2009/q4/24, + http://seclists.org/nmap-dev/2009/q4/120 + +o Upgrade to latest version of NSIS on Nmap Win build system [Fyodor]. + +o Standardize on a proper file header for the Zenmap source code. [David] + o For now, David is going to augment the templatereplacement system + to insert the normal nmap.header.tmpl, but change the comment format + to work with Python, and then replace the current Zenmap headers + with that. + +o We may want to look into if/how we support IPv6 nameservers. Here + is a bug report from someone having a problem with them: + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=539244 [Ankur] + +o Once all the man page languages are in the Nmap tarball, we should + update our install system to install them in the appropriate place. + We'll want to integrate this with configure so users can decide which + languages they want. See http://seclists.org/nmap-dev/2009/q4/249. + +o Resolve allow_ipid_match issue which can cause some malformed + replies to be ignored when we might be able to still use them. See + this thread: http://seclists.org/nmap-dev/2009/q2/665 [David] + +o Fix Zenmap 'make install' TypeError issue + (http://seclists.org/nmap-dev/2009/q4/225). [David] + +o Fix a bug in which Nmap can wrongly associate responses to SYN and + ACK host discovery probes. [David] + For example: + # nmap -sP -PS80 -PA80 australia.gov.au --packet-trace -d2 + SENT (0.0760s) TCP 192.168.0.21:60182 > 152.91.126.70:80 S ttl=43 id=13466 iplen=44 seq=4046449223 win=4096 <mss 1460> + SENT (0.0770s) TCP 192.168.0.21:60182 > 152.91.126.70:80 A ttl=48 id=39976 iplen=40 seq=4046449223 win=1024 ack=921915001 + RCVD (0.3020s) TCP 152.91.126.70:80 > 192.168.0.21:60182 SA ttl=53 id=0 iplen=44 seq=3924706636 win=5840 ack=4046449224 <mss 1380> + We got a TCP ping packet back from 152.91.126.70 port 80 (trynum = 0) + ultrascan_host_probe_update called for machine 152.91.126.70 state UNKNOWN -> HOST_UP (trynum 0 time: 226875) Changing ping technique for 152.91.126.70 to tcp to port 80; flags: A + In the example above, Nmap wrongly uses ACK as the preferred ping technique, when it should be SYN. [David] + o we're thinking about ways to encode the information better. Right + now we have pingseq and tryno, but we may want to just move to a + single probe ID and then we can look up any other information in + structures attached to that ID in memory when we get the response. + o A related problem, which we hope the fix for this will also + resolve, is that replies can currently match any probe whose tryno + is less than or equal to the tryno encoded in the reply. + o However, "fixing" this problem has been shown in the past to + cause accuracy problems. See + http://seclists.org/nmap-dev/2009/q1/387. We should figure out + whether we can still reproduce that and, if so, what is going on + before "fixing" this issue. + +o Add PJL (Printer Job Language) probes to + nmap-service-probes. Brandon wrote some in + http://seclists.org/nmap-dev/2009/q1/0560.html. Test them to see if + they cause anything to be printed out (on paper) with printers that + don't support PJL. If not, then remove the JetDirect ports from the + default exclude list. The script pjl-ready-message.nse also uses + PJL. We have concerns about the safety of this probe given + http://seclists.org/nmap-dev/2009/q4/61, but it still is probably + better to have the probe in there than not, as long as we continue + blocking the ports by default with the Exclude directive. + [We put in the probes, but are keeping the Exclude directives + because the probes still seem a bit dangerous] + +o [NSE] in_chksum in packet.lua doesn't work with an odd number of + bytes. Also make it more efficient. + +o Add --confdir option to Zenmap. See + http://seclists.org/nmap-dev/2009/q1/92 [David] + +o Update our Winpcap from 4.0.2 to 4.1.1 + (http://seclists.org/nmap-dev/2009/q4/128). This is a bit complex + because we have our own installer. See + https://nmap.org/svn/mswin32/winpcap/Upgrading-Instructions.txt. + +o Change Nmap to not show the "Host not scanned" lines in list scan + +o Change Nmap to show latency in "host is up" lines even w/o verbose + mode. + +o Update our included Libpcap from 0.9.7 to 1.0.0 + (http://www.tcpdump.org/) [David] + +o Improve Nmap output to show the forward DNS name when specified on + command line as well as rDNS where appropriate. We're also going to + reorganize output to enable some other improvements as well. See + the proposal at http://seclists.org/nmap-dev/2009/q3/814, and that + whole thread which starts at + http://seclists.org/nmap-dev/2009/q3/805 [David]. + +o [Zenmap] Solve some unusual utf8 Zenmap crashes reported in the + crash reporter. David has fixed some of them so far, but there are a + few more remaining that may be related. [David] + +o Change Nsock to give an error if you try to FD_SET a fd larger than + FD_SETSIZE. [Brandon] + o Some research from David: + We have help off on this change because of Windows portability + problems. The Windows fd_set works differently than the Unix + fd_set. In Unix, FD_SETSIZE (which is typically 1024) is both the + maximum number of file descriptors that can be in the set and one + greater than the greatest file descriptor number that can be + set. In other words, we want to bail out whenever someone tries + to FD_SET file descriptor 1060, for example. But on Windows it's + different: FD_SETSIZE is only 64, but any file descriptor + numbers, no matter how great, may be stored in the set. Windows + socket descriptors are typically greater than 1023, but you can + only have 64 of them in the set at once. + + So the fix on Unix would be + --- nsock/src/nsock_core.c (revision 15214) + +++ nsock/src/nsock_core.c (working copy) + @@ -97,6 +97,7 @@ + do { \ + assert((count) >= 0); \ + (count)++; \ + + assert((sd) < FD_SETSIZE); \ + FD_SET((sd), (fdset)); \ + (max_sd) = MAX((max_sd), (sd)); \ + return 1; \ + @@ -107,6 +108,7 @@ + assert((count) > 0); \ + (count)--; \ + if ((count) == 0) { \ + + assert((sd) < FD_SETSIZE); \ + FD_CLR((sd), (fdset)); \ + assert((iod)->events_pending > 0); \ + if ((iod)->events_pending == 1 && (max_sd) == (sd)) \ + + But that doesn't work on Windows (I just tried it) because even + the smallest socket descriptor is bigger than FD_SETSIZE, 64. + Really we're trying to accomplish two different things on the two + platforms: On Unix we must not store a file descriptor greater + than 1023, no matter how many or how few other descriptors have + been set. On Windows we must not set more than 64 descriptors at + a time, no matter what their descriptor number happens to be. + +o Add a way in NSE to set socket source addresses and port numbers. + See this thread: http://seclists.org/nmap-dev/2009/q3/821. Some + potential solutions are discussed later in the thread. + +o [Ncat] Fix --max-conns on Windows so that it only counts concurrent + connections and not long-dead ones. See this thread + (http://seclists.org/nmap-dev/2009/q3/1017.html) and particularly this + message (http://seclists.org/nmap-dev/2009/q3/1032.html) for + details. Venkat has a patch for David to review and potentially merge. + +o [Ncat] Fix 100% CPU usage with ncat -l --send-only. See this + thread: http://seclists.org/nmap-dev/2009/q2/797 and continues + further at http://seclists.org/nmap-dev/2009/q3/99. This message is + key: http://seclists.org/nmap-dev/2009/q3/308 [David] + +o [Seclists] There is currently some extra vertical space after the + first post of a thread in the thread index (example: + http://seclists.org/nmap-dev/2009/q4/index.html). + +o [NSE] Decide which scripts belong to the "safe" category (we now have 20 + which aren't either safe or intrusive), then remove the intrusive + category since people can now specify "not safe". See + http://seclists.org/nmap-dev/2009/q3/1091.html and that whole + thread. [Fyodor] + [ OK, see http://seclists.org/nmap-dev/2009/q4/0002.html] + +o [NSE] Fix http pipelining. Responses are being split on anything + that looks like HTTP/1.X which doesn't come at the beginning of a + line, and doesn't work when a line like that happens to legitimately + come in a body. Joao has an nmap-exp branch which resolves this + issue, though David found some bugs in that and sent some hard test + cases. [Joao] + +o Fix traceroute performance/algorithms. It is terribly bad in some + cases. For example, this traceroute scan took 36 minutes against a + single host(!): http://seclists.org/nmap-dev/2009/q3/0425.html . We + don't need to go up to hop 50 in such cases (maybe some heuristic + like "at least go to hop 15, and stop after 5 unresolved in a row). + And more importantly, there is no reason each hop should take 40s to + timeout. It should probably use timeout variables like we use in + port scanning. And it should parallelize as much as possible. Even + if parallel resolution means we went a little further than we had to + in incrementing the TTL, and we go to hop 15 when host is at 12 + that's no big deal (of course we would only report up to hop 12 in + the output). Once we do this, we should put back the ability to + make --traceroute work even when we haven't found a probe which + elicits a response from the target. (that feature was added in July, + but we'll probably take it out until we can fix + performance). [David] + +o Fix four Nmap bugs discovered by Ankur and analyzed a bit by + David. [Ankur] + +o [NSE] Consider HTTP request caching. + +o [NSE] Finish (or write new) favicon fingerprinting script. See + http://seclists.org/nmap-dev/2008/q4/0583.html . May need to do + some more scanning and increase the DB size a bit. May or may not + want to later combine this as part of a larger webapp fingerprinting + script. + +o [Zenmap] When the inventory is changed, the current host/service selection is + forgotten and the Ports / Hosts tab is switched to hosts mode. It should + remember your current selection and not change the view. [David/SoC] + +o Device categorization improvements + o Examine Nmap's device categorization in nmap-os-deb and + nmap-service-probes. Decide if some small categories which have + never really took off should be consolidated, or whether others + should be split off. For example, maybe there are some groups in + 'specialized' or other misc. categories which are now large enough + to split off. Personally, I wouldn't give anything its own + category unless there are at least half a dozen of them and no + other category really fits them well. We should use a combined + system for nmap-os-db and nmap-service-probes. + o Add a classification sect1 to os-detection.xml + (https://nmap.org/book/osdetect.html) to cover how Nmap handles OS + classification. It should include a list with descriptions of + each device type recognized by Nmap. Version-detection.xml should + reference (link to) it in the approprate place. + [Doug has done some initial work on this. For example, see + nmap/docs/device-types.txt] [David] + +o Consider what new UDP payloads we might want to add. David has many + ideas at: http://seclists.org/nmap-dev/2009/q3/0290.html + +o For traceroute we should give some indication that the RTT is in ms. + Changing the column header to maybe "RTT MS" or "RTT (MS)" would + probably do the trick or we could append "ms" to each value. + [David] + +o OS fingerprint should probably specify somewhow when DS=1 if it's + because target->directlyConnected is true, or because it sent the + distance probe and calculated a distance of 1. The second situation + should never happen, but often David strongly suspects that it is the + case. + +o --traceroute should probably set currenths->distance because right + now, I do an -O scan against scanme.nmap.org, and it does not figure + out the distance. So the fingerprint shows no distance element and + Nmap doesn't print "Network Distance" in the results line. That may + be OK (Nmap probably isn't receiving the probe response needed for + this, and maybe doesn't want to print the TG), but even when I do + --traceroute I get no distance printed. Yet Nmap clearly knows the + distance since the traceroute shows all the hops up to and including + the target (scanme.nmap.org). + +o Figure out best favicon to use for Nmap and related web sites + [David] + +o [Ncat] David says: "After you get EOF on stdin with --send-only, the + program hangs on until the idle timeout expires instead of terminating + immediately. I had a fix for it but it involved deleting events in + the Nsock queue and it caused an assertion failure in Nmap so I backed + it out. I have a less intrusive solution." [David] + +o We should update our config.{sub,guess} files. This Debian bug + #542079 requests that we do so: + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=542079. We last + updated on 3/15/08 and in that case we used versions from + http://cvs.savannah.gnu.org/viewvc/config/?root=config. That may or + may not be the best place to get them now (e.g. perhaps there has + been a recent official release). [David] + +o Look a bit more at default version detection timing. Particularly + deciding the number of probes to run in parallel. [ We increased + that a bit on 8/18/09] + +o [Ncat] Right now our -i (idle timeout) causes Ncat to quit if EITHER + reading or writing is idle for the given amount of time. But it is + really only idle if BOTH reading AND writing are idle for the + period. We should make the code work that way. + +o Add scripting.xml documentation on strict.lua and the avoidance of + global vars in libraries. See + http://seclists.org/nmap-dev/2009/q3/0169.html. Probably a new + section just above "Adding C Modules to "Nselib", such as "Writing + Your Own Library" or somesuch. [Patrick] + +o Update nsedoc to refer to 'libraries' rather than 'modules'. This + affects the front page (which calls them 'Libraries' on left sidebar + and 'Modules' on the list of right, and affects the url (we should + change /modules/ to /lib/ and then have Fyodor add a redirect for + people still using old URLs) and the title of the module pages like + https://nmap.org/nsedoc/modules/base64.html. [Patrick] + +o [Ncat] Prefix Ncat stderr messages with "Ncat: " to make it clear + that they are coming from Ncat and not the remote server (or typed in + by user). [David/SoC] + +o [NSE] Optimize NSE Performance--e.g. measure the current performance and + see what can be improved in terms of scheduling scan threads, + determining how many to run concurrently, looking at CPU load items, + etc. [David/Patrick] + +o Increase version scan concurrency based on Patrick's performance + testing. We decided to go to 20 for timing_level 3, 30 for 4, and 50 + for 5. + +o [NSE] Consider POST/HEAD support. See + http://seclists.org/nmap-dev/2009/q1/0889.html. + o Implemented: http://seclists.org/nmap-dev/2009/q3/0074.html + o Joao going to check in very soon soon. + +o [NSE] Consider Rob Nicholls http-enum script for incorporation: + http://seclists.org/nmap-dev/2009/q1/0889.html + [Joao tested w/his HEAD support, is going to check this in] + +o Consider the open proxy scripts more carefully + - How should we test whether the proxy attempt was successful? Right + now we look for a google-specific Server header after trying to + reach http://www.google.com through the proxy. Maybe we should let + users specify their own pattern if they specify their own URL. + [ Joao is going to check it in today (7/28)] + +o I should add code to Nmap to bail if sizeof(char) isn't 1. + Otherwise there could be security risks if it is not one on any + platforms. [ Actually, we think C standard requires this and we've + not heard of any system where sizeof(char) isn't 1. So removing + this item.] + +o [Zenmap] More complete implementation of ZenmapCommandLine/profile + editor improvement ideas. See + http://www.bamsoftware.com/wiki/Nmap/ZenmapCommandLine. [David] + +o [Ncat] Think about whether we should offer "-q secs" (quit after EOF + + delay of secs) and/or -k (set SO_KEEPALIVE on socket) (or maybe + that should be set by default). Anyway, these were suggested here: + http://lwn.net/Articles/341706/ [We're going to fix -i (added + separate item), and not worry about SO_KEEPALIVE unless we see more + demand for it. It doesn't seem that nc110 or OpenBSD nc or so-called + GNU Netcat support SO_KEEPALIVE either] + +o [Ncat] In verbose mode, I'd like to see clock time (duration) and + maybe in/out traffic stats when a client connection ends. Maybe it + could use a format similar to what Nmap provides. [David/Venkat] + +o Seriously consider making --traceroute work even when we haven't + found a probe which elicits a response from the target. We'd just + have to pick a probe in that case (probably echo request, as we + found that to be the most effective in prev. empirical testing). + This is similar to UNIX traceroute and Windows tracert.exe which + just pick a probe (high UDP port on UNIX, ICMP echo request on Win). + Even if the host is down or something, we usually get some useful + hop information. + +o [NSE] Allow spaces in script arguments without the user having to + manually quote them (beyond normal shell escape quoting). See: + http://seclists.org/nmap-dev/2009/q3/0090.html + [Patrick] + +o [Ncat] Support SCTP now that Nmap does. + - See client support patch by Daniel Roethlisberger: + http://seclists.org/nmap-dev/2009/q2/0609.html + - Server support? + - Daniel has a patch, David looking to apply once an nsock thing is fixed. + +o Look at etc/payloads.conf in unicornscan-0.4.7 and see if they have + any which we don't have, but should, for our version detection. + They have a decent collection there. KX sent some other programs we + should look at too. [David] + +o Ncat should give it's ethernet cat ASCII logo after + configure--similar to the way that Nmap, Ncrack, and Nping + do. [David/SoC] + +o [Zenmap] The Search dialogue is helpful for finding a certain scan + you've performed recently, but we should probably also offer a similar + function for searching for certain applications/hosts within a scan + (e.g. find all the hosts running Apache). This new functionality + might be a find option or some other mechanism rather than being + part of the Search dialogue proper. + +o Ncat SSLv2 issues. See + http://seclists.org/nmap-dev/2009/q1/0319.html. A big part of it is + done, which was enhanced version detection probes to detect more SSL + servers, The defect that remains is that Nsock can't connect to a + small fraction of servers (including some of the ones detected by + the new version probe). They are the servers that do only SSLv3 or + TLSv1 and don't respond to a SSLv2-compatible ClientHello. Even + though most servers don't support SSLv2, they usually respond to the + ClientHello and just don't offer any SSLv2 features. [David/Venkat + working on this] + +o Deadlock identification and correction: + o Plan of action: implement freeing of script mutexes when scripts + exit without freeing them (done and in /nmap now). And then if it + continues to be a problem we'll consider this other stuff: + o Add detection for deadlocks and print which threads are involved. + o use above results to make a strategy for automatic deadlock resolution. + o Original entry: Figure out what to do about NSE mutexes: + http://seclists.org/nmap-dev/2008/q3/0276.html . In particular, they + are not currently cleaned up if a thread dies or otherwise exits + without unlocking them and can cause endless deadlocks which are + annoying to users and can be difficult to debug :(. Patrick has + some ideas for this in his SoC09 proposal: + "Adding a cleanup system for NSE that is called periodically + similar to nsock_loop. There would be a registration system + allowing C libraries to register a Lua function that will run + periodically to check for irresolvable deadlock or simply dead + resources. For example, the nmap library would register a mutex + cleanup handler which would inspect all mutexes looking for a dead + thread or circular dependencies. The nsock library could register + a handler that checks for unused sockets. The nsock may save a + strong reference to the thread that owns the socket and inspect it + to determine if the thread is dead." + David later says: "After some discussion we decided to start more + modestly, first by ensuring that a scripts mutexes are released when + it dies for whatever reason. I have a hunch that this is the cause + of most deadlocks. It was certainly the cause of two whois.nse + deadlocks I found. Then, the next step if deadlocks continue to be a + problem, is to do automatic detection and just print out a list of + what scripts are involved. It could be that several smb scripts are + deadlocked, or as in the case I observed where whois.nse was locked + with itself." + +o Joao is auditing his Lua code to make sure all his variables are + local where appropriate. [Joao - done, should be commited very soon] + +o [NSE] We need to deal with libraries which improperly use global + variables, as that is very common (Patrick made a list: + http://batbytes.com/bad.txt). Solutions could involve augmenting + our runtime system (the "strict.lua" approach) to detect/prevent the + problem, a script we run occasionally to identify issues that we + then manually resolve, or, at the very minimum, documenting + somewhere in scripting.xml the dangers inherent in global variables + and warn people to generally declare them local instead. We have a + long history of bugs caused by non-local variables defined in NSE + libraies and often causing deadlocks. + +o The Nmap refguide (https://nmap.org/book/man-performance.html) says + "The --max-parallelism option is sometimes set to one to prevent Nmap + from sending more than one probe at a time to hosts. This can be + useful in combination with --scan-delay (discussed later), although + the latter usually serves the purpose well enough by itself." But + when you actually try it: + # ./nmap --max-parallelism 1 --scan-delay 10 scanme.nmap.org + You can't use --max-parallelism with --scan-delay. + QUITTING! + We need to either make that work or adjust the documentation. [David/SoC] + o David changed this to a warning. Note that with --scan-dealy, + --max-parallelism is essentially 1 anyway. + +o [NSE] Consider integrating HP Laserjet print PJL status-setting + script. See this thread for an example of such a script: + http://seclists.org/nmap-dev/2009/q3/0083.html (note that it is + updated during the thread). Also, see this thread: + http://seclists.org/nmap-dev/2009/q3/0092.html + +o Ndiff man page should be expanded to include sample execution/output + and more fully describe its functionality. [David] + +o David is going to reexamine the old coverity-reported issues (the + ones we previously marked as "ignore" because they weren't real bugs) + just to be sure that is (and is still) the case. + +o Make -sP work with -PN to disable both port and ping scanning. We + need to make sure the various options still work (-O, --script, + --traceroute, etc.) with this, as many currently don't as they don't + expect this behavior, which used to be unsupported and cause Nmap to + quit with an error messaqge. It may be OK to refuse -O since that + will rarely give useful results. OTOH, -O may work on some systems + with unique closed port signatures where Nmap guesses a closed + port. Users should then be able to do an NSE-only scan with "-sP -PN + --script [scripts]" We should document this -sP -PN usage in + refguide. [David] + +o Add -sn and -Pn options which are aliases for -sP and -PN. Once + they've been around long enough to be in most people's copy of Nmap, + we plan to document those as the preferred version. Those match -n, + and the main problem with -sP is that we now use it more for + "disable portscan" than ping only. For example, you still might + want to use NSE. [David] + +o [NSE] Make sure all our HTTP scripts transparently support SSL + servers too. [Joao has a solution and is testing the http scripts to + make sure they don't break.] + +o Resolve "memcpy overlap in getinterfaces(int*) (tcpip.cc:2987)". + See this thread: http://seclists.org/nmap-dev/2009/q2/0713.html + [David/Brandon] + +o [Ncat] Print a message to stderr upon connection failure even if -v + isn't specified so the user knows what went wrong. [David/SoC] + +o [Ncat] Maybe --chat should imply -l. And Maybe --broker should too? + - OTOH, we might want to extend --chat for connect mode in the + future. + [We're going to hold off on chat now, David/SoC is doing --broker] + +o Consider making it easier to tell whether scripts were specified by + name on the command-line (rather than default or by class) so they + have the option of providing extra verbosity in that case. For + example, see http://seclists.org/nmap-dev/2009/q2/0563.html. We + could either provide a special function for scripts to determine + that, or we could magically adjust nmap.verbosity() when called by + those scripts. [David] + +o [NSE] Figure out a way to support people who want to do script scan, + but not port scan or ping scan. One option would be to allow + --script to list scan (-sL), but perhaps a better option is to + provide a way to disable port scanning in the same way as we offer + -PN to disable ping scanning. As an example of this need, David had + to write special code to avoid ping/port scanning when doing a + whois.nse survey for + http://www.bamsoftware.com/wiki/Nmap/EffectivenessOfPingProbes. The + key for this task is to figure out how to do it from a user + interface perspective and then implement and document it. We've + already been going in the direction of allowing script scanning in + more types of scans--a while back we started allowing it with -sP + ping scans due to high demand. [David/SoC] + [ We decided how we're going to do it (-sP -PN to start out with; + leading to eventual -sn -Pn) and added new TODO entries for actually + doing the code/docs. ] + +o Ndiff should be able to show NSE script result changes. [David] + +o Get set up for Coverity scan of latest version to see if it catches + any important issues before stable release. [Fyodor,David] + [Found 7 new results, 3 are real bugs, and 2 have been fixed so far] + +o [nsock] Fix Makefile to handle dependencies correctly (if that turns + out to be the problem). See + http://seclists.org/nmap-dev/2009/q1/0629.html. o Or it may be + related to SVN timestampling. See + http://seclists.org/nmap-dev/2009/q1/0632.html. Diagnosed by David: + http://seclists.org/nmap-dev/2009/q2/0728.html + +o For at least our UDP ping probes, Nmap should probably notice if it + is a very well known service port such as 53, 161, or 137 and send + an appropriate probe packet (server status for DNS, public community + string query for SNMP, etc) rather than empty data in that case. + This is similar to the way our IP protocol probes automatically + include common headers such as TCP and UDP if that common protocol + is given. Good probes for these services are already available in + nmap-service-probes, though we might want to make a custom file for + this. We should probably do this for port scanning as well. [David] + +o [NSE] Make NSE work better for SSL tunneled services in general by + supporting them easily in the libraries. For example, I don't think + irc-info.nse currently works against all the servers which tunnel + over SSL. Maybe augment comm library, etc. [Joao - done, except for + http, which is already a separate TODO item] + +o Update scripts which use table args to use pseudo-table format + "name.arg" rather than requiring the user to create a Lua table + themselves. On the lua side, it's not really being stored in a + table, but just an arg named "name.arg". [Joao] + - Look at all our existing scripts which use tables + (dns-zone-transfer, whois, the proxy scripts, etc.) and change as + appropriate. Remember to change the usage throughout the script + and also change the nsedoc script arguments and example usage. + For the existing scripts, try to retain the table version check + for now to avoid breaing backward compatability if possible. Just + add the newer style check as well. + - Is taking arguments in a table specific to a script a good idea? + The example in the socks-open-proxy nsedoc of "--script-args + openproxy={host=<host>}" is a bit of a mess and I'm not sure the + best way to document that in the script argument list. Note that + this is the standard way we've handled it for some other scripts, + so it's not an open-proxy-script-specific problem. + +o [NSE] Track active sockets in the nsock library binding and don't + rely on garbage collection for reallocation. Can probably wait until + post-stable release for integration. [Patrick] + - Patrick has a patch and is waiting on dev branch to check it in. + +o [NSE] Resolve ssh2.lua buffering problems + (http://seclists.org/nmap-dev/2009/q2/0673.html) [Joao] + +o Decide what to do about ncat source code headers -- maybe just use + the Nmap ones. [David added the Nmap headers] + +o Once we go into deep stability freeze mode, create an nmap-exp + development branches for changes we plan to integrate after the + stable release. [Fyodor] + +o Update CHANGELOG for latest changes [Fyodor] + +o Release 4.85BETA10 + +o [NSE] Open proxy detection scripts + o We have http-open-proxy.nse, but we should probably either extrand + that to handle other types of proxies (such as SOCKS and HTTP + CONNECT) or create more scripts to handle those other proxy + types. [Joao, David] + o Joao has written scripts, just need to finish up, evaluate, integrate. + +o Determine whether zenmap.spec.in can currently require + "python-sqlite" rather than "python-sqlite2", or if it at least can + be easily made to do so. The former seems more compatible since + RHEL/CentOS 5.3 has a "python-sqlite" package, but not + "python-sqlite2". Meanwhile, Fedora 10 provides the "python-sqlite" + capability as long as you have the Python 2.5 package installed + (python-2.5.2-1.fc10). Fedora 10 does also make a + python-sqlite2 package available. + +o [Ncat] Solve EOF issues which crop up when piping to an external + command. See http://seclists.org/nmap-dev/2009/q2/0528.html. It + sounds like we will go with Daniel's patch [Daniel, David] + +o Look into building RPMs with SSL support. Statically linking to + OpenSSL on Linux for the RPMs didn't work for me last time I + tried. [Fyodor] + o Static linking of Nmap to OpenSSL does not seem to work on Fedora + 10 or CentOS 5.3. The problem appears to relate to the OpenSSL + krb5 support. + o Could build my own OpenSSL libraries on the build system + (w/o Kerberos support) and link to those. + o At some point, we might want to consider including OpenSSL with + Nmap tarball. The problem is that it is rather big. Would + increase Nmap .tar.bz2 size from about 9 megs to about 12. OTOH, + OpenSSL is only going to get more and more important. Maybe we + can include a stripped down version? + o If we don't integrate OpenSSL (or until we do), we might consider + a more prominent configure warning for when SSL is not detected. + We could suggest that users run "yum install libopenssl-devel" or + "apt-get install libssl-dev" commands or whatever is appropriate + and then reconfigure. Or we could point them to a page or + nmap-dev posting URL with instructions. + +o Figure out why I [Fyodor] get a bunch of "Operation not permitted" errors +when I launch a scan on SYN such as: + - I'm going to ignore this for now unless it causes me trouble + again, as this is an old machine that will be replaced soon anyway. + And we haven't been hearing of the problems from others lately. + /home/fyodor/nmap-exp/fyodor-perf/nmap -nogcc -T4 -n -v -p- --portpingfreq 250 -oA /home/fyodor/nmap-misc/logs/WorldScan/portpingfreq/logs/portpingfreq-250-1%T-%D 67.15.236.34 67.15.236.36 81.174.236.66 81.174.236.119 170.140.20.160 170.140.20.174 202.138.180.9 202.138.180.17 202.138.180.132 209.20.64.112 + The errors look like: +sendto in send_ip_packet: sendto(7, packet, 44, 0, 170.140.20.174, 16) => Operation not permitted +Offending packet: TCP 64.13.134.4:59820 > 170.140.20.174:59120 S ttl=39 id=19927 iplen=44 seq=2425535549 win=4096 <mss 1460> +sendto in send_ip_packet: sendto(7, packet, 44, 0, 67.15.236.36, 16) => Operation not permitted +Offending packet: TCP 64.13.134.4:59820 > 67.15.236.36:15030 S ttl=57 id=50640 iplen=44 seq=2425535549 win=2048 <mss 1460> +Discovered open port 49394/tcp on 170.140.20.174 +sendto in send_ip_packet: sendto(7, packet, 44, 0, 170.140.20.174, 16) => Operation not permitted +Offending packet: TCP 64.13.134.4:59819 > 170.140.20.174:8256 S ttl=48 id=38510 iplen=44 seq=2425601084 win=1024 <mss 1460> + May be related to connection tracking and high scan rates. See + http://seclists.org/nmap-dev/2008/q4/0652.html + http://www.shorewall.net/FAQ.htm#faq26 + Others have reported similar issues even without connection tracking. See + http://seclists.org/nmap-dev/2006/q3/0277.html + http://seclists.org/nmap-dev/2007/q2/0292.html + + +o -PO1 and "-sO -p1" seem to send ICMP ping packets with an ICMP ID + field of 0, which we found that a small percentage of hosts drop + (61.13% responded with 0, 62% with a random value). So we might as + well randomize them in these cases. [Josh Marlow] + +o Some of the -PS443 scans (and maybe other ones) we've been running + have been missing the Nmap line telling how many packets were + sent/received, even though we had verbose mode. [David/Josh] + +o Deal with Ncat newline problem. See this thread: + http://seclists.org/nmap-dev/2009/q2/0325.html [David,Jah] + +o Integrate SCTP scanning support. See Daniel Roethlisberger's branch + in nmap-exp/daniel/nmap-sctp. As of 4/30/09, he is nearing + completion. See http://seclists.org/nmap-dev/2009/q2/0270.html. + +o [NSE] Release mutexes upon script death to prevent certain deadlocks + [Patrick, David] + +o Consider whether to let Zenmap Topology graph export the images to + svg/png/etc. Also think about printing. Note that João Medeiros + has written a Umit patch to do this: [Joao, David] + http://trac.umitproject.org/ticket/316. + - Now he has Nmap patch: + http://seclists.org/nmap-dev/2009/q2/0409.html + - Consider integrating. + - Integrated! + +o Ensure that when I build a distribution package on UNIX (e.g. make + distro), it builds what is in the Nmap directory I am calling it + from rather than a particular SVN version. I'm going to start + building packages from a special "clean" directory which is + different than the one I do development work in. Also, I want to be + sure that any changes in that dir are included in the release, even + if they aren't check in yet. [Fyodor] + +o Nmap UNIX distro build script should regenerate script.db. [Fyodor] + o Now it is in make prerelease + +o Nmap build system should be split into [Fyodor] + o prerelease -> generates version files, man pages, script.db + etc. That has to be done on one system, and then results checked in + before doing a make release. It does this stuff based on the + directory it is run in rather than some set dirname or a pure SVN + version + o release-tarballs -> does any system-dependent building and creates + the source tarballs. It does this stuff based on the directory it + is run in rather than some set dirname or a pure SVN version + o release-rpms -> Same as above, but also uses the created tarballs + to build the Linux RPM binaries for the current platform based on the + tarballs. + +o Build x86 and x86-64 VM instances for RPM building. [Fyodor] + * I think I'll use CentOS 5.3 + +o [NSE] Script scanning does not seem to work on Fyodor's Linux + machines after being installed from latest SVN (or 4.85BETA9) and run + as a non-root user (it works fine as root). The command "nmap -sC + localhost" leads to NSE failure messages which differ based on the + exact version run. [Was a relatively simple permissions problem in + our Makefile.in -- I fixed it] + +o [NSE] Release socket locks on connection failure or + timeout. [Patrick] + +o Update Nmap entry on Linux Online - + http://www.linux.org/apps/AppId_1979.html + - Screw it, the site does not seem to be maintained at all. They + aren't taking updates as of 6/2/09, and even Firefox shows latest + update as 0.9.1. + +o [Ncat] In verbose mode, print when an SSL connection is established + successfully and give the leaf certificate hash to make it easier to + verify when connecting to a machine where you can't or don't want to + use --ssl-verify (e.g. connecting to an ncat ssl server where it + created its own key). While we're at it, we might want to print + some other information from the leaf node, such as organizationName + and maybe localityName, countryName or something. We don't want to + be too verbose, but 1 line would be great and 2-3 might be + acceptable. [David] + +o Fix NSEdoc to better escape single-quotes in fields. If we can't do + that for some reason, we need to document it better. For example, + when we initially tried generating nsedoc for + http-webdav-unicode-bypass.nse, NSEdoc was listing it as a module + named "s auxiliary module", apparently because this line exited in + the description field: + This module is based on Metasplit's auxiliary module, modules/auxiliary/scanner/http/wmap_dir_webdav_unicode_bypass.rb. + (For full example, see scripts/http-webdav-unicode-bypass.nse + r13345) [David/SoC] + +o --script-args should allow a wider range of characters, and should + give a more useful error message if it receives chars it really + can't handle for some reason. For an example, try + "--script-args=smbuser=admin,smbpass=pass^word". For more details, + see Ron's report at + http://seclists.org/nmap-dev/2009/q2/0378.html. + +o [Ncat] Have --ssl-cert and --ssl-key send a certificate in connect + mode so that client certificate auth can be done. [David/Venkat] + +o Once we're done with host discovery empirical research, add it to + host-discovery.xml. Would be great to show the best combinations to + use for a given number of probes, the efficiency of the common probes + by themselves, etc. + +o Consider making the ping scan default be more comprehensive. Note + that I got 23% more Internet boxes found out of a 50K sample (see host + enumeration chapter of my book for details). Maybe I should + experiment a bit more to ensure they are real boxes and not network + artifacts and figure out exactly which tests are helping the most. + If I do this change, I'll have to update the host enumeration + chapter. For UDP probing purposes, we should test whether including + extra data in the packet (e.g. --data-length) helps in general, and + for services such as 53 and 137, we should probably send proper + protocol headers (e.g. a DNS server status message) so that we + receive responses from listening services. + +o We should probably check for a system Lua in a "lua5.1" directory + rather than just "lua", as Debian and also my Fedora 10 systems seem + to have that. See + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=527997. [Note, + Fyodor asked the bug reporter Jan Nordholz on 5/14/09 if he could + write a patch. Jan sent in a patch, it worked, Fyodor checked it in.] + +o [NSE] Get rid of ceil so that floating point NSE runlevels work + again (some scripts, including (smb-brute) rely on this. They got + broken with the NSE core lua rewrite. [David]. + +o NSE script logical operator stuff is now documented in + scripting.xml--add to refguide.xml as well. [David/Patrick] + +o [NSE] Correct nsock_connect to unlock the socket slot if the + connection fails. When a socket is closed, it is unlocked so the + arbitrator can potentially open up a socket for another thread. But + Patrick discovered that a socket is not automatically unlocked when + a connection fails or times out, only when it is closed + explicitly. So that could hold up socket allocation for other + threads until garbage collection. May be a cause of slowness or + possibly deadlocks. [Patrick] + +o [NSE] Solve segfault issue which occurs when Nsock events call back + on a thread that has already ended (e.g. timeout, crash, early exit, + whatever) and been garbage collected. May want to just nsi_delete + all nsock sockets immediately upon thread ending. For an example of + this type of segfault, see + http://seclists.org/nmap-dev/2009/q2/0289.html. David says " I think + in the interests of getting this in a stable release, we should use + that strategy of closing all a thread's sockets. That ought to fix + all the problems above. Not to rule out a more thoughtful redesign + in the future." [David,Patrick] + +o We added the SEQ.CI value in Feb 2009 with 0 matchpoints. At some + point (once we have some real-life values) we need to evaluate whether + we want to give it points. A good time to do that would be when we + next do fingerprint integration, so we will actually have examples + of .CI in the nmap-os-db. [David] + +o [NSE] Make it a warning rather than error if a script in script.db + can't be found. [Patrick] + +o Add version detection signature for Ncat chat once we finalize the + announce format. [David] + +o Change Nmap signature files to use the .sig extension rather than + .gpg.txt, as that seems to be what gpg recommends. In fact, gpg + will automatically verify the right file if it exists after dropping + the .sig (or .asc) extension. I may need to configure .htaccess to + serve .sig files properly. Update nmap-install.xml + accordingly. Suggested by tic at eternalrealm.net by email on + 7/13/08. [Fyodor] + * Rename existing files, add symlink from the old .gpg.txt to .asc + versions + * Add appropriate .htaccess content type if needed for downloads + - not needed since I decided on .asc extension rather than .sig + * Update the generation scripts + * Update the book documentation - + https://nmap.org/book/install.html#inst-integrity + +o Ask Coverity if they'll scan latest version of Nmap. [Fyodor asked + David Maxwell on 5/14/09 ] + +o Make 4.85BETA9 release [Fyodor] + +o [Zenmap] Make a way to start a scan from the profile editor without + creating a profile, then remove the command wizard. This is partial + implementation of + http://www.bamsoftware.com/wiki/Nmap/ZenmapCommandLine. [David] + +o [Ncat] Make proxy server mode work on Windows (this is the last + remaining fork() dependency in Ncat). + +o Do an OS detection integration run -- last was based on + 1/8/09. [David] + +o [Ncat] Maybe we should create an SSL cert with no passphrase during + Ncat compilation or install process so that if someone specifies + Ncat -l and --ssl with no --ssl-cert and --ssl-key, we already have + one for them, and it is a slightly better one (since the private key + isn't known) than if we distributed a key. Obviously it is still + subject to MITM attacks since there is no domain validation going + on. But people who need that will have to buy a key from a + certificate authority in any case. We could create the key by using + the "openssl" command line tool as shown in + https://nmap.org/ncat/guide/ncat-advanced.html#ncat-ssl, or maybe + better to have a way for ncat to do it using openssl calls. [David] + +o [Zenmap] Should probably give some sort of widget indication that a + scan is running. Now that we can start multiple scans at once, the + "scan" button goes back to being unpressed while the scan is + running. As some scans take minutes or more to show output, it is + not always clear whether they are still properly running. We should + probably have some sort of widget, such as the throbber used in web + browsers, to show that Nmap is still running. It could be fore a + specific scan (kind of like how you have a separate throbber for + each tab on a web browser), or a global one which means at least one + scan is running. Or maybe a different sort of indication is in + order (like a timer). [David] + +o Further investigate Nmap Proxy patch by Zoltan Panczel and Ferenc + Spala. See http://nmap-dev.fw.hu/ and + http://seclists.org/nmap-dev/2009/q1/0255.html . [Discussed it and + then added new proxy feature item] + +o Wherever practical, fix compiler warnings when compiling Nmap with + VC++ 2008 Express SP1 (there aren't many). [David] + +o [NSE] Consider adding boolean expressions to --script arguments. For + example, see Patrick's implementation at + http://seclists.org/nmap-dev/2008/q3/0300.html . + +o Generate a list of trusted SSL certificates to ship with Ncat (by + extracting f rom Mozilla or similar), and install them with + Ncat. Decide how these certificat es should be preferred to any + system-provided certs, if any. [David] + +o [NSE] Add desired SoC09 infrastructure ideas to this TODO to the + extent they don't already exist. + +o [Ncat] Consider supporting server certificate verification when used + in client SSL mode. + o For now we document in user's guide that it is not secure. + o Maybe we can do an ssh-style approach where we just print the + fingerprint and expect the ncat client user to ensure it is the + right one? + o If we're going to verify cert's etc., we need to also make sure we + are actually using secure ciphers. We may need to update nsock to + support cipher selection, because we want fast ones for version + detection, but usually want secure ones for NSE and/or ncat. + o Do we want to check all this by default, or offer an option for + it? Doing it by default is more secure, though it can be annoying + when a certificate has expired, is self-signed, you connect to + domain.com when the certificate is for www.domain.com, etc. If it + is done by deault, we might just print an error message. Whreas + if we have a special option, it may be OK to exit and refuse the + connection. + o What certs should we allow? Same as the browsers do? Maybe get + rid of Comodo? Maybe we should fail to recognize any certs with MD5 + in the trust chain? + o What about people who are running their own SSL service and just + want to specify the cert file they use, because they generated it + themself and not from a trusted CA. + o Need to check expiration, domain, etc. if we're checking certs at + all. + o We can probably get away with not doing revocation checking, as + long as we document that we don't. + +o consider changing status field from "up" and "down" to "online" and + "offline". Actually, maybe we don't want this after all. + online/offline look pretty similar, and they're longer too. I'm + taking this out of the TODO. + +o [Ncat] When acting as an HTTP proxy, we should support GET mode as + well as CONNECT so that it works as a non-SSL proxy in browsers such + as firefox. [David] + +o Finalize GSoC applicant research, communication, and selection + [David, Fyodor] + +o Go through all the SoC applicants and decide who we want to accept + and start communicating with them. [David,Fyodor] + o Decide which applicants we want, and who would be best for + mentoring them. + +o Document that U1.RID gives "G" as long as all the data bytes in the + echoed response data are "C" as expected. This G code is still + given even when the response is truncated, including if there are 0 + bytes echoed. [David] + +o [Ndiff] Rethink the output format. David says: In particular, I + would like to always have the old state on the left and the new + state on the right: "was filtered, is open," not "is open, was + filtered." I also like the context diff output of MadHat's + nmap-diff. [David] + + +o Canonicalize the "host up" messages for port scan and ping scan so + that instead of things like "Host scanme.nmap.org (64.13.134.52) + appears to be up ... good." we standardize in both cases on + something like: "Host scanme.nmap.org (64.13.134.52) is up (.75s + latency)". Note the addition of the latency value, which is our + srtt value for the host. This will only show in ping scan and + verbose port scan because the line doesn't appear without verbose + mode. [David] + +o Ping scans always seem to say "0 [hosts] undergoing Ping Scan" when + you request stats, rather than the proper number. For an example, + try a command such as "nmap -iR 10000 -sP -n" and then press enter + during the scan. Here are some examples of the bad output: Stats: + 25:34:33 elapsed; 991232 hosts completed (72530 up), 0 undergoing + Ping Scan Ping Scan Timing: About 53.69% done; ETC: 22:49 (0:00:09 + remaining) Stats: 0:01:10 elapsed; 0 hosts completed (0 up), 0 + undergoing Ping Scan Ping Scan Timing: About 24.03% done; ETC: 22:42 + (0:03:41 remaining) Stats: 0:03:28 elapsed; 4096 hosts completed + (284 up), 0 undergoing Ping Scan Ping Scan Timing: About 3.06% done; + ETC: 22:44 (0:03:07 remaining) [David] + + +o Remove obsolete tests from nmap-os-db itself. [David] + +o Prepare for Summer of Code + * Brainstorm for ideas + * Create new ideas page + * Apply to participate in program again + * Advertise for applicants + * Evaluate applicants + +o NSEDoc script/module documentation pages should probably provide a + link to the script/module source code (except for C modules). The + link format should probably be of the form + https://nmap.org/data/scripts/[script].nse and + /data/nselib/[module].lua. NSEdoc can assume they already exist + there, as we'll probably put them there using the same system we use + to copy other stuff to the data dir. + +o [Ncat] Let people set up authenticated proxies using + --listen and --proxy-auth together (right now we don't support + that). [David] + +o When you specify multiple comma-separated arguments to --script, + those arguments seem to get lost when the Nmap command is printed in + Nmap's output files. For example, I run the command: + nmap -oN - --script=discovery,intrusive scanme.nmap.org + The output includes: + # Nmap 4.85BETA4 scan initiated Thu Mar 26 15:40:05 2009 as: ./nmap + -oN - --script=discovery scanme.nmap.org + Note the missing ",intrusive" in the script argument. [David] + +o Merge patrick/nse-lua-merge for easier-to-maintain and simpler + codebase once David and Patrick are happy with it. [David] + +o SVN check out /nmap as an external in a directory named svn or src + or nmapsvn or something under nmap.org web tree. Then redirect the + individual nmap.org/data/ files, where needed, to the nmapsvn + instead. and update nmap-dev Makefile not to copy them to the + /data/ dir anymore. Then update the nsedoc system to generate proper + links to the new script/nselib locations. [Fyodor] + +o Improvements to presentation of version detection + information. [Brandon] + o Allow longer strings. Right now it can be 128 chars for the + fullversion info, I think. But that isn't enough for this useful + information-packed string: "Apache httpd 2.0.52 ((Red Hat) + mod_perl/1.99_16 Perl/v5.8.5 DAV/2 mod_jk/1.2.19 PHP/4.3.9 + mod_python/3.1.3 Python/2.3.4 mod_ssl/2.0.52 OpenSSL/0.9.7a)". + After discussion w/Brandon, we're going to allow 160 chars total. + o Instead of omitting all information when version info string too + long, we're going to truncate and allow 157 characters, plus + ellipses (...) + o Brandon says: "my final gripe is that the full version string is + constructed as <product><space><version><space>(<extrainfo>). + but, even if product or version are blank, the spaces are still + there" + +o I need an output-autoflush option of some sort. This could be + useful to ensure I get all the --packet_trace and debug data before + Nmap crashes. Actually, I'm not sure that is so critical. + o Killing it for now, not sure that it even is needed. + +o Fix the directory function(s) in nse_fs.cc to be usable by scripts and + improve flexibility. [this entry added by Patrick] + +o [Ncat] The sys_wrap.c/.h code contains a whole bunch of capitalized + versions of system calls (Fork(), Socket(), Sscanf(), etc.) which + are mostly the same as the standard version except that they cause + ncat to quit if they are triggered. They also may be used partially + for portability. The main issues are: + 1) Because the function quits in the case of errors, it doesn't + always have the context to print a useful error message (and + even when it does, it often doesn't -- for example Fopen could + print the filename, but doesn't.) Also, sometimes these + functions are called when quitting really isn't the desired + outcome of an error. + 2) Some could be replaced by code in nbase, for example, Malloc + basically does the same thing as our safe_malloc already used + throughout Nmap. + So we should probably consider simplifying/removing this code to the + extent possible. But we need to remember to add error detection to + the callers where necessary rather than blindly switching from + (e.g.) Connect() to connect(). [Kris or David] + +o With --version-trace (may be a problem with other uses of nsock + tracing too), I often get dozens of "wait_for_events" reports in a + row in a very short period, flooding the logs. For example, with + the command "nmap -sV --version-trace www.google.com", I get: + NSOCK (22.3570s) Callback: WRITE SUCCESS for EID 283 [74.125.19.147:443] + NSOCK (22.3570s) msevent_delete (IOD #4) (EID #283) + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + NSOCK (22.3570s) wait_for_events + [Goes on for pages] + +o NSE memory issues (and gh_list assert failure) [David] + o See this thread: http://seclists.org/nmap-dev/2009/q1/0532.html + o We're taking this out for now since the new nse-lua-merge + tenatively looks like it fixes this. + +o [Ncat] Why does Ncat require enclosure in a while loop to answer + repeated UDP queries, but not TCP? For example, see the "Emulating + Diagnostic Services" section of the Ncat user's guide. + o Note: http://seclists.org/nmap-dev/2009/q1/0133.html + +o Determine what we should do about the IE.DLI OS detection test [David] + o All of the 1656 results for this test in nmap-os-db are DLI=S. + o Is the test not working right (producing the proper results + against targets), or is it just a generally useless test for + which virtually all targets respond the same way? + o Are there other "useless" tests in nmap-os-db? It is worth + checking, IMHO. + o We're going to get rid of IE.DLI, IE.SI, U1.RUL, and maybe TOS and + TOSI tests. + +o When you do ncat -h, Ncat should probably show the Nmap version + number rather than (currently) 0.2. Also ncat in -v mode should + show that same header. [David] + +o Ncat verbose mode (-v) should probably only give important messages, + such as perhaps a message once you connect successfully to a port, + or a message if the connection attempt times out. An Ncat version + banner (with URL) like Nmap has might be warranted (in verbose + mode). Currently, Ncat floods you with (mostly) useless debugging + information like this with a single -v (this output, on the other + hand, might be useful for a debugging option): [David] + # ncat -C -v scanme.nmap.org 80 + NSOCK (0.0000s) TCP connection requested to 64.13.134.52:80 (IOD #1) EID 8 + NSOCK (0.0200s) Callback: CONNECT SUCCESS for EID 8 [64.13.134.52:80] + NSOCK (0.0200s) Read request from IOD #1 [64.13.134.52:80] (timeout: -1ms) EID 18 + NSOCK (0.0200s) Read request for 0 bytes from IOD #2 (peer unspecified) EID 26 + GET / HTTP/1.0 + NSOCK (4.4280s) Callback READ SUCCESS for EID 26 (peer unspecified) (15 bytes) + NSOCK (4.4280s) Write request for 16 bytes to IOD #1 EID 35 [64.13.134.52:80] + NSOCK (4.4280s) Callback: WRITE SUCCESS for EID 35 [64.13.134.52:80] + NSOCK (4.4280s) Read request for 0 bytes from IOD #2 (peer unspecified) EID 42 + For comparison, here is what Eric Jackson's nc (The nc available in + Fedora 10's package repository) shows in verbose mode for the same + connection: + # nc -v scanme.nmap.org 80 + Connection to scanme.nmap.org 80 port [tcp/http] succeeded! + GET / HTTP/1.0 [David] + +o Final polishing of our GSoC pages. [Fyodor] + +o Advertise widely for Nmap GSoC applicants [Fyodor] + +o [Ncat] We should (maybe) consider a way for people to choose + usernames in --chat. + o Removing this for now. We can add it back if we decide we really + want this. + +o Deal with new Python 2.6 Zenmap build warnings: + C:\Python26\lib\site-packages\py2exe\build_exe.py:16: DeprecationWarning: the sets module is deprecated + import sets + http://sourceforge.net/tracker/index.php?func=detail&aid=2314799&group_id=15583&atid=115583 + [Bug in py2exe, will probably be fixed with a new version of py2exe + once it is released and we upgrade. This isn't causing us any major + problem anyway.] + +o When I scan large groups of hosts with OS detection enabled, I get + groups of warnings like: + Insufficient responses for TCP sequencing (0), OS detection may be less accurate + Insufficient responses for TCP sequencing (0), OS detection may be less accurate + Insufficient responses for TCP sequencing (0), OS detection may be less accurate + Insufficient responses for TCP sequencing (0), OS detection may be less accurate + Insufficient responses for TCP sequencing (0), OS detection may be less accurate + Note how it doesn't even tell the relevant IP address, and it isn't + included in an individual host section. We should probably either + include it in the section for an individual host, like we do with + "OSScan results may be unreliable because we could not find at least + 1 open and 1 closed port", or (not quite as + good) include the relevant IP address in the error message. And we + may or may not want to require verbose mode. + +o Ncat chat should bomine the "already connected" user list into one + line, like: + <announce> already connected: 69.232.238.42 is connected as <user5>, 206.81.65.43 as <user4>, 69.232.238.42 as <user6> + +o [Ndiff] Maybe Ndiff should display changes to version detection and + OS detection information? [David] + o Version detection done, now just needs OS detection. + +o When I start ncat chat with this tcsh command: + ncat -l --chat scanme.nmap.org < /dev/null >& /dev/null & + The first client to connect to the chat becomes user0 and doesn't + work quite right. Messages user0 type get transmitted to other + clients, but user0 does not see their messages. Nore does user0 get + the normal connection announcement upon connecting. If I quit + user0, the next client to connect becomes user0 again and has the + same problem. If I start ncat on the server with "ncat -l --chat + scanme.nmap.org" (no redirection), other clients can connect with no problems. + +o Ncat --chat should probably announce to everyone (including the new + person) when someone connects. This tells the new person their + username, and lets everyone else know about the new connection. [David] + o We should also tell the new person (and possibly everyone on the + channel) the list of existing participants. + +o SoC ideas page [Fyodor] + +o Nmap 4.85BETA4 release [Fyodor] + +o [Ncat] Wouldn't it be nice if we could support --exec (and maybe + some sort of partial-emulated --sh-exec) on Windows? [David] + o Almost working! We found some problems with "ncat.exe -v -l + --sh-exec "ncat -v scanme.nmap.org" + +o [Ncat] Can we use it as an IPv4 <-> IPv6 gateway? If so (or if we + can add it), it should be added to the ncat guide feature list. + o Yes, David tried it with --sh-exec and it worked. + +o [Ncat] We should probably make it work without OpenSSL. When I try + ./configure --without-openssl on latest svn Nmap, Ncat build fails + with: + gcc -MM -I../libpcap -DHAVE_CONFIG_H -D_FORTIFY_SOURCE=2 -I. -I.. -I../nsock/include/ -I../nbase ncat_main.c ncat_connect.c ncat_core.c ncat_listen.c ncat_proxy.c ncat_broker.c ncat_hostmatch.c ncat_ssl.c util.c sys_wrap.c > makefile.dep + make[2]: Leaving directory `/mondo/fyodor/nmap/ncat' + make[2]: Entering directory `/mondo/fyodor/nmap/ncat' + gcc -I../libpcap -DHAVE_CONFIG_H -D_FORTIFY_SOURCE=2 -I. -I.. -I../nsock/include/ -I../nbase -c ncat_main.c -o ncat_main.o + ncat_main.c: In function ‘main’: + ncat_main.c:536: error: ‘struct options’ has no member named ‘ssl’ + ncat_main.c: In function ‘ncat_listen_mode’: + ncat_main.c:646: error: ‘struct options’ has no member named ‘ssl’ + ncat_main.c:646: error: ‘struct options’ has no member named ‘sslcert’ + ncat_main.c:646: error: ‘struct options’ has no member named ‘sslkey’ + make[2]: *** [ncat_main.o] Error 1 + make[2]: Leaving directory `/mondo/fyodor/nmap/ncat' + make[1]: *** [build-ncat] Error 2 + make[1]: Leaving directory `/mondo/fyodor/nmap' + make: *** [static] Error 2 + +o [Ncat] Defensive coding review of Ncat --chat (talk) + +o [Ncat] As SSL server it should not crash when someone connects in + w/o SSL and does ^C. When David tried it during our chat, the ncat + servr "ncat --broker --ssl-key test-key.pem --ssl-cert test-cert.pem + --ssl --chat -l" crashed with: SSL_accept(): + error:00000000:lib(0):func(0):reason(0). Also, when a Windows SSL + clients joined and then left, the server died with "Broken pipe + +o [Ncat] --chat should probably only allow reasonable chars, to avoid + cntrl-chars, etc. + +o Nmap should treat ports named "unknown" in nmap-services the same + way (from a naming perspective) as it treats ports which are not + listed at all. See http://seclists.org/nmap-dev/2009/q1/0589.html. + +o Ncat user guide "Emulating Diagnostic Services" page has a very long + UDP chargen server line which causes wrapping problems in web browsers + (e.g. it widens the page substantially). It should probably be + split into multiple lines. [David] + +o Ncat user guide proxying section says "The only exception is when + listing a proxy host by IPv6 address; then the port is required." + Why would we require a port number for IPv6 rather than just use the + same defaults as we do for IPv4? + [David explained that this is because to do otherwise would be + ambiguous because IPv6 uses : for separaters, so we wouldn't know + how to handle things like FF::10:80] + +o [Ncat] Perhaps we should make --ssl work in --chat. If nothing + else, it might be useful if you want to reduce the number of people + connecting with telnet, etc. rather than ncat. + +o [Ncat] --talk should probably be changed (in the code and + documentation) to --chat, as Ncat chat has a + much nicer ring to it, IMHO. --talk should remain as an alias to + --chat, but we don't need to document it. [David] + +o Ncat Windows issue where you make a connection and then take several + seconds to type in a line to the server, Ncat wrongly times out when + trying to write your line to the remote server. [David] + +o Ncat write timeout problems cause client to quit due to write + timeout sometimes. [David] + Examples: + o yes | ncat localhost + o when we paste a few lines into the terminal window in an Ncat chat + +o Defensive coding review of ncat_proxy.* [David] + +o Process the latest version detection submissions. We now have more + than 1,700 of them queued up. [Doug] + +o Write Ncat users' guide, demonstrating all the neat stuff you can do + with it. This should probably be in DocBook XML so it can be an NNS + chapter. You might want to query nmap-dev for list of neat things + people do with ncat (or look around for what people do with nc). + Testing it out for examples might expose areas for improvement as + well. [David] + +o Look at Dario Ciccarone's email from 5/1/07 about IPID sequence + issues, and consider adding IPID sequence test for closed-port-tcp as + they apparently can be different. [David] + o Also fix bug which causes SEQ to not be printed if the TCP open + port tests fail to produce results, even though the II and + (upcoming) CI tests may have useful results. [David] + +o NSE should offer some way to sleep/yield for a given amount of + time. This would allow other scripts to run while a script has + nothing to do. Possible uses: + o Many services have rate limits (or you might just want to use them + for politeness). For example, a web site spidering application + might want to limit HTTP requests to some number per second to avoid + pissing off the target webmaster more than is necessary (or prevent + getting auto-blocked). Similarly, whois servers often will block + IPs which query them too often in a short period. Or maybe you + don't want to exceed the threshold limits of an IDS. + o Example current scripts which might benefit: sql-injection, whois + (possibly), pop3-brute, etc. + o If we don't currently have a way for a cpu-bound NSE script to + yield, then perhaps this could help us implement such a mechanism. + But maybe coroutine.yield already does the trick. + o The mechanism needs to be documented, and ideally should be + implemented in at least one of the scripts shipped with Nmap. + +o Consider adding a way for requesting timing status updates at a + given interval (such as every 5 seconds) to XML and/or normal + output. This would be useful for people who run Nmap from scripts + or other higher level applications. [David] + +o Ncat --allow/--deny bug: "--allow and --deny only support host + specification by IP address, and give no warning when you use + another form such as a host name." Should probably use same syntax + as --exclude. We also want to at least do verification at the + beginning to make sure all the entries are legitimately formed. We + probably want to do things like DNS resolution at the beginning + too. Otherwise we might have a DNS failure when we actually get a + connection and perhaps have to reject the connection wrongly, or + risk a false negative. [David] + +o Fix this overflow: + Stats: 93:57:40 elapsed; 254868 hosts completed (2048 up), 2048 undergoing UDP Scan + UDP Scan Timing: About 11.34% done; ETC: 03:21 (-688:-41:-48 remaining) + [Done by David and Henri Doreau] + +o Ncat -- perhaps connection brokering should support UDP as well as + (its existing support for) TCP? Actually this does raise issues + such as deciding what list of UDP systems to forward a packet too. + Its obviously not like TCP where you have a list of open + connections. Ncat could build such a list, but, for example, would + never know when to remove the host. For now, David is just going to + adjust the error message to encourage people to email nmap-dev + describing their usage scenario if they want this feature. + +o Ncat documentation should note that no SSL certificate verification + is done (maybe we should offer an option to do so, if OpenSSL makes + that easy). + o Done in the new Ncat user's guide + +o Fix dns-zone-transfer infinite recursion bug described at + http://seclists.org/nmap-dev/2009/q1/0317.html. It sounds like the + best approach is to use our dns.lua library rather than having + dns-zone-transfer do its own DNS packet parsing. + +o Fix XML escaping issue so that improper chars from NSE scripts or + elsewhere can't cause corrupt XML files. See + http://seclists.org/nmap-dev/2009/q1/0316.html for an example. [David] + +o Look into whether we should increase the frequency of port scan + pings. See http://seclists.org/nmap-dev/2008/q1/0096.html . Note + that Fyodor already increased them a bit in 2008. Might not need + more. [David did extensive testing of this one already] + +o Find way to document NSE library script arguments and perhaps have + them bubble up to scripts themselves. For example, I had to read + the SNMP library source code to determine the script argument to + specify the SNMP community name for snmp-sysdescr + (https://nmap.org/nsedoc/scripts/snmp-sysdescr.html). Maybe we could + just standardize on something like we do with SMB library and the + scripts which call it (https://nmap.org/nsedoc/modules/smb.html, + https://nmap.org/nsedoc/scripts/smb-check-vulns.html). [David] + +o If it wouldn't bloat things too much, it would be nice to include + ndiff in the Nmap win32 zip distribution files. + +o Reported NSE crash: + "Assertion failed - file ..\nse_main.cc line 314 + lua_gettop(L_script_scan) == 0" + o He says: "After looking at this closer, it appears the assertion + occurs if I include the IP where the scan is run from. For us, I'm + running this on IP 57, which is a VMware Windows Server image. If + I eliminate that IP from the range it successfully completed the + scan for all other devices." + o Seems to be fixed. He can no longer reproduce the problem with + 4.85BETA3. + +o Deal with GTK DLL problem with Nmap 4.85BETA1: [Fyodor] + o David's installer seems to work--he's using a different GTK + distribution. I'll try that. Works! Done! + o Details on problem: http://seclists.org/nmap-dev/2009/q1/0207.html + o Quick workaround done for 4.85BETA2, but better solution needed. + +o "SCRIPT ENGINE (250.600s): ./scripts/rpcinfo.nse against + a.b.c.d:<port> ended with error: ./nselib/datafiles.lua:114: attempt + to index global 'arg' (a nil value)" + -- http://seclists.org/nmap-dev/2009/q1/0227.html [Patrick] + +o Consider making the TODO list public + o Done: http://seclists.org/nmap-dev/2009/q1/0175.html + o Probably remove all of the "done" items since that is easier than + reviewing them. + o Might as well add to insecure.org/nmap/data/ + o Maybe a bug tracker is a better approach. + +o [NPING] Fix compilation on Solaris. See + http://seclists.org/nmap-dev/2010/q1/870. + diff --git a/todo/gorjan.txt b/todo/gorjan.txt new file mode 100644 index 0000000..3eada09 --- /dev/null +++ b/todo/gorjan.txt @@ -0,0 +1,66 @@ +===== +GSoC 2011 participation: Discovery and miscelaneous script specialist +===== + +Work in progress: + +* bgpmon-info analyze + +* bittorrent-dht-nodes + +* lldp - write script proposal +http://en.wikipedia.org/wiki/Link_Layer_Discovery_Protocol + +* disjunctive-traceroute analyze feasibility +http://ccr.sigcomm.org/online/?q=node/398 + +===== + +ToDo: + +* snmp-brute port to brute framework +There are a couple of default passwords that snmp-brute uses atm which should be +considered even when it's the brute.lua is used + +===== + +Maybe (the ones with ** aren't on the Script_Ideas Page yet) + +* Bonjour / mdns / llmnr etc. +(DNS protocols support) + backscatter into dns scripts where applicable? + +* targets-asn +John Bond is working on this. It's called asn-to-prefixes. Perhaps I could +review it, asist so it makes its way to the library faster? On the other hand +there already are a couple of people assisting. + +* targets-dhcp +dhcp-discover as a prerule, so it doesn't run by default. But it doesn't run by +default. It's discovery, intrusive, but not default. Maybe just add the prerule +there, and some way of forcibly initiating the prerule (like an argument). + +* hnap-info +* hnap-auth-bypass +A nice hnap library would be fitting, that will make these scripts a breeze. +I'd need testing equipment, or some :S implementation. + +* vuze-dht-version +* Nbstat.nse -> change to using a broadcast prerule +* SSL renegotiation +* soap.lua +* xmlrpc.lua + +===== + +Completed: + +* broadcast-ping +* nmap lib: get_ttl() and get_payload_info() +* ip-geolocation scripts +* snmp-interfaces patch related to mac-geolocation +* mac-geolocation +* stdnse.lua: in_port_range() +* backorifice-brute +* backorifice-info + +=====
\ No newline at end of file diff --git a/todo/henri.txt b/todo/henri.txt new file mode 100644 index 0000000..cca8814 --- /dev/null +++ b/todo/henri.txt @@ -0,0 +1,41 @@ +o Proper SSL support in proxy mode. + - A naive implementation relying on the current code would probably look + horrible (at least my own attempts did). I believe that nsock should + internally be able to SSLify a plain TCP connection. It doesn't have to be + exported but it should be implemented just like the other operations. Then + it would be trivial (and clean) for the library to SSLify the channel + established by the proxy hooks. + - When redesigning nsock SSL code, keep in mind the ability to establish a SSL + session and still expose the raw TCP. That can be convenient when auditing + the SSL/TLS layer. +o Don't drop pending writes when deleting the corresponding IOD. For nsock to + behave a bit like standard BSD sockets we should flush writes on close. (OTOH + anything which isn't ack'ed has no meaning, caller can still cancel it + typically...) +o Give IODs their own methods to streamline the code and get rid of all + the special cases in nsock_core.c. This would also make it easier to + hook operations (typically: override the default iod_connect() method + to establish a proxy chain). +o Fix the read API (!) +o Profile the pcap code. It needs cleanup (for sure) and optimizations (maybe). +o Proxy authentication +o Handle socks4a + - This requires to figure out how to trigger proxy code without + resolving target hostname first. The problem is that the proxy code + is supposed to be a transparent hook of connect()... Extending the + exported API will probably be needed :( + - Async hostname resolution available from within nsock would let us + try clever tricks... I'm not sure whether nsock should provide it + or if it should simply provide an API to plug an external system. +o Socks5 support +o Some code is copied from ncat. I should move it to nbase. +o Replace event lists by more efficient data structures. Consider using + a radix tree to map event IDs to pointers. Another solution would + be to put them all into a single RB-tree (TODO: validate BSD_HACK_MODE + & stuff). Encoding the event type in the ID's MSB would let us do inorder + traversal with connect events first, then read, then write... + {NOTE: It'd be cool for the beauty of it, but my tests reveal that as of Oct. + 2013 there's no big bottleneck there.} +o Rework the filespace code to avoid unneeded data copy. Scatter/gather + I/O might be useful there. Same task can also be expressed as: "profile and + optimize the usual nmap nsock I/O patterns." diff --git a/todo/nmap.txt b/todo/nmap.txt new file mode 100644 index 0000000..76fe1ff --- /dev/null +++ b/todo/nmap.txt @@ -0,0 +1,638 @@ +TODO $Id: TODO 11866 2009-01-24 23:10:05Z fyodor $ -*-text-*- + +o Work on Nmap on Mobile devices, particularly Android. Would be + great to get it in Google Play store, for example. An official + version with a workable GUI. For now, people have to do manual work + and it isn't as well tested either: + https://secwiki.org/w/Nmap/Android . If this is successful, we could + consider iOS. + +o Nmap performance work. Particularly with --min-rate. + +o Consider re-architecting Nmap to have more of a scanning pipeline +approach rather than fixed sets of hosts which start and finish one +phase and then move into the next in parallel. This could potentially +allow us to add hosts one by one to a phase as other hosts finish that +phase and, ideally, the phases could run in parallel too. + +o Nmap Network Scanning, 2nd Edition work [placeholder] + +o Organize nselib into a hierarchy. A library "dirname/filename.lua" can be + required as "dirname.filename". We would need to ensure the installers + (Makefile, OS X, Windows, RPM) can handle this. See + http://seclists.org/nmap-dev/2014/q3/364 + +o We should work to reduce Zenmap's memory consumption. We used to + commonly get error reports from people who load so many systems that + Zenmap gives an out of memory error and crashes. For example, see + this thread: http://seclists.org/nmap-dev/2014/q2/46 + After committing patch at http://seclists.org/nmap-dev/2014/q2/429, + we no longer get the error report but the problem still exists. + The problem seems to lie in a very large Nmap Output being stored + in memory and a possible fix seems to be to use a file based paging + system. + +o Consider making a version of Nmap for Apple's official Mac App + Store. A particular concern with the downloadable Mac version of + Nmap is that Apple's new "Mountain Lion" release may require users + to jump through hoops to install unsigned non-app-store content per + their "Gatekeeper" "feature". Though maybe signing the app will be + enough. There may also be an issue with the "Sandboxing" + requirement for App Store apps starting June 2012. Will Nmap be + able to request all the permissions it needs? Ignoring the + technical challenges for the moment, what will users prefer? + +o Do a roll up on (state, TTL) pair instead of just state so that TTL + info is not lost when doing roll up on port states. + See thread at http://seclists.org/nmap-dev/2014/q3/93 + +o Consider looking into differring TTL values during OS detection + phase and choose a port that is (hopefully) not firewalled to get + a better chance at correct result. See thread at + http://seclists.org/nmap-dev/2014/q3/33 + +o [Zenmap] Look into and refactor code which uses the (very slow) += operation + on strings. http://seclists.org/nmap-dev/2014/q2/432 helped improve speeds + for opening files (from hours to seconds) and it seems like more speedups + can be done in other places. + +o Look into moving our Mac building/testing system into a virtual + machine or leased server sort of environment so that multiple Nmap + developers can access it and nobody has to keep a stack of Mac Minis + in their closet. + +o INFRASTRUCTURE: Upgrade seclists to use Mailman 3, which apparently + has many improvements. + +o We should fix nsedoc generation so it doesn't fail when blocks like + @usage, @output, etc. are followed by a local declaration. See + http://seclists.org/nmap-dev/2014/q2/331. If for some reason this + just can't be fixed, we will have to document the heck out of it, I + suppose. + +o When scanning your own IP from Windows, Nmap currently recognizes + the problem (can't do a raw scan like that on Windows) and skips the + SYN scan, leading to Nmap printing a bunch of ports in "unknown" + state at the end. Nmap should probably act like unprivileged mode + in this case (e.g. do a connect scan, etc.). See + http://seclists.org/nmap-dev/2013/q3/519 + +o Investigate Checkmarx static analysis report of Nmap source tree + that someone sent us on Feb 12. It looks like mostly false positives, + but we should go through to check for any real bugs or even possible + security issues. Fyodor has the report. + +o INFRASTRUCTURE: Consider updating our svn-mailer.py (and conf file) + to the latest official version. First check whether there is a + later official version and whether it has material changes. We're + currently using one from + subversion-1.4.2/tools/hook-scripts/mailer/mailer.py. + +o Consider a two-stage model for IPv6 subnet/pattern support + o Right now you can try to scan a /64, for example, and Nmap will try + to iterate through them all (and of course never complete). So + perhaps Nmap should first look at a specification and decide if it + should use other techniques like multicast discovery instead. + +o Move advanced IPv6 host discovery features from NSE into core Nmap. + We'll probably add the functionality of + targets-ipv6-multicast-invalid-dst, targets-ipv6-multicast-echo, and + maybe targets-ipv6-multicast-slaac. + - The idea is that Nmap does them automatically if it gets a large + target specification and sees that it is local so can be multicast + pinged. + +o We should figure out why (at least with Nping) raw ethernet frame + sends seem to be taking significantly longer than raw socket sends + (e.g. using --send-ip or the OS-provided ping utility). This has + been reproduced on Linux and Windows. Here's a thread: + http://seclists.org/nmap-dev/2012/q4/424 + o Note that David and I tried to reproduce this on his machine and + on 'web' and 'research' machines and could not reproduce. Still + happens with Fyodor's machine connected with WiFi. Fyodor should + test on the same machine using wired and see if that changes anything. + +o Implement some improvements to dns-ip6-arpa.nse, as describe at + http://seclists.org/nmap-dev/2012/q2/45. + - Also consider a move to "fire and forget" logic. Just blast out + the queries that we know we have to make, and then read any replies + that may happen to come back. (but still try not to introduce + inaccuracy (missed hosts) by flooding the network. + +o Treat the input to the escape function in xml.cc as UTF-8, not just + ASCII. Good UTF-8 should survive into the output; i.e., "\xe2\x98\xbb" + should become "\xe2\x98\xbb" in the output, not "☻". + If the input happens not to be UTF-8, (like the file name in + http://seclists.org/nmap-dev/2013/q1/180), I suppose we can + individually encode each byte of each invalid sequence: "\xba\xda\xbf" + becomes "ºÚ¿". Can probably do this with simple + byte->rune and rune->byte functions as in + http://plan9.bell-labs.com/sys/doc/utf.html. + +o We should probably redo the Nmap header (e.g. on https://nmap.org) to + make it more attractive. Or, at a minimum we should update the + screenshots and think about which links we really need (some of those + pages aren't really updated any more). + +o Test a hierarchical classifier for IPv6 OS detection. Our classifier + currently treats, for example, some localhost Linux fingerprints as + separate classes from remote Linux fingerprints, simply because we + lose precision if we lump them together (for example TCP window size + differs across certain Linux versions when measured remotely, but + not on localhost). This leads to the linear classifier having to use + narrow margins between fingerprints that are really very similar. I + want to try a tree of classification where each non-leaf node is a + separately trained classifier and each leaf node is a final + classification. The first layer of the hierarchy would be something + like + (linux windows solaris aix ... other) + where "linux" would contain *all* the Linux fingerprints in a single + class. Lower levels would be like + (linux-2.4 linux-2.6) + (windows-xp windows-vista windows-7) + Lower levels will include only those fingerprints in their parent + class, so we don't even think about Windows when classifying + Linux. Probably three or four levels will be sufficient. There may + be a principled or automatic way to build this hierarchy, but I + suspect playing it by ear will be sufficient. Talk to David for more + of his thinking on this topic. + +o Maybe we should rename dns-brute to dns-brute-enum since it is so different + from our traditional brute force authentication cracking -brute scripts? + +o NSE WORK (note that this is mostly infrastructure because script + ideas are generally put on the script ideas page instead: + https://secwiki.org/w/Nmap_Script_Ideas) + o Review NSE-based port scanning and RST idle scan. + http://seclists.org/nmap-dev/2011/q2/307. [Henri and Hani?] + +o Maybe we should add an analysis or reporting or intelligence (or + different name) for our NSE scripts which don't send any packets, but + simply analyze Nmap's existing data and report when useful. + +o Install some sort of svnview webapp for svn.nmap.org which is + wrapped in Insecure chrome, allows people to click link for direct + file download, probably shows revision history and allows users to + see older versions, etc. + +o Process Nmap survey and send out results [Fyodor] + +o Nping (we think) will stop after 2^32 rounds even when "-c 0" is + given. We should probably make this a 64-bit integrer so that "-c + 0" will go essentially forever and so that users can give values + higher than 4 billion. + +o Nscan work [placeholder] + - Hosted Nmap system + +o Add CPE entries to OS fingerpting DB entries which still lack them. + This is a gradual process since almost all of the missing ones + aren't in the official CPE dictionary either and it can take a lot + of research to decide on an appropriate entry. Milestones so far: + - 3/21/12: We have entries for 2,601 of 3,572 fingerprints (971 + missing; 73% coverage) + - 11/5/12: We have entries for 3,285 of 3,907 fingerpritns (622 + missing; 84% coverage) + - 11/12/12: We have entries for 3,558 of 3,946 fingerprints (388 + missing; 90% coverage). + +o [Zenmap] should actually parse and use script results. See + http://seclists.org/nmap-dev/2010/q1/1108 + - We have an initial prototype, but probably need to redo because it + doesn't present the results in the way we'd like yet due to + problems implementing such a presentation with GTK, etc. + +o Make Zenmap settings get upgraded when the Zenmap executable is + upgraded. The per-user configuration files such as scan_profile.usp + and zenmap.conf are never overwritten once installed by Zenmap, so + changes and fixes to those files don't reach anyone who has + installed Zenmap already. This is most noticeable with changes to + profiles and highlight definitions are notably affected. This fix + may involve hard-coding settings that are not normally configured by + users (like highlighting) or updating the per-user files at startup + (only those parts that haven't been changed by the user). + +o We should offer partial results when a host timeouts. I (Fyodor) + have been against this in the past, but maybe the value is + sufficient to be worth the maintenance headaches. Many users have + asked for this. If we do implement this, we may want to only print + results for the COMPLETED phases (e.g. host discovery, port + scanning, version detection, traceroute, NSE, etc.) Trying to print + partial results of a port scan or NSE or the like might be a pain. + And if we print some results for a host which timeouts, we should + give a very clear warning that the results for that host are + incomplete. As an example, here is someone who hacked Nmap source + code to achieve this: http://seclists.org/pen-test/2010/Mar/108. + o Another benefit would be that it would allow us to clean + up/regularize the host output code. Right now there are I think + three places where a host's final output can be printed. If, + instead, that code just looked at what information was available and + printed that out only, we could potentially isolate it in just one + place. + o This also might let us provide a feature for skipping the rest of + an Nmap phase which is going too slowly (I think that has its own + Nmap TODO item). + +o [Nsock] Some SSL connections that used to work now fail; find out + why. http://seclists.org/nmap-dev/2010/q4/788. Narrowed down to + r19801 in http://seclists.org/nmap-dev/2011/q1/12. + +o [NSE] Consider a system where scripts can tell if any other scripts + depend on them. They could then use that to determine whether they + should bother storing information in the registry. For example, + snmp-interfaces could store the discovered table if another script + (such as a mac address geolocator script) depends on it. + +o [NSE] Consider whether we need script.db for performance reasons at + all or should just read through all the scripts and parse on the fly. + See: [http://seclists.org/nmap-dev/2009/q2/0221.html] + +o A couple minor nsedoc issues (see + http://seclists.org/nmap-dev/2011/q1/1095): + o After the ssh-hostkey portrule was added, nsedoc seems to be + generating a blank "Script types" filed for the script: + http://localhost:8082/nsedoc/scripts/ssh-hostkey.html + o This is happening because "portrule" and "hostrule" appear later in + the script, and NSEDoc thinks it is their definition, and there is + no NSEDoc there. + local ActionsTable = { + -- portrule: retrieve ssh hostkey + portrule = portaction, + -- postrule: look for duplicate hosts (same hostkey) + postrule = postaction + } + o ssh-hostkey and rmi-dumpregistry each have two @output sections, + and NSEDoc is only showing the second one. We should probably just + combine them into one @output section, and maybe make nsedoc give a + warning in this case. Or we could make nsedoc handle multiple + @outputs. + +o Add general regression unit testing system to Nmap + o David has created a system for Ncat which could serve as a + model. + +o Make version detection and NSE timing system more dynamic so that + the concurrency can change based on network conditions/ability. + After all, beefy systems on fast connections should be able to handle + far more parallel connections than slower systems. + o At a minimum, this at least warrants more benchmark testing. + +o We should run at least one SCTP service on scanme. Daniel + Roethlisberger has made available dummy services which support IPv4 + and IPv6 (see http://seclists.org/nmap-dev/2011/q2/450). + Alternatively, we could run some sort of "real" SCTP application(s) + (preferably one which is relatively simple, easy to install, secure, + and supports IPv6). + +o Create new default username list: + http://seclists.org/nmap-dev/2010/q1/798 + o Could be a SoC Ncrack task, though should prove useful for Nmap + too + o We probably want to support several lists. Like an admin/default + list like "root", "admin", "administrator", "web", "user", "test", + and also a general list which we obtain from spidering from + emails, etc. + +o Improve Nsock proxies system + - Add SSL support + - Add proxy authentication + - Switch Ncat to using Nsock proxy system rather than it's own + built-in support. + - Move the code which is shared with ncat to nbase (URL parsing code, + for instance). + - Add socks4a/socks5 support. This requires to figure out how to + enter the nsock proxy code w/o having the target IP address. No huge + technical blocker there though, only design choices to make. + - Nping could potentially use it as well (could be useful for + measuring latency and reliability of a given proxy chain, for + example). + - Add proxy support to connect() scan. This would mean moving + connect scan to nsock. + +o [NCAT] Send one line at a time when --delay is in effect. This is + cumbersome to do until Nsock supports buffered reading. + +o [NCAT] Make the HTTP proxy support the chunked transfer encoding, + then change it to be HTTP/1.1 and support pipelining. + +o [NCAT] Drop privileges once it has started up, bound the ports it + needs to, etc. + +o [NCAT] Work as a SOCKS4a/SOCKSv5 proxy. + +o [NCAT] Resolve names through the proxy when possible. + http://seclists.org/nmap-dev/2012/q2/768 + +o [NSE] Script writing contest (something to think about) + +o We should document an official way to compile/test refguide.xml so + people can more easily test their changes to it. This will probably + involve moving legal-notices.xml into /nmap/docs, among other + things. + o Note that nping has its own /nmap/nping/docs/genmanpage.sh - we + could look at how that could apply to Nmap. + +o Move Zenmap man page from nmap/docs/ to nmap/zenmap/docs to match + the man page location for ncat and ndiff. + o Don't break packaging/build system + o Don't break the system for posting html to web site. + o Consider standardizing names for nping and ncrack man pages as well. + [Fyodor] + +o [NSE] MSRPC - Improve domain support all around -- in particular, + let the user give the domain in the format DOMAIN\username or + username@DOMAIN anywhere that usernames are accepted. Suggested + at http://seclists.org/nmap-dev/2010/q2/389 + +o [NSE] Combine similar MSRPC scripts, especially the "get info" + stuff. See this thread on combining + (http://seclists.org/nmap-dev/2010/q1/1023). This was suggested by + Ron at http://seclists.org/nmap-dev/2010/q2/389. + +o [Zenmap] Investigate getting new OS icon art. See + http://seclists.org/nmap-dev/2010/q1/1090 + +o We should probably enhance scan stats--maybe we can add a full-scan + completion time estimate? Some ideas here: + http://seclists.org/nmap-dev/2010/q1/1007 + +o [NSE] Do some benchmarking of our brute.nse. We should check the + performance with different levels of thread parallelism. Our + initial results show that it isn't helping much for vnc-brute or for + drda-brute (which is currently using the multi-thread feature + directly rather than through brute.nse library). We should figure + out why the threads aren't helping more, and whether there is + something we can do to fix it. It would also be interesting to + compare speed with Ncrack for services we have in common. + +o Start project to make Nmap a Featured Article on Wikipedia. + - See http://seclists.org/nmap-dev/2010/q1/614 + +o Add Nmap web board/forum + - First step is looking at the available software for this. + - Nmap subreddit exists: https://www.reddit.com/r/nmap + +o [Zenmap] Consider a couple ideas from Norris Carden + (http://seclists.org/nmap-dev/2010/q2/228): + - remember last save and/or open location for new saves and/or opens + - default save location option + +o [Nsock] Consider adding server support to Nsock so it can accept + multiple connections and multiplex the SD's, like it does for + clients. This could potentially be used by Ncat and Nping echo + mode. Currently Ncat server doesn't use Nsock at all, while Nping + echo mode basically polls, repeating a loop of 1s in nsock_loop + followed by a nonblocking accept(). Then Nping gives the SD's to + Nsock to manage. + +o Consider implementing both global and per-host congestion control in + the IPv6 OS detection engine. Currently it handles congestion globally + (one CWND and SSTHRESH shared by all hosts). This works fine but it + may not be the most efficient approach: if the congestion is not + in our network segment but in a target's and we are os-scanning + hosts in different networks, then all hosts get "penalized" because + there is congestion in another network, not in theirs. + +o [Nsock] Consider implementing a nsock_pcap_close() function or making + nsp_delete() call pcap_close() when pcap IODs are used. Currently valgrind + warns about a socket descriptor left opened (at least in Nping). + ==10526== at 0x62F77A7: socket (syscall-template.S:82) + ==10526== by 0x4E348A5: ??? (in /usr/lib/libpcap.so.1.0.0) + ==10526== by 0x4E36819: pcap_activate (in /usr/lib/libpcap.so.1.0.0) + ==10526== by 0x4E375FC: pcap_open_live (in /usr/lib/libpcap.so.1.0.0) + ==10526== by 0x4311A9: nsock_pcap_open (nsock_pcap.c:64) + ==10526== by 0x428078: ProbeMode::start() (ProbeMode.cc:329) + +o Consider rethinking Nmap's -s* syntax for specifing scan types + o Current problems with this -s syntax: + o We already use like 20 of the 26 letters, so we end up with + things like SCTP scan using -sY + o Can make Nmap command lines hard to read, particularly given + that we often need to improvise to find a letter which isn't + taken. + o Problematic for scan types -sI and -b which require arguments + o Inconsistencies. For example, -sC and -sV do script scan and + version detection, respectively, and yet for OS detection we use + -O. Also, control flow (-sP, -sL) is used with -s, which further + overloads the options. + o Possible solution: + o We are enabling -Pn and -sn as preferred notations for -PN and + -sP which mean "no ping" and "no port scan". Those match the + already existing -n for "no DNS". The problem with -sP is that it + implies "ping only", when what it really should mean is "disable + port scan" because you may want to do NSE, OS detection, + traceroute, etc. still. + o We might want to just give them normal option strings, so you + could do --maimon instead of -sM, for example. For extremely + common options such as SYN scan, UDP scan, version detection, we + could perhaps find good single letter options as an alias to the + longer one. + o Another idea is to use something like --scantype syn,udp,sctp, + which is a lot longer for single-type scans, but shorter when + you're combining mulitiple ones. Doesn't allow for individual + scan arguments easily. I (Fyodor) think I prefer the idea above + of just givem them top level arguments. + o If we keep -s*, we could just give it one defined function, such + as selecting port scan type, or control flow. + o Obviously this will take some discussion/brainstorming on nmap-dev. + +o Do -p- Internet UDP scans. + +o Scanning through proxies + o Nmap should be able to scan through proxy servers, particularly now + that we have an NSE script for detectiong open proxies and now that + Ncat can act as proxy client or server. + o Requirements: + o Would be nice to be able to chain through multiple proxy servers of + different types. + o Would be nice to be able to spread the load amongst multiple + proxies. + o Should support port scanning, version detection, and NSE. In + other words, nsock should support proxies. + o Support IPv4 and v6 + o Need to figure out how to get good performance. Pool of + connections to proxy or proxies for concurrency? HTTP pipelining? + o Support the different varieties of proxies: socks4, socks4a, + socks5, HTTP GET (if possible), HTTP CONNECT. Note that GET + proxies present some challenges since the error messages may not + be standard, etc. + o Maybe auto-detect the proxy type so that Nmap can try the most + efficient scanning method first? + o I've been asked to support basic, ntlm, and digest authentication + if possible. + o Implementation ideas: + o There is a patch by Zoltan Panczel (http://nmap-dev.fw.hu) and it + has been improved by Jacob Appelbaum in nmap-exp/ioerror/ . This + patch doesn't handle things like parallelization, but it may be a + good proof of concept. + o This might not be appropriate for ultra_scan ... perhaps would be + better to write a general scanning engine for abusing + applications for port scanning purposes. This could handle + scanning through proxies and the existing FTP bounce scan would + also be ported to this engine (or, frankly, we could probably get + away with removing FTP bounce). rembrandt at jpberlin.de tells me + that you can also do this with the "forwarding" commands on IMAP + servers. Whoever does this should probably start by reading the + code for the main port scanning engine (ultra_scan()) and also + the version detection code (service_scan()). And the version + detection paper at https://nmap.org/book/vscan.html. If you + understand all that, you may be ready for this project :). This + is important, because it is easy to do poorly. The tough part is + high performance and clean code which is general enough that all + these different applications can be scanned through using the + same basic engine. You should run your ideas by nmap-dev in as + much detail as possible before starting. + o David: I'm starting to think about building proxy support into + Nsock and then implementing -sT with Nsock instead of ultra_scan. + +o [Web] Consider adding training/introduction videos to the Nmap site + o Would be great to have a (5 minute or less) promotional video + introduction to each tool (Nmap, Zenmap, Ncat, Ndiff) on its web + page. + o They need to be good to be useful--the sort of the quality you see + in Laura Chappell's Wireshark videos or James Messer's Nmap videos + or Irongeek's videos (http://www.irongeek.com). + o Besides the promotional videos, users would probably enjoy more + in-depth video instructions (e.g. covering the Nmap Network + Scanning topics). + o Here's an example product page with lots of videos (we may not go + that far): http://www.splunk.com/product + +o The Zenmap translation system + (https://nmap.org/book/zenmap-lang.html) has been pretty successful + so far. We should consider doing the same for Nmap. After all, we + already have the reference guide in 16 languages at + https://nmap.org/docs.html. We should definitely try to use the same + translation methods for Zenmap as we do for Nmap. In fact, maybe we + can create a combined PO file Nmap, Zenmap, Ncat, and Ndiff so that + they can all be translated and maintained together. Something to + consider: calling setlocale can change the behavior of functions like + isalpha. Locale-dependent functions need to be checked for security + risks. + +o [NSE] Consider whether we should include some sort of NSE debugger. Or we + could include something simpler. For example, Nmap now provides a + traceback (with sufficient debugging/verbosity) when a script ends + in error. For some inspiration/ideas, look at Diman's NSE + debugger (http://seclists.org/nmap-dev/2008/q1/0228.html). + +o [NSE] Support routing http requests through proxies. + +o [NSE] Would be great if NSE scripts could be made to NOT + run as root if they don't have to. + +o [NSE] Security Review + o Consider what, if any, vulnerabilities or security risks NSE has + with respect to buffer overflows, format string bugs, any other + maliciously formatted responses from target systems, etc. Maybe + address the known risk of malicious scripts too. + o Consider that NSE runs scripts as root + +o More security auditing of Nmap code (it never hurts to do more proactive + security auditing). + +o Figure out and document (in at least the Ncat user's guide) the best + way to use Ncat for chaining through proxies. One option is this + sort of thing: + ncat -l localhost 1234 --sh-exec "ncat --proxy A.A.A.A B.B.B.B" + ncat --proxy localhost:1234 C.C.C.C + If you had two proxies A.A.A.A and B.B.B.B, connecting to C.C.C.C. + With another listener/--sh-exec pair for each additional proxy. + But perhaps we can make it easier by adding it to the syntax. + +o Look into whether we should loosen/change the global congestion + control system to address possible cases of one target host with many + dropped packets slowing down the whole group. See + http://seclists.org/nmap-dev/2008/q1/0096.html . + * Related possibility: Fix --nogcc to gracefully handle ping scans. + Right now it seems to go WAY TOO FAST (e.g. several thousand + packets per second on my DSL line). + * [12/22/09] David says: It still is in one case that I've + documented on my wiki. I had an idea to fix it, but on testing it + it didn't work. The idea was to treat the global congestion limit + differently. Instead of dropping it down to the minimum level on a + drop as is done currently, I thought about only dropping it by the + amount that the individual host limit drops. For example, if a + host had a drop and its limit fell from 25 to 1, then the global + limit would change (if it was at 100 to begin with) to 76, not all + the way down to 2 or whatever it is. The idea being that the + global limit is most important at the beginning of a scan, when + there's no information to set host limits, and every host wants to + send all its first probes at once. See + http://www.bamsoftware.com/wiki/Nmap/PerformanceNotesArchive2#global-cc. I + am convinced, though, that some sort of global control is + necessary. There's a reason that a web browser limits the number + of connections it will make, and doesn't try to download every + image file at once and count on the fairness of TCP to sort it + out. + +o libnmap organization for UNIX and Windows + o Then change Nmap and Zenmap to simply call this library + o It is interesting to look at: http://www.gnupg.org/gpgme.html + +o Deal with UDP retransmission for version detection (I think I + should just do a second run of all probes for UDP if it fails to + match anything). The advantage there is that no retransmissions are + neccessary if the service is found. Then again, per-probe + retransmission would let us redo the most likely probes (the one(s) + that match the port number) quickly. Lost packets should probably + affect ideal_parallelism. + +o Make RPM relocatable (requires somehow avoiding storing paths in the + binary) + - That may be easier now that David has made some big improvements + in detecting where the binary is cross-platform and then looking for + data files based on that location. + +o Nmaprc-related - Create a system to store Nmap defaults/preferences + in an nmaprc file. + o nmaprc should be in ~/.nmap on UNIX + o On Windows, we may need a registry key to find the .nmaprc + o Perhaps Lua could be used as the format? + o .nmaprc for keeping defaults, etc. + o Nmaprc infrastructure, hook to new timing variables + o Nmaprc man page + o Default timing mode + o Default NSE arguments, such as user agent + o Maybe Default source IP (-S) argument + o should be a way to specify your own .nmaprc + o Maybe lets you add a directory and template for saving all + scans. + o Maybe let you define "scan profiles" like is done with Zenmap. + There would then be a command-line option to select the profile used. + +o Get new Zenmap logo + o consider putting back on top-right of command constructor wizard + (there used to be umit logo there). + o Maybe that can be done after the release by soliciting ideas. + +o Create or collect some great ./configure ascii art. + +o Look at all the pcap functions, there are some like + pcap_findalldevs() which could be quite useful. There are mails to + the Nmap list relating to suggested improvements -- + http://seclists.org/lists/nmap-dev/2004/Apr-Jun/0024.html . + Actually I do indirectly use that for Windows. I wonder if they work + for UNIX? + +o perhaps each 'match' line in nmap-service-probes should have a + maximum lines, bytes, and/or time by which a response should be + available. Once that much time (or many bytes or lines) have passed, + that match can be considered 'failed' and ignored in subsequent runs. + Once all matches are considered failed, that probe is done. This + could be a useful optimization and is arguably better than the less + granular 'totalwaitms'. Or I could just have a simple function that + looks at whether a given regex could possibly match something + starting with the received data (not too hard since almost all of + the current regexes are anchored). But before doing this, I should + look long and hard at how many of the probes have every match + capable of doing this. In particular, many of the softmatch lines + don't offer many chars anchored at the front. + +o Separate nbase into its own Windows library in the same way as Andy did + with iphlpapi . + +o Nmap / Nmap-hackers FAQ + +o random tip database + diff --git a/todo/nping.txt b/todo/nping.txt new file mode 100644 index 0000000..c1130cf --- /dev/null +++ b/todo/nping.txt @@ -0,0 +1,799 @@ +/***************************************************************************** + * * + * o * + * o * + * o * + * o o * + * o o * + * o o * + * o o o * + * o o o * + * 888b 888 o o o * + * 8888b 888 o o o * + * 888Y88 888 o o o * + * 888Y88b 888 o * + * 888 Y88b888 o * + * 888 Y88888 * + * 888 Y8888 * + * 888 Y888 * + * * + * --[NPING TO-DO LIST]-- * + * * + *****************************************************************************/ + + This file contains Nping's to-do list. Items are listed in order of priority + (high priority items are listed first). Feel free to work on any of the items + on the list. However, if you'd like to work on something that is not trivial + to implement you may want to send a message to the nmap-dev list before you + start so other developers can see what you are planning to do. Make sure you + explain exactly what you are trying to fix/implement and how you are planning + to do it. It's always better to discuss bugfixes and new feature additions in + advance because they may actually have bigger implications than you think and + you may not get your patch accepted. + + Please keep in mind that contributed code must: + * Be written in C++. + * Include comments so anyone can understand immediately what it does. + * Work on Linux, Mac OS and MS Windows. It's OK if you have not tested + the code in all those platforms, but at least keep portability in mind when + you write it and include a list of systems you've tested it on along with + your patch. + + Questions, comments and patches should be sent to the Nmap development + mailing list (nmap-dev). To suscribe: + <https://nmap.org/mailman/listinfo/dev> + + +/***************************************************************************** + * Things that have NOT been done yet * + *****************************************************************************/ + +* Improve IPv6 support. Currently it doesn't work well. The situation should be + analyzed in detail because right now Nping has code to send packets at raw + transport level (letting the OS craft the IPv6 header), and at raw ethernet + level. None of them seems to work well, though. + +* Investigate an IPv6-related core dump reported by Vasiliy Kulikov. + More info: http://seclists.org/nmap-dev/2011/q3/567 + +* Consider using Nmap's proto-dependant payloads for UDP packets. According + to David's tests, better results are obtained when sending UDP probes with a + payload specific to the protocol. + +* Consider adding the possibility to see the RTT in the RECV line. Something + similar to the way the traditional ping tool prints the RTT (time=XXX ms) + + $ ping nmap.org + PING nmap.org (173.255.243.189) 56(84) bytes of data. + 64 bytes from nmap.org (173.255.243.189): icmp_req=1 ttl=48 time=169 ms + 64 bytes from nmap.org (173.255.243.189): icmp_req=2 ttl=48 time=177 ms + 64 bytes from nmap.org (173.255.243.189): icmp_req=3 ttl=48 time=179 ms + ^C + --- nmap.org ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2000ms + rtt min/avg/max/mdev = 169.097/175.137/179.152/4.347 ms + + + This was requested by Jacek Wielemborek. More info: + http://seclists.org/nmap-dev/2013/q3/533 + +* Currently, Nping determines the maximum number of open descriptors + (in TCP connect and UDP unprivileged modes), from the value returned + by libnetutil::get_max_open_descriptors(). However, it is often the + case that such function returns a value higher than FD_SETSIZE, which + is the maximum number of descriptors that select(2) can handle. + Currently Nsock uses select(2) so we have to limit the number of + descriptor to FD_SETSIZE, and not to the value returned bu + get_max_open_descriptors(). However, Henri Doreau is working on a new + nsock-engines branch which will provide Nsock engines based on + better I/O syscalls like poll() and epoll(). I've asked Henri if he + could implement a function in Nsock that provides the maximum number + of descriptors that can be handled at the same time, based on the + nsock engine being used. So, if that function gets implemented and + his nsock-engines branch merged into trunk, we should consider + updating Nping's code to use it. + More info here: + http://seclists.org/nmap-dev/2011/q4/550 + +* A few ideas for the Echo protocol: + - Add an authenticated NEP_BYE message, so session termination is explicit + and both ends can determine if the session was ended because the other end + requested it or if it was due to some error at the network or transport + layer. Suggested by David. + + - Add examples for encryption and hmac to the RFC. This would help in + debugging implementations. Suggested by Toni Ruottu. + + - RFC. Improve description of how the IVs work. Suggested by Toni Ruottu. + + - RFC. Improve description of encryptionless sessions. Suggested by Toni + Ruottu. + + - Currently, the echo server zeroes any application layer data before + transmission in a NEP_ECHO message. This minimizes the impact of + errors in the server's packet matching engine or malicious attacks that + attempt to trick the server into echoing packets that do not belong to + a particular user. This works well but in the future, if one day we + create a NEPv2 specification, we may want to consider extending NEP_ECHO + packets to allow stripped-packet transport. This is, to allow echo servers + to remove application layer data before transmission, and include + additional information in the NEP_ECHO message so clients can determine + that the payload part was stripped and how long was it. + + - Consider making the echo server bind to all IPv4 AND IPv6 interfaces. + + - Add a description of the security implications of running a public echo + server (failures in the packet matching algorithm, etc), to either the + RFC or the man page. Suggested by Toni Ruottu. + + - Test the new --safe-payloads option with a packet fuzzer to make sure + the packet parser behaves correctly. + +* When running Nping echo client with the --no-capture parameter, the last + packet's CAPT line is not displayed. + + nping --ec public echo.nmap.org -p90 --tcp --count 1 --no-capture + + luis@Aberdeen:~$ sudo nping --ec public echo.nmap.org -p90-92 --tcp --count 1 --no-capture + + Starting Nping 0.5.52.IPv6.Beta2 ( https://nmap.org/nping ) at 2011-07-05 12:53 CEST + SENT (7.3302s) TCP 163.117.203.253:18554 > 74.207.244.221:90 S ttl=64 + CAPT (7.4625s) TCP 163.117.203.253:18554 > 74.207.244.221:90 S ttl=54 + SENT (8.3309s) TCP 163.117.203.253:18554 > 74.207.244.221:91 S ttl=64 + CAPT (8.4429s) TCP 163.117.203.253:18554 > 74.207.244.221:91 S ttl=54 + SENT (9.3310s) TCP 163.117.203.253:18554 > 74.207.244.221:92 S ttl=64 + + Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A + Raw packets sent: 3 (120B) | Rcvd: 0 (0B) | Lost: 3 (100.00%)| Echoed: 2 (80B) + Tx time: 2.00181s | Tx bytes/s: 59.95 | Tx pkts/s: 1.50 + Rx time: 2.00193s | Rx bytes/s: 0.00 | Rx pkts/s: 0.00 + Nping done: 1 IP address pinged in 9.33 seconds + +* Sometimes Nping displays a couple of error messages (related to cleanup of + Nsock events), even though everything went fine. + + luis@Aberdeen:~$ sudo nping --ec public echo.nmap.org -p90 --tcp --count 1 + + Starting Nping 0.5.52.IPv6.Beta2 ( https://nmap.org/nping ) at 2011-07-05 12:51 CEST + SENT (1.8965s) TCP 163.117.203.253:64288 > 74.207.244.221:90 S ttl=64 + CAPT (2.0293s) TCP 163.117.203.253:64288 > 74.207.244.221:90 S ttl=54 + RCVD (2.1233s) TCP 74.207.244.221:90 > 163.117.203.253:64288 RA ttl=51 + nping_event_handler(): READ-PCAP killed: Resource temporarily unavailable + nping_event_handler(): TIMER killed: Resource temporarily unavailable + + Max rtt: 226.762ms | Min rtt: 226.762ms | Avg rtt: 226.762ms + Raw packets sent: 1 (40B) | Rcvd: 1 (40B) | Lost: 0 (0.00%)| Echoed: 1 (40B) + Tx time: 0.00136s | Tx bytes/s: 29411.76 | Tx pkts/s: 735.29 + Rx time: 1.00082s | Rx bytes/s: 39.97 | Rx pkts/s: 1.00 + Nping done: 1 IP address pinged in 2.93 seconds + +* Investigate about warning on old version of gcc like g++ 4.1.2 20080704 + (Red Hat 4.1.2-48). No warnings are shown on newer version but it would be + nice to get rid of them if possible. There are some of them: + + ARPHeader.h:169: warning: ‘class ARPHeader’ has virtual functions but + non-virtual destructor + RawData.h:99: warning: ‘class RawData’ has virtual functions but + non-virtual destructor + +* Decide more on rDNS + - Do we want to rDNS resolve all target IPs? If so, where should we + show the name? At the final report (even when just one host + scanned, which omits that line now)? In the individual packet + trace lines? When a CNAME (or a name which forward resolves but + does the IP doesn't reverse resolve) is specified on the command + line, should we use that version, or the official rDNS, if any? + - Some more discussion on this topic on nmap-dev may be warranted. + +* Improve output for negative verbosity levels. Currently, one can't + even tell how many hosts replied, just how many responses were + received, which could be all from the same host. If there is only + one target, then the current behavior is fine. However, when pinging + more targets, we should be able to provide a better output; at least + how many hosts were alive. This was suggested by Dan Farmer. + +* Consider adding more examples of setting fields/payloads to the man + page. This was suggested by Dan Farmer. + +* Consider adding support for XML output. + +* From: David Lam <david@thedavid.net>, "Some general questions about + Nping/Ncat" + + In TCP traceroute mode, would it be possible to ask Nping to + stop once it gets an SYN-ACK response back from the destination host rather + than continuously hitting the host until the max TTL? + +* Make broadcast ping work. Currently the following command does not + show any captured packets: + nping 192.168.0.255 --dest-mac ff:ff:ff:ff:ff:ff -c 1 + The cause is probably the BPF filter, which only allows replies from + 192.168.0.255. + Also, look into official multicast addresses like 224.0.0.1. Can we + received replies to that probe? + + +* Do some performance testing. + Fyodor: + <<Nping should be able to send packets quickly, at least comparable to + "ping -f" and hping. If it can't send as many packets per second as those, + then it warrants looking into whym figuring out what the bottlenecks are. + It would be good to compare nping with other tools such as hping in + terms of how high the values of packets per second can get and still + work reliably.>> + +* Stats for ARP packets. + +* Do more testing on Mac + +* Support pre defined probe rates: --fast, --faster, --flood, --slow, + --slower, --paranoid... + +* Think about --establish feature, which uses raw packets to establish + a connection and can then send data on the connected stream (Luis + already has a proof-of-concept implementation). + +* Make privileged and unprivileged TCP/UDP mode specification consistent. + +> - User is unprivileged and did not supply mode: --> Use TCP-Connect +> - User is unprivileged and supplied --tcp --> Use TCP-Connect +> - User is unprivileged and supplied --upd --> User UDP unprivileged +> - User is root and did not supply mode --> Use ICMP Echo +> - User is root and supplied --tcp --> Use raw sockets TCP +> - User is root and supplied --udp --> User raw sockets UDP +> - User is root and wants to use TCP-Connect --> User needs to either +> pass --tcp-connect or --unprivileged +> - User is root and want unprivileged UDP --> User needs to pass +> --unprivileged or --udp-XXXXX (any suggestions?. --udp-sendto() may not +> be the best idea because when we use raw sockets we also use sendto() to +> transmit the data). + +* Support reverse DNS resolution in --traceroute + +* Implement TCP options + +* Implement hping-like ability to change the port/ttl using the keyboard + during a scan. + +* Disable ARP resolution when --source-mac is specified. + +* Implement --data-file option. What should we do if file is big? Read the + first X bytes? Send consecutive chunks? + +* Implement ICMP address mask + +* Implement entire ICMP Traceroute message opts. + +* Research on default IP Identification value. Kernel does not seem to like + value 0 because when set to zero, kernel changes it to some other value. When + we set it to something !=0, the kernel leaves our value untouched. + +* At some point in the future, implement weird ICMP Types. I think this would + let us make a difference to the rest of pings and packet creation tools + because anyone wanting to send weirds packes would have to download our + Nping ;-) + ( http://www.iana.org/assignments/icmp-parameters ) + 6 Alternate Host Address [JBP] + 31 Datagram Conversion Error [RFC1475] + 32 Mobile Host Redirect [David Johnson] + 33 IPv6 Where-Are-You [Bill Simpson] + 34 IPv6 I-Am-Here [Bill Simpson] + 35 Mobile Registration Request [Bill Simpson] + 36 Mobile Registration Reply [Bill Simpson] + 39 SKIP [Markson] + 40 Photuris [RFC2521] + +* Implement checks in function that handles received packets: + Fyodor: + <<You can't assume that the filter always works right, so you do need to + validate the information anyway. For example, on windows in some cases + we have to change the filter to "" because it doesn't work otherwise + so, in actuality, I often end up with rather broad pcap filters and then + do the checking by hand, but tightening the pcap filter can improve + performance a bit.>> + +* Implement "-iL inputfilename (Input from list) " and the case where "-" is + supplied and target specs need to be read from stdin. + +* Consider adding option to allow sending NO packets but act as a + simple sniffer. Users could use --bpf-filter to specify a + tcpdump-like filter and get every receive packet printed to + stdout. Maybe with "-c 0"? "-c none"? We need to have some flag in + NpingOps so we don't terminate Nping but wait undefinitely. + +* At some point we should support nmap-like MAC specification. + +* When implementing IPv6, check MAX_TCP_PAYLOAD_LEN constant and method + TCPHeader::setSum(). Because with IPv6 the max payload length should be 20 + bytes less than with the IPv4 header. + +* When using payloads, take into account that the IP and TCP headers may + contain options and therefore, the maximum payload len should be + 65535 - 20(ip header) - 40 (ip options) -20(tcp header) -20(tcp options); + +* Make sure randomnly generated checksums in IPv6-TCP/UDP are in fact invalid + and don't match the correct checksum. + +* Fyodor: + <<in some cases it might be nice to have an option which sends all + probes (all ports to all hosts) at the same time.>> + +* ARP mode does not support payload specification. However, users may + want to do things like appending null bytes at the end of an ARP + packet to test some device behaviour, etc. Adding support for + payload to this mode is really trivial, would make the payload spec + more consistent with the rest of the modes, and may be a nice to have + feature. + +* [EM] For CAPT packets, decide if we want to print the full info or + just the fields that have changed in transit (or both). Note that + printing differences would be complicated by the fact that nping + doesn't currently associate captured packets with the original send. + +* Decide if we want to allow things like "1074628148" or "0x400d8634" to + be treated as valid IP addresses. + +* Check out if --ip-options "RTUS 1.1.1.1 2.2.2.2" makes sense. It now + fails. + +* It may be nice to let users set the IP header lenght field. Maybe they + want to stress tcp/stacks with this. + +* Investigate on ICMP preference levels. It's not clear whether there is + a standard encoding or not. The logic that parses this in Nping needs + to be reviewed. + +* Split up libnetutil.cc into different source files. + +* Investigate on nping's version of devname2ipaddr. Think about side + effects on using that in Nmap. + +* Consider adding multi-packet support. + o Example: tell nping to send 4 tcp packets, 5 icmp packets and 3 udp packets + +* Consider adding RFC-style output for send/recv packets. + +* Consider adding more detailed stats for the Echo Mode. + +* [EM] Handle DLT types. Currently the server always sets the null DLT value + that indicates that no data link header is included. + +/***************************************************************************** + * Things that have been solved already * + *****************************************************************************/ + +[DONE] Add default target port for TCP-Connect and TCP modes :: Port 80 + +[DONE] Add default target port for UDP mode :: Port 40125 + +[DONE] Add default UDP Source port: 53 + JUSTIFICATION: From David's EffectivenessOfPingProbes + http://www.bamsoftware.com/wiki/Nmap/EffectivenessOfPingProbes + "The best individual UDP probes are still those to a random high port, + with a source port of 53 and a non-empty payload. Even without the source + port and payload, the ports 40125 and 40126 that I picked out of the air + are better choices than the current default of 31338, finding around 400 + additional hosts." + +[DONE] Change resolution for the inter-ping delay. (Fyodor: btw, usleep() will + probably do the trick for you as it let's you sleep with microsecond + precision) + +[DONE] Use int send_ip_packet(int sd, struct eth_nfo *eth, u8 *packet, unsigned int + packetlen) instead of ip_open(); + +[DONE] Add protocol to BPF filterstring because It is possible that when in TCP mode + a UDP packet destined to the TCP source, arrives to the net iface and gets + printed. + +[DONE] Implement multiple port specification. + +[DONE] Implement ICMP router advertisement entries + +[DONE] Default probe mode: ICMP echo + +[DONE] Test ICMPv4Header::addRouterAdEntry() and check entries are being added + correctly. + +[DONE] Determine source IP address automatically + +[DONE] Determine network interface to be used for packet capture automatically + +[DONE] Add support for cached DNS requests + +[DONE] Start user documentation (mainly man page) + +[DONE] Change output to include timing information + +[DONE] Implement controls in payload options parsing to prevent specifying lengths + that cannot be carried by a single TCP/UDP packet. + +[DONE] Start implementing unprivileged UDP pings. + +[DONE] When sending ICMP packets, checksum is not being computed correcly if + --data-length, and options like that, are specified. + +[DONE] Find a bug that under some circumstances produces a segfault. It is probably + related to the way option -e is being handled. + +[DONE] Fix a bug in option "-e iface" that results on IP 2.0.0.0 being used as a + source address. + +[DONE] Update --help display to include new ICMP flags. Check also commandline syntax + docs. + +[DONE] Use nsock approach instead of threads. + +[DONE] Finish ARP/RARP support. + +[DONE] Change doc for option --count. We don't stop after N probes, we stop after + N rounds. + +[DONE] Ask Fyodor what tool is used to convert from nmap-man.xml to nmap.1 + +[DONE] Check all outPrint()s and outError()s to ensure they specify the correct + verbosity/debug level. + +[DONE] Document format specified in ArgParser::atoICMPType(). + +[DONE] Document format specified in ArgParser::atoICMPCode(). + +[DONE] Finish implementing unprivileged UDP pings. + +[DONE] Finish Ethernet frame creation. + +[DONE] Find a way to convert the nping.xml into man page. + +[DONE] Check what happens if payload is specified and we are not sending TCP/UDP + but ICMP or other proto packets. [Sometimes it may not make sense to include + payloads (e.g. ARP) but we still allow it just in case users want to play + around]. + +[DONE] Ask Fyodor whether we want to display elapsed time (like nmap) or we prefer to + display rtt time as other ping utilities do. [This is probably fine for now] + +[DONE] Fix the warnings produced by Fyodor's gcc. + +---------------+ + NpingTargets.cc: In member function ‘int NpingTargets::processSpecs()’: + NpingTargets.cc:315: warning: comparison between signed and unsigned integer expressions + NpingTargets.cc: In member function ‘NpingTarget* NpingTargets::getNextTarget()’: + NpingTargets.cc:333: warning: comparison between signed and unsigned integer expressions + +---------------+ + In file included from /usr/include/string.h:640, + from nbase/nbase.h:158, + from nping.h:107, + from utils.cc:95: + In function ‘void* memset(void*, int, size_t)’, + inlined from ‘int getNetworkInterfaceName(sockaddr_storage*, char*)’ at utils.cc:689: + /usr/include/bits/string3.h:85: warning: call to void* __builtin___memset_chk(void*, int, long unsigned int, long unsigned int) will always overflow destination buffer + +---------------+ + + +[DONE] Redesign verbosity levels: + * Put verbosity levels 2 into level 1 + * Use level 2 for error. + * Use level 3 to print everything but not sent/rcv packets. + * Level 4 the usual + +[DONE] Add stats at the end of nping execution. + +[DONE] Add options to disable viewing of sent packets. + +[DONE] Add option to to disable packet capture. + +[DONE] Add a section to the man page explaining how we iterate over targets, + ports, etc. + +[DONE] Beta-testing email to the list. + +[DONE] Change default round count to 5. + +[DONE] Fix a segfault detected by Fyodor in trg=o.targets.findTarget(...). + +[DONE] Send an email to the list telling about the nping.exe file. + +[DONE] Support CTRL-C statistics. + +[DONE] Change "solution" file in mswin32/nmap.sln to nping.sln + +[DONE] In man page and -h: move Ethernet section so it appears after network + layer info. + +[DONE] Make rx time more accurate taking into account that we wait for a bit after + the last probe is sent. + +[DONE] Fix bug: add ICMP dest unreachable, etc to the BPF filter so we can get + icmp error messages when TTLs expire, etc. + +[DONE] Disable all ethernet related code when sendEth is false. + +[DONE] Finish porting Nping to Windows. + +[DONE] Find an OS X box to test Nping. + +[DONE] Reorganize verbosity levels (again ;-) [-3, +3]. + +[DONE] Finish documentation for options --source-mac and --dest-mac + +[DONE] Make sure --ether-type supports specifying types in hex. + +[DONE] Implement verbosity level 3: in this level, sent and recv packets are + hexdumped to stdout. + +[DONE] Write and check in nping/index.html web site + - Include SVN checkout/install instructions + - include tarballs when available + +[DONE] Create Windows installer (maybe can copy a lot of stuff from what + Ithilgore has done with Ncrack) + +[DONE] Create Nping release tarball for UNIX systems + +[DONE] Release Nping 0.1BETA2 + +[DONE] Man page should say Nping is currently in Alpha stage. + +[DONE] Support -vvv, -qqq and -ddd syntax. [Requested by Dirk Loss] + +[DONE] Create Mac OS X installer (also can probably copy a lot of stuff + from what Ithilgore has done with Ncrack. David can usually help + with installer building). + +[DONE] Move nping to /nping in SVN rather than being in nmap-exp + +[DONE] Set up automatic conversion from nping XML man page to HTML for + https://nmap.org/nping/man.html [Fyodor working on this] + +[DONE] Include signature files in new releases. [Requested by Henri Salo] +[DONE] It would be nice to have Bzip2 packages. [Requested by Henri Salo] + (These last two don't make sense anymore as Nping is now distributed + with Nmap). + +[DONE] Do small fix in nmap's send_ip_packet_sd() + - res = Sendto("send_ip_packet", sd, packet, packetlen, 0, + + res = Sendto("send_ip_packet_sd", sd, packet, packetlen, 0, + +[DONE] Correct BPF filter specs, to make the condition about the source + address apply everywhere. + +[DONE] Fix possible bug in BPF filter specification. More details in + http://seclists.org/nmap-dev/2010/q2/252 + +[DONE] Work on nping&nmap code merge. + +[DONE] For options that take numbers we need to allow users to specify them + also in hex with the format 0xNNNN... + +[DONE] Replace this pattern: + if ( isNumber_u32(optarg) ){ + u32 aux32 = strtoul( optarg, NULL, 10); + ... + } + with a function that checks for syntax and returns the value (i.e., a wrapper + around strtoul). There is nowhere that isNumber_u* is called without it being + immediately followed by a strtoul, outside of utils.cc. + +[DONE] Bug in --icmp-advert-entry. Specified IPs are being set in host byte + order instead if in network byte order. + +[DONE] Investigate why ARP replies are not being received. Wireshark shows + replies but they don't get captured by Nping. The bpf filter looks + ok: "arp and arp[6]==0x00 and arp[7]==0x02" + +[DONE] Investigate into this: + sudo nping --icmp scanme.nmap.org -vvv -d1 --icmp-type ra --icmp-advert-entry 256.257.258.259,222 + Invalid Router Advertising Entry specification: Unable to resolve 6628128 + Apparently the call to outFatal() is specifying %d instead of %s, but + that's not being detected properly by the compiler, because we don't + get a warning. We have to do something like this: + void fatal(const char *fmt, ...) + __attribute__ ((noreturn)) + __attribute__ ((format (printf, 1, 2))); + TODO: Look at the documentation to see what the numbers mean. + Probably one of the is the index of the format argument, and the + other is where the varargs start. + +[DONE] Fix division by zero exception: + sudo nping --icmp scanme.nmap.org -vvv -d1 --icmp-type echo --rate 0 + ./test_nping.sh: line 83: 11690 Floating point exception"$@" + +[DONE] Fix little problem in TIMING_5. We need to detect the bogus time + before we actually pass the value to NpingOps. Nping is giving an + error but the bogus input is getting to far. + +[DONE] Document that badsum-ip may not always work because the kernel may + correct the sum. + +[DONE] Change overloaded functions in libnetutil that were refactored to + make them compile in C. Go back to the overloaded version if possible. + +[DONE] Move grab_next_host_spec() and pals to netutil. + +[DONE] Control the case when user passes "--mtu 0". An assertion fails but + Nping should print a nicer message. + +[DONE] Improve error message for --mtu. We should probably allow mtu's bigger + than 2^16 but take that as a "dont fragment" request. Also, make + "rand" produce only valid MTUs (multiple of 8, etc). + +[DONE] When passing "--tcp-flags 0x100" the error is not very accurate. + This is because parser_u8() fails and then Nping tries to resolve the + value letter by letter. Maybe we can parse_u32() it, and then check + if n<255 and print a better error message. + +[DONE] Document what happens with the IP header length when user wants to + add uneven bytes of IP options. We are truncating the result, because + the header length is expressed in 32 bit words. + +[DONE] Check if there is any problem with -e "". Maybe we shouldn't let users + supply a NULL name, but make them use the "any" specifier. Add doc + about this and update the test description (MISC_12). + +[DONE] Update documentation for option --delay, including that now, time + specification as float numbers is supported (eg: --delay 0.1 meaning 100ms) + +[DONE] Change info about TODO file in https://nmap.org/nping web page. + - If you wish to contribute code to Nping there is a TO-DO list you can have + - a look at (file "TODO" in the source package). + + If you wish to contribute code to Nping there is a TO-DO list you can have + + a look at (file "todo/nping.txt" in nmap's source package). + +[DONE] Make sure randomnly generated checksums are in fact invalid and don't match + the correct checksum. There is a 1/65535 chance of this happening. + +[DONE] After merging nmap-dedup, change send_frag_ip_packet() to take "u32 mtu" + and fix the printf below to use "%u" instead of "%i". + +[DONE] [EM] Update EchoProtoRFC.txt and any of the other design files as + appropriate and send to nmap-dev for comments + +[DONE] [EM] Pick a default port number + +[DONE] [EM] Make a mockup of the desired standard output in a regular echo mode + execution, like nping -c 2 --tcp --flags SYN -p 80 scanme.nmap.org (let's + assume there are some differences found, like a NAT is in place) + o A key aspect of this task is determining what diffs are going + to look like. + +[DONE] [EM] Things to decide on: + o Decide on packet specifiers that can be passed to the server so it + can recognize packets sent by the client even if a number of headers + have changed and pass them back. (see Fyodor/Luis IM discussion logs + from 6/28/10). + +[DONE] [EM] Improve client error handling. Currently it doesn't behave well when + the server crashes. + +[DONE] [EM] Make the client timeout if the server does not send data during + handshake. Currently the client waits forever. + +[DONE] [EM] Make the server detect when a client disconnects and delete its context + data. + +[DONE] [EM] Get rid of some messages that are currently displayed in the client. + Print them only if debugging level is high enough. + +[DONE] [EM] Make sure -h help screen includes info about the echo mode. + +[DONE] [EM] Add echo mode to the man page. + +[DONE] [EM] Add received echoed packet to the final statistics. + +[DONE] [EM] Multi-client support + +[DONE] [EM] Delay RECV message printing so the CAPT messages are shown in order. + +[DONE] [EM] Use NEP_QUIT only if necessary, just close connection if possible. + +[DONE] [EM] Implement crypto + +[DONE] [EM] Consider whether the CAPT line should (or should have an + option to) display the time based on capture time from the server. + Obviously this can be problematic because not all machines run + ntpd. One option is to just make it an option so that people should + only use it if both the client and server are running ntpd. Luis is + adding a precision timestamp to NEP_ECHO packets so we could easily + add it in the future. Another approach would be to do NTP-style + handshaking to compute time offsets between the two machines during + the echo side-channel handshaking. Then the client could remember + how far off it is. A third approach is to guess about the CAPT time + that it was 1/2 the time between packet send and when we received + the NEP_ECHO back notifying us of receipt. + NOTE: We finally decided to take the third approach. CAPT_time=RTT/2. + +[DONE] [EM] Consider whether we should delay RCVD packet printing + slightly so that CAPT packets received just slightly afterward could + be printed before the RCVD. This might make the most sense if we do + the previous feature where we show the time that a packet was + actually captured by echo server. If we did it in normal cases, it + might make it easier to compare SENT and CAPT packets, but would + also be a bit strange to see the timeline out-of-order. + +[DONE] Fix Windows rtt values. Right now Nsock does not seem to be giving + the callback at the proper time, or something. + +[DONE] Add --no-crypto to -h output. + +[DONE] Make sure nping does not allow generating packets with tcp src port or + tcp dst port 9929 (or --echo-port N, if that is set), because 1) the + echo server does not capture those packets and 2) to avoid messing up the + established side-channel tcp connection. + +[DONE] Add support for custom IP binding: if user supplies -S then + the echo side-channel connection and connections in TCP-Connect mode should be + established from that IP. This includes the echo server binding to that IP. + +[DONE] Make nping issue a warning when user supplies a payload in TCP-Connect + mode. + +[DONE] [EM] Echo server should print which interface is using to capture packets. + +[DONE] In some cases, when using nping through a VPN connection, nsi_pcap_linktype() + returns something different to DLT_EN10MB, and Nping fatals. Investigate + why this happens to nping and is not a problem for Nmap. Also, determine + why this doesn't happen all the time. What does it change between these + two?: sudo nping --udp 1.1.1.1 -g 999 -p998 + sudo nping --udp 1.1.1.1 -g 999 -p999 + The first one works, and the other one fatals with the "Currently only + Ethernet is supported." (error message @ nping.cc:1717). + - Note this also happens when Fyodor uses Nping tethering through + his cell phone (ppp0) + +[DONE] [EM] Make the server stop capturing packets when all connected clients + finish their session. + +[DONE] [EM] Some things to keep in mind for the implementation and to update + our design docs accordingly: + o Implement different "modes" for the server: complete access, + one-time-access, and restricted. + +[DONE] Do more testing on MS Windows. + +[DONE] [EM] Investigate why the echo server does not send NEP_ECHO messages when the + client sends probes at a very high rate, like in : + ./nping -c 1000 --rate 1000 --echo-client "pass" --icmp -v echo.nmap.org + +[DONE] [EM] Add echo mode to the man page + + +[DONE] [EM] Do some extensive testing of the Echo mode once it is working + to try and flesh out any bugs before merging. + +[DONE] Make Nping call nsi_delete() on pcap IODs, IODs in TCP-Connect mode and maybe + in IODs of other modes. See http://seclists.org/nmap-dev/2010/q3/587 + +[DONE] Fix bug that causes Nping to fail when sending UDP packets to a broadcast + address. More info: <http://seclists.org/nmap-dev/2010/q3/752> + +[DONE] When doing ICMP echo traceroute (with --traceroute), unless the user + supplies a custom round count (-c/--count), Nping only sends 5 packets + (default round count). This is usually not enough to reach hosts + on the internet. What should be the default behaviour? Stick with the + default round count of 5 or increment it when --traceroute is set? + - We should probably set -c 32 when --traceroute is specified, + unless user specifies their own -c explicitly. + +[DONE] Try to reduce the size of the internal buffer in the EchoHeader class. + Currenltly it allocates a big buffer that is able to hold the theoretical + maximum size of a NEP message (normal use does not require so much space). + When this is done, check if we still need to increase the stack size + in the project properties in Visual Studio. + +[DONE] [Fixed by Vasiliy Kulikov] When running Nping in ARP mode, hexdump of + ARP replies is not shown with -vvv, only for requests. Here's the output: + +sudo nping --arp 192.168.240.139 -vvv -d1 + +Starting Nping 0.5.59BETA1 ( https://nmap.org/nping ) at 2011-07-11 12:32 CEST +BPF-filter: arp and arp[6]==0x00 and arp[7]==0x02 +SENT (0.0562s) ARP who has 192.168.240.139? Tell 192.168.240.1 +0000 ff ff ff ff ff ff 00 50 56 c0 00 01 08 06 00 01 .......PV....... +0010 08 00 06 04 00 01 00 50 56 c0 00 01 c0 a8 f0 01 .......PV....... +0020 00 00 00 00 00 00 c0 a8 f0 8b .......... +RCVD (0.0568s) ARP reply 192.168.240.139 is at 00:0C:29:E4:90:CD +SENT (1.0580s) ARP who has 192.168.240.139? Tell 192.168.240.1 +0000 ff ff ff ff ff ff 00 50 56 c0 00 01 08 06 00 01 .......PV....... +0010 08 00 06 04 00 01 00 50 56 c0 00 01 c0 a8 f0 01 .......PV....... +0020 00 00 00 00 00 00 c0 a8 f0 8b .......... + + diff --git a/todo/patrick.txt b/todo/patrick.txt new file mode 100644 index 0000000..7508afd --- /dev/null +++ b/todo/patrick.txt @@ -0,0 +1,77 @@ +=== + +Currently working on: + +-- LPEG in NSE. + +-- HTTP Library in LPeg. + +=== + +Maybe: + +-- NSE Debugger. Look at Diman's implementation: + http://seclists.org/nmap-dev/2008/q1/0228.html + http://www.keplerproject.org/remdebug/ + +-- Review NSE Nsock Socket Allocation: + o Dynamically increase socket slots if nothing has been done + in the last ~5 seconds. Also decrease once traffic is working again. + This resolves any sort of socket deadlock. + +-- Deadlock identification and correction: + o Add detection for deadlocks and print which threads are involved. + o use above results to make a strategy for automatic deadlock resolution. + +-- Look into moving Packet Module to C. + +=== + +Done: + +-- Review and Improve NSE Nsock Library. + o Move away from C pointer references and allocation over to Lua. + If a function ends in error, all the userdata will be collected. + We would otherwise need to use pcalls everywhere to clean up + and free malloc()'d memory. + o Use thread calling nsock_loop (or currently running thread) + for restoring waiting threads to the running queue. + Making a function call on a yielded thread is a hack and + could cause problems in the future. + o Get rid of the static nsock_pool and use a dynamically allocated + structure on a per-host-group basis. + o Prepare for Lua 5.2 --> Change to real errors. + +-- Update NSE Book Implementation Section. + +-- Added boolean operator patch. + +-- Update NSE --script section (book) to include Boolean operators. + +-- Fix ceil for runlevels. + +-- Solve Brandon's Segfault for thread's sockets and close them when + the thread ends. + +-- Change the error on finding the name of a nonexistent file in script.db + into a non-fatal warning. + +-- Correct nsock_connect to unlock the socket slot if the connection fails. + +-- Remove packet.hextobin and packet.bintohex. Fix scripts that used them + to instead use bin.(un)pack. + +-- Commit --script-args patch and update the relevant section in the book. + +-- Deadlock identification and correction: + o Release mutexes upon script death. + +-- Review NSE Nsock Socket Allocation: + o Release socket locks on connection failure or timeout. + o Track active sockets in the nsock library and don't rely on + garbage collection for reallocation. + +-- HTTP Caching: + o Add ability to use a proxy to http.lua. + o Test http.lua performance using local caching proxy. + o Implement a cache in http.lua. diff --git a/todo/paulino.calderon.txt b/todo/paulino.calderon.txt new file mode 100644 index 0000000..6f88523 --- /dev/null +++ b/todo/paulino.calderon.txt @@ -0,0 +1,4 @@ +TODO: + +-Update wiki page. +-Fix: http-enum does not work on windows. UNIX paths are hardcoded into the script. It also fails when running from a directory with spaces in the name.
\ No newline at end of file diff --git a/todo/sctp.txt b/todo/sctp.txt new file mode 100644 index 0000000..55bf042 --- /dev/null +++ b/todo/sctp.txt @@ -0,0 +1,49 @@ +TODO.sctp $Id$ -*-text-*- + +o Further investigate SCTP functionality, as some people reported + problems (see this thread: + http://seclists.org/nmap-dev/2009/q2/0669.html) + +o Add support for UDP encapsulated SCTP (9899/udp). + Basically just wrap the SCTP packets into a UDP packet. + Think about how to add support for this to libdnet first. + See this Internet Draft by Michael Tuexen for the specs: + http://tools.ietf.org/html/draft-tuexen-sctp-udp-encaps + This is actually quite a challenging task due to the + current architecture of the scan engine. How to best + differentiate a UDP packet related to a UDP scan from a + UDP wrapped SCTP packet? How to unpack the UDP wrapped + SCTP packet in order not to duplicate a lot of code? + A good solution will be non-trivial. + +o Verify ICMP response handling for SCTP. Make sure all + ICMP types are handled in an optimal way (esp. destination + unreachable: protocol unreachable). + +o Consider removing 9899/sctp from the default port list. + 9899/udp is used for UDP encapsulated SCTP. One reason + to keep 9899/sctp is likely misconfigurations. + +o Investigate whether it makes sense to store scan state in + the itag/itsn fields for INIT scans. + +o Investigate the suitability of other SCTP chunks for port + scanning and implement more scan types if they turn out to + be worthwhile. One unverified idea is to experiment with + undefined chunk types and their first two magic bits to + provoke ERROR responses. + +o Add SCTP based service probing. + +o [Ncat] Consider implementing SCTP broker mode. + +o [NSE] Add SCTP support to NSE. + +o Investigate on differences between SCTP stacks and + implement SCTP based OS detection probes based on the + results. For example, BSD systems send the ASCII string + KAME-BSD in INIT-ACK chunks. + +o SCTP-enable scanme.nmap.org in order to make scanme.roe.ch + obsolete. + diff --git a/todo/shinnok.txt b/todo/shinnok.txt new file mode 100644 index 0000000..b294e42 --- /dev/null +++ b/todo/shinnok.txt @@ -0,0 +1,150 @@ +In progress: +============ + +o We should offer partial results when a host + timeouts. I (Fyodor) have been against this in the past, but maybe + the value is sufficient to be worth the maintenance headaches. Many + users have asked for this. If we do implement this, we may want to + only print results for the COMPLETED phases (e.g. host discovery, + port scanning, version detection, traceroute, NSE, etc.) Trying to + print partial results of a port scan or NSE or the like might be a + pain. And if we print some results for a host which timeouts, we + should give a very clear warning that the results for that host are + incomplete. As an example, here is someone who hacked Nmap source + code to achieve this: http://seclists.org/pen-test/2010/Mar/108. + o Another benefit would be that it would allow us to clean + up/regularize the host output code. Right now there are I think + three places where a host's final output can be printed. If, + instead, that code just looked at what information was available and + printed that out only, we could potentially isolate it in just one + place. + o This also might let us provide a feature for skipping the rest of + an Nmap phase which is going too slowly (I think that has its own + Nmap TODO item). + +Hanging(waiting for further input, etc..): +========================================== + +o Nmap *poor's man* test suite by expanding on what I already have in + /nmap-exp/shinnok/nmap-test-script. + +o NMAP reports different service results every so often with the same port. + http://seclists.org/nmap-dev/2011/q2/815 + +o Review latest revision of Marek's ncat_proxy.patch - DONE + http://seclists.org/nmap-dev/2011/q2/573 + o Commit approval pending + +Pending: +======== + +Pending (low priority): +======================= + +o E-mail nmap-dev with GProfiles /ncrack + o Create new default username list: + http://seclists.org/nmap-dev/2010/q1/798 + o Could be a SoC Ncrack task, though should prove useful for Nmap + too + o We probably want to support several lists. Like an admin/default + list like "root", "admin", "administrator", "web", "user", "test", + and also a general list which we obtain from spidering from + emails, etc. + +Potential: +========== + +COMPLETED: +========== + +o Add a --append-output option to ncat. [DONE - r25737] + +o libpcre/pcre.h - is cleared upon make distclean thus leaving the SVN + working directory dirty + http://seclists.org/nmap-dev/2011/q2/708 + +o De-duplicate code by unifying ncat_broker.c and ncat_listen.c code paths, + either as a single file in ncat_listen.c or merge duplicate code in + ncat_listen.c and keep only broker specific code in ncat_broker.c(it it's a + lot of code, otherwise ncat_listen.c would do just fine). + +o Nmap should defer address parsing in arguments until it has read + through all the args. Otherwise you get an error if you use like -S + with an IPv6 address before you put -6 in the command line. You + get a similar problem (on David's IPv6 branch) if you do "-A -6" + (but "-6 -A works properly). + +o Delve into Lua and NSE and try to write some scripts to get the hang + of it and gain a better understanding of the NSE engine in Nmap. + o Written two NSE scripts, http-reverse-ip and http-google-email that + can be found in /nmap-exp/shinnok/nse. + +o E-mail nmap-dev with QtCreator usage steps for Nmap + +-- +o Ncat hangs on ssl -> REFACTORING + some refactoring left to be done to reduce code duplication + http://seclists.org/nmap-dev/2011/q2/842 + o Commit current switch/ifdef refactoring patch. + o Research code deduplication even further. + +o Ncat chat (at least in ssl mode) no longer gives the banner greeting + when I connect. This worked in r23918, but not in r24185, which is + the one running on chat.nmap.org as of 6/20/11. Verify by running + "ncat --ssl -v chat.nmap.org" + +o Pending uncompleted SSL handshakes when in --exec* listening mode make + Ncat consume 100% cpu(core/thread). + Possible solutions: + o Listen on the union of the two sets in ncat_listen.c composed of the + current set and a secondary one, ssl_pending which should include the + pending ssl hanshake sockets. + o Timeout ssl handshakes. + o Delay adding the exec output pipes to fselect/WaitForMultipleObjects + until the ssl handshake has been completed. + http://seclists.org/nmap-dev/2011/q2/988 +--- + +o Fix ncat.xml(the input for the man page) examples section. - David came up + with the final right fix on this one. + +o Ncat should close its socket and refuse further connections after the first + one, if invoked without --keep-open. That's what traditional netcat does + too. - DONE [r24197] + http://seclists.org/nmap-dev/2011/q2/944 + o Add TEST in ncat-test.pl - DONE [r24373] + +o Closing Zenmap without stopping the scan first will leave nmap running in + the process list on Windows. [r24308] + [Actually, Zenmap was unable to kill the nmap scan processes at all on + Windows] + +o Zenmap should wait for the return exit code of the nmap scanning subprocess + upon killing it(canceled scan), otherwise the subprocesses will enter a + defunct(zombie) state.[r24235] + +o Fix build_icmp_raw and build_igmp_raw filling the packet data payload + with zeroes instead of the supplied random data, when nmap is invoked + with --data-length.[r24127] + +o Investigate and document how easy it is to drop Ncat.exe by itself + on other systems and have it work. [r24242] + http://seclists.org/nmap-dev/2011/q2/1090 + + o We should also look into the dependencies of Nmap and Zenmap. + It may be instructive to look at "Portable Firefox" + (http://portableapps.com/apps/internet/firefox_portable) which is + built using open source technology from portableapps.com, or look at + "The Network Toolkit" by Cace + (http://www.cacetech.com/products/network_toolkit.html). + +o --max-conns is broken in latest svn -> fixed in r24130, other two + bugs discovered: + o --max-conns 0 kills ncat with a glibc assertion error on calloc with + zero as nmemb(??) at: + init_fdlist(&broadcast_fdlist, o.conn_limit); + o When killing the first initiated connection on --max-conns > 1 Ncat: + Ncat: Program bug: fd (5) not on list. QUITTING. + [DONE]The previous two bugs were introduced in r24130, they are now fixed + in r24193. + |