Surprise, FastFind NX is coming (very) soon !

Chris and I came up with an idea for a way to increase the crawling speed of FastFind that I started playing around with at home. While I was doing that I came up with a few different ideas for features and how I could implement them into FastFind so I started coding them up and realised that they were significant enough that it was worth doing a new release for. I think they will make a big difference to FastFind, hopefully you will too.

Out of this list the only thing left to do is to finish off the PDF parser but after that there will be lots of testing and polish but I am hoping to at least have a beta out in about a month so it's actually pretty close to being done.

• New control panel interface bringing it in line with our other NX products
• Improved crawler so that it only crawls the site once. The previous crawler would crawl a site twice, once for links and a second time for content. This would allow it to display a progress bar. FastFind NX will only do a crawl once and use the previous amount of pages as a guide for displaying the progress bar. As a result the progress bar may not seem to behave correctly when you have added / removed a large amount of pages to your site, however crawls should be at least roughly twice as fast as the previous version.
• Improved crawler that can greatly reduce the amount of content transferred and the speed of a crawl. If you have the php-curl module installed then FastFind can now detect if a page has changed since the last crawl (assuming your server returns the correct headers) and if it hasn't it can skip transferring the file again. This will be especially beneficial for sites with large amounts of static content.
  

• Modular Parser System
  
  A FastFind NX parser is a way to extract some content from a file that you want to be able to search.The new parser system allows you to create a simple php object which extends a base Parser class and you just have to implement the Parse() function. Then you can assign it using the FastFind NX web interface to any url/file extension you like. FastFind NX will ship with the following parsers
  
  • Default html parser
  • New PDF parser 
  
  Other potential parsers could include
  
  • A parser to extract the exif information from photos (I will probably create this one as a way to document the process of writing the parser)
    
  • A parser to extract the ID3 information from MP3's
  • A parser to extract the text from word documents
  • A parser to allow you to pass the content to an external application to extract the text for you
  • Get a list of filenames from a zip file
  • And many more
    
    
• Modular Reporting System
  The new reporting system allows you to write your own reports by creating a file in the reports directory with 2 functions, Intro() which defines a welcome page which explains what the report is going to do and takes any input required from the user and Report() which generates the actual report. FastFind NX will ship with at least the following reports
  
  • Broken Links Report
  • Page Rank Report
    
  
  Other potential reports could include
  
  • Keyword density
  • Check for spelling errors on pages
  • Meta Keyword phrase generator
  • Sitemap generators
  • And many more
    

• Search Weighting system
  
  Control the ordering of your search results by applying weights (importance) to various parts of each page. Using this you can greatly improve the relevance of your search result ordering. The metrics that you can adjust for this are
  
  • Keyword Score
  • If the keyword is the page name
  • If the keyword is in the page name
  • If the keyword is in the url
  • If the keyword is in the title
  • The number of internal backlinks to the page
  • The length of the url
  • The depth of the url from the intial crawl page
  • The size of the page content
    

• Ability to honour robots.txt file which should make getting started with FastFind on an existing site even easier
• Ability to clear the search stats from the settings page
• Ability to show a new icon next to pages that have been added recently (length of time a page is considered recent is configured on the settings page). If all pages are recent it won't show the new icon.
• Ability to restrict a crawl to not go higher than the starting path specified on the settings page allowing FastFind to properly index only a section of your site.
• Ability for us to ship with different languages (only includes English at present though)
  

If you have any other ideas for features that I might be able to slip into the release, just reply to this blog post.